Installation of Elastic Search and Mongo DB

Installation of Elastic Search and MongoDB

Preparation:

#sudo apt-get update

#sudo apt-get upgrade

Installing OpenJDk and its dependencies: It will take some time to complete full installation

#sudo apt-get install python-software-properties

#sudo add-apt-repository ppa:ferramroberto/java  ### show repository

#sudo apt-get update

#sudo apt-get install sun-java6-jre sun-java6-plugin openjdk-6-jre

Installing Elasticsearch:

#curl http://cloud.github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.4.tar.gz | tar zxv

#ln -s elasticsearch-0.19.4/ elasticsearch

Downloading and uncompressing Service Wrapper:

#curl -k -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz

Moving service wrapper to bin folder:

#mv *servicewrapper*/service elasticsearch/bin/

Removing Service Wrapper file:

#rm -Rf *servicewrapper*

Configuring ElasticSearch:

#ln -s `readlink -f elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl

#sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /opt/elasticsearch/config/elasticsearch.yml

#/etc/init.d/elasticsearch start

Verifying Elasticsearch status:

#curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Output should be as given below

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{

  "cluster_name" : "graylog2",

  "status" : "green",

  "timed_out" : false,

  "number_of_nodes" : 1,

  "number_of_data_nodes" : 1,

  "active_primary_shards" : 0,

  "active_shards" : 0,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0

}

--------------------------------------------------------------------------------------------------------------

elasticsearch-head

A web front end for an ElasticSearch cluster

http://mobz.github.com/elasticsearch-head

Installing and Running

There are two ways of running and installing elasticsearch-head

Running as a plugin of ElasticSearch (this is the preferred method)

If you’ve installed the .deb package, then the plugin exectuable will be available at /usr/share/elasticsearch/bin/plugin.

• sudo elasticsearch/bin/plugin -install mobz/elasticsearch-head
• open http://localhost:9200/_plugin/head/

This will automatically download the latest version of elasticsearch-head from github and run it as a plugin within the elasticsearch cluster. In this mode;

• elasticsearch provides a simple webserver to run head
• elasticsearch-head automatically connects to the node that is running it
• is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)
• Will not work with elasticsearch prior to version 0.17

Running as a standalone webapp

• git clone git://github.com/mobz/elasticsearch-head.git
• cd elasticsearch-head
• open index.html

This will manually download the latest version of elasticsearch-head from github and run it standalone. In this mode;

• elasticsearch-head is running from the file system and some features will not work
• may have trouble connecting to the node if the node is protected by some security measures
• works with any version of elasticsearch

Alternatives

elastisearch-head is a standalone webapp written in good-ol’ html5. This means, you can put it up on any webserver, run it directly from the filesystem, use it on an ipad, or put it on a floppy disk and carry it with you.
------------------------------------------------------------------------------------------------------

How to backup ElasticSearch with rsync

http://karussell.wordpress.com/2011/07/10/how-to-backup-elasticsearch-with-rsync/

Although there is a gateway feature implemented in ElasticSearch which basically recovers your index on start if it is corrupted or similar it is wise to create backups if there are bugs in Lucene or ElasticSearch (assuming you have set the fs gateway). The backup script looks as follows and uses the possibility to enable and disable the flushing for a short time:

# TO_FOLDER=/something

# FROM=/your-es-installation

DATE=`date +%Y-%m-%d_%H-%M`

TO=$TO_FOLDER/$DATE/

echo "rsync from $FROM to $TO"

# the first times rsync can take a bit long - do not disable flusing

rsync -a $FROM $TO

# now disable flushing and do one manual flushing

$SCRIPTS/es-flush-disable.sh true

$SCRIPTS/es-flush.sh

# ... and sync again

rsync -a $FROM $TO

$SCRIPTS/es-flush-disable.sh false

# now remove too old backups

rm -rf `find $TO_FOLDER -maxdepth 1 -mtime +7` &> /dev/null

E.g. you could call the backup script regularly (even hourly) from cron and it will create new backups. By the way – if you want to take a look on the settings of all indices (e.g. to check the disable flushing stuff) this might be handy:

curl -XGET 'localhost:9200/_settings?pretty=true'

MONGODB Installation Steps:

#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

To ensure database software you download is actually the software published by 10gen.

Now we have to add 10gen repository location in the system by adding the following line to the end of /etc/apt/sources.list

#nano /etc/apt/sources.list

Add this line to the bottom of the file

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

Save the file and exit

Update the repository list and install MongoDB

#sudo apt-get update

#sudo apt-get install mongodb-10gen

* This will take some time to complete depending upon your internet connection and the system performance.

When MongoDB has finished installing, run mongo just to check whether MongoDB is running correctly.

If you get the same as above it means MongoDB is successfully installed.

MongoDB configuration steps:

Now we need to create an admin user for mongo and add a user to mongo for the ‘graylog2’ collection. I like to use a Secure Password Generator (https://secure.pctools.com/guides/password/) to generate passwords.

Click on "Generate Password(s)" button for generating the number of secure Password(s).

For the sake of this documentation I have used some simple passwords.

use admin

db.addUser('admin', 'MongoDB@123')

db.auth('admin', 'MongoDB@123')

use graylog2

db.addUser('grayloguser', 'password@123')

db.auth('grayloguser', 'password@123')

Now you are ready with Elastic Search and MongoDB for installing Graylog2 Log management tool www.graylog2.org.

MongoDB connectivity problem Troubleshooting:

Error: couldn't connect to server 127.0.0.1 shell/mongo.js:79
exception: connect failed

Logs from  /var/log/mongodb/mongodb.log

***** SERVER RESTARTED *****


Mon Jul 30 08:39:27
Mon Jul 30 08:39:27 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Jul 30 08:39:27
Mon Jul 30 08:39:27 [initandlisten] MongoDB starting : pid=886 port=27017 dbpath=/var/lib/mongodb 32-bit host=ubuntu-OptiPlex-990
Mon Jul 30 08:39:27 [initandlisten]
Mon Jul 30 08:39:27 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Jul 30 08:39:27 [initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Jul 30 08:39:27 [initandlisten] **       with --journal, the limit is lower
Mon Jul 30 08:39:27 [initandlisten]
Mon Jul 30 08:39:27 [initandlisten] db version v2.0.6, pdfile version 4.5
Mon Jul 30 08:39:27 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Mon Jul 30 08:39:27 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Mon Jul 30 08:39:27 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
**************
Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
*************
Mon Jul 30 08:39:27 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Mon Jul 30 08:39:27 dbexit:
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to close listening sockets...
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to flush diaglog...
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to close sockets...
Mon Jul 30 08:39:27 [initandlisten] shutdown: waiting for fs preallocator...
Mon Jul 30 08:39:27 [initandlisten] shutdown: closing all files...
Mon Jul 30 08:39:27 [initandlisten] closeAllFiles() finished
Mon Jul 30 08:39:27 dbexit: really exiting now


To fix this please follow the step by step approach:
First try connecting with the below given command
1)
#mongod --config /etc/mongodb.conf --dbpath /var/lib/mongodb
Where /etc/mongodb.conf is your mongodb configuration file and /var/lib/mongodb is your database path.

If it gives the same connection error. Then remove the lock file of mongodb by issuing the command.
2)
Locate the "mongod.lock" fie
#rm -rf /var/lib/mongodb/mongod.lock

3)
Issue the Database repair command:
#mongod --repair --config /etc/mongodb.conf --dbpath /var/lib/mongodb

4)
Restart the mongod and mongo services.
#service mongod stop
#service mongod start

#service mongo stop
#service mongo start

5)
 Issue the Database connecting command.
#mongod --config /etc/mongodb.conf --dbpath /var/lib/mongodb&

6)
Try connecting
root@ubuntu# mongo
MongoDB shell version: 2.0.6
connecting to: test
>
You should be Happy once you see the above prompt.


Backup and Restoration of MongoDB.
Refer the link below for the steps and methods for backup of MongoDB.
http://docs.mongodb.org/manual/administration/backups/

mongodump syntax and options:
# mongodump --help
Export MongoDB data to BSON files.

options:
  --help                   produce help message
  -v [ --verbose ]         be more verbose (include multiple times for more
                           verbosity e.g. -vvvvv)
  --version                print the program's version and exit
  -h [ --host ] arg        mongo host to connect to ( /s1,s2 for
                           sets)
  --port arg               server port. Can also use --host hostname:port
  --ipv6                   enable IPv6 support (disabled by default)
  -u [ --username ] arg    username
  -p [ --password ] arg    password
  --dbpath arg             directly access mongod database files in the given
                           path, instead of connecting to a mongod  server -
                           needs to lock the data directory, so cannot be used
                           if a mongod is currently accessing the same path
  --directoryperdb         if dbpath specified, each db is in a separate
                           directory
  --journal                enable journaling
  -d [ --db ] arg          database to use
  -c [ --collection ] arg  collection to use (some commands)
  -o [ --out ] arg (=dump) output directory or "-" for stdout
  -q [ --query ] arg       json query
  --oplog                  Use oplog for point-in-time snapshotting
  --repair                 try to recover a crashed database
  --forceTableScan         force a table scan (do not use $snapshot)

# mongorestore --help
usage: mongorestore [options] [directory or filename to restore from]
options:
  --help                  produce help message
  -v [ --verbose ]        be more verbose (include multiple times for more
                          verbosity e.g. -vvvvv)
  --version               print the program's version and exit
  -h [ --host ] arg       mongo host to connect to ( /s1,s2 for sets)
  --port arg              server port. Can also use --host hostname:port
  --ipv6                  enable IPv6 support (disabled by default)
  -u [ --username ] arg   username
  -p [ --password ] arg   password
  --dbpath arg            directly access mongod database files in the given
                          path, instead of connecting to a mongod  server -
                          needs to lock the data directory, so cannot be used
                          if a mongod is currently accessing the same path
  --directoryperdb        if dbpath specified, each db is in a separate
                          directory
  --journal               enable journaling
  -d [ --db ] arg         database to use
  -c [ --collection ] arg collection to use (some commands)
  --objcheck              validate object before inserting
  --filter arg            filter to apply before inserting
  --drop                  drop each collection before import
  --oplogReplay           replay oplog for point-in-time restore
  --keepIndexVersion      don't upgrade indexes to newest version


eg:-
#mongodump -h 127.0.0.1 --dbpath /var/lib/mongodb -o /usr/bin/dump/2012-08-06
#mongorestore -h 127.0.0.1 /usr/bin/dump/2012-08-06

Now you are done with the Installation and configuration of Elasticsearch and MongoDB for Graylog2.

                Request everyone to put a comment about this blog and give your suggestions for improvement.
 --------------------------------------------------------------------------------------------------------------


Disclaimer:

I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content.

I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site.

Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law.

All product and firm names on www.erco.com are proprietary names of their corresponding owners

 All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.