Installation of Elastic Search
and MongoDB
Preparation:
#sudo apt-get update
#sudo apt-get upgrade
Installing OpenJDk
and its dependencies: It will take some time to complete full installation
#sudo apt-get install python-software-properties
#sudo add-apt-repository ppa:ferramroberto/java ### show repository
#sudo apt-get update
#sudo apt-get install
sun-java6-jre sun-java6-plugin openjdk-6-jre
Installing Elasticsearch:
#curl
http://cloud.github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.4.tar.gz
| tar zxv
#ln -s
elasticsearch-0.19.4/ elasticsearch
Downloading and
uncompressing Service Wrapper:
#curl -k -L
http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master |
tar -xz
Moving service wrapper to bin folder:
#mv
*servicewrapper*/service elasticsearch/bin/
Removing Service Wrapper file:
#rm -Rf *servicewrapper*
Configuring ElasticSearch:
#ln -s `readlink -f
elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl
#sed -i -e 's|#
cluster.name: elasticsearch|cluster.name: graylog2|'
/opt/elasticsearch/config/elasticsearch.yml
#/etc/init.d/elasticsearch
start
Verifying Elasticsearch status:
#curl -XGET
'http://localhost:9200/_cluster/health?pretty=true'
Output should be as given below
curl -XGET
'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog2",
"status" : "green",
"timed_out" : false,
"number_of_nodes"
: 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
--------------------------------------------------------------------------------------------------------------
elasticsearch-head
A web front end for an ElasticSearch cluster
Installing and Running
There are two ways of running and installing elasticsearch-head
Running as a plugin of ElasticSearch (this is the preferred method)
If you’ve installed the .deb package, then the plugin exectuable will be available at /usr/share/elasticsearch/bin/plugin.
sudo elasticsearch/bin/plugin -install mobz/elasticsearch-headopen http://localhost:9200/_plugin/head/
This will automatically download the latest version of
elasticsearch-head from github and run it as a plugin within the
elasticsearch cluster. In this mode;
- elasticsearch provides a simple webserver to run head
- elasticsearch-head automatically connects to the node that is running it
- is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)
- Will not work with elasticsearch prior to version 0.17
Running as a standalone webapp
git clone git://github.com/mobz/elasticsearch-head.gitcd elasticsearch-headopen index.html
This will manually download the latest version of elasticsearch-head from github and run it standalone. In this mode;
- elasticsearch-head is running from the file system and some features will not work
- may have trouble connecting to the node if the node is protected by some security measures
- works with any version of elasticsearch
Alternatives
elastisearch-head is a standalone webapp written in good-ol’ html5.
This means, you can put it up on any webserver, run it directly from the
filesystem, use it on an ipad, or put it on a floppy disk and carry it
with you.
------------------------------------------------------------------------------------------------------
How
to backup ElasticSearch with rsync
http://karussell.wordpress.com/2011/07/10/how-to-backup-elasticsearch-with-rsync/
Although there is a gateway feature implemented in ElasticSearch
which basically recovers your index on start if it is corrupted or similar it
is wise to create backups if there are bugs in Lucene or ElasticSearch
(assuming you have set the fs gateway). The backup script looks as follows and
uses the possibility to enable and disable the flushing for a short time:
#
TO_FOLDER=/something
#
FROM=/your-es-installation
DATE=`date
+%Y-%m-%d_%H-%M`
TO=$TO_FOLDER/$DATE/
echo
"rsync from $FROM to $TO"
#
the first times rsync can take a bit long - do not disable flusing
rsync
-a $FROM $TO
#
now disable flushing and do one manual flushing
$SCRIPTS/es-flush-disable.sh
true
$SCRIPTS/es-flush.sh
#
... and sync again
rsync
-a $FROM $TO
$SCRIPTS/es-flush-disable.sh
false
#
now remove too old backups
rm
-rf `find $TO_FOLDER -maxdepth 1 -mtime +7` &> /dev/null
E.g. you could call the backup
script regularly (even hourly) from cron and it will create new backups. By the
way – if you want to take a look on the settings of all indices (e.g. to check
the disable flushing stuff) this might be handy:
curl
-XGET 'localhost:9200/_settings?pretty=true'
MONGODB Installation Steps:
#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv
7F0CEB10
To ensure database software you download is actually the
software published by 10gen.
Now we have to add 10gen repository location in the
system by adding the following line to the end of /etc/apt/sources.list
#nano /etc/apt/sources.list
Add this line to the bottom of the file
deb
http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
Save the file and exit
Update the repository list and install MongoDB
#sudo apt-get update
#sudo apt-get install mongodb-10gen
* This will take some time to complete depending upon
your internet connection and the system performance.
When MongoDB has finished installing, run mongo just to
check whether MongoDB is running correctly.
If you get the same as above it means MongoDB is
successfully installed.
MongoDB configuration steps:
Now we need to create an admin user for mongo and add a user
to mongo for the ‘graylog2’ collection. I like to use a Secure Password
Generator (https://secure.pctools.com/guides/password/) to generate passwords.
Click on "Generate Password(s)" button for
generating the number of secure Password(s).
For the sake of this documentation I have used some
simple passwords.
use admin
db.addUser('admin', 'MongoDB@123')
db.auth('admin', 'MongoDB@123')
use graylog2
db.addUser('grayloguser', 'password@123')
db.auth('grayloguser', 'password@123')
Now you are ready with Elastic Search and MongoDB for installing Graylog2 Log management tool www.graylog2.org.
MongoDB connectivity problem Troubleshooting:
Error: couldn't connect to server 127.0.0.1 shell/mongo.js:79
exception: connect failed
Logs from /var/log/mongodb/mongodb.log
***** SERVER RESTARTED *****
Mon Jul 30 08:39:27
Mon Jul 30 08:39:27 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Jul 30 08:39:27
Mon Jul 30 08:39:27 [initandlisten] MongoDB starting : pid=886 port=27017 dbpath=/var/lib/mongodb 32-bit host=ubuntu-OptiPlex-990
Mon Jul 30 08:39:27 [initandlisten]
Mon Jul 30 08:39:27 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Jul 30 08:39:27 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Jul 30 08:39:27 [initandlisten] ** with --journal, the limit is lower
Mon Jul 30 08:39:27 [initandlisten]
Mon Jul 30 08:39:27 [initandlisten] db version v2.0.6, pdfile version 4.5
Mon Jul 30 08:39:27 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Mon Jul 30 08:39:27 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Mon Jul 30 08:39:27 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
**************
Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
*************
Mon Jul 30 08:39:27 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Mon Jul 30 08:39:27 dbexit:
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to close listening sockets...
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to flush diaglog...
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to close sockets...
Mon Jul 30 08:39:27 [initandlisten] shutdown: waiting for fs preallocator...
Mon Jul 30 08:39:27 [initandlisten] shutdown: closing all files...
Mon Jul 30 08:39:27 [initandlisten] closeAllFiles() finished
Mon Jul 30 08:39:27 dbexit: really exiting now
To fix this please follow the step by step approach:
First try connecting with the below given command
1)
#mongod --config /etc/mongodb.conf --dbpath /var/lib/mongodb
Where /etc/mongodb.conf is your mongodb configuration file and /var/lib/mongodb is your database path.
If it gives the same connection error. Then remove the lock file of mongodb by issuing the command.
2)
Locate the "mongod.lock" fie
#rm -rf /var/lib/mongodb/mongod.lock
3)
Issue the Database repair command:
#mongod --repair --config /etc/mongodb.conf --dbpath /var/lib/mongodb
4)
Restart the mongod and mongo services.
#service mongod stop
#service mongod start
#service mongo stop
#service mongo start
5)
Issue the Database connecting command.
#mongod --config /etc/mongodb.conf --dbpath /var/lib/mongodb&
6)
Try connecting
root@ubuntu# mongo
MongoDB shell version: 2.0.6
connecting to: test
>
You should be Happy once you see the above prompt.
Backup and Restoration of MongoDB.
Refer the link below for the steps and methods for backup of MongoDB.
http://docs.mongodb.org/manual/administration/backups/
mongodump syntax and options:
# mongodump --help
Export MongoDB data to BSON files.
options:
--help produce help message
-v [ --verbose ] be more verbose (include multiple times for more
verbosity e.g. -vvvvv)
--version print the program's version and exit
-h [ --host ] arg mongo host to connect to (
/s1,s2 for
sets)
--port arg server port. Can also use --host hostname:port
--ipv6 enable IPv6 support (disabled by default)
-u [ --username ] arg username
-p [ --password ] arg password
--dbpath arg directly access mongod database files in the given
path, instead of connecting to a mongod server -
needs to lock the data directory, so cannot be used
if a mongod is currently accessing the same path
--directoryperdb if dbpath specified, each db is in a separate
directory
--journal enable journaling
-d [ --db ] arg database to use
-c [ --collection ] arg collection to use (some commands)
-o [ --out ] arg (=dump) output directory or "-" for stdout
-q [ --query ] arg json query
--oplog Use oplog for point-in-time snapshotting
--repair try to recover a crashed database
--forceTableScan force a table scan (do not use $snapshot)
# mongorestore --help
usage: mongorestore [options] [directory or filename to restore from]
options:
--help produce help message
-v [ --verbose ] be more verbose (include multiple times for more
verbosity e.g. -vvvvv)
--version print the program's version and exit
-h [ --host ] arg mongo host to connect to ( /s1,s2 for sets)
--port arg server port. Can also use --host hostname:port
--ipv6 enable IPv6 support (disabled by default)
-u [ --username ] arg username
-p [ --password ] arg password
--dbpath arg directly access mongod database files in the given
path, instead of connecting to a mongod server -
needs to lock the data directory, so cannot be used
if a mongod is currently accessing the same path
--directoryperdb if dbpath specified, each db is in a separate
directory
--journal enable journaling
-d [ --db ] arg database to use
-c [ --collection ] arg collection to use (some commands)
--objcheck validate object before inserting
--filter arg filter to apply before inserting
--drop drop each collection before import
--oplogReplay replay oplog for point-in-time restore
--keepIndexVersion don't upgrade indexes to newest version
eg:-
#mongodump -h 127.0.0.1 --dbpath /var/lib/mongodb -o /usr/bin/dump/2012-08-06
#mongorestore -h 127.0.0.1 /usr/bin/dump/2012-08-06
Now you are done with the Installation and configuration of
Elasticsearch and MongoDB for Graylog2.
Request everyone to put a comment about this blog and give your suggestions for improvement.
--------------------------------------------------------------------------------------------------------------
Disclaimer:
I cannot assume any liability for the content of
external pages. Solely the operators of those linked pages are
responsible for their content.
I make every reasonable effort to ensure that the
content of this Web site is kept up to date, and that it is accurate
and complete. Nevertheless, the possibility of errors cannot be entirely
ruled out. I do not give any warranty in respect of the timeliness,
accuracy or completeness of material published on this Web site, and
disclaim all liability for (material or non-material) loss or damage
incurred by third parties arising from the use of content obtained from
the Web site.
Registered trademarks and proprietary names, and
copyrighted text and images, are not generally indicated as such on my
Web pages. But the absence of such indications in no way implies the
these names, images or text belong to the public domain in the context
of trademark or copyright law.
All product and firm names on
www.erco.com are proprietary names of their corresponding owners
All products and firm names used in this site are proprietary names of
their corresponding owners. All rights are reserved which are not
explicitly granted here.
