Friday, August 17, 2012

How to Install and configure Graylog2 complete suite on Fedora 14 & 17

Graylog2 complete suite on Fedora

Installing Elasticsearch on Fedora

1) Download the latest package from the site given below:

http://www.elasticsearch.org/download/

Package name:

elasticsearch-0.1x.x.tar

2) Unzip the file and save to the folder /opt/

#tar -zxvf elasticsearch-0.19.8.tar /opt/

If the unzipped elasticsearch folder to the location “/opt/”

#mv elasticsearch-0.19.8 /opt/

3) Create a symbolic link by applying the below command (change the version as per the package)
#cd /opt

#ln -s /opt/elasticsearch-0.19.8/ elasticsearch

4) Download the service wrapper for elasticsearch from the link given below:

http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master

Save the file and unzip

5) #tar -zxvf elasticsearch-elasticsearch-servicewrapper-d47d048.tar

6) Move the folder “service” from the unzipped folder to /opt/elasticsearch-0.19.8/bin

#mv *servicewrapper*/service /opt/elasticsearch/bin/

#mv *servicewrapper*/service /opt/elasticsearch-0.19.8/bin/

7) Remove the folder for service wrapper

#rm -Rf *servicewrapper*

Configuring Elasticsearch:

#ln -s `readlink -f /opt/elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl

#sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /opt/elasticsearch/config/elasticsearch.yml

# cd /opt/elasticsearch/bin/service/

Starting Elasticsearch

#./elasticsearch start

Checking the health status of Elasticsearch Cluster:

Verifying Elasticsearch status:

#curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Output should be as given below

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "graylog2",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 1,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0

}

Elasticsearch-head

A web front end for an ElasticSearch cluster

http://mobz.github.com/elasticsearch-head

Installing and Running

There are two ways of running and installing elasticsearch-head

Running as a plugin of ElasticSearch (this is the preferred method)

If you’ve installed the .deb package, then the plugin exectuable will be available at /usr/share/elasticsearch/bin/plugin.

#sudo elasticsearch/bin/plugin -install mobz/elasticsearch-head

open http://localhost:9200/_plugin/head/

This will automatically download the latest version of elasticsearch-head from github and run it as a plugin within the elasticsearch cluster. In this mode;

elasticsearch provides a simple webserver to run head

elasticsearch-head automatically connects to the node that is running it

is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)

Will not work with elasticsearch prior to version 0.17

Running as a standalone webapp

git clone git://github.com/mobz/elasticsearch-head.git

cd elasticsearch-head

open index.html

This will manually download the latest version of elasticsearch-head from github and run it standalone. In this mode;

elasticsearch-head is running from the file system and some features will not work

may have trouble connecting to the node if the node is protected by some security measures

works with any version of elasticsearch

Alternatives

elastisearch-head is a standalone webapp written in good-old html5. This means, you can put it up on any webserver, run it directly from the filesystem, use it on an ipad, or put it on a floppy disk and carry it with you.

Mongodb installation on Fedora

Go to the link given below for installation instructions for mongodb:

http://www.mongodb.org/downloads

Go to the bottom of the page and under packages, select For Fedora and CentOS, see CentOS and Fedora packages. Click on the link

Updating yum repository for mongoDB

Create a file with the below command and paste the contents as per your platform.

#nano /etc/yum.repos.d/10gen.repo

[10gen]

name=10gen Repository

baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686

gpgcheck=0

enabled=1

Save the file and exit

Issue the command for updating the yum repository and updating the pacakages.

#yum update

#yum install mongo-10gen mongo-10gen-server

# service mongod start

# mongo

MongoDB shell version: 2.0.7

connecting to: test

Type “exit” to come out of MongoDB prompt

Check the logs in the file /var/log/mongo/mongod.log

#tail /var/log/mongo/mongod.log

#mongo

MongoDB shell version: 2.0.7

connecting to: test

>use admin

switched to db admin

> db.addUser('admin', 'M@NgoDB@123')

{ "n" : 0, "connectionId" : 2, "err" : null, "ok" : 1 }

{

"user" : "admin",

"readOnly" : false,

"pwd" : "7dc9af43dc114adaf5da03a53e72d22a",

"_id" : ObjectId("502a2f4beaf6a7c7177a4a9a")

}

Testing authentication:

> db.auth('admin', 'M@NgoDB@123')

>use graylog2

> db.addUser('grayloguser', 'Graylog2@123')

{ "n" : 0, "connectionId" : 2, "err" : null, "ok" : 1 }

{

"user" : "grayloguser",

"readOnly" : false,

"pwd" : "6be58c343ffe01fa2ea3605bff5a10bc",

"_id" : ObjectId("502a3004eaf6a7c7177a4a9b")

}

Installation of Graylog2 on Fedora

Download the latest stable package from the site given below:

https://github.com/Graylog2/graylog2-server/downloads

https://github.com/Graylog2/graylog2-web-interface/downloads

Update Java

#yum install java-1.6.0-openjdk-1.6.0.0-44.1.9.1.fc14.i686

Unzipping the files:

#tar -zxvf graylog2-server-0.9.6p1.tar.gz

Moving the folder to “/opt”

#mv graylog2-server-0.9.6p1 /opt

# tar -zxvf graylog2-web-interface-0.9.6p1.tar.gz

#mv graylog2-web-interface-0.9.6p1 /opt

#cd /opt

Creating symbolic links:

#ln -s /opt/graylog2-server-0.9.6p1/ graylog2-server

#ln -s /opt/graylog2-web-interface-0.9.6p1/ graylog2-web-interface

Change directory to Graylog2 server

#cd graylog2-server

Copy the sample graylog.conf file to /etc/

#sudo cp graylog2.conf.example /etc/graylog2.conf

Reflect the MongoDB configurations in “graylog2.conf” as per your Database settings and

configure the UDP listening port for Graylog2 in the same file .

# On which port (UDP) should we listen for Syslog messages? (Standard: 514)

syslog_listen_port = 5140

syslog_protocol = udp

# ElasticSearch URL (default: http://localhost:9200/)

elasticsearch_url = http://localhost:9200/

elasticsearch_index_name = graylog2

# Always try a reverse DNS lookup instead of parsing hostname from syslog message?

force_syslog_rdns = false

# Set time to NOW if parsing date/time from syslog message failed instead of rejecting it?

allow_override_syslog_date = true

# MongoDB Configuration

mongodb_useauth = true

mongodb_user = grayloguser

mongodb_password = Graylog2@123

mongodb_host = localhost

#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019

mongodb_database = graylog2

mongodb_port = 27017

Start the Graylog2 server with the following command:

# cd /opt/graylog2/bin/
# ./graylog2ctl start

For stopping Graylog2 issues the below given command:

# ./graylog2ctl stop

Installation of Graylog2 Web Interface:

Graylog2 Web interface is running under Ruby, we need to install the latest version of Ruby. Before starting remove all your previous Ruby installation because old release of Ruby is not compatible with Graylog2 and other dependencies.

# yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs

Install all the required development tools :

# yum install openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel

Download and Install Ruby latest version:

Browse to the below given link and as per your requirement install the latest Ruby with the methods mentioned over there. I am installing Ruby from Source code by compiling.

http://www.ruby-lang.org/en/downloads/

Package name:

ruby-1.9.3-p194.tar Stable Release as on 16-08-2012

After downloading the package unzip the Tar by issuing the below given command:

#tar –zxvf ruby-1.9.3-p194.tar

Move the ruby unzipped ruby folder to “/opt”

#mv ruby-1.9.3-p194 /opt

#cd /opt/ruby-1.9.3-p194

* How to compile and install Ruby

This is what you need to do to compile and install Ruby:

1. If ./configure does not exist or is older than configure.in,

run autoconf to (re)generate configure.

This will take some time to complete based upon your system and architecture.

2. Run ./configure, which will generate config.h and Makefile.

#./configure

Some C compiler flags may be added by default depending on your

environment. Specify optflags=.. and warnflags=.. as necessary

to override them.

3. Edit defines.h if you need. Usually this step will not be needed.

4. Remove comment mark(#) before the module names from ext/Setup (or

add module names if not present), if you want to link modules

statically.

If you don't want to compile non static extension modules

(probably on architectures which does not allow dynamic loading),

remove comment mark from the line "#option nodynamic" in

ext/Setup.

5. Run make.
#make
#make test (optional)

6. Optionally, run 'make test' to check whether the compiled Ruby

interpreter works well. If you see the message "test succeeded",

your ruby works as it should (hopefully).

7. Run 'make install'

#make install

This command will create following directories and install files

onto them.

Test the installation of Ruby

#ruby –v

#gem --version

If it gives somes notifications for installing any other packages, install those and reinstall Ruby from starting.

Update all the gems, install git and rake:

#gem update && gem install git rake

Unzip the latest stable version of Graylog2 Web Interface.

#tar –zxvf graylog2-web-interface-0.9.6p1.tar.gz

Move the unzipped folder to “/opt”

#mv graylog2-web-interface-0.9.6p1

Now change the directory to Graylog2-Web Interface

#cd /opt/graylog2-web-interface-0.9.6p1

Install Gem bundler:

#gem install bundler
#bundle install

For showing the gems installed:

#bundle show

Edit all “*.yml” configuration files in “/opt/graylog2-web-interface/config/” folder as per your requirement and environment.

Mongoid.yml

#nano /opt/graylog2-web-interface/config/mongoid.yml

# Use environment variables

production:

host: <%= ENV['MONGOID_HOST'] %>

port: <%= ENV['MONGOID_PORT'] %>

username: <%= ENV['MONGOID_USERNAME'] %>

password: <%= ENV['MONGOID_PASSWORD'] %>

database: <%= ENV['MONGOID_DATABASE'] %>

# or specify values manually
production: ###### No space before Production #########

host: localhost

port: 27017

username:

password:

database: graylog2

*** With Username and Password given Graylog2 is not able to connect.

# Values below are for development and testing.

# You may comment out them or just ignore.

development:

host: localhost

database: graylog2_development

test:

host: localhost

database: graylog2_test

email.yml

production:

via: smtp # via: sendmail

host: smtp.example.org

enable_starttls_auto: true

port: 25

auth: plain # plain, login, cram_md5 - Comment out or remove to use no auth

user: somebody

password: yourpass

domain: example.org # the HELO domain provided by the client to the server

Set the Index Name:

Index.yml

production:

url: http://localhost:9200/

index_name: graylog2

To access server Graylog2 Web interface through Apache and Passenger.
To install Passanger just run the following command :

# gem install passenger

# passenger-install-apache2-module

Welcome to the Phusion Passenger Apache 2 module installer, v3.0.15.

This installer will guide you through the entire installation process. It

shouldn't take more than 3 minutes in total.

Here's what you can expect from the installation process:

1. The Apache 2 module will be installed for you.

2. You'll learn how to configure Apache.

3. You'll learn how to deploy a Ruby on Rails application.

Don't worry if anything goes wrong. This installer will advise you on how to

solve any problems.

--------------------------------------------

The Apache 2 module was successfully installed.

Please edit your Apache configuration file, and add these lines:

LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15/ext/apache2/mod_passenger.so

PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15

PassengerRuby /usr/local/bin/ruby

After you restart Apache, you are ready to deploy any number of Ruby on Rails

applications on Apache, without any further Ruby on Rails-specific

configuration!

Press Enter to continue, or Ctrl-C to abort.

--------------------------------------------

Deploying a Ruby on Rails application: an example

Suppose you have a Rails application in /somewhere. Add a virtual host to your

Apache configuration file and set its DocumentRoot to /somewhere/public:

ServerName 10.x.x.x

# !!! Be sure to point DocumentRoot to 'public'!

DocumentRoot /opt/graylog2-web-interface/public

# This relaxes Apache security settings.

AllowOverride all

# MultiViews must be turned off.

Options -MultiViews

ErrorLog /var/log/httpd/Graylog2_error.log

LogLevel warn

CustomLog /var/log/httpd/Graylog2_access.log combined

Listen 10.x.x.x:8080

And that's it! You may also want to check the Users Guide for security and

optimization tips, troubleshooting and other useful information:

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15/doc/Users guide Apache.html

Create a “passenger.conf” file in “/etc/httpd/conf.d/” directory and add the following entries :

LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15/ext/apache2/mod_passenger.so

PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15

PassengerRuby /usr/local/bin/ruby

Then in your “/etc/httpd/conf/httpd.conf” file include the “passenger.conf” file.
Or check for the entry in the file as given below:

# Load config files from the config directory "/etc/httpd/conf.d".

Include conf.d/*.conf

Giving ownership for apache to Graylog2 Web Interface folders

# chown -R apache:apache /opt/graylog2-web-interface-0.9.6p1 ### Check the package name

# chown -R apache:apache /opt/graylog2-web-interface ### For the soft link

Now you will be able to connect you on your vhost, configure the Graylog2 first user and connect into the Web interface.

Graylog2 Email notification Configuration:

To get email notification for Alarms and Subscriptions you need to enable email notification in Graylog2.

Main configuration files:

/opt/graylog2-web-interface/config/general.yml

/opt/graylog2-web-interface/config/email.yml

------------------------------------------------------------------------------------------------------------

Step 1)

Editing to be done in the files according to your SMTP server and domain name:

#nano /opt/graylog2-web-interface/config/general.yml

######## Contents of the file ######################

general:

external_hostname: "your-graylog2.example.org" # Used for example to generate permalinks. Don't add 'http://' or trailing slashes.

date_format: "%d.%m.%Y - %H:%M:%S" # http://ruby-doc.org/core/classes/Time.html#M000298 (strftime syntax)

allow_deleting: false # Allowing deleting of messages negatively impacts performance

allow_version_check: true # Enables manual (/versioncheck/index) and automatic (every 30min from overview page) version checking against graylog2.org via HTTP.

# custom_cookie_name: graylog2_staging1 # Set an own cookie name - Useful for multiple deployments on same host like example.org/staging1/graylog2, example.org/staging2/grayl$

# Settings for stream subscription emails.

subscriptions:

from: graylog2subscriptions@example.org

subject: "[graylog2] Subscription"

# Settings for stream alarm emails.

streamalarms:

from: graylog2alarms@example.org

subject: "[graylog2] Stream alarm!"

hoptoad:

enabled: false

ssl: false

api_key: 123

#nano /opt/graylog2-web-interface/config/email.yml

################Copy from here and paste it in the file ############

production:

via: smtp # via: sendmail

host: 10.x.x.x

enable_starttls_auto: true

port: 25

####### If you don’t want authentication comment out the next three lines ######

# auth: plain # plain, login, cram_md5 - Comment out or remove to use no auth

# user: somebody

# password: yourpass

domain: example.org # the HELO domain provided by the client to the server

#########################################################

---------------------------------------------------------------------------------------------------------------

Step 2)

Create a script file for executing Stream Alarm check and Stream Subscription check.

#nano run_rake_tasks_Alarms.sh

###### Copy the file and paste it into the file #############

#!/bin/bash

cd /opt/graylog2-web-interface/

PATH=/usr/local/bin

rake RAILS_ENV=production streamalarms:send

########################################################

### Path of the rake command could be found by issuing the command #locate rake | more #####

#nano run_rake_tasks_ Subscriptions.sh

###### Copy the file and paste it into the file #############

#!/bin/bash

cd /opt/graylog2-web-interface/

PATH=/usr/local/bin

rake RAILS_ENV=production subscriptions:send

########################################################

--------------------------------------------------------------------------------------------------------------

Step 3)

Create a cron file for executing your scripts based on your requirement.

I am pasting the configuration for executing the scripts every one minute.

#crontab –e

######## Copy and paste the below items to the cron file bottom##########

* * * * * /opt/graylog2-web-interface/cron/run_rake_tasks_Subscription.sh

* * * * * /opt/graylog2-web-interface/cron/run_rake_tasks_Alarms.sh

--------------------------------------------------------------------------------------------------------------

Verify whether the below given screenshot is displayed in the Home screen right corner showing the services as running.

***For steps on polling network devices logs, please refer my old post.

Disclaimer:

I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content.
I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site.
Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law.
All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.

Thursday, July 26, 2012

Ruby Regex Syntax

Regular-expression modifiers:

Regular expression literals may include an optional modifier to control various aspects of matching. The modifier is specified after the second slash character, as shown previously and may be represented by one of these characters:

Modifier	Description
i	Ignore case when matching text.
o	Perform #{} interpolations only once, the first time the regexp literal is evaluated.
x	Ignores whitespace and allows comments in regular expressions
m	Matches multiple lines, recognizing newlines as normal characters
u,e,s,n	Interpret the regexp as Unicode (UTF-8), EUC, SJIS, or ASCII. If none of these modifiers is specified, the regular expression is assumed to use the source encoding.

Like string literals delimited with %Q, Ruby allows you to begin your regular expressions with %r followed by a delimiter of your choice. This is useful when the pattern you are describing contains a lot of forward slash characters that you don't want to escape:

# Following matches a single slash character, no escape required
%r|/|               

# Flag characters are allowed with this syntax, too
%r[</(.*)>]i

Regular-expression patterns:

Except for control characters, (+ ? . * ^ $ ( ) [ ] { } | \), all characters match themselves. You can escape a control character by preceding it with a backslash.

Following table lists the regular expression syntax that is available in Ruby.

Pattern	Description
^	Matches beginning of line.
$	Matches end of line.
.	Matches any single character except newline. Using m option allows it to match newline as well.
[...]	Matches any single character in brackets.
[^...]	Matches any single character not in brackets
re*	Matches 0 or more occurrences of preceding expression.
re+	Matches 1 or more occurrence of preceding expression.
re?	Matches 0 or 1 occurrence of preceding expression.
re{ n}	Matches exactly n number of occurrences of preceding expression.
re{ n,}	Matches n or more occurrences of preceding expression.
re{ n, m}	Matches at least n and at most m occurrences of preceding expression.
a\| b	Matches either a or b.
(re)	Groups regular expressions and remembers matched text.
(?imx)	Temporarily toggles on i, m, or x options within a regular expression. If in parentheses, only that area is affected.
(?-imx)	Temporarily toggles off i, m, or x options within a regular expression. If in parentheses, only that area is affected.
(?: re)	Groups regular expressions without remembering matched text.
(?imx: re)	Temporarily toggles on i, m, or x options within parentheses.
(?-imx: re)	Temporarily toggles off i, m, or x options within parentheses.
(?#...)	Comment.
(?= re)	Specifies position using a pattern. Doesn't have a range.
(?! re)	Specifies position using pattern negation. Doesn't have a range.
(?> re)	Matches independent pattern without backtracking.
\w	Matches word characters.
\W	Matches nonword characters.
\s	Matches whitespace. Equivalent to [\t\n\r\f].
\S	Matches nonwhitespace.
\d	Matches digits. Equivalent to [0-9].
\D	Matches nondigits.
\A	Matches beginning of string.
\Z	Matches end of string. If a newline exists, it matches just before newline.
\z	Matches end of string.
\G	Matches point where last match finished.
\b	Matches word boundaries when outside brackets. Matches backspace (0x08) when inside brackets.
\B	Matches nonword boundaries.
\n, \t, etc.	Matches newlines, carriage returns, tabs, etc.
\1...\9	Matches nth grouped subexpression.
\10	Matches nth grouped subexpression if it matched already. Otherwise refers to the octal representation of a character code.

Regular-expression Examples:

Literal characters:

Example	Description
/ruby/	Match "ruby".
¥	Matches Yen sign. Multibyte characters are suported in Ruby 1.9 and Ruby 1.8.

Character classes:

Example	Description
/[Rr]uby/	Match "Ruby" or "ruby"
/rub[ye]/	Match "ruby" or "rube"
/[aeiou]/	Match any one lowercase vowel
/[0-9]/	Match any digit; same as /[0123456789]/
/[a-z]/	Match any lowercase ASCII letter
/[A-Z]/	Match any uppercase ASCII letter
/[a-zA-Z0-9]/	Match any of the above
/[^aeiou]/	Match anything other than a lowercase vowel
/[^0-9]/	Match anything other than a digit

Special Character Classes:

Example	Description
/./	Match any character except newline
/./m	In multiline mode . matches newline, too
/\d/	Match a digit: /[0-9]/
/\D/	Match a nondigit: /[^0-9]/
/\s/	Match a whitespace character: /[ \t\r\n\f]/
/\S/	Match nonwhitespace: /[^ \t\r\n\f]/
/\w/	Match a single word character: /[A-Za-z0-9_]/
/\W/	Match a nonword character: /[^A-Za-z0-9_]/

Repetition Cases:

Example	Description
/ruby?/	Match "rub" or "ruby": the y is optional
/ruby*/	Match "rub" plus 0 or more ys
/ruby+/	Match "rub" plus 1 or more ys
/\d{3}/	Match exactly 3 digits
/\d{3,}/	Match 3 or more digits
/\d{3,5}/	Match 3, 4, or 5 digits

Nongreedy repetition:

This matches the smallest number of repetitions:

Example	Description
/<.*>/	Greedy repetition: matches "<ruby>perl>"
/<.*?>/	Nongreedy: matches "<ruby>" in "<ruby>perl>"

Grouping with parentheses:

Example	Description
/\D\d+/	No group: + repeats \d
/(\D\d)+/	Grouped: + repeats \D\d pair
/([Rr]uby(, )?)+/	Match "Ruby", "Ruby, ruby, ruby", etc.

Backreferences:

This matches a previously matched group again:

Example	Description
/([Rr])uby&\1ails/	Match ruby&rails or Ruby&Rails
/(['"])(?:(?!\1).)*\1/	Single or double-quoted string. \1 matches whatever the 1st group matched . \2 matches whatever the 2nd group matched, etc.

Alternatives:

Example	Description
/ruby\|rube/	Match "ruby" or "rube"
/rub(y\|le))/	Match "ruby" or "ruble"
/ruby(!+\|\?)/	"ruby" followed by one or more ! or one ?

Anchors:

This need to specify match position

Example	Description
/^Ruby/	Match "Ruby" at the start of a string or internal line
/Ruby$/	Match "Ruby" at the end of a string or line
/\ARuby/	Match "Ruby" at the start of a string
/Ruby\Z/	Match "Ruby" at the end of a string
/\bRuby\b/	Match "Ruby" at a word boundary
/\brub\B/	\B is nonword boundary: match "rub" in "rube" and "ruby" but not alone
/Ruby(?=!)/	Match "Ruby", if followed by an exclamation point
/Ruby(?!!)/	Match "Ruby", if not followed by an exclamation point

Special syntax with parentheses:

Example	Description
/R(?#comment)/	Matches "R". All the rest is a comment
/R(?i)uby/	Case-insensitive while matching "uby"
/R(?i:uby)/	Same as above
/rub(?:y\|le))/	Group only without creating \1 backreference

http://www.tutorialspoint.com/ruby/ruby_regular_expressions.htm -------------------------------------------------------------------------------------------------------------- Disclaimer: I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content. I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site. Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law. All product and firm names on www.erco.com are proprietary names of their corresponding owners All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.

Friday, June 22, 2012

Installation of Elastic Search and Mongo DB

Installation of Elastic Search and MongoDB

Preparation:

#sudo apt-get update

#sudo apt-get upgrade

Installing OpenJDk and its dependencies: It will take some time to complete full installation

#sudo apt-get install python-software-properties

#sudo add-apt-repository ppa:ferramroberto/java ### show repository

#sudo apt-get update

#sudo apt-get install sun-java6-jre sun-java6-plugin openjdk-6-jre

Installing Elasticsearch:

#curl http://cloud.github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.4.tar.gz | tar zxv

#ln -s elasticsearch-0.19.4/ elasticsearch

Downloading and uncompressing Service Wrapper:

#curl -k -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz

Moving service wrapper to bin folder:

#mv *servicewrapper*/service elasticsearch/bin/

Removing Service Wrapper file:

#rm -Rf *servicewrapper*

Configuring ElasticSearch:

#ln -s `readlink -f elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl

#sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /opt/elasticsearch/config/elasticsearch.yml

#/etc/init.d/elasticsearch start

Verifying Elasticsearch status:

#curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Output should be as given below

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "graylog2",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 1,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0

}

--------------------------------------------------------------------------------------------------------------

elasticsearch-head

A web front end for an ElasticSearch cluster

http://mobz.github.com/elasticsearch-head

Installing and Running

There are two ways of running and installing elasticsearch-head

Running as a plugin of ElasticSearch (this is the preferred method)

If you’ve installed the .deb package, then the plugin exectuable will be available at /usr/share/elasticsearch/bin/plugin.

sudo elasticsearch/bin/plugin -install mobz/elasticsearch-head
open http://localhost:9200/_plugin/head/

This will automatically download the latest version of elasticsearch-head from github and run it as a plugin within the elasticsearch cluster. In this mode;

elasticsearch provides a simple webserver to run head
elasticsearch-head automatically connects to the node that is running it
is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)
Will not work with elasticsearch prior to version 0.17

Running as a standalone webapp

git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
open index.html

This will manually download the latest version of elasticsearch-head from github and run it standalone. In this mode;

elasticsearch-head is running from the file system and some features will not work
may have trouble connecting to the node if the node is protected by some security measures
works with any version of elasticsearch

Alternatives

elastisearch-head is a standalone webapp written in good-ol’ html5. This means, you can put it up on any webserver, run it directly from the filesystem, use it on an ipad, or put it on a floppy disk and carry it with you.
------------------------------------------------------------------------------------------------------

How to backup ElasticSearch with rsync

http://karussell.wordpress.com/2011/07/10/how-to-backup-elasticsearch-with-rsync/

Although there is a gateway feature implemented in ElasticSearch which basically recovers your index on start if it is corrupted or similar it is wise to create backups if there are bugs in Lucene or ElasticSearch (assuming you have set the fs gateway). The backup script looks as follows and uses the possibility to enable and disable the flushing for a short time:

# TO_FOLDER=/something

# FROM=/your-es-installation

DATE=`date +%Y-%m-%d_%H-%M`

TO=$TO_FOLDER/$DATE/

echo "rsync from $FROM to $TO"

# the first times rsync can take a bit long - do not disable flusing

rsync -a $FROM $TO

# now disable flushing and do one manual flushing

$SCRIPTS/es-flush-disable.sh true

$SCRIPTS/es-flush.sh

# ... and sync again

rsync -a $FROM $TO

$SCRIPTS/es-flush-disable.sh false

# now remove too old backups

rm -rf `find $TO_FOLDER -maxdepth 1 -mtime +7` &> /dev/null

E.g. you could call the backup script regularly (even hourly) from cron and it will create new backups. By the way – if you want to take a look on the settings of all indices (e.g. to check the disable flushing stuff) this might be handy:

curl -XGET 'localhost:9200/_settings?pretty=true'

MONGODB Installation Steps:

#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

To ensure database software you download is actually the software published by 10gen.

Now we have to add 10gen repository location in the system by adding the following line to the end of /etc/apt/sources.list

#nano /etc/apt/sources.list

Add this line to the bottom of the file

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

Save the file and exit

Update the repository list and install MongoDB

#sudo apt-get update

#sudo apt-get install mongodb-10gen

* This will take some time to complete depending upon your internet connection and the system performance.

When MongoDB has finished installing, run mongo just to check whether MongoDB is running correctly.

If you get the same as above it means MongoDB is successfully installed.

MongoDB configuration steps:

Now we need to create an admin user for mongo and add a user to mongo for the ‘graylog2’ collection. I like to use a Secure Password Generator (https://secure.pctools.com/guides/password/) to generate passwords.

Click on "Generate Password(s)" button for generating the number of secure Password(s).

For the sake of this documentation I have used some simple passwords.

use admin

db.addUser('admin', 'MongoDB@123')

db.auth('admin', 'MongoDB@123')

use graylog2

db.addUser('grayloguser', 'password@123')

db.auth('grayloguser', 'password@123')

Now you are ready with Elastic Search and MongoDB for installing Graylog2 Log management tool www.graylog2.org.

MongoDB connectivity problem Troubleshooting:

Error: couldn't connect to server 127.0.0.1 shell/mongo.js:79
exception: connect failed

Logs from /var/log/mongodb/mongodb.log

***** SERVER RESTARTED *****

Mon Jul 30 08:39:27
Mon Jul 30 08:39:27 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Jul 30 08:39:27
Mon Jul 30 08:39:27 [initandlisten] MongoDB starting : pid=886 port=27017 dbpath=/var/lib/mongodb 32-bit host=ubuntu-OptiPlex-990
Mon Jul 30 08:39:27 [initandlisten]
Mon Jul 30 08:39:27 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Jul 30 08:39:27 [initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Jul 30 08:39:27 [initandlisten] **       with --journal, the limit is lower
Mon Jul 30 08:39:27 [initandlisten]
Mon Jul 30 08:39:27 [initandlisten] db version v2.0.6, pdfile version 4.5
Mon Jul 30 08:39:27 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Mon Jul 30 08:39:27 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Mon Jul 30 08:39:27 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
**************
Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
*************
Mon Jul 30 08:39:27 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Mon Jul 30 08:39:27 dbexit:
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to close listening sockets...
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to flush diaglog...
Mon Jul 30 08:39:27 [initandlisten] shutdown: going to close sockets...
Mon Jul 30 08:39:27 [initandlisten] shutdown: waiting for fs preallocator...
Mon Jul 30 08:39:27 [initandlisten] shutdown: closing all files...
Mon Jul 30 08:39:27 [initandlisten] closeAllFiles() finished
Mon Jul 30 08:39:27 dbexit: really exiting now

To fix this please follow the step by step approach:
First try connecting with the below given command
1)
#mongod --config /etc/mongodb.conf --dbpath /var/lib/mongodb
Where /etc/mongodb.conf is your mongodb configuration file and /var/lib/mongodb is your database path.

If it gives the same connection error. Then remove the lock file of mongodb by issuing the command.
2)
Locate the "mongod.lock" fie
#rm -rf /var/lib/mongodb/mongod.lock

3)
Issue the Database repair command:
#mongod --repair --config /etc/mongodb.conf --dbpath /var/lib/mongodb

4)
Restart the mongod and mongo services.
#service mongod stop
#service mongod start

#service mongo stop
#service mongo start

5)
Issue the Database connecting command.
#mongod --config /etc/mongodb.conf --dbpath /var/lib/mongodb&

6)
Try connecting
root@ubuntu# mongo
MongoDB shell version: 2.0.6
connecting to: test
>
You should be Happy once you see the above prompt.

Backup and Restoration of MongoDB.
Refer the link below for the steps and methods for backup of MongoDB.
http://docs.mongodb.org/manual/administration/backups/

mongodump syntax and options:
# mongodump --help
Export MongoDB data to BSON files.

options:
--help                   produce help message
-v [ --verbose ]         be more verbose (include multiple times for more
                           verbosity e.g. -vvvvv)
--version                print the program's version and exit
-h [ --host ] arg        mongo host to connect to ( /s1,s2 for
                           sets)
--port arg               server port. Can also use --host hostname:port
--ipv6                   enable IPv6 support (disabled by default)
-u [ --username ] arg    username
-p [ --password ] arg    password
--dbpath arg             directly access mongod database files in the given
                           path, instead of connecting to a mongod server -
                           needs to lock the data directory, so cannot be used
                           if a mongod is currently accessing the same path
--directoryperdb         if dbpath specified, each db is in a separate
                           directory
--journal                enable journaling
-d [ --db ] arg          database to use
-c [ --collection ] arg collection to use (some commands)
-o [ --out ] arg (=dump) output directory or "-" for stdout
-q [ --query ] arg       json query
--oplog                  Use oplog for point-in-time snapshotting
--repair                 try to recover a crashed database
--forceTableScan         force a table scan (do not use $snapshot)

# mongorestore --help
usage: mongorestore [options] [directory or filename to restore from]
options:
--help                  produce help message
-v [ --verbose ]        be more verbose (include multiple times for more
                          verbosity e.g. -vvvvv)
--version               print the program's version and exit
-h [ --host ] arg       mongo host to connect to ( /s1,s2 for sets)
--port arg              server port. Can also use --host hostname:port
--ipv6                  enable IPv6 support (disabled by default)
-u [ --username ] arg   username
-p [ --password ] arg   password
--dbpath arg            directly access mongod database files in the given
                          path, instead of connecting to a mongod server -
                          needs to lock the data directory, so cannot be used
                          if a mongod is currently accessing the same path
--directoryperdb        if dbpath specified, each db is in a separate
                          directory
--journal               enable journaling
-d [ --db ] arg         database to use
-c [ --collection ] arg collection to use (some commands)
--objcheck              validate object before inserting
--filter arg            filter to apply before inserting
--drop                  drop each collection before import
--oplogReplay           replay oplog for point-in-time restore
--keepIndexVersion      don't upgrade indexes to newest version

eg:-
#mongodump -h 127.0.0.1 --dbpath /var/lib/mongodb -o /usr/bin/dump/2012-08-06
#mongorestore -h 127.0.0.1 /usr/bin/dump/2012-08-06

Now you are done with the Installation and configuration of Elasticsearch and MongoDB for Graylog2.

Request everyone to put a comment about this blog and give your suggestions for improvement.
--------------------------------------------------------------------------------------------------------------

Disclaimer:

I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content.

I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site.

Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law.

All product and firm names on www.erco.com are proprietary names of their corresponding owners

All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.