Graylog2 complete suite on Fedora

Installing Elasticsearch on Fedora

1) Download the latest package from the site given below:

http://www.elasticsearch.org/download/

Package name:

elasticsearch-0.1x.x.tar

2) Unzip the file and save to the folder /opt/

#tar -zxvf elasticsearch-0.19.8.tar /opt/

If the unzipped elasticsearch folder to the location “/opt/”

#mv elasticsearch-0.19.8 /opt/

3) Create a symbolic link by applying the below command (change the version as per the package)
#cd /opt

#ln -s /opt/elasticsearch-0.19.8/ elasticsearch

4) Download the service wrapper for elasticsearch from the link given below:

http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master

Save the file and unzip

5) #tar -zxvf elasticsearch-elasticsearch-servicewrapper-d47d048.tar

6) Move the folder “service” from the unzipped folder to /opt/elasticsearch-0.19.8/bin

#mv *servicewrapper*/service /opt/elasticsearch/bin/

#mv *servicewrapper*/service /opt/elasticsearch-0.19.8/bin/

7) Remove the folder for service wrapper

#rm -Rf *servicewrapper*

Configuring Elasticsearch:

#ln -s `readlink -f /opt/elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl

#sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /opt/elasticsearch/config/elasticsearch.yml

# cd /opt/elasticsearch/bin/service/

Starting Elasticsearch

#./elasticsearch start

Checking the health status of Elasticsearch Cluster:

Verifying Elasticsearch status:

#curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Output should be as given below

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "graylog2",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 1,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0

}

Elasticsearch-head

A web front end for an ElasticSearch cluster

http://mobz.github.com/elasticsearch-head

Installing and Running

There are two ways of running and installing elasticsearch-head

Running as a plugin of ElasticSearch (this is the preferred method)

If you’ve installed the .deb package, then the plugin exectuable will be available at /usr/share/elasticsearch/bin/plugin.

#sudo elasticsearch/bin/plugin -install mobz/elasticsearch-head

open http://localhost:9200/_plugin/head/

This will automatically download the latest version of elasticsearch-head from github and run it as a plugin within the elasticsearch cluster. In this mode;

elasticsearch provides a simple webserver to run head

elasticsearch-head automatically connects to the node that is running it

is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)

Will not work with elasticsearch prior to version 0.17

Running as a standalone webapp

git clone git://github.com/mobz/elasticsearch-head.git

cd elasticsearch-head

open index.html

This will manually download the latest version of elasticsearch-head from github and run it standalone. In this mode;

elasticsearch-head is running from the file system and some features will not work

may have trouble connecting to the node if the node is protected by some security measures

works with any version of elasticsearch

Alternatives

elastisearch-head is a standalone webapp written in good-old html5. This means, you can put it up on any webserver, run it directly from the filesystem, use it on an ipad, or put it on a floppy disk and carry it with you.

Mongodb installation on Fedora

Go to the link given below for installation instructions for mongodb:

http://www.mongodb.org/downloads

Go to the bottom of the page and under packages, select For Fedora and CentOS, see CentOS and Fedora packages. Click on the link

Updating yum repository for mongoDB

Create a file with the below command and paste the contents as per your platform.

#nano /etc/yum.repos.d/10gen.repo

[10gen]

name=10gen Repository

baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686

gpgcheck=0

enabled=1

Save the file and exit

Issue the command for updating the yum repository and updating the pacakages.

#yum update

#yum install mongo-10gen mongo-10gen-server

# service mongod start

# mongo

MongoDB shell version: 2.0.7

connecting to: test

Type “exit” to come out of MongoDB prompt

Check the logs in the file /var/log/mongo/mongod.log

#tail /var/log/mongo/mongod.log

#mongo

MongoDB shell version: 2.0.7

connecting to: test

>use admin

switched to db admin

> db.addUser('admin', 'M@NgoDB@123')

{ "n" : 0, "connectionId" : 2, "err" : null, "ok" : 1 }

{

"user" : "admin",

"readOnly" : false,

"pwd" : "7dc9af43dc114adaf5da03a53e72d22a",

"_id" : ObjectId("502a2f4beaf6a7c7177a4a9a")

}

Testing authentication:

> db.auth('admin', 'M@NgoDB@123')

>use graylog2

> db.addUser('grayloguser', 'Graylog2@123')

{ "n" : 0, "connectionId" : 2, "err" : null, "ok" : 1 }

{

"user" : "grayloguser",

"readOnly" : false,

"pwd" : "6be58c343ffe01fa2ea3605bff5a10bc",

"_id" : ObjectId("502a3004eaf6a7c7177a4a9b")

}

Installation of Graylog2 on Fedora

Download the latest stable package from the site given below:

https://github.com/Graylog2/graylog2-server/downloads

https://github.com/Graylog2/graylog2-web-interface/downloads

Update Java

#yum install java-1.6.0-openjdk-1.6.0.0-44.1.9.1.fc14.i686

Unzipping the files:

#tar -zxvf graylog2-server-0.9.6p1.tar.gz

Moving the folder to “/opt”

#mv graylog2-server-0.9.6p1 /opt

# tar -zxvf graylog2-web-interface-0.9.6p1.tar.gz

#mv graylog2-web-interface-0.9.6p1 /opt

#cd /opt

Creating symbolic links:

#ln -s /opt/graylog2-server-0.9.6p1/ graylog2-server

#ln -s /opt/graylog2-web-interface-0.9.6p1/ graylog2-web-interface

Change directory to Graylog2 server

#cd graylog2-server

Copy the sample graylog.conf file to /etc/

#sudo cp graylog2.conf.example /etc/graylog2.conf

Reflect the MongoDB configurations in “graylog2.conf” as per your Database settings and

configure the UDP listening port for Graylog2 in the same file .

# On which port (UDP) should we listen for Syslog messages? (Standard: 514)

syslog_listen_port = 5140

syslog_protocol = udp

# ElasticSearch URL (default: http://localhost:9200/)

elasticsearch_url = http://localhost:9200/

elasticsearch_index_name = graylog2

# Always try a reverse DNS lookup instead of parsing hostname from syslog message?

force_syslog_rdns = false

# Set time to NOW if parsing date/time from syslog message failed instead of rejecting it?

allow_override_syslog_date = true

# MongoDB Configuration

mongodb_useauth = true

mongodb_user = grayloguser

mongodb_password = Graylog2@123

mongodb_host = localhost

#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019

mongodb_database = graylog2

mongodb_port = 27017

Start the Graylog2 server with the following command:

# cd /opt/graylog2/bin/
# ./graylog2ctl start

For stopping Graylog2 issues the below given command:

# ./graylog2ctl stop

Installation of Graylog2 Web Interface:

Graylog2 Web interface is running under Ruby, we need to install the latest version of Ruby. Before starting remove all your previous Ruby installation because old release of Ruby is not compatible with Graylog2 and other dependencies.

# yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs

Install all the required development tools :

# yum install openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel

Download and Install Ruby latest version:

Browse to the below given link and as per your requirement install the latest Ruby with the methods mentioned over there. I am installing Ruby from Source code by compiling.

http://www.ruby-lang.org/en/downloads/

Package name:

ruby-1.9.3-p194.tar Stable Release as on 16-08-2012

After downloading the package unzip the Tar by issuing the below given command:

#tar –zxvf ruby-1.9.3-p194.tar

Move the ruby unzipped ruby folder to “/opt”

#mv ruby-1.9.3-p194 /opt

#cd /opt/ruby-1.9.3-p194

* How to compile and install Ruby

This is what you need to do to compile and install Ruby:

1. If ./configure does not exist or is older than configure.in,

run autoconf to (re)generate configure.

This will take some time to complete based upon your system and architecture.

2. Run ./configure, which will generate config.h and Makefile.

#./configure

Some C compiler flags may be added by default depending on your

environment. Specify optflags=.. and warnflags=.. as necessary

to override them.

3. Edit defines.h if you need. Usually this step will not be needed.

4. Remove comment mark(#) before the module names from ext/Setup (or

add module names if not present), if you want to link modules

statically.

If you don't want to compile non static extension modules

(probably on architectures which does not allow dynamic loading),

remove comment mark from the line "#option nodynamic" in

ext/Setup.

5. Run make.
#make
#make test (optional)

6. Optionally, run 'make test' to check whether the compiled Ruby

interpreter works well. If you see the message "test succeeded",

your ruby works as it should (hopefully).

7. Run 'make install'

#make install

This command will create following directories and install files

onto them.

Test the installation of Ruby

#ruby –v

#gem --version

If it gives somes notifications for installing any other packages, install those and reinstall Ruby from starting.

Update all the gems, install git and rake:

#gem update && gem install git rake

Unzip the latest stable version of Graylog2 Web Interface.

#tar –zxvf graylog2-web-interface-0.9.6p1.tar.gz

Move the unzipped folder to “/opt”

#mv graylog2-web-interface-0.9.6p1

Now change the directory to Graylog2-Web Interface

#cd /opt/graylog2-web-interface-0.9.6p1

Install Gem bundler:

#gem install bundler
#bundle install

For showing the gems installed:

#bundle show

Edit all “*.yml” configuration files in “/opt/graylog2-web-interface/config/” folder as per your requirement and environment.

Mongoid.yml

#nano /opt/graylog2-web-interface/config/mongoid.yml

# Use environment variables

production:

host: <%= ENV['MONGOID_HOST'] %>

port: <%= ENV['MONGOID_PORT'] %>

username: <%= ENV['MONGOID_USERNAME'] %>

password: <%= ENV['MONGOID_PASSWORD'] %>

database: <%= ENV['MONGOID_DATABASE'] %>

# or specify values manually
production: ###### No space before Production #########

host: localhost

port: 27017

username:

password:

database: graylog2

*** With Username and Password given Graylog2 is not able to connect.

# Values below are for development and testing.

# You may comment out them or just ignore.

development:

host: localhost

database: graylog2_development

test:

host: localhost

database: graylog2_test

email.yml

production:

via: smtp # via: sendmail

host: smtp.example.org

enable_starttls_auto: true

port: 25

auth: plain # plain, login, cram_md5 - Comment out or remove to use no auth

user: somebody

password: yourpass

domain: example.org # the HELO domain provided by the client to the server

Set the Index Name:

Index.yml

production:

url: http://localhost:9200/

index_name: graylog2

To access server Graylog2 Web interface through Apache and Passenger.
To install Passanger just run the following command :

# gem install passenger

# passenger-install-apache2-module

Welcome to the Phusion Passenger Apache 2 module installer, v3.0.15.

This installer will guide you through the entire installation process. It

shouldn't take more than 3 minutes in total.

Here's what you can expect from the installation process:

1. The Apache 2 module will be installed for you.

2. You'll learn how to configure Apache.

3. You'll learn how to deploy a Ruby on Rails application.

Don't worry if anything goes wrong. This installer will advise you on how to

solve any problems.

--------------------------------------------

The Apache 2 module was successfully installed.

Please edit your Apache configuration file, and add these lines:

LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15/ext/apache2/mod_passenger.so

PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15

PassengerRuby /usr/local/bin/ruby

After you restart Apache, you are ready to deploy any number of Ruby on Rails

applications on Apache, without any further Ruby on Rails-specific

configuration!

Press Enter to continue, or Ctrl-C to abort.

--------------------------------------------

Deploying a Ruby on Rails application: an example

Suppose you have a Rails application in /somewhere. Add a virtual host to your

Apache configuration file and set its DocumentRoot to /somewhere/public:

ServerName 10.x.x.x

# !!! Be sure to point DocumentRoot to 'public'!

DocumentRoot /opt/graylog2-web-interface/public

# This relaxes Apache security settings.

AllowOverride all

# MultiViews must be turned off.

Options -MultiViews

ErrorLog /var/log/httpd/Graylog2_error.log

LogLevel warn

CustomLog /var/log/httpd/Graylog2_access.log combined

Listen 10.x.x.x:8080

And that's it! You may also want to check the Users Guide for security and

optimization tips, troubleshooting and other useful information:

/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15/doc/Users guide Apache.html

Create a “passenger.conf” file in “/etc/httpd/conf.d/” directory and add the following entries :

LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15/ext/apache2/mod_passenger.so

PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.15

PassengerRuby /usr/local/bin/ruby

Then in your “/etc/httpd/conf/httpd.conf” file include the “passenger.conf” file.
Or check for the entry in the file as given below:

# Load config files from the config directory "/etc/httpd/conf.d".

Include conf.d/*.conf

Giving ownership for apache to Graylog2 Web Interface folders

# chown -R apache:apache /opt/graylog2-web-interface-0.9.6p1 ### Check the package name

# chown -R apache:apache /opt/graylog2-web-interface ### For the soft link

Now you will be able to connect you on your vhost, configure the Graylog2 first user and connect into the Web interface.

Graylog2 Email notification Configuration:

To get email notification for Alarms and Subscriptions you need to enable email notification in Graylog2.

Main configuration files:

/opt/graylog2-web-interface/config/general.yml

/opt/graylog2-web-interface/config/email.yml

------------------------------------------------------------------------------------------------------------

Step 1)

Editing to be done in the files according to your SMTP server and domain name:

#nano /opt/graylog2-web-interface/config/general.yml

######## Contents of the file ######################

general:

external_hostname: "your-graylog2.example.org" # Used for example to generate permalinks. Don't add 'http://' or trailing slashes.

date_format: "%d.%m.%Y - %H:%M:%S" # http://ruby-doc.org/core/classes/Time.html#M000298 (strftime syntax)

allow_deleting: false # Allowing deleting of messages negatively impacts performance

allow_version_check: true # Enables manual (/versioncheck/index) and automatic (every 30min from overview page) version checking against graylog2.org via HTTP.

# custom_cookie_name: graylog2_staging1 # Set an own cookie name - Useful for multiple deployments on same host like example.org/staging1/graylog2, example.org/staging2/grayl$

# Settings for stream subscription emails.

subscriptions:

from: graylog2subscriptions@example.org

subject: "[graylog2] Subscription"

# Settings for stream alarm emails.

streamalarms:

from: graylog2alarms@example.org

subject: "[graylog2] Stream alarm!"

hoptoad:

enabled: false

ssl: false

api_key: 123

#nano /opt/graylog2-web-interface/config/email.yml

################Copy from here and paste it in the file ############

production:

via: smtp # via: sendmail

host: 10.x.x.x

enable_starttls_auto: true

port: 25

####### If you don’t want authentication comment out the next three lines ######

# auth: plain # plain, login, cram_md5 - Comment out or remove to use no auth

# user: somebody

# password: yourpass

domain: example.org # the HELO domain provided by the client to the server

#########################################################

---------------------------------------------------------------------------------------------------------------

Step 2)

Create a script file for executing Stream Alarm check and Stream Subscription check.

#nano run_rake_tasks_Alarms.sh

###### Copy the file and paste it into the file #############

#!/bin/bash

cd /opt/graylog2-web-interface/

PATH=/usr/local/bin

rake RAILS_ENV=production streamalarms:send

########################################################

### Path of the rake command could be found by issuing the command #locate rake | more #####

#nano run_rake_tasks_ Subscriptions.sh

###### Copy the file and paste it into the file #############

#!/bin/bash

cd /opt/graylog2-web-interface/

PATH=/usr/local/bin

rake RAILS_ENV=production subscriptions:send

########################################################

--------------------------------------------------------------------------------------------------------------

Step 3)

Create a cron file for executing your scripts based on your requirement.

I am pasting the configuration for executing the scripts every one minute.

#crontab –e

######## Copy and paste the below items to the cron file bottom##########

* * * * * /opt/graylog2-web-interface/cron/run_rake_tasks_Subscription.sh

* * * * * /opt/graylog2-web-interface/cron/run_rake_tasks_Alarms.sh

--------------------------------------------------------------------------------------------------------------

Verify whether the below given screenshot is displayed in the Home screen right corner showing the services as running.

***For steps on polling network devices logs, please refer my old post.

Disclaimer:

I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content.
I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site.
Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law.
All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.

sharmith's blog

Friday, August 17, 2012

How to Install and configure Graylog2 complete suite on Fedora 14 & 17

Graylog2 complete suite on Fedora

Installing Elasticsearch on Fedora

Elasticsearch-head

Mongodb installation on Fedora

Installation of Graylog2 on Fedora

1 comment: