How to send Logs from Network Device > Syslog-ng > Logstash > Graylog2

Network Device > Syslog-ng > Logstash > Graylog2

Step 1

Cisco Device configuration for enabling Logging to a syslog server:

Global configuration:

(config)#logging trap debugging

logging origin-id hostname

logging facility syslog

logging host {ip_address_of_syslog_server} transport udp port {1000}      #####custom port for UDP

logging host {ip_address_of_syslog_server} transport tcp port {1540}      #####custom port for TCP

logging on

Juniper Device configuration for enabling Logging to a syslog server:
From Home screen Configuration>Report Settings>Syslog:

Step 2

Syslog Server configuration:

Install syslog-ng on the linux

#apt-get install syslog-ng

Step 3

Configure the syslog-ng.conf file for collecting and creating log files

# locate syslog-ng.conf

#nano  /etc/syslog-ng/syslog-ng.conf                 #######Default file location

Syslog-ng configuration File:

 -------------------------------------------------------------------------------------------------------

Paste the below contents in syslog-ng.conf:

###### Source Network definition ################

source s_network_udp {

  udp(ip(0.0.0.0) port(1000));

 };

source s_network_tcp {

  tcp(ip(0.0.0.0) port(1000) max-connections(5000));

};

########## Destination file Definition #########

########## Create the path if not already there /var/log/HOSTS/ #########

destination d_clients { file("/var/log/HOSTS/$HOST"); };

######### Collecting logs from devices and writing to files ##########

##### Source:s_network_udp and destination:d_clients defined in earlier steps

log { source(s_network_udp); destination(d_clients); };

-----------------------------------------------------------------------------------------------------------

Save the syslog-ng.conf and restart the service.

# service syslog-ng start | Restart

_____________________________________________________
Above step will create log entry files for all the devices sending syslog messages on port 1000 to the directory /var/log/HOSTS/

Making directory:

# mkdir /var/log/HOSTS/

# chmod XXX /var/log/HOSTS/

Verify it with command:

# ls -ll /var/log/HOSTS

eg:- /var/log/HOSTS/x.x.x.x.log
      /var/log/HOSTS/hostname.log

Step 4

Download latest Logstash file logstash-x.x.x-monolithic.jar from the site http://logstash.net/

Step 5

Create a new configuration file  "mylogstash.conf". The configuration file name could be anything eg:-sharmith.conf :)

Logstash configuration:

Logstash configuration file Name: mylogstash.conf

# vi mylogstash.conf

-----------------------------------------------------------------------------------------------------------

###INPUT###

input {

file {

 type => "Network"

 path => [ "/var/log/HOSTS/**/*" ]

 }

}

######## For reflecting the real Hostnames in Graylog2 interface ##########

filter {

  grok { match => [ "@source_path", "/var/log/HOSTS/%{HOSTNAME:host}" ] }

  mutate { replace => [ "@source_host", "%{host}" ] }

}

output {

  stdout {

  }

 elasticsearch {

      embedded => true

  }

  gelf {

    chunksize => 1420

#    facility => "logstash-gelf"              #########Default Setting ##########

    facility => "%{@type}"

    host => "127.0.0.1"

#    level => "INFO"                             #########Default Setting ##########

    level => "%{level}"                        

    port => 12201

    sender => "%{@source_host}"

  }

}

---------------------------------------------------------------------------------------------------------------

Step 6

Save the configuration and execute

# java -jar logstash-1.1.0-monolithic.jar agent -f mylogstash.conf -- web --backend elasticsearch:///?local&

To check the status of logstash service and start it automatically if it is not running.

1)      Create a Directory called "cron" in the folder /opt/logstash

               #mkdir cron

2)      Create a script file named "logstash_check.sh" inside directory "/opt/logstash/cron"

#nano logstash_check.sh

#### Paste the below portion to the file. Save it and exit. ##########

#!/bin/bash

cd /opt/logstash

PATH=/usr/bin:/bin:/opt/logstash:/usr/sbin/:/sbin

clear

echo "Checking Logstash service"

while true

do

    sleep 5  # check the process every 5 seconds

echo "Sleep timer passed"

    if [ `pgrep -c -f logstash-1.1.1-monolithic.jar` = 0 ]

echo "pgrep command executed"

    then

        echo "problem!"

        sendmail sharmithr@fss.co.in < /opt/logstash/email_text.txt

         java -jar logstash-1.1.1-monolithic.jar agent -f mylogstash.conf -- web --backend elasticsearch:///?local&

        echo "Mail sent to administrator"

        break

        echo "After Break"

        killall logstash_check.sh

else

         echo "Logstash working"

                killall logstash_check.sh

   fi

done

3)      check the script by running it manually

#./logstash_check.sh

In the paths directory mention the location of the commands by using the "locate" command.

eg:- locate echo

                 locate sendmail

                 locate logstash-1.1.1-monolithic.jar

                locate killall

4)      Issue the command on the command prompt before creating the file. This will show the status output.

# pgrep -c -f logstash-1.1.1-monolithic.jar

1

5)      Make the script run in the background automatically with the help of cron process.

#crontab -e

#### Paste the below portion to the file. Save it and exit. ##########

30 8 * * *       /opt/logstash/cron/logstash_check.sh

***I have mentioned those “echo” commands for verifying the working of the shell script. Upon your requirement “echo” commands could be edited or removed.

Logstash Tips:

http://petes-brain.com/2011/12/my-logging-setup-rsyslog-logstash-and-graylog2/

http://sphughes.com/2012/01/01/a-more-secure-logstash-install/

Step 7

Graylog2 Server, Graylog2 Web Interface, Mongo DB and Elastic search configuration.

Refer the Step by Step Guide mentioned in the below links.

http://sharmith.blogspot.in/2012/08/installing-elasticsearch-on-fedora-1.html 
Graylog2 on Fedora

http://blog.dean.io/2012/04/getting-started-with-graylog2-for-logging-updated-for-0-9-6/

http://blog.milford.io/2012/03/installing-graylog2-0-9-6-elasticsearch-0-18-7-mongodb-2-0-3-on-centos-5-with-rvm/

#nano /opt/graylog2-web-interface/config/mongoid.yml

Make sure that there is no space before the "production:" Space will lead to errors.

Creating Virtual Host in Apache 2

Paste the below lines in /etc/apache2/apache2.conf at the bottom of the file.

###############

   LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.$

   PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.12

   PassengerRuby /usr/local/bin/ruby

################

Create a file called graylog2 in /etc/apache2/sites-available/ folder

Add the below lines as per your custom port in /etc/apache2/ports.conf file:

-----------------------------------------------------------------------------

NameVirtualHost *:8080          ######## Custom Port for Graylog2 #######

Listen 8080

 ----------------------------------------------------------------------------

Restart Apache2 service

# service apache2 restart | start

For verifying whether the logs are coming to your server, run the below command: tcpdump package required.
 Troubleshooting  & fine tuning tips:

#tcpdump dst port {graylog2_Listen_port}

#tcpdump dst port 514

To make Graylog2 listen to custom port:
Required if you are running multiple services on the same server.

First verify the port status.

# ps aux | grep {port}

If there are no services running on the specific port, configure the port in /etc/graylog2.conf file as given below:

------------------------------------------------------------------------------------------------

# On which port (UDP) should we listen for Syslog messages? (Standard: 514)

syslog_listen_port =xxx

#syslog_listen_port = 5400         ###########example configuration

syslog_protocol = udp

------------------------------------------------------------------------------------------------

Restart Graylog2 Service

# service graylog2-server restart | start | stop

Any queries or difficulties in configuring this please feel free to send a note to sharmith@gmail.com. Enjoy :)

I got an error message while trying to restart Graylog2 server after a system reboot. It gave me the error message "FATAL: org.graylog2.Main - Could not start syslog server core thread. Do you have permissions to listen on port 514?"
I checked whether any other ports are listening to that port or not but found none and tried changing the port to another but with no luck.

Fixed the issue by killing the java process.
#ps aux | grep java
It listed a Java process as given below
root      1469  1.0  1.9 1049788 62860 ?       Sl   10:10   0:19 /usr/bin/java -jar /opt/graylog2-server/graylog2-server.jar

Issued the command "kill 1469" to terminate the process and started the graylog2-server process and it worked fine.

Graylog2 Email notification Configuration:

To get email notification for Alarms and Subscriptions you need to enable email notification in Graylog2.

Main configuration files:

/opt/graylog2-web-interface/config/general.yml

/opt/graylog2-web-interface/config/email.yml

 ------------------------------------------------------------------------------------------------------------

Step 1)

Editing to be done in the files according to your SMTP server and domain name:

#nano /opt/graylog2-web-interface/config/general.yml

######## Contents of the file ######################

general:

  external_hostname: "your-graylog2.example.org" # Used for example to generate permalinks. Don't add 'http://' or trailing slashes.

  date_format: "%d.%m.%Y - %H:%M:%S" # http://ruby-doc.org/core/classes/Time.html#M000298 (strftime syntax)

  allow_deleting: false # Allowing deleting of messages negatively impacts performance

  allow_version_check: true # Enables manual (/versioncheck/index) and automatic (every 30min from overview page) version checking against graylog2.org via HTTP.

  # custom_cookie_name: graylog2_staging1 # Set an own cookie name - Useful for multiple deployments on same host like example.org/staging1/graylog2, example.org/staging2/grayl$

# Settings for stream subscription emails.

subscriptions:

  from: graylog2subscriptions@example.org

  subject: "[graylog2] Subscription"

# Settings for stream alarm emails.

streamalarms:

  from: graylog2alarms@example.org

  subject: "[graylog2] Stream alarm!"

hoptoad:

  enabled: false

  ssl: false

  api_key: 123

 

#nano /opt/graylog2-web-interface/config/email.yml

################Copy from here and paste it in the file ############

production:

  via: smtp # via: sendmail

  host: 10.x.x.x

  enable_starttls_auto: true

  port: 25

####### If you don’t want authentication comment out the next three lines ######

#  auth: plain # plain, login, cram_md5 - Comment out or remove to use no auth

#  user: somebody

#  password: yourpass

  domain: example.org # the HELO domain provided by the client to the server

#########################################################

---------------------------------------------------------------------------------------------------------------

Step 2)

Create a script file for executing Stream Alarm check and Stream Subscription check.

#nano run_rake_tasks_Alarms.sh

###### Copy the file and paste it into the file #############

#!/bin/bash

cd /opt/graylog2-web-interface/

PATH=/usr/local/bin

rake RAILS_ENV=production streamalarms:send

########################################################

### Path of the rake command could be found by issuing the command

#locate rake | more

#nano run_rake_tasks_ Subscriptions.sh

###### Copy the file and paste it into the file #############

#!/bin/bash

cd /opt/graylog2-web-interface/

PATH=/usr/local/bin

rake RAILS_ENV=production subscriptions:send

########################################################

 --------------------------------------------------------------------------------------------------------------

Step 3)

Create a cron file for executing your scripts based on your requirement.

I am pasting the configuration for executing the scripts every one minute.

#crontab –e

######## Copy and paste the below items to the cron file bottom##########

* * * * *       /opt/graylog2-web-interface/cron/run_rake_tasks_Subscription.sh

* * * * *       /opt/graylog2-web-interface/cron/run_rake_tasks_Alarms.sh

 --------------------------------------------------------------------------------------------------------------

Verify whether the below given screenshot is displayed in the Home screen right corner showing the services as running.

Request everyone to put a comment about this blog and give your suggestions for improvement.
 --------------------------------------------------------------------------------------------------------------


Disclaimer:

I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content.

I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site.

Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law.

All product and firm names on www.erco.com are proprietary names of their corresponding owners

 All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.

All rights are reserved which are not explicitly granted here.

All product and firm names on www.erco.com are proprietary names of their corresponding owners

All product and firm names on www.erco.com are proprietary names of their corresponding owners