Monday, May 28, 2012

How to send Logs from Network Device > Syslog-ng > Logstash > Graylog2


Network Device > Syslog-ng > Logstash > Graylog2




Step 1

Cisco Device configuration for enabling Logging to a syslog server:
Global configuration:
(config)#logging trap debugging
logging origin-id hostname
logging facility syslog
logging host {ip_address_of_syslog_server} transport udp port {1000}      #####custom port for UDP
logging host {ip_address_of_syslog_server} transport tcp port {1540}      #####custom port for TCP
logging on

Juniper Device configuration for enabling Logging to a syslog server:
From Home screen Configuration>Report Settings>Syslog:


Step 2

Syslog Server configuration:


Install syslog-ng on the linux
#apt-get install syslog-ng

Step 3


Configure the syslog-ng.conf file for collecting and creating log files
# locate syslog-ng.conf
#nano  /etc/syslog-ng/syslog-ng.conf                 #######Default file location

Syslog-ng configuration File:
 -------------------------------------------------------------------------------------------------------
Paste the below contents in syslog-ng.conf:
###### Source Network definition ################
source s_network_udp {
  udp(ip(0.0.0.0) port(1000));
 };
source s_network_tcp {
  tcp(ip(0.0.0.0) port(1000) max-connections(5000));
};
########## Destination file Definition #########
########## Create the path if not already there /var/log/HOSTS/ #########
destination d_clients { file("/var/log/HOSTS/$HOST"); };
######### Collecting logs from devices and writing to files ##########
##### Source:s_network_udp and destination:d_clients defined in earlier steps
log { source(s_network_udp); destination(d_clients); };
-----------------------------------------------------------------------------------------------------------
Save the syslog-ng.conf and restart the service.
# service syslog-ng start | Restart
_____________________________________________________
Above step will create log entry files for all the devices sending syslog messages on port 1000 to the directory /var/log/HOSTS/

Making directory:
# mkdir /var/log/HOSTS/
# chmod XXX /var/log/HOSTS/

Verify it with command:
# ls -ll /var/log/HOSTS
eg:- /var/log/HOSTS/x.x.x.x.log
      /var/log/HOSTS/hostname.log

Step 4

Download latest Logstash file logstash-x.x.x-monolithic.jar from the site http://logstash.net/


Step 5

Create a new configuration file  "mylogstash.conf". The configuration file name could be anything eg:-sharmith.conf :)

Logstash configuration:
Logstash configuration file Name: mylogstash.conf
# vi mylogstash.conf
-----------------------------------------------------------------------------------------------------------
###INPUT###
input {
file {
 type => "Network"
 path => [ "/var/log/HOSTS/**/*" ]
 }
}
######## For reflecting the real Hostnames in Graylog2 interface ##########
filter {
  grok { match => [ "@source_path", "/var/log/HOSTS/%{HOSTNAME:host}" ] }
  mutate { replace => [ "@source_host", "%{host}" ] }
}
output {
  stdout {
  }

 elasticsearch {
      embedded => true
  }
  gelf {
    chunksize => 1420
#    facility => "logstash-gelf"              #########Default Setting ##########
    facility => "%{@type}"
    host => "127.0.0.1"
#    level => "INFO"                             #########Default Setting ##########
    level => "%{level}"                        
    port => 12201
    sender => "%{@source_host}"
  }
}
---------------------------------------------------------------------------------------------------------------

Step 6


Save the configuration and execute
# java -jar logstash-1.1.0-monolithic.jar agent -f mylogstash.conf -- web --backend elasticsearch:///?local&

To check the status of logstash service and start it automatically if it is not running.

1)      Create a Directory called "cron" in the folder /opt/logstash
               #mkdir cron

2)      Create a script file named "logstash_check.sh" inside directory "/opt/logstash/cron"
#nano logstash_check.sh
#### Paste the below portion to the file. Save it and exit. ##########
#!/bin/bash
cd /opt/logstash
PATH=/usr/bin:/bin:/opt/logstash:/usr/sbin/:/sbin
clear
echo "Checking Logstash service"

while true
do
    sleep 5  # check the process every 5 seconds
echo "Sleep timer passed"
    if [ `pgrep -c -f logstash-1.1.1-monolithic.jar` = 0 ]
echo "pgrep command executed"
    then
        echo "problem!"
        sendmail sharmithr@fss.co.in < /opt/logstash/email_text.txt
         java -jar logstash-1.1.1-monolithic.jar agent -f mylogstash.conf -- web --backend elasticsearch:///?local&
        echo "Mail sent to administrator"
        break
        echo "After Break"
        killall logstash_check.sh
else
         echo "Logstash working"
                killall logstash_check.sh
   fi
done

3)      check the script by running it manually
#./logstash_check.sh
In the paths directory mention the location of the commands by using the "locate" command.
eg:- locate echo
                 locate sendmail
                 locate logstash-1.1.1-monolithic.jar
                locate killall
4)      Issue the command on the command prompt before creating the file. This will show the status output.
# pgrep -c -f logstash-1.1.1-monolithic.jar
1


5)      Make the script run in the background automatically with the help of cron process.

#crontab -e

#### Paste the below portion to the file. Save it and exit. ##########
30 8 * * *       /opt/logstash/cron/logstash_check.sh



***I have mentioned those “echo” commands for verifying the working of the shell script. Upon your requirement “echo” commands could be edited or removed.

Logstash Tips:

Step 7

Graylog2 Server, Graylog2 Web Interface, Mongo DB and Elastic search configuration.
Refer the Step by Step Guide mentioned in the below links.

http://sharmith.blogspot.in/2012/08/installing-elasticsearch-on-fedora-1.html 
Graylog2 on Fedora


#nano /opt/graylog2-web-interface/config/mongoid.yml
Make sure that there is no space before the "production:" Space will lead to errors.



Creating Virtual Host in Apache 2

Paste the below lines in /etc/apache2/apache2.conf at the bottom of the file.
###############

   LoadModule passenger_module /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.$
   PassengerRoot /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.12
   PassengerRuby /usr/local/bin/ruby
################

Create a file called graylog2 in /etc/apache2/sites-available/ folder


Add the below lines as per your custom port in /etc/apache2/ports.conf file:
-----------------------------------------------------------------------------
NameVirtualHost *:8080          ######## Custom Port for Graylog2 #######
Listen 8080
 ----------------------------------------------------------------------------
Restart Apache2 service
# service apache2 restart | start


For verifying whether the logs are coming to your server, run the below command: tcpdump package required.
 Troubleshooting  & fine tuning tips:
#tcpdump dst port {graylog2_Listen_port}
#tcpdump dst port 514

To make Graylog2 listen to custom port:
Required if you are running multiple services on the same server.

First verify the port status.
# ps aux | grep {port}
If there are no services running on the specific port, configure the port in /etc/graylog2.conf file as given below:
------------------------------------------------------------------------------------------------
# On which port (UDP) should we listen for Syslog messages? (Standard: 514)
syslog_listen_port =xxx
#syslog_listen_port = 5400         ###########example configuration
syslog_protocol = udp
------------------------------------------------------------------------------------------------

Restart Graylog2 Service
# service graylog2-server restart | start | stop


Any queries or difficulties in configuring this please feel free to send a note to sharmith@gmail.com. Enjoy :)

I got an error message while trying to restart Graylog2 server after a system reboot. It gave me the error message "FATAL: org.graylog2.Main - Could not start syslog server core thread. Do you have permissions to listen on port 514?"
I checked whether any other ports are listening to that port or not but found none and tried changing the port to another but with no luck.

Fixed the issue by killing the java process.
#ps aux | grep java
It listed a Java process as given below
root      1469  1.0  1.9 1049788 62860 ?       Sl   10:10   0:19 /usr/bin/java -jar /opt/graylog2-server/graylog2-server.jar

Issued the command "kill 1469" to terminate the process and started the graylog2-server process and it worked fine.

Graylog2 Email notification Configuration:
To get email notification for Alarms and Subscriptions you need to enable email notification in Graylog2.
Main configuration files:
/opt/graylog2-web-interface/config/general.yml
/opt/graylog2-web-interface/config/email.yml
 ------------------------------------------------------------------------------------------------------------

Step 1)

Editing to be done in the files according to your SMTP server and domain name:
#nano /opt/graylog2-web-interface/config/general.yml
######## Contents of the file ######################
general:
  external_hostname: "your-graylog2.example.org" # Used for example to generate permalinks. Don't add 'http://' or trailing slashes.
  date_format: "%d.%m.%Y - %H:%M:%S" # http://ruby-doc.org/core/classes/Time.html#M000298 (strftime syntax)
  allow_deleting: false # Allowing deleting of messages negatively impacts performance
  allow_version_check: true # Enables manual (/versioncheck/index) and automatic (every 30min from overview page) version checking against graylog2.org via HTTP.
  # custom_cookie_name: graylog2_staging1 # Set an own cookie name - Useful for multiple deployments on same host like example.org/staging1/graylog2, example.org/staging2/grayl$

# Settings for stream subscription emails.
subscriptions:
  from: graylog2subscriptions@example.org
  subject: "[graylog2] Subscription"

# Settings for stream alarm emails.
streamalarms:
  from: graylog2alarms@example.org
  subject: "[graylog2] Stream alarm!"

hoptoad:
  enabled: false
  ssl: false
  api_key: 123

 
#nano /opt/graylog2-web-interface/config/email.yml
################Copy from here and paste it in the file ############
production:
  via: smtp # via: sendmail
  host: 10.x.x.x
  enable_starttls_auto: true
  port: 25
####### If you don’t want authentication comment out the next three lines ######
#  auth: plain # plain, login, cram_md5 - Comment out or remove to use no auth
#  user: somebody
#  password: yourpass
  domain: example.org # the HELO domain provided by the client to the server
#########################################################
---------------------------------------------------------------------------------------------------------------

Step 2)


Create a script file for executing Stream Alarm check and Stream Subscription check.
#nano run_rake_tasks_Alarms.sh
###### Copy the file and paste it into the file #############
#!/bin/bash
cd /opt/graylog2-web-interface/
PATH=/usr/local/bin
rake RAILS_ENV=production streamalarms:send

########################################################

### Path of the rake command could be found by issuing the command
#locate rake | more
#nano run_rake_tasks_ Subscriptions.sh
###### Copy the file and paste it into the file #############
#!/bin/bash
cd /opt/graylog2-web-interface/
PATH=/usr/local/bin
rake RAILS_ENV=production subscriptions:send
########################################################
 --------------------------------------------------------------------------------------------------------------

Step 3)

Create a cron file for executing your scripts based on your requirement.
I am pasting the configuration for executing the scripts every one minute.
#crontab –e
######## Copy and paste the below items to the cron file bottom##########

* * * * *       /opt/graylog2-web-interface/cron/run_rake_tasks_Subscription.sh
* * * * *       /opt/graylog2-web-interface/cron/run_rake_tasks_Alarms.sh
 --------------------------------------------------------------------------------------------------------------
Verify whether the below given screenshot is displayed in the Home screen right corner showing the services as running.


Request everyone to put a comment about this blog and give your suggestions for improvement.
 --------------------------------------------------------------------------------------------------------------


Disclaimer:
I cannot assume any liability for the content of external pages. Solely the operators of those linked pages are responsible for their content.
I make every reasonable effort to ensure that the content of this Web site is kept up to date, and that it is accurate and complete. Nevertheless, the possibility of errors cannot be entirely ruled out. I do not give any warranty in respect of the timeliness, accuracy or completeness of material published on this Web site, and disclaim all liability for (material or non-material) loss or damage incurred by third parties arising from the use of content obtained from the Web site.
Registered trademarks and proprietary names, and copyrighted text and images, are not generally indicated as such on my Web pages. But the absence of such indications in no way implies the these names, images or text belong to the public domain in the context of trademark or copyright law.
All product and firm names on www.erco.com are proprietary names of their corresponding owners
 All products and firm names used in this site are proprietary names of their corresponding owners. All rights are reserved which are not explicitly granted here.
All rights are reserved which are not explicitly granted here.
All product and firm names on www.erco.com are proprietary names of their corresponding owners

All product and firm names on www.erco.com are proprietary names of their corresponding owners

17 comments:

  1. Logstash automatic startup script added

    ReplyDelete
  2. Solution for Error" FATAL: org.graylog2.Main - Could not start syslog server core thread. Do you have permissions to listen on port 514?"

    ReplyDelete
  3. Script for Email notification for Stream Alarms and Stream Alerts added

    ReplyDelete
    Replies
    1. I have done properly as per your instructions but still I am not able to get Email notifications.

      Delete
  4. Steps for verifying whether traffic is reaching the Graylog2 server added

    ReplyDelete
  5. Juniper and Cisco device syslog configuration added

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Elastic Search and MongoDB installation steps link
    http://sharmith.blogspot.in/2012/06/installation-of-elastic-search-and.html

    ReplyDelete
  8. Graylog2 complete Suite on Fedora
    http://sharmith.blogspot.in/2012/08/installing-elasticsearch-on-fedora-1.html

    ReplyDelete
  9. This is great, thank you for you work!

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Great help!

    But now i want to install rsyslog and graylog2 on the same server, is this possible?
    I has centralize loging including: rsyslog 5.8.7 + loganalyzer 3.4.3 is fine.
    But now i want to change frontend for system by use only graylog2-web-interface replaced for loganalyzer, can i do it?

    Regards,

    ReplyDelete
  12. Hi, thanks for this got mine going well with your instructions. However like you I too would like to create custom grok patterns for cisco, apache etc, as you mentioned and posted on Grokbase. I was wondering if you have had any success on this, or any tips for creating them?

    thanks!

    ReplyDelete
  13. No, it's my question
    If any one has already system as i mentioned, please share your experience

    ReplyDelete
  14. is it possible to have network device > syslog ng > logstash > graylog2 but all three run on one debain 6 box?

    ReplyDelete
  15. I have install Graylog2 server but not able to fetch network device logs,can anyone help me in this regards...??

    ReplyDelete