19 November 2018

pfSense v2.4.4+|Kibana+Elasticsearch+Logstash [ELK] v6.5+|Ubuntu 18.04+



Prerequisites 
Ubuntu Server v18.04+
pfSense v2.4.4+

Navigate to the following within pfSense
Status>>System Logs [Settings]
Provide 'Server 1' address (this is the IP address of the ELK your installing - example: 192.168.1.60:5140)
Select "Firewall events"


Preparation

Add Oracle Java Repository
sudo add-apt-repository ppa:webupd8team/java

Download and install the public GPG signing key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Download and install apt-transport-https package (Debian)
sudo apt-get install apt-transport-https

Add Elasticsearch|Logstash|Kibana Repositories (version 6+
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
Update
sudo apt-get update

Install Java8 (At this time Java9 will not work with this configuration)
sudo apt-get install oracle-java8-installer


Install
Elasticsearch v6.5+ | Kibana v6.5+ | Logstash v6.5+
ELK Stack




Install Elasticsearch|Kibana|Logstash
sudo apt-get install elasticsearch && sudo apt-get install kibana && sudo apt-get install logstash

Configure Kibana|v6.5+

Configure Kibana
sudo nano /etc/kibana/kibana.yml

Amend host file (/etc/kibana/kibana.yml)
server.port: 5601
server.host: "0.0.0.0"
Configure Logstash|v6.5+

Change Directory (preparation for configuration files)
cd /etc/logstash/conf.d
Download the following configuration files
sudo wget https://raw.githubusercontent.com/a3ilson/pfelk/master/01-inputs.conf
sudo wget https://raw.githubusercontent.com/a3ilson/pfelk/master/10-syslog.conf
sudo wget https://raw.githubusercontent.com/a3ilson/pfelk/master/11-pfsense.conf
sudo wget https://raw.githubusercontent.com/a3ilson/pfelk/master/30-outputs.conf
Make Patterns Folder

sudo mkdir /etc/logstash/conf.d/patterns
Navigate to Patterns Folder

cd /etc/logstash/conf.d/patterns/
Download the following configuration file

sudo wget https://raw.githubusercontent.com/a3ilson/pfelk/master/pfsense_2_4_2.grok
Edit (10-syslog.conf)


sudo nano /etc/logstash/conf.d/10-syslog.conf
Revise/Update w/pfsense IP address (10-syslog.conf)
filter {  
  if [type] == "syslog" {
    #change to pfSense ip address
    if [host] =~ /192\.168\.1\.1/ {
      mutate {
        add_tag => ["PFSense", "Ready"]
      }
    }
    if "Ready" not in [tags] {
      mutate {
        add_tag => [ "syslog" ]
      }
    }
  }
}
filter {  
  if [type] == "syslog" {
    mutate {
      remove_tag => "Ready"
    }
  }
}
filter {  
  if "syslog" in [tags] {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM  dd HH:mm:ss" ]
      locale => "en"
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
#    if "_grokparsefailure" in [tags] {
#      drop { }
#    }
  }
}
Edit (11-pfsense.conf)
sudo nano /etc/logstash/conf.d/11-pfsense.conf
Update TimeZone (11-pfsense.conf)
//Update the timezone as needed - http://joda-time.sourceforge.net/timezones.html //
filter {  
  if "PFSense" in [tags] {
    grok {
      add_tag => [ "firewall" ]
      match => [ "message", "<(?<evtid>.*)>(?<datetime>(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\s+(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:[0-5][0-9])) (?<prog>.*?): (?<msg>.*)" ]
    }
    mutate {
      gsub => ["datetime","  "," "]
    }
    date {
      match => [ "datetime", "MMM dd HH:mm:ss" ]
      timezone => "America/New_York"
    }
    mutate {
      replace => [ "message", "%{msg}" ]
    }
    mutate {
      remove_field => [ "msg", "datetime" ]
    }
}
if [prog] =~ /^filterlog$/ {  
    mutate {
      remove_field => [ "msg", "datetime" ]
    }
    grok {
      patterns_dir => "/etc/logstash/conf.d/patterns"
      match => [ "message", "%{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}",
                 "message", "%{PFSENSE_LOG_DATA}%{PFSENSE_IPv4_SPECIFIC_DATA_ECN}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}",
                 "message", "%{PFSENSE_LOG_DATA}%{PFSENSE_IPv6_SPECIFIC_DATA}"]
    }
    mutate {
      lowercase => [ 'proto' ]
    }
    geoip {
      add_tag => [ "GeoIP" ]
      source => "src_ip"
      # Optional GeoIP database
      # Comment out the below if you do not wise to utilize and omit last three steps dealing with (recommended) suffix
      database => "/etc/logstash/GeoLite2-City.mmdb"
    }
  }
}

Download and install the MaxMind GeoIP database (recommended)
cd /etc/logstash
Download and install the MaxMind GeoIP database (recommended)
sudo wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
Download and install the MaxMind GeoIP database (recommended)
sudo gunzip GeoLite2-City.mmdb.gz
Configure Services


Automatic Start (on boot)
Start Services on Boot as Services (you'll need to reboot or start manually to proceed)
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl enable kibana.service
sudo /bin/systemctl enable logstash.service

Manual Start

Start Services Manually
sudo -i service elasticsearch start
sudo -i service kibana start
sudo -i service logstash start

Point browser to url:5601 (ex: 192.168.1.1:5601)
Select @timestamp and click 'Create'
*You may have to wait a few minutes...allowing log retrieval 


Testing/Troubleshooting



Elasticsearch
curl -X GET http://localhost:9200 { "name" : "NYLJDFe", "cluster_name" : "elasticsearch", "cluster_uuid" : "7krQg2MzR0irVJ6gNAB7fg", "version" : { "number" : "5.6.3", "build_hash" : "253032b", "build_date" : "2017-10-31T05:11:34.737Z", "build_snapshot" : false, "lucene_version" : "6.6.1" }, "tagline" : "You Know, for Search" }


Status (Elasticsearch)
systemctl status elasticsearch.service elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2017-10-31 20:53:51 EDT; 13h ago Docs: http://www.elastic.co Main PID: 6121 (java) Tasks: 74 Memory: 2.4G CPU: 7min 46.327s CGroup: /system.slice/elasticsearch.service └─6121 /usr/bin/java -Xms2g -Xmx32g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=7 Oct 31 20:53:51 logs systemd[1]: Starting Elasticsearch... Oct 31 20:53:51 logs systemd[1]: Started Elasticsearch.

Status (Kibana)
systemctl status kibana.service kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2017-10-31 20:54:09 EDT; 13h ago Main PID: 6205 (node) Tasks: 10 Memory: 82.2M CPU: 2min 51.950s CGroup: /system.slice/kibana.service └─6205 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c Oct 31 10:43:16 logs kibana[6205]: {"type":"response","@timestamp":"2017-10-031T14:43:16Z","tags":[],"pid":

Status (Logstash)
systemctl status logstash.service logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2017-10-31 08:52:27 EDT; 1h 58min ago Main PID: 32366 (java) Tasks: 43 Memory: 405.6M CPU: 4min 43.959s CGroup: /system.slice/logstash.service └─32366 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFracti Oct 31 08:52:27 logs systemd[1]: Started logstash.

Logstash Log's
/var/log/logstash
#cat/nano/vi the files within this location to view Logstash logs



TUTORIAL VIDEO


Optional
X-Pack Plugin Installation



Install x-pack plugin
sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
sudo /usr/share/logstash/bin/logstash-plugin install x-pack
sudo /usr/share/kibana/bin/kibana-plugin install x-pack

Kibana Configuration (/etc/kibana/kibana.yml)
sudo nano /etc/kibana/kibana.yml

Kibana.yml
# Uncomment and revise the following lines:
elasticsearch.username: "elastic"
elasticsearch.password: "changeme"

Logstash Configuration (/etc/logstash/logstash.yml)
sudo nano /etc/logstash/logstash.yml

Logstash.yml
# Add the following
xpack.monitoring.elasticsearch.url: "localhost:9200"
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "changeme"

Configure 30-outputs.conf (/etc/logstash/conf.d/30-outputs.conf)
output {
        elasticsearch {
              hosts => ["http://localhost:9200"]
              user => "elastic"
              password => "changeme"
              index => "logstash-%{+YYYY.MM.dd}" }
#             stdout {codex => rubydebug }
}

Restart ELK Services
systemctl restart elasticsearch.service
systemctl restart logstash.service
systemctl restart kibana.service

Changing Passwords

[Ensure You Adjust Any Changed Passwords In the Previously Configured Files]


Change Passwords
     Login to kibana (http://##.##.##.##:5601)

     Management>>Users>>

          Add New User(s)

          Modify Default Password(s)