18 October 2020

pf (Firewall Logs) + Elasticsearch + Logstash + Kibana

  pfSense/OPNsense | Elastic Stack v7.9+ | Ubuntu 20.04+ 

pfELKCore installation. 

Ubuntu Server v20.04+

pfSense v2.4+ or OPNsense 19+

 Primary Installation Method
Scripted Install

A. Download Script

wget https://raw.githubusercontent.com/3ilson/pfelk/master/pfelk-install-6.0.sh

B. Make Script Executable

chmod +x pfelk-install-6.0.sh

C. Run Script Installer

sudo ./pfelk-install-6.0.sh

 D. Proceed to Section e below:

Alternate Installation Method
Manual Install

(a) - Preparation

a1. Configure Date/Time Zone

sudo timedatectl set-timezone EST

a2. Disable Swap

sudo swapoff -a

(b) - Prerequisites
MaxMind, apt-transport, ELK repositories, ELK, GPG signing key, Java 14 LTS 

Add MaxMind Repository

b1. Add MaxMind Repository

sudo add-apt-repository ppa:maxmind/ppa

Add Elastic Stack Repository

b2. Add Elastic Stack (Elasticsearch, Logstash and Kibana) Repository

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

b3. Download and install the public GPG signing key (Elastic Stack)

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Update System Repositories

b4. Update Repositories

sudo apt update

Install apt-transport

b5. Download and install apt-transport-https package

sudo apt install apt-transport-https

Install Java

b6. Install Java 14 LTS

sudo apt install openjdk-14-jre-headless

Install MaxMind

b7. Install MaxMind

sudo apt install geoipupdate

Install Elastic Stack

b8. Install Elastic Stack (Elasticsearch, Logstash and Kibana

sudo apt install elasticsearch; sudo apt install kibana; sudo apt install logstash

(c) - Configuration
MaxMind Kibana and Logstash 

Configure MaxMind

c1. Configure MaxMind

  • Create a MaxMind Account @ https://www.maxmind.com/en/geolite2/signup
  • Login to your MaxMind Account; navigate to "My License Key" under "Services" and Generate new license key
sudo nano /etc/GeoIP.conf
  • Modify lines 7 & 8 as follows (without < >):
AccountID <Input Your Account ID>
LicenseKey <Input Your LicenseKey>
  • Modify line 13 as follows:
EditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN
  • Modify line 18 as follows:
DatabaseDirectory /usr/share/GeoIP/

c2. Add MaxMind cron

sudo nano /etc/cron.weekly/geoipupdate
  • Add the following and save/exit (i.e. automatically update MaxMind every Sunday at 1700)
00 17 * * 0 geoipupdate

c3. Download MaxMind Databases

sudo geoipupdate

Configure Kibana

c4. Add MaxMind Repository

sudo nano /etc/kibana/kibana.yml

c5. Modify host file (/etc/kibana/kibana.yml)

server.port: 5601
server.host: ""

Configure Logstash

c6. Create the following directories

sudo mkdir /etc/logstash/conf.d/{databases,patterns,templates}

c7. Download conf.d files

sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/01-inputs.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/02-types.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/03-filter.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/05-firewall.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/10-apps.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/30-geoip.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/35-rules-desc.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/36-ports-desc.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/45-cleanup.conf -P /etc/logstash/conf.d/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/50-outputs.conf -P /etc/logstash/conf.d/

c8. Download pfelk.grok pattern

sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/patterns/pfelk.grok -P /etc/logstash/conf.d/patterns/

c9. Download pfelk databases

sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/databases/rule-names.csv -P /etc/logstash/conf.d/databases/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/databases/service-names-port-numbers.csv -P /etc/logstash/conf.d/databases/

c10. Create pfELK Directory

sudo mkdir -p /etc/pfELK/logs/

c11. Download pfELK Error Log Script (Optional)

sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/error-data.sh -P /etc/pfELK/

c12. Make pfELK Error Log Script Executable (Optional)

sudo chmod +x /etc/pfELK/error-data.sh

(d) - Configure Services
Elasticsearch, Kibana and Logstash 

d1. Automatic Start (Start Servies on Boot)

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl enable kibana.service
sudo /bin/systemctl enable logstash.service

d2. Manual Start (Start Servies Manually)

sudo -i service elasticsearch start
sudo -i service kibana start
sudo -i service logstash start

(e) - Finalize

e1. Import the required templates (NOTE: Must be done prior to receiving logs. Otherwise, you'll need to rebuild your indices)

e2. Import the saved objects (i.e. Dashboards, visualizations etc...)
  • The Saved Objects may be downloaded here (NOTE: Download v6.0, previous versions are not compatible)
  • Click on the in the upper left corner
  • Click on Stack Management located near the bottom under the Management heading
  • Click on Saved Objects located under the Kibana heading
  • Click "Import" (YouTube Tutorial Video 1 and Video 2)
e3. Configure pfSense and/or OPNsense to send logs
  •  pfSense - Navigate to Status >> System Logs [Settings] and configure as depicted below:
    • Enable Remote Logging
    • Provide "Server 1" address (this is the IP address of the ELK installation [e.g.])
    • Select "Firewall events"
  •  OPNsense - Navigate to System >> Settings >> Logging / targets and configure as depicted below:
    • The Hostname is the IP address of where you installed ELK
    • The port should be set to 5140

(f) - Troubleshooting

f1. Check Status of each process

systemctl status elasticsearch.service
systemctl status kibana.service
systemctl status logstash.service 

f2. Review Logstash Logs for errors

cat /var/log/logstash/logstash-plain.log

f3. Generate pfELK log

sudo ./etc/pfELK/error-data.sh
  • This will generate a log file within /etc/pfELK/logs/
  • Utilize the log file to aid in troubleshooting

f4. Need additional assistance, visit pfELK wiki page via GitHub

f5. Video installation tutorial via 3ilson YouTube Channel

f6. Submit an Issues via here, leave a comment below or send an email

f7. Discuss, collaborate, troubleshoot, etc... within the pfELK community on Gitter

If this helped, feel free to make a contribution:

No comments:

Post a Comment