pf (Firewall Logs) + Elasticsearch + Logstash + Kibana

  pfSense/OPNsense | Elastic Stack v7.9+ | Ubuntu 20.04+ 

pfELKCore installation. 

Requirements
Ubuntu Server v20.04+
pfSense v2.4+ or OPNsense 19+

 Primary Installation Method
Scripted Install

- YouTube Tutorial

A. Download Script

wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-dashboard-installer.sh

B. Make Script Executable

chmod +x pfelk-installer.sh

C. Run Script Installer

sudo ./pfelk-installer.sh

 D. Proceed to Section e3 below:

Alternate Installation Method
Manual Install

- YouTube Tutorial

(a) - Preparation

a1. Configure Date/Time Zone

sudo timedatectl set-timezone EST

a2. Disable Swap

sudo swapoff -a

(b) - Prerequisites
MaxMind (optional), apt-transport, ELK repositories, ELK, GPG signing key, Java 14 LTS 

Add Elastic Stack Repository

b1. Add Elastic Stack (Elasticsearch, Logstash and Kibana) Repository

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

b2. Download and install the public GPG signing key (Elastic Stack)

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Update System Repositories

b3. Update Repositories

sudo apt update

Install apt-transport

b4. Download and install apt-transport-https package

sudo apt install apt-transport-https

Install Elastic Stack 

b5. Install Elastic Stack (Elasticsearch, Logstash and Kibana

sudo apt install elasticsearch; sudo apt install kibana; sudo apt install logstash

Configure Kibana

c1. Configure Kibana

sudo nano /etc/kibana/kibana.yml

c2. Modify host file (/etc/kibana/kibana.yml)

server.port: 5601
server.host: "0.0.0.0"

Configure Logstash

c3. Download the pipelines.yml

sudo wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/logstash/pipelines.yml -P /etc/logstash/

c4. Create the following directories

sudo mkdir -p /etc/pfelk/{conf.d,config,logs,databases,patterns,scripts,templates}

c5. Download conf.d files

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/01-inputs.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/02-types.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/03-filter.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/05-apps.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/20-interfaces.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/30-geoip.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/35-rules-desc.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/36-ports-desc.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/45-cleanup.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/50-outputs.conf -P /etc/pfelk/conf.d/

c6. Download pfelk.grok pattern

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/pfelk.grok -P /etc/pfelk/patterns/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/openvpn.grok -P /etc/pfelk/patterns/

c7. Download pfelk databases

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/private-hostnames.csv -P /etc/pfelk/databases/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/rule-names.csv -P /etc/pfelk/databases/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/service-names-port-numbers.csv -P /etc/pfelk/databases/

c8. Download pfELK Error Log Script (Optional)

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main
/etc/pfelk/scripts/error-data.sh -P /etc/pfelk/scripts/

c9. Make pfELK Error Log Script Executable (Optional)

sudo chmod +x /etc/pfelk/scripts/error-data.sh

(d) - Configure Services
Elasticsearch, Kibana and Logstash 

d1. Automatic Start (Start Servies on Boot)

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl enable kibana.service
sudo /bin/systemctl enable logstash.service

d2. Manual Start (Start Servies Manually)

sudo -i service elasticsearch start
sudo -i service kibana start
sudo -i service logstash start

(e) - Finalize
pfELK 

- YouTube Tutorial

e1. Import the required templates (NOTE: Must be done prior to receiving logs. Otherwise, you'll need to rebuild your indices)

• Click ☰ in the upper left corner
• Click on Dev Tools located near the bottom under the Management heading
• Paste the contents of each file located here (repeat for each file) or the following:
• Component Templates
• pfelk-settings and pfelk-mappings-ecs must be installed first as the subsequent index patterns are dependent upon these Component Templates.
• pfelk-settings - Must be installed first
• pfelk-mappings-ecs - Must be installed second
• pfelk-ilm 
• Index Templates
• pfelk
• pfelk-dhcp
• pfelk-haproxy
• pfelk-suricata
• Click the green triangle after pasting the contents (one at a time) into the console

 

 

e2. Import the saved objects (i.e. Dashboards, visualizations etc...)

• The Saved Objects may be downloaded here 
• Click on the ☰ in the upper left corner
• Click on Stack Management located near the bottom under the Management heading
• Click on Saved Objects located under the Kibana heading
• Click "Import" (YouTube Tutorial Video 1 and Video 2)

e3. Configure pfSense and/or OPNsense to send logs

•  pfSense - Navigate to Status >> System Logs [Settings] and configure as depicted below:
  

 

• Enable Remote Logging
  
• Provide "Server 1" address (this is the IP address of the ELK installation [e.g. 192.168.1.60:5140])
• Select "Firewall events"
  

•  OPNsense - Navigate to System >> Settings >> Logging / targets and configure as depicted below:

 

• The Hostname is the IP address of where you installed ELK
• The port should be set to 5140
  

(f) - Troubleshooting
pfELK

f1. Check Status of each process

systemctl status elasticsearch.service

systemctl status kibana.service

systemctl status logstash.service 

f2. Review Logstash Logs for errors

cat /var/log/logstash/logstash-plain.log

f3. Generate pfELK log

sudo ./etc/pfELK/error-data.sh

• This will generate a log file within /etc/pfELK/logs/
• Utilize the log file to aid in troubleshooting
  

f4. Need additional assistance, visit pfELK wiki page via GitHub

f5. Video installation tutorial via 3ilson YouTube Channel

f6. Submit an Issues via here, leave a comment below or send an email

f7. Discuss, collaborate, troubleshoot, etc... within the pfELK community on Gitter

If this helped, feel free to make a contribution: