18 October 2020

pf (Firewall Logs) + Elasticsearch + Logstash + Kibana

  pfSense/OPNsense | Elastic Stack v7.9+ | Ubuntu 20.04+ 

pfELKCore installation. 

Requirements
Ubuntu Server v20.04+

pfSense v2.4+ or OPNsense 19+




 Primary Installation Method
Scripted Install

A. Download Script

wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-dashboard-installer.sh

B. Make Script Executable

chmod +x pfelk-installer.sh

C. Run Script Installer

sudo ./pfelk-installer.sh

 D. Proceed to Section e3 below:

Alternate Installation Method
Manual Install

(a) - Preparation

a1. Configure Date/Time Zone

sudo timedatectl set-timezone EST

a2. Disable Swap

sudo swapoff -a

(b) - Prerequisites
MaxMind (optional), apt-transport, ELK repositories, ELK, GPG signing key, Java 14 LTS 

Add Elastic Stack Repository

b1. Add Elastic Stack (Elasticsearch, Logstash and Kibana) Repository

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

b2. Download and install the public GPG signing key (Elastic Stack)

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Update System Repositories

b3. Update Repositories

sudo apt update

Install apt-transport

b4. Download and install apt-transport-https package

sudo apt install apt-transport-https

Install Elastic Stack 

b5. Install Elastic Stack (Elasticsearch, Logstash and Kibana

sudo apt install elasticsearch; sudo apt install kibana; sudo apt install logstash

Configure Kibana

c1. Configure Kibana

sudo nano /etc/kibana/kibana.yml

c2. Modify host file (/etc/kibana/kibana.yml)

server.port: 5601
server.host: "0.0.0.0"

Configure Logstash

c3. Download the pipelines.yml

sudo wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/logstash/pipelines.yml -P /etc/logstash/

c4. Create the following directories

sudo mkdir -p /etc/pfelk/{conf.d,config,logs,databases,patterns,scripts,templates}

c5. Download conf.d files

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/01-inputs.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/02-types.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/03-filter.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/05-apps.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/20-interfaces.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/30-geoip.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/35-rules-desc.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/36-ports-desc.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/45-cleanup.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/50-outputs.conf -P /etc/pfelk/conf.d/

c6. Download pfelk.grok pattern

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/pfelk.grok -P /etc/pfelk/patterns/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/openvpn.grok -P /etc/pfelk/patterns/

c7. Download pfelk databases

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/private-hostnames.csv -P /etc/pfelk/databases/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/rule-names.csv -P /etc/pfelk/databases/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/service-names-port-numbers.csv -P /etc/pfelk/databases/

c8. Download pfELK Error Log Script (Optional)

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main
/etc/pfelk/scripts/error-data.sh -P /etc/pfelk/scripts/

c9. Make pfELK Error Log Script Executable (Optional)

sudo chmod +x /etc/pfelk/scripts/error-data.sh

(d) - Configure Services
Elasticsearch, Kibana and Logstash 

d1. Automatic Start (Start Servies on Boot)

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl enable kibana.service
sudo /bin/systemctl enable logstash.service

d2. Manual Start (Start Servies Manually)

sudo -i service elasticsearch start
sudo -i service kibana start
sudo -i service logstash start

(e) - Finalize
pfELK 

e1. Import the required templates (NOTE: Must be done prior to receiving logs. Otherwise, you'll need to rebuild your indices)

  • Click in the upper left corner
  • Click on Dev Tools located near the bottom under the Management heading
    • Paste the contents of each file located here (repeat for each file) or the following:
 
e2. Import the saved objects (i.e. Dashboards, visualizations etc...)
  • The Saved Objects may be downloaded here 
  • Click on the in the upper left corner
  • Click on Stack Management located near the bottom under the Management heading
  • Click on Saved Objects located under the Kibana heading
  • Click "Import" (YouTube Tutorial Video 1 and Video 2)
e3. Configure pfSense and/or OPNsense to send logs
  •  pfSense - Navigate to Status >> System Logs [Settings] and configure as depicted below:
 
    • Enable Remote Logging
    • Provide "Server 1" address (this is the IP address of the ELK installation [e.g. 192.168.1.60:5140])
    • Select "Firewall events"
  •  OPNsense - Navigate to System >> Settings >> Logging / targets and configure as depicted below:
 
    • The Hostname is the IP address of where you installed ELK
    • The port should be set to 5140

(f) - Troubleshooting
pfELK

f1. Check Status of each process

systemctl status elasticsearch.service
systemctl status kibana.service
systemctl status logstash.service 

f2. Review Logstash Logs for errors

cat /var/log/logstash/logstash-plain.log

f3. Generate pfELK log

sudo ./etc/pfELK/error-data.sh
  • This will generate a log file within /etc/pfELK/logs/
  • Utilize the log file to aid in troubleshooting

f4. Need additional assistance, visit pfELK wiki page via GitHub

f5. Video installation tutorial via 3ilson YouTube Channel

f6. Submit an Issues via here, leave a comment below or send an email

f7. Discuss, collaborate, troubleshoot, etc... within the pfELK community on Gitter


If this helped, feel free to make a contribution:
Donate

5 comments:

  1. Hi there is it possible to have multi pfSense sending the info to the same pfelk server? but separate by site A and Site B?

    ReplyDelete
    Replies
    1. Absolutely! It is currently configured/ready for two instances via port 5140 and 5141.

      For the example provided, you'll configure site A to send logs to pfELK via port 5140 and site B via port 5141.


      Furthermore, you can add additional sites as desired but will need to amend 01-inputs.conf and 02-types.conf to do so. Let me know if you need assistance - I can certainly help amend those files as needed for your setup.

      Delete
  2. #! Deprecation: [types removal] Specifying include_type_name in put index template requests is deprecated. The parameter will be removed in the next major version.
    {
    "acknowledged" : true
    }

    ReplyDelete
    Replies
    1. They'll still work and we'll update in the future (likely Elastic v8) and possibly incorporate non legacy templates.

      Delete
  3. Hi! So in this scenario, if pfsense and ubuntu server are two virtual machines? can you shed some details on the configuration and setup of those?

    ReplyDelete