pfELK (pfSense or OPNsense)
Visit https://pfelk.3ilson.dev for Scripted, Ansible and Docker installations
Prerequisites
Ubuntu Server v18.04+
pfSense v2.4.4+ or OPNsense 20.1+
Navigate to the following within pfSense
Status>>System Logs [Settings]
1) Enable Remote Logging
2) Provide 'Server 1' address (this is the IP address of the ELK installation - ex: 192.168.1.60:5140)
2) Provide 'Server 1' address (this is the IP address of the ELK installation - ex: 192.168.1.60:5140)
3) Select "Firewall events"
Preparation
1. Create Directory Structure
sudo mkdir /data
sudo mkdir /data/pfELK
sudo mkdir /data/cofigurations
sudo mkdir /data/patterns
sudo mkdir /data/templates
sudo mkdir /data/GeoIP
sudo mkdir /data/logs
sudo chmod 777 /data/logs
sudo add-apt-repository ppa:maxmind/ppa
3. Download and Install the Public GPG Signing Key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
4. Download and Install apt-transport-https Package (Debian)
sudo apt-get install apt-transport-https
5. Add Elasticsearch|Logstash|Kibana Repositories (version 7+)
6. Update
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update
7. Install Java 11 LTS
sudo apt install openjdk-11-jre
8. Install MaxMind
sudo apt-get install geoipupdate
- Create a Max Mind Account @ https://www.maxmind.com/en/geolite2/signup
- Login to your Max Mind Account; navigate to "My License Key" under "Services" and Generate new license key
sudo nano /etc/GeoIP.confAccountID <Input Your Account ID>
LicenseKey <Input Your LicenseKey>
11. Modify line 13 as Follows:
EditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN
12. Download MaxMind Databases:
sudo geoipupdate -d /data/pfELK/GeoIP/
13. Add cron (automatically updates MaxMind everyweek on Sunday at 1700hrs):
sudo nano /etc/cron.weekly/geoipupdate
14. Add the Following and Save/Exit
00 17 * * 0 geoipupdate -d /data/pfELK/GeoIP
Install
Elasticsearch v7.6+ | Kibana v7.6+ | Logstash v7.6+
Elastic Stack
Elastic Stack
15. Install Elasticsearch|Kibana|Logstash
sudo apt-get install elasticsearch && sudo apt-get install kibana && sudo apt-get install logstash
Configure Kibana|v7.6+
16. Configure Kibana
sudo nano /etc/kibana/kibana.yml
17. Amend Host File (/etc/kibana/kibana.yml)
server.port: 5601
server.host: "0.0.0.0"
Configure Logstash|v7.6+
18. Change Directory (preparation for configuration files)
cd /data/pfELK/configurations
19. Download the Following Configuration Files
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/01-inputs.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/05-firewall.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/30-geoip.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/50-outputs.confsudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/10-others.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/20-suricata.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/25-snort.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/35-rules-desc.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/40-dns.conf sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/configurations/45-cleanup.conf
20. Navigate to Patterns Folder
cd /data/pfELK/patterns/
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/patterns/pfelk.grok
22. Navigate to Templates Folder
cd /data/pfELK/templatessudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/data/templates/pf-geoip-template.json
24. Navigate to Logstash Folder
cd /etc/logstashsudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/pipelines.ymlsudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/logstash.yml
26. Edit (01-inputs.conf)
sudo nano /etc/logstash/conf.d/01-inputs.conf
# 01-inputs.conf
input {
udp {
port => 5140
}
}
filter {
#Adjust to match the IP address of pfSense or OPNsense
if [host] =~ /172\.22\.33\.1/ {
mutate {
add_tag => [ "pf", "Ready" ]
add_field => [ "[observer][type]", "firewall" ]
}
}
#To enable or ingest multiple pfSense or OPNsense instances uncomment the below section
##############################
#if [host] =~ /172\.2\.22\.1/ {
# mutate {
# add_tag => ["pf", "Ready"]
# add_field => [ "[observer][type]", "firewall" ]
# }
#}
##############################
if "pf" in [tags] {
grok {
# OPNsense - Enable/Disable the line below based on firewall platform
match => { "message" => "%{SYSLOGTIMESTAMP:[event][created]} %{SYSLOGHOST:[observer][name]} %{DATA:labels}(?:\[%{POSINT:pf_pid}\])?: %{GREEDYDATA:pf_message}" }
########################################################################################################################################
# pfSense - Enable/Disable the line below based on firewall platform
# match => { "message" => "%{SYSLOGTIMESTAMP:[event][created]} %{DATA:labels}(?:\[%{POSINT:[event][id]}\])?: %{GREEDYDATA:pf_message}" }
}
mutate {
rename => { "[message]" => "[event][original]"}
remove_tag => "Ready"
}
}
}
28. Naviaget to pfELK
cd /data/pfELK
29. Download error-data.sh
sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/error-data.sh
30. Make error-data.sh Executable
sudo chmod +x /data/pfELK/error-data.sh
31. Disable Swap
sudo swapoff -a
//Update the timezone as needed - http://joda-time.sourceforge.net/timezones.html //
sudo timedatectl set-timezone EST
Configure Services
Automatic Start (on boot)
33. Start Services on Boot as Services (you'll need to reboot or start manually to proceed)
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl enable kibana.service
sudo /bin/systemctl enable logstash.service
Manual Start
34. Start Services Manually
sudo -i service elasticsearch start
sudo -i service kibana start
sudo -i service logstash start
Point browser to url or IP:5601 (ex: 192.168.1.1:5601)
Select @timestamp and click 'Create'
*You may have to wait a few minutes to allow log retrieval
*You may have to wait a few minutes to allow log retrieval
35. Configuring Patterns
- Click the gear icon (management) in the lower left
- Click Kibana >> Index Patters
- Click Create New Index Pattern
- Type "pf-*" into the input box, then click Next Step
- In your web browser go to the ELK local IP using port 5601 (ex: 192.168.0.1:5601)
- Click Management -> Saved Objects
- You can import the dashboards found in the
Dashboardfolder via the Import buttom in the top-right corner.
Testing/Troubleshooting
37. Elasticsearch
curl -X GET http://localhost:9200 { "name" : "NYLJDFe", "cluster_name" : "elasticsearch", "cluster_uuid" : "7krQg2MzR0irVJ6gNAB7fg", "version" : { "number" : "7.6.2", "build_hash" : "253032b", "build_date" : "2017-10-31T05:11:34.737Z", "build_snapshot" : false, "lucene_version" : "6.6.1" }, "tagline" : "You Know, for Search" }
38. Status (Elasticsearch)
systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2020-03-13 20:53:51 EDT; 13h ago Docs: http://www.elastic.co Main PID: 6121 (java) Tasks: 74 Memory: 2.4G CPU: 7min 46.327s CGroup: /system.slice/elasticsearch.service └─6121 /usr/bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=7 Mar 13 20:53:51 logs systemd[1]: Starting Elasticsearch... Mar 13 20:53:51 logs systemd[1]: Started Elasticsearch.
39. Status (Kibana)
systemctl status kibana.service ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2020-03-13 20:54:09 EDT; 13h ago Main PID: 6205 (node) Tasks: 10 Memory: 82.2M CPU: 2min 51.950s CGroup: /system.slice/kibana.service └─6205 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c Mar 13 10:43:16 logs kibana[6205]: {"type":"response","@timestamp":"2020-03-131T14:43:16Z","tags":[],"pid":
40. Status (Logstash)
systemctl status logstash.service ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2020-03-13 08:52:27 EDT; 1h 58min ago Main PID: 32366 (java) Tasks: 43 Memory: 405.6M CPU: 4min 43.959s CGroup: /system.slice/logstash.service └─32366 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFracti Mar 13 08:52:27 logs systemd[1]: Started logstash.
41. Logstash Log's
/var/log/logstash
#cat/nano/vi the files within this location to view Logstash logs
