18 October 2020

pf (Firewall Logs) + Elasticsearch + Logstash + Kibana

  pfSense/OPNsense | Elastic Stack v7.9+ | Ubuntu 20.04+ 

pfELKCore installation. 

Requirements
Ubuntu Server v20.04+

pfSense v2.4+ or OPNsense 19+




 Primary Installation Method
Scripted Install

A. Download Script

wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-installer.sh

B. Make Script Executable

chmod +x pfelk-installer.sh

C. Run Script Installer

sudo ./pfelk-installer.sh

 D. Proceed to Section e3 below:

Alternate Installation Method
Manual Install

(a) - Preparation

a1. Configure Date/Time Zone

sudo timedatectl set-timezone EST

a2. Disable Swap

sudo swapoff -a

(b) - Prerequisites
MaxMind (optional), apt-transport, ELK repositories, ELK, GPG signing key, Java 14 LTS 

Add Elastic Stack Repository

b1. Add Elastic Stack (Elasticsearch, Logstash and Kibana) Repository

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

b2. Download and install the public GPG signing key (Elastic Stack)

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Update System Repositories

b3. Update Repositories

sudo apt update

Install apt-transport

b4. Download and install apt-transport-https package

sudo apt install apt-transport-https

Install Elastic Stack 

b5. Install Elastic Stack (Elasticsearch, Logstash and Kibana

sudo apt install elasticsearch; sudo apt install kibana; sudo apt install logstash

Configure Kibana

c1. Configure Kibana

sudo nano /etc/kibana/kibana.yml

c2. Modify host file (/etc/kibana/kibana.yml)

server.port: 5601
server.host: "0.0.0.0"

Configure Logstash

c3. Create the following directories

sudo mkdir -p /etc/pfelk{conf.d,config,logs,databases,patterns,scripts,templates}

c4. Download conf.d files

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/01-inputs.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/05-apps.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/20-interfaces.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/30-geoip.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/35-rules-desc.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/36-ports-desc.conf -P /etc/pfelk/conf.d/sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/45-cleanup.conf -P /etc/pfelk/conf.d/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/conf.d/50-outputs.conf -P /etc/pfelk/conf.d/

c5. Download pfelk.grok pattern

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/pfelk.grok -P /etc/pfelk/patterns/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/openvpn.grok -P /etc/pfelk/patterns/

c6. Download pfelk databases

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/rule-names.csv -P /etc/pfelk/databases/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/service-names-port-numbers.csv -P /etc/pfelk/databases/ sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/databases/private-hostnames.csv -P /etc/pfelk/databases

c7. Download pfELK Error Log Script (Optional)

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/error-data.sh -P /etc/pfelk/scripts/

c8. Make pfELK Error Log Script Executable (Optional)

sudo chmod +x /etc/pfelk/scripts/error-data.sh

(d) - Configure Services
Elasticsearch, Kibana and Logstash 

d1. Automatic Start (Start Servies on Boot)

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl enable kibana.service
sudo /bin/systemctl enable logstash.service

d2. Manual Start (Start Servies Manually)

sudo -i service elasticsearch start
sudo -i service kibana start
sudo -i service logstash start

(e) - Finalize
pfELK 

e1. Import the required templates (NOTE: Must be done prior to receiving logs. Otherwise, you'll need to rebuild your indices)

  • Click in the upper left corner
  • Click on Dev Tools located near the bottom under the Management heading
    • Paste the contents of each file located here (repeat for each file) or the following:
 
e2. Import the saved objects (i.e. Dashboards, visualizations etc...)
  • The Saved Objects may be downloaded here 
  • Click on the in the upper left corner
  • Click on Stack Management located near the bottom under the Management heading
  • Click on Saved Objects located under the Kibana heading
  • Click "Import" (YouTube Tutorial Video 1 and Video 2)
e3. Configure pfSense and/or OPNsense to send logs
  •  pfSense - Navigate to Status >> System Logs [Settings] and configure as depicted below:
 
    • Enable Remote Logging
    • Provide "Server 1" address (this is the IP address of the ELK installation [e.g. 192.168.1.60:5140])
    • Select "Firewall events"
  •  OPNsense - Navigate to System >> Settings >> Logging / targets and configure as depicted below:
 
    • The Hostname is the IP address of where you installed ELK
    • The port should be set to 5140

(f) - Troubleshooting
pfELK

f1. Check Status of each process

systemctl status elasticsearch.service
systemctl status kibana.service
systemctl status logstash.service 

f2. Review Logstash Logs for errors

cat /var/log/logstash/logstash-plain.log

f3. Generate pfELK log

sudo ./etc/pfELK/error-data.sh
  • This will generate a log file within /etc/pfELK/logs/
  • Utilize the log file to aid in troubleshooting

f4. Need additional assistance, visit pfELK wiki page via GitHub

f5. Video installation tutorial via 3ilson YouTube Channel

f6. Submit an Issues via here, leave a comment below or send an email

f7. Discuss, collaborate, troubleshoot, etc... within the pfELK community on Gitter


If this helped, feel free to make a contribution:
Donate

17 comments:

  1. Hi there is it possible to have multi pfSense sending the info to the same pfelk server? but separate by site A and Site B?

    ReplyDelete
    Replies
    1. Absolutely! It is currently configured/ready for two instances via port 5140 and 5141.

      For the example provided, you'll configure site A to send logs to pfELK via port 5140 and site B via port 5141.


      Furthermore, you can add additional sites as desired but will need to amend 01-inputs.conf and 02-types.conf to do so. Let me know if you need assistance - I can certainly help amend those files as needed for your setup.

      Delete
  2. #! Deprecation: [types removal] Specifying include_type_name in put index template requests is deprecated. The parameter will be removed in the next major version.
    {
    "acknowledged" : true
    }

    ReplyDelete
    Replies
    1. They'll still work and we'll update in the future (likely Elastic v8) and possibly incorporate non legacy templates.

      Delete
  3. Hi! So in this scenario, if pfsense and ubuntu server are two virtual machines? can you shed some details on the configuration and setup of those?

    ReplyDelete
  4. Getting stuck @ step 1:

    $ wget https://raw.githubusercontent.com/pfelk/pfelk/master/pfelk-installer.sh
    --2021-12-12 01:34:35-- https://raw.githubusercontent.com/pfelk/pfelk/master/pfelk-installer.sh
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
    HTTP request sent, awaiting response... 404 Not Found
    2021-12-12 01:34:35 ERROR 404: Not Found.

    Is there an updated path for the install script?

    ReplyDelete
  5. fun88 - Welcome bonus codes & promotions
    Enjoy the link 12bet excitement of the online gaming scene. Play slot games online with real money. Claim fun88 soikeotot your welcome bonus today! ➤ Review by Vienne Garcia.

    ReplyDelete
  6. 1Xbet – Up to $1000 in Welcome Bonus + 1xbet Wagering
    With 1xbet 먹튀 1Xbet, titanium pan you can titanium earrings enjoy up titanium plumbing to titanium belly rings $1000 in welcome bonus and free bets up to $1,000 in welcome bonus. You'll notice that the bonus is

    ReplyDelete
  7. New to this. I got it working with the script but got an error when trying to use my maxmind credentials. I followed the instructions for maxmind in the wiki and restart all the services. How do I know if elastic is now using maxmind? the instructions btw are confusing as there seems to be a mix of older instruction/youtube videos that don't match your github, or I just confused?

    Also any tutorials that explain how to use these graphs to analyze my pfsense logs

    ReplyDelete
  8. 지난달 31일 청와대 국민청원 게시판에는 ‘62명의 사상자를 냈음에도 반성과 사과조차 없는 화물 차량 운전자와 관계자들 처벌을 청원합니다’라는 제목의 글이 게시됐다. 해당 07카지노 청원은 1일 오전 10시 기준 1300명 넘는 이들의 동의를 받았다. 정답은 없지만 그에 맞는 게임 환경을 가지고 있느냐가 중요한것 같습니다.

    ReplyDelete
  9. However, there is a big inequality of the variety of placed bets amongst gamblers. We therefore wonder whether these distributions we get hold of end result from the inequality of variety of bets amongst individuals. To take away the consequences of 바카라 사이트 this inequality, we randomly sample in every dataset the identical variety of bets from heavy gamblers.

    ReplyDelete
  10. The formability of CNC machining aluminum is low as compared with automotive metal. Still it considered one of the|is amongst the|is probably certainly one of the} promising materials to be utilized in automotive outcome of} mild weight. It has been discovered that effect of mesh measurement is prominent in simulation experiments. Plotted FLDs for gentle carbon metal using hemispherical punch take a look at.

    ReplyDelete
  11. Designers want to adhere to machinery pointers when designing for bending. Some critical dimensions that have to be thought of when establishing sheet metal in CAD software program are sheet metal thickness, the k-factor and bend radius. One needs to examine that these elements are in keeping with} the tooling day dreamer lounger that shall be utilized in manufacturing. This guide provides necessary pointers for good design practice. There is a wide array|a extensive selection|a big selection} of VIEWKOTE™ pre-coated steel sheets, together with the high forming metallic-type.

    ReplyDelete
  12. There are more than one thousand titles select from|to choose from} 50 + providers which makes Luckster an actual gem for slots followers. Of course, might also|you can even} discover live sellers and desk games on high of well-liked sports events. Those who've any questions or issues can access the help team at any time. Customer help is on the market in 11 languages so you will get get} 다파벳 assist in your native language. The Game of Thrones slots sees players delve into the darkish fantasy world created within the well-liked TV collection and is one of the|is amongst the|is likely one of the} high casino slots at Jackpot City in accordance with participant suggestions.

    ReplyDelete
  13. The hood, the fender, the side panels, and the roof are all produced from sheet metal that MOTORCYCLE GEAR GLOVES has been cut by laser and punching operations and formed in stamping processes. The body and exhaust are each roll formed then bent into form by CNC tube benders. There is a very huge range of automotive parts being made that sheet metal fabrication paved finest way|the means in which} for. Sheet metal encompasses an intensive range of supplies, from mild metal and stainless-steel to aluminum, brass, and copper.

    ReplyDelete
  14. For progressive jackpot enjoyers, the placement offers a big assortment with fan-favourite names like Dolphin Gold, Fruit Fiesta, or Divine Fortune. In case you get uninterested in spinning, you probably can have some pleasant with the stay casino games from top-notch suppliers. When it entails withdrawing your funds, CasiGO has your again. The withdrawal processing is extraordinarily fast, and players might have entry to their winnings virtually instantly. As for the charge strategies, you’ll discover a method to deposit and withdraw winnings using Visa, MasterCard, or Maestro credit rating and debit playing cards, Trustly, Interac, 클레오카지노 Paysafe, and Sofort.

    ReplyDelete
  15. This appears to be the primary patent describing Stockings for Women 3D printing with speedy prototyping and managed on-demand manufacturing of patterns. PCMag.com is quantity one} authority on technology, delivering lab-based, independent evaluations of the latest services and products. Our skilled industry evaluation and practical solutions assist you to make higher shopping for choices and get extra from technology. The Mini features a beautifully produced consumer information and entry to Prusa Labs’ prodigious help resources and boards. It comes with the company’s PrusaSlicer software, various to|an alternative alternative to|a substitute for} the well-known Cura open-source 3D-printing program.

    ReplyDelete