Top Button Left Second Button Left Third Button Left Fourth Button Left Fifth Button Left Sixth Button left Seventh Button left Eight Button Left

Monitoring PF Firewalls For Health And Performance


The PF (packet filter) firewall package was introduced in OpenBSD 3.0, and has since been ported to the FreeBSD and NetBSD Operating Systems. PF contains a stateful packet inspection engine, the ability to replicate state information to a backup firewall, a flexible self optimizing rule engine, QOS support, and the ability to collect performance metrics. These metrics can be useful for gauging the performance of a firewall platform, and provide a way to trend firewall performance over time. This article will describe several utilities that can be used to monitor the health and performance of a PF firewall.

Viewing performance counters with pfctl


The pfctl utility provides a command line interface to monitor and control the PF firewall module. pfctl contains options to display firewall configuration information and a variety of performance and status counters. To display the status and performance counters, the "info" parameter can be passed to pfctl's "-s" (show values) option:

$ sudo pfctl -s info
Status: Enabled for 2 days 10:33:04           Debug: Urgent

Hostid: 0xdf1a2b73

Interface Stats for tun0              IPv4             IPv6
  Bytes In                       577197106                0
  Bytes Out                       30295580                0
  Packets In
    Passed                          564503                0
    Blocked                           1494                0
  Packets Out
    Passed                          386562                0
    Blocked                              0                0

State Table                          Total             Rate
  current entries                        1
  searches                         2164707           10.3/s
  inserts                            23601            0.1/s
  removals                           23600            0.1/s
Counters
  match                            1226533            5.8/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s

In addition to displaying the counters listed above, the show values option can be used to display firewall rulesets, NAT table entries, state table entries, and ALTQ queues. To get detailed usage data for each show values option, pfctl can be invoked with the "-v" (verbose) option. The following example shows how to get detailed usage data for each firewall rule:

$ sudo pfctl -v -s rules
@0 scrub in all fragment reassemble
  [ Evaluations: 3884714   Packets: 1917431   Bytes: 0           States: 0     ]

@1 block drop in log on tun0 all
  [ Evaluations: 1200737   Packets: 1368      Bytes: 170289      States: 0     ]

The pfctl(8) man page contains the full list of values that can be be passed to the show values option, and covers pfctl's advanced options.

Viewing active connections with pftop


The pftop utility provides a "top" like view of the PF state table. pftop displays source and destination IP addresses, TCP and UDP port numbers, packets and bytes transmitted, the age of a connection, and the time left until a connection will be removed from the state table. pftop contains several options to control how data is displayed, and is invoked by running the "pftop" executable from the command line:

$ sudo pftop

pftop can be installed from the OpenBSD ports collection, or downloaded from the pftop website.

A sample pftop display is included in figure 1.

Monitoring logged packets with tcpdump


PF can record network packet headers and data when the log key word is used with a rule. Log files can be valuable for debugging rules, understanding traffic flows, and finding performance bottlenecks. When a packet matches a rule with the log key word, the headers and packet body are sent to the pflog pseudo-device. Once a packet is logged to the pflog pseudo-device, The tcpdump utility can be used to print the packet's contents in real time:

$ sudo tcpdump -i pflog0 -o -ttt -vv -e -n

Nov 26 00:58:32.001036 rule 4/0(match): block in on tun0: 1.2.3.4.4087 > 5.6.7.8.www: S [tcp sum ok] (src OS: Windows XP SP
1, Windows 2000 SP2+) 3615243359:3615243359(0) win 16384 <mss 1440,nop,nop,sackOK> (DF) (ttl 119, id 7135)
        ........

This example prints protocol headers, timestamps, and an OS fingerprint (using the signatures available in /etc/pf.os) for each packet logged to the pflog pseudo-device. tcpdump's "-r" option can be used to process PF log files written to the file system by the pflogd daemon:

$ sudo tcpdump -r /var/log/pflog -o -ttt -vv -e -n dst port 80
Nov 26 00:58:32.001036 rule 4/0(match): block in on tun0: 1.2.3.4.4087 > 5.6.7.8.www: S [tcp sum ok] (src OS: Windows XP SP
1, Windows 2000 SP2+) 3615243359:3615243359(0) win 16384 <mss 1440,nop,nop,sackOK> (DF) (ttl 119, id 7135)
        ........

This example uses tcpdump's filtering capabilities to limit the results to connections with a destination port of 80. tcpdump also allows connections to be filtered by IP address, hostname, Ethernet address, TCP flags, etc.

Summarizing log files with fwanalog


fwanalog is a firewall log file analysis tool that can be used to parse and summarize the log files from several firewall packages (e.g., PF, Checkpoint, ipfilter, iptables, Cisco PIX ). fwanalog uses the analog log file analysis program, and a simple straight forward configuration file to generate reports. The software is available as source code from the fwanalog website.

The fwanalog package contains several sample configuration files, each used to represent a different firewall platform. The "fwanalog.opts.openbsd3" configuration file contains the definitions required to process PF log files on OpenBSD 3.X platforms. When the fwanalog.sh script is invoked, it will process the contents of the configuration file named "fwanalog.opts" by default. The fwanalog documentation recommends creating a symbolic link to point to the platform specific configuration file:

$ ln -s fwanalog.opts.openbsd3 fwanalog.opts

The fwanalog.opts configuration file contains a series of variables to define the log file locations, the format of each log file, and the location to store the reports. The customizable variables are summarized below:

outdir           - Location to store reports
logformat        - Format of the log file to analyze
inputfiles_mask  - The log file name (supports wildcards)
inputfiles_dir   - The directory with the log files
inputfiles_mtime - The maximum age of the log files to parse

Once the configuration file is modified to match your desired layout and platform ( a sample configuration file to parse PF log files on the OpenBSD platform is included in figure 2), fwanalog.sh can be executed to generate a report:

$ fwanalog.sh

This will place the reports in the directory assigned to the "outdir" variable. The reports can be viewed with a web browser, or emailed as part of a routine reporting process.

Graphing PF Performance Data with pfstat


The pfstat utility can be used to collect and graph statistics exported through the /dev/pf pseudo-device. pfstat can be installed from the OpenBSD ports collection, or downloaded from the pfstat website.

pfstat requires the PF "loginterface" global configuration directive to be set in the pf.conf configuration file. This directive enables statistics collection for one of the physical interfaces in the firewall. The following pf.conf entry will collect statistics on the hme0 interface:

set loginterface hme0

Once statistics collecting is enabled, the pfstat utility can be invoked with the "-q" option. This will query the current value of each statistics counter, and printed the result to standard out:

$ pfstat -q
1101400143 1101219586 483226347 25637411 0 0 496899 3866 325988 0 0 0 0 0 6 1692642 17030 17024 879499 0 2 0 0 0

pfstat uses this data to generate historical utilization graphs, so the data should be collected at periodic intervals if graphs are desired. The following cron job will collect statistics every five minutes, and write the results to "/var/log/pfstat/pfstat":

*/5 * * * * /usr/local/bin/pfstat -q >> /var/log/pfstat/pfstat

To graph the data that is collected, a pfstat configuration file needs to be created. This file describes the graphs to generate, how to display the data, and where to store the output. The following example shows the pfstat configuration required to graph state table data:

image "/home/matty/pfstat/images/state_table.jpg" {
     from 3 months to now
     width 800 height 300
     left
        graph states_entries  label "state table entries"   color 0 255 0,
        graph states_searches label "state table searches"  color 255 0 0,
        graph states_inserts  label "state table insertions"   color 0 0 255,
        graph states_removals label "state table removals"  color 0 0 0
}

The pfstat configuration file contains one or more "image" directives. Each image directive is followed by the file name of the image to generate, and a set of curly braces to control the attributes of the image. The "from" and "to" keywords select the time interval to graph. The value that follows the "from" keyword contains an integer value and a time frame (minutes, hours, days, weeks, months, years) to control how far back pfstat will go when processing data. The "to" keyword controls how pfstat processes new data elements, and the special key word "now" indicates the current time.

The "height" and "width" directives set the size, in pixels, for the height and width of the image to be output. Two directives determine the horizontal alignment of text descriptions: "left" aligns text on the left side of the graph and "right" aligns text on the right side. The "graph" statements control which data is graphed, the label assigned to the graph, and the colors used to create the entries on the graph. As of pfstat version 1.7, pfstat can graph packets, bytes, state table information, and several miscellaneous packet counters. The full list of options that can be passed to the "graph" directive are described in the pfstat(8) man page.

Once the configuration file is created, we can execute pfstat, and pass the configuration and data file as arguments to the "-c" and "-d" options:

$ pfstat -c /etc/pfstat/pfstat.conf -d /var/log/pfstat/pfstat >/dev/null

Figure 3 contains a sample pfstat.conf configuration file to graph IPv4, IPv6, and state table information. Figures 4, 5 and 6 include sample pfstat graphs from a production firewall.

Conclusion


This article provided an introduction to several tools that can be used to monitor a PF firewall. The PF FAQ and application manual pages contain additional details on advanced topic, and additional options that can be used to customize each application. If you have questions or comments on the article, please feel free to E-mail the author.

References


The following references were used while writing this article:

Acknowledgements


Ryan would like to thank the OpenBSD, PF, pftop, pfstat, tcpdump, analog and fwanalog developers for their awesome work!

* Originally published in the September '05 issue of SysAdmin Magazine