Parser Example

pfSense

Overview

Parser xử lý pfSense firewall logs trong JSON Wrapped Format - logs được ship qua Filebeat/Fluentd/Logstash với message field chứa syslog message và CSV fields.

Input Format:

{"message":"<134>1 2026-03-16T17:33:00+07:00 hostname filterlog pid - - CSV_FIELDS", "timestamp":"2026-03-16T10:33:00.759Z", "host":"10.6.24.10"}

Message Structure:

Syslog header: <priority>version timestamp hostname
Application: filterlog
CSV fields: Comma-separated values (format khác nhau giữa IPv4 và IPv6)

Log Types:

IPv4 traffic: TCP, UDP, ICMP
IPv6 traffic: TCP, UDP, ICMPv6, VRRP, other protocols
Actions: pass, block, reject, match
Directions: in (inbound), out (outbound)

pfSense FilterLog CSV Format

IPv4 Format (28+ fields)

rulenr,subrulenr,anchorname,rid,interface,reason,action,dir,ipversion,tos,ecn,ttl,id,offset,flags,protoid,proto,length,src,dst,srcport,dstport,...

Key Field Positions (IPv4):

Field 4: Interface name
Field 6: Action (pass/block/reject)
Field 7: Direction (in/out)
Field 8: IP version (4)
Field 16: Protocol number (6=TCP, 17=UDP, 1=ICMP)
Field 18: Source IP
Field 19: Destination IP
Field 20: Source port (TCP/UDP)
Field 21: Destination port (TCP/UDP)

IPv6 Format (Different field positions)

rulenr,subrulenr,anchorname,rid,interface,reason,action,dir,ipversion,class,flowlabel,hlim,proto,protoid,length,src,dst,...

Key Field Positions (IPv6):

Field 4: Interface name
Field 6: Action (pass/block/reject)
Field 7: Direction (in/out)
Field 8: IP version (6)
Field 12: Protocol (tcp/udp/icmp6/VRRP/etc)
Field 15: Source IP
Field 16: Destination IP
Field 17: Source port (TCP/UDP)
Field 18: Destination port (TCP/UDP)

Sample Logs (JSON Wrapped Format)

1. IPv4 - TCP Block

{"@timestamp":"2026-03-16T10:30:00.123Z","message":"<134>1 2026-03-16T17:30:00+07:00 pfsense-gw filterlog 12345 - - 100,,,1234567890abcdef,em0,match,block,in,4,0x0,0,64,12345,0,none,6,tcp,60,192.168.1.100,10.0.0.50,54321,22,0,S,1024,,,","timestamp":"2026-03-16T10:30:00.123Z","host":"10.6.24.10","vendor":"pfSense"}

2. IPv4 - UDP Allow

{"@timestamp":"2026-03-16T10:31:00.234Z","message":"<134>1 2026-03-16T17:31:00+07:00 pfsense-gw filterlog 12346 - - 200,,,2345678901bcdefg,em1,match,pass,out,4,0x0,0,128,23456,0,none,17,udp,100,10.0.0.100,8.8.8.8,61234,53,80","timestamp":"2026-03-16T10:31:00.234Z","host":"10.6.24.10","vendor":"pfSense"}

3. IPv4 - ICMP Block

{"@timestamp":"2026-03-16T10:32:00.345Z","message":"<134>1 2026-03-16T17:32:00+07:00 pfsense-gw filterlog 12347 - - 300,,,3456789012cdefgh,em0,match,block,in,4,0x0,0,64,34567,0,none,1,icmp,84,203.0.113.50,192.168.1.1","timestamp":"2026-03-16T10:32:00.345Z","host":"10.6.24.10","vendor":"pfSense"}

4. IPv6 - TCP Allow

{"@timestamp":"2026-03-16T10:33:00.456Z","message":"<134>1 2026-03-16T17:33:00+07:00 pfsense-gw filterlog 12348 - - 400,,,4567890123defghi,vtnet0,match,pass,out,6,0x00,0x00000,64,tcp,6,80,2001:db8::1,2001:db8::100,443,55123,20,PA,4096,,,","timestamp":"2026-03-16T10:33:00.456Z","host":"10.6.24.10","vendor":"pfSense"}

5. IPv6 - UDP Block

{"@timestamp":"2026-03-16T10:34:00.567Z","message":"<134>1 2026-03-16T17:34:00+07:00 pfsense-gw filterlog 12349 - - 500,,,567890124efghij,vtnet1,match,block,in,6,0x00,0x00000,128,udp,17,200,fe80::1,ff02::1,5353,5353","timestamp":"2026-03-16T10:34:00.567Z","host":"10.6.24.10","vendor":"pfSense"}

6. IPv6 - VRRP

{"@timestamp":"2026-03-16T10:35:00.678Z","message":"<134>1 2026-03-16T17:35:00+07:00 pfsense-gw filterlog 12350 - - 600,,,67890125fghijk,vtnet2,match,pass,in,6,0x00,0x00000,255,VRRP,112,58,fe80::22ed:4702:91ea:2fd5,ff02::12,carp","timestamp":"2026-03-16T10:35:00.678Z","host":"10.6.24.10","vendor":"pfSense"}

7. IPv6 - ICMPv6

{"@timestamp":"2026-03-16T10:36:00.789Z","message":"<134>1 2026-03-16T17:36:00+07:00 pfsense-gw filterlog 12351 - - 700,,,7890126ghijkl,em2,match,block,in,6,0x00,0x00000,64,icmp6,58,64,2001:db8:1::1,ff02::1","timestamp":"2026-03-16T10:36:00.789Z","host":"10.6.24.10","vendor":"pfSense"}

8. IPv4 - TCP Reject

{"@timestamp":"2026-03-16T10:37:00.890Z","message":"<134>1 2026-03-16T17:37:00+07:00 pfsense-gw filterlog 12352 - - 800,,,890127hijklm,em0,match,reject,in,4,0x0,0,64,45678,0,none,6,tcp,60,45.67.89.123,192.168.1.10,12345,80,0,S,512,,,","timestamp":"2026-03-16T10:37:00.890Z","host":"10.6.24.10","vendor":"pfSense"}

Parser Configuration

#regex


#conditional
event_timestamp = ""
if (ts, err = parse_timestamp(.timestamp, "%Y-%m-%dT%H:%M:%S%.3fZ"); err == null) { 
  if (formatted, err = format_timestamp(ts, "%Y-%m-%d %H:%M:%S"); err == null) {
    event_timestamp = formatted
  }
}

log_message = ""
if (msg, err = to_string(.message); err == null) { 
  log_message = msg
}

host_str = ""
if (h, err = to_string(.host); err == null) { host_str = h }

event_module = "pfsense"
event_category = "network"
event_action = ""
event_outcome = ""
event_type = "connection"
log_level = "info"

source_ip = ""
source_port = ""
destination_ip = ""
destination_port = ""
network_protocol = ""
network_transport = ""
network_iana_number = ""
network_direction = ""

observer_vendor = "Netgate"
observer_hostname = ""
observer_product = "pfSense"
observer_interface = ""

rule_id = ""

if contains(log_message, "filterlog") {
  observer_hostname = host_str
  
  if (csv_m, err = parse_regex(log_message, r'filterlog \d+ - - (?P<csv>.+)$'); err == null) {
    csv_str = to_string(csv_m.csv)
    csv_parts = split(csv_str, ",")
    
    rule_id = to_string(csv_parts[0])
    observer_interface = to_string(csv_parts[4])
    event_action = to_string(csv_parts[6])
    
    dir_val = to_string(csv_parts[7])
    if dir_val == "in" { 
      network_direction = "inbound" 
      event_type = "denied"
    }
    if dir_val == "out" { 
      network_direction = "outbound"
      event_type = "allowed"
    }
    
    ipver = to_string(csv_parts[8])
    
    if ipver == "4" {
      proto_num = to_string(csv_parts[16])
      network_iana_number = proto_num
      
      if proto_num == "6" { 
        network_transport = "tcp"
        network_protocol = "tcp"
      }
      if proto_num == "17" { 
        network_transport = "udp"
        network_protocol = "udp"
      }
      if proto_num == "1" { 
        network_transport = "icmp"
        network_protocol = "icmp"
      }
      
      source_ip = to_string(csv_parts[18])
      destination_ip = to_string(csv_parts[19])
      
      if proto_num == "6" {
        source_port = to_string(csv_parts[20])
        destination_port = to_string(csv_parts[21])
      }
      if proto_num == "17" {
        source_port = to_string(csv_parts[20])
        destination_port = to_string(csv_parts[21])
      }
    }
    
    if ipver == "6" {
      proto_val = to_string(csv_parts[12])
      proto_num = to_string(csv_parts[13])
      
      network_iana_number = proto_num
      
      if proto_val == "tcp" { 
        network_transport = "tcp"
        network_protocol = "tcp"
      }
      if proto_val == "udp" { 
        network_transport = "udp"
        network_protocol = "udp"
      }
      if proto_val == "icmp6" { 
        network_transport = "icmp6"
        network_protocol = "icmp6"
      }
      if proto_val == "VRRP" { 
        network_protocol = "vrrp"
        network_transport = "vrrp"
      }
      
      source_ip = to_string(csv_parts[15])
      destination_ip = to_string(csv_parts[16])
      
      if proto_val == "tcp" {
        source_port = to_string(csv_parts[17])
        destination_port = to_string(csv_parts[18])
      }
      if proto_val == "udp" {
        source_port = to_string(csv_parts[17])
        destination_port = to_string(csv_parts[18])
      }
    }
    
    act_str = to_string(event_action)
    if act_str == "pass" {
      event_action = "allowed"
      event_outcome = "success"
    }
    if act_str == "block" {
      event_action = "denied"
      event_outcome = "success"
      event_type = "denied"
    }
    if act_str == "reject" {
      event_action = "denied"
      event_outcome = "success"
      event_type = "denied"
    }
  }
}

if event_action == "" { event_action = "info" }
if event_outcome == "" { event_outcome = "unknown" }

#normalize
timestamp: event_timestamp
event.module: event_module
event.category: event_category
event.action: event_action
event.outcome: event_outcome
event.type: event_type
log.level: log_level
message: log_message
source.ip: source_ip
source.port: source_port
destination.ip: destination_ip
destination.port: destination_port
network.transport: network_transport
network.protocol: network_protocol
network.iana_number: network_iana_number
network.direction: network_direction
observer.vendor: observer_vendor
observer.hostname: observer_hostname
observer.product: observer_product
observer.ingress.interface.name: observer_interface
rule.id: rule_id

Event Categories

Action	Direction	Event Action	Event Type	Event Outcome
pass	in/out	allowed	allowed	success
block	in/out	denied	denied	success
reject	in/out	denied	denied	success
match + pass	out	allowed	allowed	success
match + block	in	denied	denied	success

Field Mapping

pfSense CSV Field	Position (IPv4)	Position (IPv6)	ECS Field	Field Set
Rule number	0	0	rule.id	Rule
Interface	4	4	observer.ingress.interface.name	Observer
Action	6	6	event.action	Event
Direction	7	7	network.direction	Network
IP version	8	8	-	-
Protocol (IPv4)	16	-	network.iana_number, network.transport	Network
Protocol (IPv6)	-	12	network.protocol, network.transport	Network
Protocol ID (IPv6)	-	13	network.iana_number	Network
Source IP (IPv4)	18	-	source.ip	Source
Source IP (IPv6)	-	15	source.ip	Source
Dest IP (IPv4)	19	-	destination.ip	Destination
Dest IP (IPv6)	-	16	destination.ip	Destination
Source port (IPv4)	20	-	source.port	Source
Source port (IPv6)	-	17	source.port	Source
Dest port (IPv4)	21	-	destination.port	Destination
Dest port (IPv6)	-	18	destination.port	Destination

Notes

Input Format: JSON object với field .message chứa syslog message và CSV fields.

Parsing Strategy:

Detect "filterlog" keyword trong message
Extract CSV fields sau "filterlog pid - - "
Split CSV by comma
Detect IP version (field 8) để xác định field positions
Extract source/dest IP và ports theo IP version
Map protocol numbers: 6=tcp, 17=udp, 1=icmp
Map actions: pass→allowed, block/reject→denied

Protocol Handling:

IPv4: Protocol number in field 16 (numeric)
- 6 = TCP, 17 = UDP, 1 = ICMP
IPv6: Protocol name in field 12 (string)
- tcp, udp, icmp6, VRRP, etc.

Direction Logic:

in = inbound (thường là block/deny)
out = outbound (thường là pass/allow)

VRL Functions Used:

to_string() - Type coercion
parse_regex() - Extract CSV portion from syslog message
split() - Split CSV fields by comma
contains() - Detect filterlog keyword

Vendor Identification:

JSON field: "vendor": "pfSense" (optional)
Message content: Contains keyword filterlog
Observer: vendor=Netgate, product=pfSense

Test Results: Xem TEST_RESULTS.md cho chi tiết test cases.

Chỉnh sửa trang nàyhoặcBáo cáo sự cố

Windows Event Channel

Supported Conditional Functions

Comprehensive reference for parser conditional functions

CMC Comprehensive Security Platform

pfSense

pfSense

Overview

pfSense FilterLog CSV Format

IPv4 Format (28+ fields)

IPv6 Format (Different field positions)

Sample Logs (JSON Wrapped Format)

1. IPv4 - TCP Block

2. IPv4 - UDP Allow

3. IPv4 - ICMP Block

4. IPv6 - TCP Allow

5. IPv6 - UDP Block

6. IPv6 - VRRP

7. IPv6 - ICMPv6

8. IPv4 - TCP Reject

Parser Configuration

Event Categories

Field Mapping

Notes

CMC Telecom
Aspire to Inspire the Digital World

pfSense

pfSense

CMC TelecomAspire to Inspire the Digital World

CMC Telecom
Aspire to Inspire the Digital World