Parser Example

FortiGate

FortiGate

Overview

Parser xử lý FortiGate logs trong JSON Wrapped Format - logs được ship qua Filebeat/Fluentd/Logstash với message field chứa key-value pairs theo tài liệu chính thức Fortinet 7.0.12.

Input Format:

{"message":"date=2019-05-10 time=11:37:47 logid=\"0000000013\" type=\"traffic\"..."}

Log Types:

  • Traffic logs: type="traffic", subtype="forward/local/multicast/sniffer"
  • UTM logs: type="utm", subtype="virus/webfilter/app-ctrl/ips/anomaly/dlp/dns/ssh/ssl"
  • Event logs: type="event", subtype="system/user/vpn/router/sdwan/ha/wireless"

Message Content: key=value key="quoted value" key=number

Sample Logs (JSON Wrapped Format)

1. Traffic - Forward Accept

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-10 time=11:37:47 logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"vdom1\" eventtime=1557513467 srcip=10.1.100.11 srcport=58012 srcintf=\"port12\" srcintfrole=\"lan\" dstip=23.59.154.35 dstport=80 dstintf=\"port11\" dstintfrole=\"wan\" poluuid=\"ccb269e0-5735-51e9-a218-a397dd08b7eb\" sessionid=105048 proto=6 action=\"accept\" policyid=1 policytype=\"policy\" service=\"HTTP\" dstcountry=\"Canada\" srccountry=\"Reserved\" duration=116 sentbyte=1188 rcvdbyte=1224 sentpkt=17 rcvdpkt=16 hostname=\"example.com\" user=\"alice\" group=\"users\""}

2. Traffic - Forward Deny

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-10 time=14:25:30 logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"warning\" vd=\"vdom1\" eventtime=1557523530 srcip=192.168.1.200 srcport=45678 srcintf=\"port12\" srcintfrole=\"lan\" dstip=10.0.0.50 dstport=22 dstintf=\"port11\" dstintfrole=\"wan\" sessionid=105050 proto=6 action=\"deny\" policyid=2 policytype=\"policy\" service=\"SSH\" dstcountry=\"Reserved\" srccountry=\"Reserved\" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=1 rcvdpkt=0"}

3. UTM - IPS (Intrusion Prevention)

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-15 time=17:56:41 logid=\"0419016384\" type=\"utm\" subtype=\"ips\" eventtype=\"signature\" level=\"alert\" vd=\"root\" eventtime=1557968201 severity=\"critical\" srcip=10.1.100.22 srccountry=\"Reserved\" dstip=172.16.200.55 srcintf=\"port10\" srcintfrole=\"lan\" dstintf=\"port9\" dstintfrole=\"wan\" sessionid=4017 action=\"dropped\" proto=6 service=\"HTTP\" policyid=1 attack=\"Adobe.Flash.newfunction.Handling.Code.Execution\" srcport=46810 dstport=80 hostname=\"172.16.200.55\" url=\"/ips/sig1.pdf\" direction=\"incoming\" attackid=23305 profile=\"block-critical-ips\" msg=\"IPS attack detected\" crscore=50"}

4. UTM - Antivirus

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-13 time=11:45:03 logid=\"0211008192\" type=\"utm\" subtype=\"virus\" eventtype=\"infected\" level=\"warning\" vd=\"vdom1\" eventtime=1557773103 msg=\"File is infected.\" action=\"blocked\" service=\"HTTP\" sessionid=359260 srcip=10.1.100.11 dstip=172.16.200.55 srcport=60446 dstport=80 srcintf=\"port12\" srcintfrole=\"lan\" dstintf=\"port11\" dstintfrole=\"wan\" policyid=4 proto=6 direction=\"incoming\" filename=\"eicar.com\" virus=\"EICAR_TEST_FILE\" dtype=\"Virus\" virusid=2172 url=\"http://172.16.200.55/virus/eicar.com\" profile=\"g-default\" crscore=50"}

5. UTM - Web Filter

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-13 time=16:29:45 logid=\"0316013056\" type=\"utm\" subtype=\"webfilter\" eventtype=\"ftgd_blk\" level=\"warning\" vd=\"vdom1\" eventtime=1557790184 policyid=1 sessionid=381780 srcip=10.1.100.11 srcport=44258 srcintf=\"port12\" srcintfrole=\"lan\" dstip=185.244.31.158 dstport=80 dstintf=\"port11\" dstintfrole=\"wan\" proto=6 service=\"HTTP\" hostname=\"morrishittu.ddns.net\" profile=\"test-webfilter\" action=\"blocked\" reqtype=\"direct\" url=\"/\" sentbyte=84 rcvdbyte=0 direction=\"outgoing\" msg=\"URL belongs to a denied category in policy\" method=\"domain\" cat=26 catdesc=\"Malicious Websites\" crscore=30"}

6. UTM - Application Control

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-15 time=18:03:36 logid=\"1059028704\" type=\"utm\" subtype=\"app-ctrl\" eventtype=\"app-ctrl-all\" level=\"information\" vd=\"root\" eventtime=1557968615 appid=40568 srcip=10.1.100.22 dstip=195.8.215.136 srcport=50798 dstport=443 srcintf=\"port10\" srcintfrole=\"lan\" dstintf=\"port9\" dstintfrole=\"wan\" proto=6 service=\"HTTPS\" direction=\"outgoing\" policyid=1 sessionid=4414 applist=\"block-social.media\" appcat=\"Web.Client\" app=\"HTTPS.BROWSER\" action=\"pass\" hostname=\"www.dailymotion.com\" url=\"/\" msg=\"Web.Client: HTTPS.BROWSER,\" apprisk=\"medium\""}

7. UTM - DNS Query

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-15 time=15:05:49 logid=\"1501054802\" type=\"utm\" subtype=\"dns\" eventtype=\"dns-response\" level=\"notice\" vd=\"vdom1\" eventtime=1557957949 policyid=1 sessionid=6887 srcip=10.1.100.22 srcport=50002 srcintf=\"port12\" srcintfrole=\"lan\" dstip=172.16.100.100 dstport=53 dstintf=\"port11\" dstintfrole=\"wan\" proto=17 profile=\"dnsfilter_fgd\" xid=57945 qname=\"changelogs.ubuntu.com\" qtype=\"AAAA\" qtypeval=28 qclass=\"IN\" ipaddr=\"2001:67c:1560:8008::11\" msg=\"Domain is monitored\" action=\"pass\" cat=52 catdesc=\"Information Technology\""}

8. Event - System Authentication Success

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-13 time=11:20:54 logid=\"0100032001\" type=\"event\" subtype=\"system\" level=\"information\" vd=\"vdom1\" eventtime=1557771654 logdesc=\"Admin login successful\" user=\"admin\" ui=\"ssh(172.16.200.254)\" method=\"ssh\" srcip=172.16.200.254 dstip=172.16.200.2 action=\"login\" status=\"success\" reason=\"none\" profile=\"super_admin\" msg=\"Administrator admin logged in successfully from ssh(172.16.200.254)\""}

9. Event - User Authentication Failed

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-13 time=15:55:56 logid=\"0102043009\" type=\"event\" subtype=\"user\" level=\"warning\" vd=\"root\" eventtime=1557788156 logdesc=\"Authentication failed\" srcip=10.1.100.11 dstip=172.16.200.55 policyid=1 interface=\"port10\" user=\"bob\" group=\"local-group1\" authproto=\"TELNET(10.1.100.11)\" action=\"authentication\" status=\"failure\" reason=\"invalid_password\" msg=\"User bob failed in authentication\""}

10. UTM - Anomaly (DoS)

{"timestamp":"2026-01-07T07:55:06.703Z","message":"date=2019-05-13 time=17:05:59 logid=\"0720018433\" type=\"utm\" subtype=\"anomaly\" eventtype=\"anomaly\" level=\"alert\" vd=\"vdom1\" eventtime=1557792359 severity=\"critical\" srcip=10.1.100.11 srccountry=\"Reserved\" dstip=172.16.200.55 srcintf=\"port12\" srcintfrole=\"undefined\" sessionid=0 action=\"clear_session\" proto=1 service=\"PING\" count=1 attack=\"icmp_flood\" attackid=16777316 policyid=1 policytype=\"DoS-policy\" msg=\"anomaly: icmp_flood, 51 > threshold 50\" crscore=50"}

Parser Configuration

#regex


#conditional
log_message = ""
if (msg, err = to_string(.message); err == null) { 
  log_message = msg
}

event_timestamp = ""
log_date = ""
log_time = ""
event_module = "fortigate"
log_id = ""
log_type = ""
log_subtype = ""
event_category = ""
event_action = ""
event_outcome = ""
log_level = "info"

source_ip = ""
source_port = ""
source_interface = ""
destination_ip = ""
destination_port = ""
destination_interface = ""
network_protocol = ""
network_transport = ""

user_name = ""
group_name = ""

policy_id = ""
policy_uuid = ""
session_id = ""
action = ""

threat_name = ""
threat_category = ""
attack_name = ""
attack_id = ""

url_full = ""
url_domain = ""
file_name = ""

server_address = ""
client_address = ""

if (date_m, err = parse_regex(log_message, r'date=(?P<date>\d{4}-\d{2}-\d{2})'); err == null) {
  log_date = to_string(date_m.date)
}

if (time_m, err = parse_regex(log_message, r'time=(?P<time>\d{2}:\d{2}:\d{2})'); err == null) {
  log_time = to_string(time_m.time)
}


if (logid_m, err = parse_regex(log_message, r'logid="(?P<id>[^"]+)"'); err == null) {
  log_id = to_string(logid_m.id)
}

if (type_m, err = parse_regex(log_message, r'\stype="(?P<type>[^"]+)"'); err == null) {
  log_type = to_string(type_m.type)
}

if (subtype_m, err = parse_regex(log_message, r'subtype="(?P<subtype>[^"]+)"'); err == null) {
  log_subtype = to_string(subtype_m.subtype)
}

if (level_m, err = parse_regex(log_message, r'level="(?P<level>[^"]+)"'); err == null) {
  log_level = to_string(level_m.level)
}

if (srcip_m, err = parse_regex(log_message, r'srcip=(?P<ip>[0-9.]+)'); err == null) {
  source_ip = to_string(srcip_m.ip)
}

if (srcport_m, err = parse_regex(log_message, r'srcport=(?P<port>\d+)'); err == null) {
  source_port = to_string(srcport_m.port)
}

if (srcintf_m, err = parse_regex(log_message, r'srcintf="(?P<intf>[^"]+)"'); err == null) {
  source_interface = to_string(srcintf_m.intf)
}

if (dstip_m, err = parse_regex(log_message, r'dstip=(?P<ip>[0-9.]+)'); err == null) {
  destination_ip = to_string(dstip_m.ip)
}

if (dstport_m, err = parse_regex(log_message, r'dstport=(?P<port>\d+)'); err == null) {
  destination_port = to_string(dstport_m.port)
}

if (dstintf_m, err = parse_regex(log_message, r'dstintf="(?P<intf>[^"]+)"'); err == null) {
  destination_interface = to_string(dstintf_m.intf)
}

if (proto_m, err = parse_regex(log_message, r'\sproto=(?P<proto>\d+)'); err == null) {
  proto_num = to_string(proto_m.proto)
  if proto_num == "6" { network_transport = "tcp" }
  if proto_num == "17" { network_transport = "udp" }
  if proto_num == "1" { network_transport = "icmp" }
}

if (service_m, err = parse_regex(log_message, r'service="(?P<service>[^"]+)"'); err == null) {
  network_protocol = downcase(to_string(service_m.service))
}

if (policyid_m, err = parse_regex(log_message, r'policyid=(?P<id>\d+)'); err == null) {
  policy_id = to_string(policyid_m.id)
}

if (poluuid_m, err = parse_regex(log_message, r'poluuid="(?P<uuid>[^"]+)"'); err == null) {
  policy_uuid = to_string(poluuid_m.uuid)
}

if (sessionid_m, err = parse_regex(log_message, r'sessionid=(?P<id>\d+)'); err == null) {
  session_id = to_string(sessionid_m.id)
}

if (action_m, err = parse_regex(log_message, r'action="(?P<action>[^"]+)"'); err == null) {
  action = to_string(action_m.action)
}

if (user_m, err = parse_regex(log_message, r'\suser="(?P<user>[^"]+)"'); err == null) {
  user_name = to_string(user_m.user)
}

if (group_m, err = parse_regex(log_message, r'group="(?P<group>[^"]+)"'); err == null) {
  group_name = to_string(group_m.group)
}

if (hostname_m, err = parse_regex(log_message, r'hostname="(?P<host>[^"]+)"'); err == null) {
  url_domain = to_string(hostname_m.host)
  server_address = to_string(hostname_m.host)
}

if (url_m, err = parse_regex(log_message, r'\surl="(?P<url>[^"]+)"'); err == null) {
  url_full = to_string(url_m.url)
}

if (filename_m, err = parse_regex(log_message, r'filename="(?P<file>[^"]+)"'); err == null) {
  file_name = to_string(filename_m.file)
}

if (attack_m, err = parse_regex(log_message, r'attack="(?P<attack>[^"]+)"'); err == null) {
  attack_name = to_string(attack_m.attack)
  threat_name = attack_name
}

if (attackid_m, err = parse_regex(log_message, r'attackid=(?P<id>\d+)'); err == null) {
  attack_id = to_string(attackid_m.id)
}

if (virus_m, err = parse_regex(log_message, r'virus="(?P<virus>[^"]+)"'); err == null) {
  threat_name = to_string(virus_m.virus)
}

if (catdesc_m, err = parse_regex(log_message, r'catdesc="(?P<cat>[^"]+)"'); err == null) {
  threat_category = to_string(catdesc_m.cat)
}

log_type_str = to_string(log_type)
log_subtype_str = to_string(log_subtype)
action_str = to_string(action)

if log_type_str == "traffic" {
  event_category = "network"
  event_module = "fortigate.traffic"
  
  if action_str == "accept" {
    event_action = "allowed"
    event_outcome = "success"
  }
  if action_str == "deny" {
    event_action = "denied"
    event_outcome = "failure"
  }
  if action_str == "close" {
    event_action = "connection_closed"
    event_outcome = "success"
  }
}

if log_type_str == "utm" {
  if log_subtype_str == "ips" {
    event_category = "intrusion_detection"
    event_module = "fortigate.ips"
    
    if action_str == "dropped" {
      event_action = "blocked"
      event_outcome = "success"
    }
    if action_str == "detected" {
      event_action = "detected"
      event_outcome = "success"
    }
  }
  
  if log_subtype_str == "virus" {
    event_category = "malware"
    event_module = "fortigate.antivirus"
    
    if action_str == "blocked" {
      event_action = "blocked"
      event_outcome = "success"
    }
    if action_str == "detected" {
      event_action = "detected"
      event_outcome = "success"
    }
  }
  
  if log_subtype_str == "webfilter" {
    event_category = "web"
    event_module = "fortigate.webfilter"
    
    if action_str == "blocked" {
      event_action = "blocked"
      event_outcome = "success"
    }
    if action_str == "passthrough" {
      event_action = "allowed"
      event_outcome = "success"
    }
  }
  
  if log_subtype_str == "app-ctrl" {
    event_category = "network"
    event_module = "fortigate.application"
    
    if action_str == "blocked" {
      event_action = "blocked"
      event_outcome = "success"
    }
    if action_str == "pass" {
      event_action = "allowed"
      event_outcome = "success"
    }
  }
  
  if log_subtype_str == "dns" {
    event_category = "network"
    event_module = "fortigate.dns"
    
    if action_str == "pass" {
      event_action = "allowed"
      event_outcome = "success"
    }
    if action_str == "blocked" {
      event_action = "blocked"
      event_outcome = "success"
    }
  }
  
  if log_subtype_str == "anomaly" {
    event_category = "intrusion_detection"
    event_module = "fortigate.anomaly"
    threat_name = attack_name
    
    if action_str == "clear_session" {
      event_action = "blocked"
      event_outcome = "success"
    }
  }
}

if log_type_str == "event" {
  if log_subtype_str == "system" {
    event_category = "authentication"
    event_module = "fortigate.event"
    
    if contains(log_message, "login") {
      if contains(log_message, "success") {
        event_action = "logon_success"
        event_outcome = "success"
      }
      if contains(log_message, "failed") {
        event_action = "logon_failed"
        event_outcome = "failure"
      }
    }
  }
  
  if log_subtype_str == "user" {
    event_category = "authentication"
    event_module = "fortigate.event"
    
    if contains(log_message, "authentication") {
      if contains(log_message, "success") {
        event_action = "logon_success"
        event_outcome = "success"
      }
      if contains(log_message, "failure") {
        event_action = "logon_failed"
        event_outcome = "failure"
      }
    }
  }
  
  if log_subtype_str == "vpn" {
    event_category = "network"
    event_module = "fortigate.vpn"
    event_action = "vpn_event"
    event_outcome = "success"
  }
  
  if log_subtype_str == "router" {
    event_category = "network"
    event_module = "fortigate.router"
    event_action = "routing_event"
    event_outcome = "success"
  }
}

if source_ip != "" { client_address = source_ip }

if event_category == "" { event_category = "network" }
if event_action == "" { event_action = "info" }
if event_outcome == "" { event_outcome = "unknown" }

#normalize
timestamp: format_timestamp!(parse_timestamp!(.timestamp, "%Y-%m-%dT%H:%M:%S.%3fZ"), "%Y-%m-%d %H:%M:%S")
event.module: event_module
event.category: event_category
event.action: event_action
event.outcome: event_outcome
log.level: log_level
message: log_message
source.ip: source_ip
source.port: source_port
destination.ip: destination_ip
destination.port: destination_port
network.transport: network_transport
network.protocol: network_protocol
user.name: user_name
group.name: group_name
url.full: url_full
url.domain: url_domain
file.name: file_name
server.address: server_address
client.address: client_address
threat.name: threat_name
labels.log_id: log_id
labels.log_type: log_type
labels.log_subtype: log_subtype
labels.policy_id: policy_id
labels.policy_uuid: policy_uuid
labels.session_id: session_id
labels.action: action
labels.source_interface: source_interface
labels.destination_interface: destination_interface
labels.threat_category: threat_category
labels.attack_name: attack_name
labels.attack_id: attack_id

Event Categories

Log TypeSubtypeActionEvent CategoryEvent ActionEvent Outcome
trafficforwardacceptnetworkallowedsuccess
trafficforwarddenynetworkdeniedfailure
trafficforwardclosenetworkconnection_closedsuccess
utmipsdroppedintrusion_detectionblockedsuccess
utmipsdetectedintrusion_detectiondetectedsuccess
utmvirusblockedmalwareblockedsuccess
utmwebfilterblockedwebblockedsuccess
utmapp-ctrlblockednetworkblockedsuccess
utmapp-ctrlpassnetworkallowedsuccess
utmdnspassnetworkallowedsuccess
utmanomalyclear_sessionintrusion_detectionblockedsuccess
eventsystemlogin successauthenticationlogon_successsuccess
eventsystemlogin failedauthenticationlogon_failedfailure
eventuserauthentication successauthenticationlogon_successsuccess
eventuserauthentication failureauthenticationlogon_failedfailure

Field Mapping

FortiGate FieldECS FieldField Set
date + timetimestampBase
srcipsource.ip, client.addressSource
srcportsource.portSource
srcintflabels.source_interfaceLabels
dstipdestination.ipDestination
dstportdestination.portDestination
dstintflabels.destination_interfaceLabels
protonetwork.transportNetwork
servicenetwork.protocolNetwork
useruser.nameUser
groupgroup.nameGroup
hostnameurl.domain, server.addressURL, Server
urlurl.fullURL
filenamefile.nameFile
attackthreat.nameThreat
virusthreat.nameThreat
catdesclabels.threat_categoryLabels
policyidlabels.policy_idLabels
sessionidlabels.session_idLabels
actionlabels.actionLabels

Notes

Input Format: JSON object với field .message chứa FortiGate key-value pairs.

Parsing Strategy:

  • Extract .message field từ JSON wrapper
  • Parse key-value pairs với regex key="value" hoặc key=number
  • Combine date + time thành timestamp
  • Map proto numbers: 6=tcp, 17=udp, 1=icmp
  • Normalize log.level và event fields theo ECS

VRL Functions Used:

  • to_string() - Type coercion
  • parse_regex() - Key-value extraction
  • contains() - String matching
  • downcase() - Lowercase conversion

Test Results: Xem TEST_RESULTS.md cho chi tiết test cases.

logo
CMC Telecom
Aspire to Inspire the Digital World