This guide explains how to configure a parser for a SecOps system to process raw log data into ECS-compliant format. The parser uses three blocks: #regex, #conditional, and #normalize. Configurations are written in plain text, processing data sequentially.

Preparation

Before configuring a parser, you need to:

1. Understand Input Data: Identify the log format (e.g., Nginx "combined" log: IP - - [time] "METHOD URL HTTP/1.1" STATUS SIZE "REFERER" "AGENT").
2. Supported Functions & ECS Fields: Refer to the full list at ECS Schema Field Supported.

Configuration Structure

The parser configuration consists of three sequential blocks:

Blocks: #regex → #conditional → #normalize

Syntax Rules

• Use .field for data access
• Use var = value for variables
• Use if-else for logic

Block Details

#regex: Parse Raw Data to JSON

Purpose: Use regex to extract fields from raw logs into JSON.

Syntax: field_name = regex_pattern (use capture groups for values)

Example:

#regex
([^"]+)

#conditional: Transform Data

Purpose: Apply logic, use built-in functions (e.g., split!, to_string), and handle errors.

Syntax: if (condition, {block}, else {block}) or var = expression

Example:

#conditional
method = ""
url_full = ""
url_path = ""
url_query = ""
if (., err = parse_nginx_log(.message, "combined"); err == null) {
  method = split!(.request, " ")[0]
  url_full = split!(.request, " ")[1]
  url_path = split(to_string(url_full), "?")[0]
  url_query = split(to_string(url_full), "?")[1]
} else {
  . = .message
}

#normalize: Map to ECS Schema

Purpose: Map processed data to ECS fields.

Syntax: ecs_field: source_var_or_field

Example:

#normalize
http.request.referrer: .referer
client.ip: .client
http.request.method: method
url.full: url_full
url.path: url_path
url.query: url_query
http.response.status: .status
agent.name: .agent
http.request.body.bytes: .size
error.message: err

Sample Configuration (Nginx Combined Log)

Here's a complete example of parsing Nginx combined log format:

#regex
([^"]+)

#conditional
method = ""
url_full = ""
url_path = ""
url_query = ""
if (., err = parse_nginx_log(.message, "combined"); err == null) {
  method = split!(.request, " ")[0]
  url_full = split!(.request, " ")[1]
  url_path = split(to_string(url_full), "?")[0]
  url_query = split(to_string(url_full), "?")[1]
} else {
  . = .message
}

#normalize
http.request.referrer: .referer
client.ip: .client
http.request.method: method
url.full: url_full
url.path: url_path
url.query: url_query
http.response.status: .status
agent.name: .agent
http.request.body.bytes: .size
error.message: err

Key Points to Remember

• Sequential Processing: Data flows through #regex → #conditional → #normalize in order
• Error Handling: Always check for errors using err variable and provide fallback logic
• ECS Compliance: Ensure all mapped fields comply with the ECS schema
• Built-in Functions: Leverage built-in parsers like parse_nginx_log when available to simplify configuration
• Testing: Always test your parser with sample logs before deploying to production