Lightweight

Kịch bản 4 Multiple receivers → Multiple exporters

Hướng dẫn cấu hình nhiều nguồn log gửi tới nhiều destinations

Tổng quan

Đây là kịch bản cấu hình nâng cao, cho phép định tuyến dòng dữ liệu (Data Pipeline) một cách độc lập và linh hoạt. Sensor sẽ thu thập từ nhiều nguồn log khác nhau, sau đó phân phối chuẩn xác từng loại log đến các điểm nhận (Exporters) chuyên biệt hoặc phân tách thành các luồng chạy song song để đảm bảo tính sẵn sàng cao (High Availability).

Cấu hình chi tiết

1. Receivers (Bộ nhận) - Nhiều nguồn

receivers:
  filelog/app:                    # Receiver 1: Application logs
    include:
      - /home/bacnt/logs/*.log    # Collect từ thư mục ứng dụng
    start_at: end
    include_file_path: true       # Thêm đường dẫn file
    include_file_name: false

  filelog/syslog:                 # Receiver 2: System logs
    include:
      - /var/log/syslog           # Collect từ syslog
    start_at: end
    include_file_path: true
    include_file_name: false

  filelog/auth:                   # Receiver 3: Authentication logs
    include:
      - /var/log/auth.log         # Collect từ auth log
    start_at: end
    include_file_path: true
    include_file_name: false

2. Processors (Bộ xử lý)

Đọc chi tiết

3. Exporters (Bộ gửi) - Nhiều điểm đến

exporters:
  otlphttp/primary:               # Exporter 1: Platform chính
    endpoint: ${PRIMARY_ENDPOINT}
    logs_endpoint: ${PRIMARY_ENDPOINT}
    headers:
      x-api-key: ${PRIMARY_API_KEY}
      Content-Type: "application/json"
    timeout: 30s
    encoding: json

  otlphttp/backup:                # Exporter 2: Platform backup
    endpoint: ${BACKUP_ENDPOINT}
    logs_endpoint: ${BACKUP_ENDPOINT}
    headers:
      x-api-key: ${BACKUP_API_KEY}
      Content-Type: "application/json"
    timeout: 30s
    encoding: json

  syslog:                         # Exporter 3: Syslog server
    endpoint: syslog-server.company.com:514
    protocol: rfc5424
    format: json

4. Service Pipeline (Đường ống xử lý)

extensions:
  health_check:
    endpoint: localhost:13133

service:
  extensions: [health_check]
  
  pipelines:
    # Application logs → Production + Backup (HA)
    logs/app-ha:
      receivers: [filelog/app]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production, otlphttp/backup]

    # System logs → Production only
    logs/system:
      receivers: [filelog/system]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production]

    # Security logs → Production + Staging (audit trail)
    logs/security:
      receivers: [filelog/security]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production, otlphttp/staging]

    # Aggregated logs → All backends (mirroring)
    logs/mirror:
      receivers: [filelog/app, filelog/system, filelog/security]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production, otlphttp/backup, otlphttp/staging]

Ví dụ thực tế

Ví dụ: Gửi Security log tới Platform chính + Backup + Archive

receivers:
  # Collect from application logs
  filelog/app:
    include:
      - /home/bacnt/logs/app/*.log
      - /home/bacnt/logs/api/*.log
    start_at: end
    include_file_path: true
    include_file_name: true
    multiline_line_start_pattern: '^\d{4}-\d{2}-\d{2}'

  # Collect from system logs
  filelog/system:
    include:
      - /var/log/syslog
      - /var/log/auth.log
    start_at: end
    include_file_path: true

  # Collect from security logs
  filelog/security:
    include:
      - /var/log/audit/audit.log
    start_at: end
    include_file_path: true

processors:
  batch:
    send_batch_size: 10
    send_batch_max_size: 100
    timeout: 5s

  # Optional: Add attributes to identify logs from each source
  resource_detection:
    detectors: [system]

exporters:
  # Primary backend - Production
  otlphttp/production:
    endpoint: ${PROD_ENDPOINT}
    logs_endpoint: ${PROD_ENDPOINT}
    headers:
      x-api-key: ${PROD_API_KEY}
      Content-Type: "application/json"
    timeout: 30s
    encoding: json

  # Secondary backend - Backup/HA
  otlphttp/backup:
    endpoint: ${BACKUP_ENDPOINT}
    logs_endpoint: ${BACKUP_ENDPOINT}
    headers:
      x-api-key: ${BACKUP_API_KEY}
      Content-Type: "application/json"
    timeout: 30s
    encoding: json

  # Staging backend - Testing
  otlphttp/staging:
    endpoint: ${STAGING_ENDPOINT}
    logs_endpoint: ${STAGING_ENDPOINT}
    headers:
      x-api-key: ${STAGING_API_KEY}
      Content-Type: "application/json"
    timeout: 30s
    encoding: json

extensions:
  health_check:
    endpoint: localhost:13133

service:
  extensions: [health_check]
  
  pipelines:
    # Application logs → Production + Backup (HA)
    logs/app-ha:
      receivers: [filelog/app]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production, otlphttp/backup]

    # System logs → Production only
    logs/system:
      receivers: [filelog/system]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production]

    # Security logs → Production + Staging (audit trail)
    logs/security:
      receivers: [filelog/security]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production, otlphttp/staging]

    # Aggregated logs → All backends (mirroring)
    logs/mirror:
      receivers: [filelog/app, filelog/system, filelog/security]
      processors: [batch, resource_detection]
      exporters: [otlphttp/production, otlphttp/backup, otlphttp/staging]

Kịch bản 3 Single receiver → Multiple exporters

Hướng dẫn cấu hình một nguồn log gửi tới nhiều destinations

Data ingestion pipeline

CMC Telecom
Aspire to Inspire the Digital World