Log Sources

You need sources that generate logs, such as:

• Linux syslogs
• Windows Event Logs
• NGINX/Apache logs
• Firewall logs
• IDS/IPS logs (e.g., Snort, Suricata)

Log Shipping

Use Filebeat from Elastic (compatible with OpenSearch) to ship logs.

sudo apt install filebeat

Data Ingestion and Enrichment

Use Logstash to parse and transform logs.

              input {
                beats {
                  port => 5044
                }
              
                file {
                  path => "/var/log/suricata/eve.json"
                  codec => json
                  type => "suricata"
                }
              
                file {
                  path => "/var/log/ufw.log"
                  start_position => "beginning"
                  sincedb_path => "/dev/null"
                  type => "firewall"
                }
              }
              
              filter {
                ### -- Syslog Logs --
                if "syslog" in [tags] {
                  grok {
                    match => { "message" => "%{SYSLOGLINE}" }
                  }
                  date {
                    match => ["timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
                  }
                }
              
                ### -- Web Server Logs --
                if "nginx" in [tags] or "apache" in [tags] {
                  grok {
                    match => { "message" => "%{COMBINEDAPACHELOG}" }
                  }
                  date {
                    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
                  }
                  geoip {
                    source => "clientip"
                  }
                }
              
                ### -- Suricata Logs --
                if [type] == "suricata" and [event_type] == "alert" {
                  mutate {
                    add_tag => ["suricata_alert"]
                  }
                }
              
                ### -- Windows Event Logs --
                if "winlogbeat" in [agent][type] or "windows" in [tags] {
                  mutate {
                    add_tag => ["windows_event"]
                  }
                }
              
                ### -- Firewall Logs --
                if [type] == "firewall" {
                  grok {
                    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{IP:src_ip} .*DPT=%{NUMBER:dest_port}" }
                  }
                  date {
                    match => ["timestamp", "MMM dd HH:mm:ss"]
                  }
                }
              }
              
              output {
                if "syslog" in [tags] {
                  opensearch {
                    hosts => ["http://localhost:9200"]
                    index => "syslog-%{+YYYY.MM.dd}"
                  }
                } else if "nginx" in [tags] or "apache" in [tags] {
                  opensearch {
                    hosts => ["http://localhost:9200"]
                    index => "web-logs-%{+YYYY.MM.dd}"
                  }
                } else if "suricata_alert" in [tags] {
                  opensearch {
                    hosts => ["http://localhost:9200"]
                    index => "suricata-%{+YYYY.MM.dd}"
                  }
                } else if "windows_event" in [tags] {
                  opensearch {
                    hosts => ["http://localhost:9200"]
                    index => "windows-logs-%{+YYYY.MM.dd}"
                  }
                } else if [type] == "firewall" {
                  opensearch {
                    hosts => ["http://localhost:9200"]
                    index => "firewall-logs-%{+YYYY.MM.dd}"
                  }
                } else {
                  # Default catch-all
                  opensearch {
                    hosts => ["http://localhost:9200"]
                    index => "misc-logs-%{+YYYY.MM.dd}"
                  }
                }
              }
              

Then you can add the tags in the config of filebeat:

              filebeat.inputs:
                - type: log
                  paths:
                    - /var/log/syslog
                  tags: ["syslog"]
              
                - type: log
                  paths:
                    - /var/log/nginx/access.log
                  tags: ["nginx"]
              
                - type: log
                  paths:
                    - /var/log/apache2/access.log
                  tags: ["apache"]
              

Then restart logstash service.

sudo systemctl restart logstash

Index in OpenSearch

Create indices with appropriate mappings for fields such as:

• timestamp
• source_ip, destination_ip
• log_level
• event_type
• threat_indicator

This allows fast querying and filtering of logs for threat detection.

Dashboards

Use OpenSearch Dashboards to:

• Visualize common threat indicators (e.g., failed logins, traffic from malicious IPs)
• Set up filters and custom visualizations
• Create dashboards showing:
• Top source IPs
• GeoIP heatmaps
• Number of failed login attempts
• Suspicious processes started

Real-Time Threat Detection

Use OpenSearch Alerting Plugin (installed by default in OpenSearch Dashboards).

Example: Alert on failed SSH logins

Monitor query:

              {
                "size": 0,
                "query": {
                  "bool": {
                    "must": [
                      { "match": { "event_type": "ssh_login_failed" } },
                      { "range": { "@timestamp": { "gte": "now-1m" } } }
                    ]
                  }
                }
              }

Trigger condition:

return ctx.results[0].hits.total.value > 10;