Expanding Your Detection Horizon: Key Data Sources Across the IT Ecosystem

Overview

In modern cybersecurity, relying solely on endpoint detection—such as antivirus or EDR tools—leaves critical blind spots. Attackers often move laterally through networks, exploit cloud misconfigurations, or abuse identity systems without ever touching an endpoint. This tutorial, inspired by Unit 42's guidance on comprehensive security strategies spanning every IT zone, will help you identify and integrate essential data sources beyond the endpoint. You'll learn how to collect, normalize, and analyze logs from network, cloud, identity, and application layers to build a holistic detection capability.

Prerequisites

Before diving in, ensure you have:

• Basic understanding of cybersecurity concepts (e.g., threat detection, log analysis).
• Access to a centralized logging platform (e.g., SIEM like Splunk, ELK stack, or cloud-native solutions).
• Administrative privileges on at least one network device, cloud account, or identity provider for configuration.
• Familiarity with common log formats (e.g., syslog, JSON, Windows Event Log).

Step-by-Step Instructions

1. Identify the IT Zones

Start by mapping your IT environment into four primary zones beyond endpoints:

• Network Zone: Firewalls, routers, switches, DNS servers, proxies.
• Cloud Zone: Cloud service provider logs (AWS CloudTrail, Azure Activity Log, GCP Audit Logs).
• Identity Zone: Active Directory, Azure AD, Okta, LDAP, SSO providers.
• Application Zone: Web servers, databases, API gateways, custom applications.

For this tutorial, we'll focus on one example per zone.

2. Configure Network Data Collection

Network devices generate flow logs, firewall rule hits, and DNS queries. For a network firewall (e.g., pfSense), configure syslog forwarding:

1. Go to Status > System Logs > Settings.
2. Enable Remote Logging and enter your SIEM's IP and port (e.g., 514 for UDP syslog).
3. Choose log types: firewall rules, DNS queries, and VPN events.
4. Example pfSense syslog configuration snippet (in XML config):

   <syslog>
     <enable>true</enable>
     <logguid>none</logguid>
     <remoteserver>192.168.1.100</remoteserver>
     <remoteport>514</remoteport>
     <sourceip>0.0.0.0</sourceip>
     <ipproto>ipv4</ipproto>
   </syslog>

Once logs arrive, parse them in your SIEM to extract source/destination IPs, ports, and actions (allow/block).

3. Integrate Cloud Logs

For AWS, enable CloudTrail for all regions and management events. To push logs to your SIEM:

1. Create an S3 bucket (e.g., my-cloudtrail-logs) with appropriate permissions.
2. In CloudTrail, create a trail pointing to that bucket.
3. Configure your SIEM to ingest from S3 (e.g., using Lambda or a connector).
4. Sample CloudTrail event in JSON:

   {
     "eventVersion": "1.08",
     "userIdentity": { "arn": "arn:aws:iam::123456789012:user/Admin" },
     "eventTime": "2025-03-10T14:30:00Z",
     "eventSource": "ec2.amazonaws.com",
     "eventName": "RunInstances",
     "awsRegion": "us-east-1",
     "sourceIPAddress": "203.0.113.50",
     "responseElements": {
       "instancesSet": { "items": [{ "instanceId": "i-0abcd1234efgh5678" }] }
     }
   }

Focus on eventName values like CreateUser, ModifyNetworkAcl, or UnauthorizedOperation.

4. Capture Identity Logs

Active Directory generates event IDs for logon, privilege escalation, and group changes. Use Windows Event Forwarding (WEF) to centralize:

1. On Domain Controllers, enable logging for security events (e.g., 4624 – successful logon, 4732 – user added to security group).
2. Configure WEF collector and push events to SIEM via wevtutil commands.
3. Example PowerShell to query a specific event:

   Get-WinEvent -FilterHashtable @{LogName='Security'; ID=4624} |
     Select-Object TimeCreated, @{n='User';e={$_.Properties[5].Value}},
                  @{n='SourceIP';e={$_.Properties[18].Value}} |
     Export-Csv -Path 'C:\logs\4624.csv' -NoTypeInformation

Look for patterns like brute-force (many 4625) or lateral movement (logon type 3 from unusual IPs).

5. Application Logs

For a web application (e.g., Nginx), access logs contain request details. Configure structured logging:

1. Edit /etc/nginx/nginx.conf to use JSON log format:

   log_format json escape=json '{ "timestamp": "$time_iso8601", '
     '"remote_addr": "$remote_addr", '
     '"request": "$request", '
     '"status": "$status", '
     '"body_bytes_sent": "$body_bytes_sent", '
     '"http_user_agent": "$http_user_agent" }';
   
   access_log /var/log/nginx/access.json json;

2. Reload Nginx: nginx -s reload.
3. Forward via syslog or filebeat to SIEM.

Detect SQL injection by pattern matching ' OR 1=1 in the request field.

6. Correlate and Detect

With all data sources flowing, create detection rules. Example for a SIEM rule (pseudo-code):

alert when
  (from network firewall: action = 'block' AND dest_port = 3389)
  AND
  (from identity logs: eventID = 4625 AND targetUser = 'admin')
  AND
  (from cloud logs: eventName = 'ConsoleLogin' AND sourceIP in list_of_blocked_IPs)
  then
  alert: 'Potential brute-force across zones'

This rule correlates firewall blocks with failed logins and cloud console access from same IP.

Common Mistakes

• Ignoring timestamp synchronization: Without NTP across devices, correlation fails. Always enable NTP on all log sources.
• Over-collecting without normalization: Sending raw logs without field extraction leads to noise. Use parsing rules or log shippers early.
• Missing context fields: Ensure logs include user identity, source/destination IPs, and action. Without these, detection rules are blind.
• Only enabling default logging: Many services disable verbose logging by default. For example, Azure AD Premium P2 is needed for Identity Protection logs.
• Forgetting data retention: Regulatory requirements (e.g., SOC 2, GDPR) often mandate 90–365 days. Plan storage and archiving accordingly.

Summary

Building detection beyond the endpoint requires gathering data from network, cloud, identity, and application zones. By configuring syslog, CloudTrail, Windows Event Forwarding, and structured web logs—and correlating them—you can detect attacks that span the entire IT ecosystem. Start small with one zone, then expand. Remember to normalize, timestamp, and retain logs for forensic use.