Collecting Open-Source Intelligence

When to Use

Use this skill when:

• Investigating external infrastructure associated with a phishing campaign targeting your organization
• Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
• Conducting authorized attack surface discovery to understand your organization's external exposure

Do not use this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.

Prerequisites

• Maltego CE or commercial license for graph-based link analysis
• Shodan API key (https://shodan.io) for internet-wide device/service discovery
• OSINT Framework familiarity (https://osintframework.com) for tool selection
• SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation

Workflow

Step 1: Define Collection Requirements

Establish the intelligence requirement (IR) before collecting. Document:

• Target: threat actor group, malicious domain, IP range, or organization
• Priority Intelligence Requirements (PIRs): What specific questions need answering?
• Legal authority: Passive OSINT is legal; active probing requires authorization
• Data handling: TLP classification for collected intelligence

Step 2: Passive DNS and WHOIS Investigation

# Passive DNS via SecurityTrails API
curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
  -H "apikey: YOUR_KEY"

# WHOIS history via ARIN / RIPE
whois -h whois.arin.net evil-domain.com

# Certificate transparency logs (no API key required)
curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'

Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.

Step 3: Shodan Infrastructure Mapping

import shodan

api = shodan.Shodan("YOUR_SHODAN_API_KEY")

# Search for specific C2 framework signatures (Cobalt Strike beacon)
results = api.search('product:"Cobalt Strike" port:443')
for r in results['matches']:
    print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))

# Find infrastructure associated with a known threat actor's ASN
results = api.search('asn:AS12345 http.title:"Redirector"')

Correlate Shodan results with passive DNS to build infrastructure clusters.

Step 4: Maltego Graph Analysis

In Maltego, use these built-in transforms for threat actor infrastructure mapping:

1. Start with a known malicious domain (Entity: Domain)
2. Run "To IP Address [DNS]" → identifies hosting IPs
3. Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
4. Run "To DNS Name [Reverse DNS]" → identifies PTR records
5. Run "To Whois" → identifies registrant email/organization
6. Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email

Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.

Step 5: Dark Web and Paste Site Monitoring

Use SpiderFoot HX or manual searches for:

• Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
• Dark web forums: via Tor browser with appropriate operational security
• GitHub/GitLab: search for exposed credentials or organization-specific strings

# SpiderFoot CLI for automated OSINT
python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
  -o TF -R result.json

Key Concepts

Term	Definition
Passive OSINT	Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data
PIR	Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering
Certificate Transparency	Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh
Pivoting	Using one data point (IP, email, registrant name) to discover related infrastructure or accounts
ASN	Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure
Co-hosted Domains	Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure

Tools & Systems

• Maltego: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
• Shodan: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
• SpiderFoot: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
• Recon-ng: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
• crt.sh: Free certificate transparency search engine for subdomain and certificate discovery
• OSINT Framework (osintframework.com): Curated directory of OSINT tools organized by intelligence category

Common Pitfalls

• Leaving digital footprints: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
• Confirmation bias in graph analysis: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
• Outdated data: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
• Attribution overconfidence: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
• Legal boundaries: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.

Verification Criteria

Confirm successful execution by validating:

• [ ] All prerequisite tools and access requirements are satisfied
• [ ] Each workflow step completed without errors
• [ ] Output matches expected format and contains expected data
• [ ] No security warnings or misconfigurations detected
• [ ] Results are documented and evidence is preserved for audit

Compliance Framework Mapping

This skill supports compliance evidence collection across multiple frameworks:

• SOC 2: CC7.1 (Monitoring), CC7.2 (Anomaly Detection)
• ISO 27001: A.6.1 (Threat Intelligence), A.16.1 (Security Incident Management)
• NIST 800-53: PM-16 (Threat Awareness), RA-3 (Risk Assessment), SI-5 (Security Alerts)
• NIST CSF: ID.RA (Risk Assessment), DE.AE (Anomalies & Events)

Claw GRC Tip: When this skill is executed by a registered agent, compliance evidence is automatically captured and mapped to the relevant controls in your active frameworks.

Deploying This Skill with Claw GRC

Agent Execution

Register this skill with your Claw GRC agent for automated execution:

# Install via CLI
npx claw-grc skills add collecting-open-source-intelligence

# Or load dynamically via MCP
grc.load_skill("collecting-open-source-intelligence")

Audit Trail Integration

When executed through Claw GRC, every step of this skill generates tamper-evident audit records:

• SHA-256 chain hashing ensures no step can be modified after execution
• Evidence artifacts (configs, scan results, logs) are automatically attached to relevant controls
• Trust score impact — successful execution increases your agent's trust score

Continuous Compliance

Schedule this skill for recurring execution to maintain continuous compliance posture. Claw GRC monitors for drift and alerts when re-execution is needed.