CG
SkillsDetecting Rootkit Activity
Start Free
Back to Skills Library
Malware Analysis๐Ÿ”ด Advanced

Detecting Rootkit Activity

Detects rootkit presence on compromised systems by identifying hidden processes, hooked system calls, modified kernel structures, hidden files, and covert network connections using memory forensics, cross-view detection, and integrity checking techniques.

7 min read8 code examples

Prerequisites

  • Volatility 3 for memory forensics and kernel structure analysis
  • GMER or Rootkit Revealer (Windows) for live system scanning
  • rkhunter and chkrootkit (Linux) for filesystem and process integrity checks
  • Sysinternals tools (Process Explorer, Autoruns, RootkitRevealer) for Windows analysis
  • Memory dump from the suspected system (WinPmem, LiME)
  • Clean baseline of the OS for comparison (known-good kernel module hashes)

Detecting Rootkit Activity

When to Use

  • System shows signs of compromise but standard tools (Task Manager, netstat) show nothing abnormal
  • Antivirus/EDR detects rootkit signatures but cannot identify the specific hiding mechanism
  • Memory forensics reveals discrepancies between kernel data structures and user-mode tool output
  • Investigating a persistent threat that survives remediation attempts and system reboots
  • Validating system integrity after a suspected kernel-level compromise

Do not use as a first-line detection method; start with standard malware triage and escalate to rootkit analysis when hiding behavior is suspected.

Prerequisites

  • Volatility 3 for memory forensics and kernel structure analysis
  • GMER or Rootkit Revealer (Windows) for live system scanning
  • rkhunter and chkrootkit (Linux) for filesystem and process integrity checks
  • Sysinternals tools (Process Explorer, Autoruns, RootkitRevealer) for Windows analysis
  • Memory dump from the suspected system (WinPmem, LiME)
  • Clean baseline of the OS for comparison (known-good kernel module hashes)

Workflow

Step 1: Cross-View Detection for Hidden Processes

Compare process lists from different data sources to find discrepancies:

# Volatility: Compare process enumeration methods
# pslist - walks ActiveProcessLinks (EPROCESS linked list - what rootkits manipulate)
vol3 -f memory.dmp windows.pslist > pslist_output.txt

# psscan - scans physical memory for EPROCESS pool tags (rootkit-resistant)
vol3 -f memory.dmp windows.psscan > psscan_output.txt

# Compare outputs to find hidden processes
python3 << 'PYEOF'
pslist_pids = set()
psscan_pids = set()

with open("pslist_output.txt") as f:
    for line in f:
        parts = line.split()
        if len(parts) > 1 and parts[1].isdigit():
            pslist_pids.add(int(parts[1]))

with open("psscan_output.txt") as f:
    for line in f:
        parts = line.split()
        if len(parts) > 1 and parts[1].isdigit():
            psscan_pids.add(int(parts[1]))

hidden = psscan_pids - pslist_pids
if hidden:
    print(f"[!] HIDDEN PROCESSES DETECTED (in psscan but not pslist):")
    for pid in hidden:
        print(f"    PID: {pid}")
else:
    print("[*] No hidden processes detected via cross-view analysis")
PYEOF

Step 2: Detect System Call Hooking

Identify hooks in the System Service Descriptor Table (SSDT) and Import Address Tables:

# Check SSDT for hooked system calls
vol3 -f memory.dmp windows.ssdt

# Identify hooks pointing outside ntoskrnl.exe or win32k.sys
vol3 -f memory.dmp windows.ssdt | grep -v "ntoskrnl\|win32k"

# Check for Inline hooks (detour patching)
vol3 -f memory.dmp windows.apihooks --pid 4  # System process

# IDT (Interrupt Descriptor Table) analysis
vol3 -f memory.dmp windows.idt

# Check for IRP (I/O Request Packet) hooking on drivers
vol3 -f memory.dmp windows.driverscan
vol3 -f memory.dmp windows.driverirp
Types of Rootkit Hooks:
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
SSDT Hook:         Modifies System Service Descriptor Table entries to redirect
                   system calls through rootkit code (filters process/file listings)

IAT Hook:          Patches Import Address Table of a process to intercept API calls
                   before they reach the kernel

Inline Hook:       Overwrites the first bytes of a function with a JMP to rootkit code
                   (detour/trampoline technique)

IRP Hook:          Intercepts I/O Request Packets to filter disk/network operations
                   at the driver level

DKOM:              Direct Kernel Object Manipulation - unlinking structures like
                   EPROCESS from the ActiveProcessLinks list without hooking

Step 3: Analyze Kernel Modules and Drivers

Identify unauthorized kernel drivers that may be rootkit components:

# List all loaded kernel modules
vol3 -f memory.dmp windows.modules

# Scan for drivers in memory (including hidden/unlinked)
vol3 -f memory.dmp windows.driverscan

# Compare module lists to find hidden drivers
vol3 -f memory.dmp windows.modscan > modscan.txt
vol3 -f memory.dmp windows.modules > modules.txt

# Check driver signatures and verify against known-good baselines
vol3 -f memory.dmp windows.verinfo

# Dump suspicious driver for static analysis
vol3 -f memory.dmp windows.moddump --base 0xFFFFF80012340000 --dump

Step 4: Detect File and Registry Hiding

Identify files and registry keys hidden by the rootkit:

# Linux rootkit detection with rkhunter
rkhunter --check --skip-keypress --report-warnings-only

# chkrootkit scanning
chkrootkit -q

# Windows: Compare filesystem views
# Live system file listing vs Volatility filescan
vol3 -f memory.dmp windows.filescan > mem_files.txt

# Check for hidden registry keys
vol3 -f memory.dmp windows.registry.hivelist
vol3 -f memory.dmp windows.registry.printkey --key "SYSTEM\CurrentControlSet\Services"

# Look for hidden services (loaded but not in service registry)
vol3 -f memory.dmp windows.svcscan | grep -i "kernel"

Step 5: Network Connection Analysis

Find hidden network connections and backdoors:

# Memory-based network connection enumeration
vol3 -f memory.dmp windows.netscan

# Compare with live netstat (if available) to find hidden connections
# Hidden connections: present in memory but not shown by netstat

# Look for raw sockets (often used by rootkits for covert communication)
vol3 -f memory.dmp windows.netscan | grep RAW

# Check for network filter drivers (NDIS hooks)
vol3 -f memory.dmp windows.driverscan | grep -i "ndis\|tcpip\|afd"

# Analyze callback routines registered by drivers
vol3 -f memory.dmp windows.callbacks

Step 6: Integrity Verification

Verify system file and kernel integrity:

# Check kernel code integrity (compare in-memory kernel to on-disk copy)
vol3 -f memory.dmp windows.moddump --base 0xFFFFF80070000000 --dump
# Compare SHA-256 of dumped ntoskrnl.exe with known-good copy

# Windows: System File Checker (on live system)
sfc /scannow

# Linux: Package integrity verification
rpm -Va  # RPM-based systems
debsums -c  # Debian-based systems

# Compare critical system binaries
find /bin /sbin /usr/bin /usr/sbin -type f -exec sha256sum {} \; > current_hashes.txt
# Compare against baseline: diff baseline_hashes.txt current_hashes.txt

# YARA scan for known rootkit signatures
vol3 -f memory.dmp yarascan.YaraScan --yara-file rootkit_rules.yar

Key Concepts

TermDefinition
RootkitMalware designed to maintain persistent, privileged access while hiding its presence from system administrators and security tools
DKOMDirect Kernel Object Manipulation; technique of modifying kernel data structures (e.g., unlinking EPROCESS) to hide objects without hooking
SSDT HookingReplacing entries in the System Service Descriptor Table to intercept and filter system call results (hide processes, files, connections)
Inline HookingPatching the first instructions of a function with a jump to rootkit code; the rootkit can filter the function output before returning
Cross-View DetectionComparing results from multiple enumeration methods (linked list walk vs memory scan) to identify discrepancies caused by hiding
Kernel DriverCode running in kernel mode (Ring 0) with full system access; rootkits use malicious drivers to gain kernel-level control
BootkitsRootkits that infect the boot process (MBR, VBR, or UEFI firmware) to load before the operating system and security tools

Tools & Systems

  • Volatility: Memory forensics framework providing cross-view detection, SSDT analysis, and kernel structure inspection for rootkit detection
  • GMER: Free Windows rootkit detection tool scanning for SSDT hooks, IDT hooks, IRP hooks, and hidden processes/files/registry
  • rkhunter: Linux rootkit detection tool checking for known rootkit signatures, suspicious files, and system binary modifications
  • chkrootkit: Linux tool for detecting rootkit presence through signature-based and anomaly-based checks
  • Sysinternals RootkitRevealer: Microsoft tool comparing Windows API results with raw filesystem/registry scans to find discrepancies

Common Scenarios

Scenario: Investigating a System Where Standard Tools Show No Compromise

Context: An endpoint shows network beaconing to a known C2 IP in firewall logs, but the local EDR, Task Manager, and netstat show no suspicious processes or connections. A memory dump has been acquired for analysis.

Approach:

  1. Run Volatility psscan and compare with pslist to identify processes hidden via DKOM
  2. Run windows.ssdt to check for system call hooks that filter process and network listings
  3. Run windows.malfind to detect injected code in legitimate processes
  4. Run windows.netscan to find network connections hidden from user-mode tools
  5. Run windows.driverscan to identify malicious kernel drivers enabling the hiding
  6. Dump the rootkit driver and analyze with Ghidra to understand its hooking mechanism
  7. Check for boot persistence (MBR/VBR modifications, UEFI firmware implants)

Pitfalls:

  • Running detection tools on the live compromised system (rootkit may hide from or subvert them)
  • Assuming kernel integrity because no SSDT hooks are found (rootkit may use DKOM or inline hooks instead)
  • Not checking for both user-mode and kernel-mode rootkit components (many rootkits have both)
  • Trusting the rootkit scanner results on a live system; always verify with offline memory forensics

Output Format

ROOTKIT DETECTION ANALYSIS REPORT
====================================
Dump File:        memory.dmp
System:           Windows 10 21H2 x64
Analysis Tool:    Volatility 3.2

CROSS-VIEW DETECTION
Process List Comparison:
  pslist processes:  127
  psscan processes:  129
  [!] HIDDEN PROCESSES: 2
    PID 6784: sysmon64.exe (hidden rootkit component)
    PID 6812: netfilter.exe (hidden network filter)

SSDT HOOK ANALYSIS
[!] Entry 0x004A (NtQuerySystemInformation) hooked -> driver.sys+0x1200
[!] Entry 0x0055 (NtQueryDirectoryFile) hooked -> driver.sys+0x1400
[!] Entry 0x0119 (NtDeviceIoControlFile) hooked -> driver.sys+0x1600
Hook Target: driver.sys at 0xFFFFF800ABCD0000 (unsigned, suspicious)

KERNEL DRIVER ANALYSIS
[!] driver.sys - No digital signature, loaded at 0xFFFFF800ABCD0000
    Size: 45,056 bytes
    SHA-256: abc123def456...
    IRP Hooks: IRP_MJ_CREATE, IRP_MJ_DEVICE_CONTROL
    Registry: HKLM\SYSTEM\CurrentControlSet\Services\MalDriver

HIDDEN NETWORK CONNECTIONS
PID 6812: 10.1.5.42:49152 -> 185.220.101.42:443 (ESTABLISHED)
  - Not visible via netstat or user-mode tools
  - Filtered by NtDeviceIoControlFile SSDT hook

ROOTKIT CAPABILITIES
- Process hiding (DKOM + SSDT)
- File hiding (NtQueryDirectoryFile hook)
- Network connection hiding (NtDeviceIoControlFile hook)
- Kernel-mode persistence (driver service)

REMEDIATION
- Boot from clean media for offline remediation
- Remove malicious driver from offline registry
- Verify MBR/VBR/UEFI integrity for boot persistence
- Full system rebuild recommended for kernel-level compromise

Verification Criteria

Confirm successful execution by validating:

  • [ ] All prerequisite tools and access requirements are satisfied
  • [ ] Each workflow step completed without errors
  • [ ] Output matches expected format and contains expected data
  • [ ] No security warnings or misconfigurations detected
  • [ ] Results are documented and evidence is preserved for audit

Compliance Framework Mapping

This skill supports compliance evidence collection across multiple frameworks:

  • SOC 2: CC7.2 (Anomaly Detection), CC7.4 (Incident Response)
  • ISO 27001: A.12.2 (Malware Protection), A.16.1 (Security Incident Management)
  • NIST 800-53: SI-3 (Malicious Code Protection), IR-4 (Incident Handling)
  • NIST CSF: DE.CM (Continuous Monitoring), RS.AN (Analysis)

Claw GRC Tip: When this skill is executed by a registered agent, compliance evidence is automatically captured and mapped to the relevant controls in your active frameworks.

Deploying This Skill with Claw GRC

Agent Execution

Register this skill with your Claw GRC agent for automated execution:

# Install via CLI
npx claw-grc skills add detecting-rootkit-activity

# Or load dynamically via MCP
grc.load_skill("detecting-rootkit-activity")

Audit Trail Integration

When executed through Claw GRC, every step of this skill generates tamper-evident audit records:

  • SHA-256 chain hashing ensures no step can be modified after execution
  • Evidence artifacts (configs, scan results, logs) are automatically attached to relevant controls
  • Trust score impact โ€” successful execution increases your agent's trust score

Continuous Compliance

Schedule this skill for recurring execution to maintain continuous compliance posture. Claw GRC monitors for drift and alerts when re-execution is needed.

Use with Claw GRC Agents

This skill is fully compatible with Claw GRC's autonomous agent system. Deploy it to any registered agent via MCP, and every execution will be logged in the tamper-evident audit trail.

// Load this skill in your agent
npx claw-grc skills add detecting-rootkit-activity
// Or via MCP
grc.load_skill("detecting-rootkit-activity")

Tags

malwarerootkitdetectionkernel-analysismemory-forensics

Related Skills

Malware Analysis

Detecting Process Injection Techniques

8mยทintermediate
Malware Analysis

Analyzing Bootkit and Rootkit Samples

8mยทadvanced
Malware Analysis

Analyzing Memory Dumps with Volatility

7mยทadvanced
Malware Analysis

Detecting Fileless Malware Techniques

8mยทintermediate
Malware Analysis

Extracting Iocs from Malware Samples

7mยทintermediate
Malware Analysis

Analyzing Command and Control Communication

7mยทadvanced

Skill Details

Domain
Malware Analysis
Difficulty
advanced
Read Time
7 min
Code Examples
8

On This Page

When to UsePrerequisitesWorkflowKey ConceptsTools & SystemsCommon ScenariosOutput FormatVerification CriteriaCompliance Framework MappingDeploying This Skill with Claw GRC

Deploy This Skill

Add this skill to your Claw GRC agent and start automating.

Get Started Free โ†’