Thereat Hunting

2022-01-25

Threat Hunting Simulator -- Technical Documentation

1. Overview

The Threat Hunting Simulator is TryHackMe's interactive training platform for developing proactive threat hunting skills. Unlike the SOC Simulator (which focuses on reactive alert triage), Threat Hunting places users in a proactive investigator role: they receive a hypothesis and threat intelligence briefing, then must independently hunt through SIEM logs, reconstruct a full attack chain using the MITRE ATT&CK framework, write adversary step descriptions, and generate an AI-powered threat report.

The system provides VMs (SIEM + analyst workstation), evaluates the user's timeline against expected attack stages, uses GPT-4o to assess the quality of adversary descriptions, and generates an overall performance report.

2. High-Level Architecture

3. Directory Structure

Backend (API)

apps/api/app/api/v2/src/
├── routes/threat-hunting/
│   ├── index.ts                     # Main router
│   ├── runs.ts                      # Run lifecycle routes
│   ├── timeline.ts                  # Timeline stage CRUD routes
│   ├── content.ts                   # CMS content routes
│   ├── vms.ts                       # VM management routes
│   └── public/                      # Unauthenticated routes
├── controllers/threat-hunting/
│   ├── runs.ts                      # Run request handlers
│   ├── timeline.ts                  # Timeline stage handlers
│   ├── content.ts                   # Content handlers
│   ├── stats.ts                     # Stats handlers
│   ├── leaderboard.ts               # Leaderboard handlers
│   └── vms.ts                       # VM handlers
├── services/threat-hunting/
│   ├── manage-runs.ts               # Run init, scoring, pause/resume
│   ├── vms.ts                       # VM deployment (wraps soc-sim)
│   ├── runs.ts                      # Run CRUD operations
│   ├── report.ts                    # AI report generation (GPT-4o)
│   ├── content.ts                   # Scenario content retrieval
│   ├── run-access.ts                # Access control
│   ├── run-public-summary.ts        # Public sharing
│   ├── stats.ts                     # Statistics
│   └── leaderboard.ts               # Leaderboard logic
├── models/threat-hunting/
│   └── run.ts                       # ThreatHuntingRun Mongoose model
├── common/
│   ├── interfaces/threat-hunting/    # TypeScript interfaces
│   ├── enums/threat-hunting/         # Status enums
│   └── constants/mitre/              # MITRE ATT&CK tactics & techniques
└── services/openai/
    └── index.ts                     # Shared OpenAI service (GPT-4o)

Frontend

apps/frontend/src/features/threat-hunting/
├── threat-hunting.tsx                # Root component
├── threat-hunting.slice.ts           # Core RTK Query API slice
├── threat-hunting.types.ts           # TypeScript types
├── threat-intel/                     # Hypothesis & threat intelligence
├── timeline/                         # Attack chain timeline builder
├── threat-report/                    # AI-generated threat report
├── summary/                          # Run summary & scoring
├── scenario-onboarding/              # Onboarding flow
├── scenario-overview/                # Scenario preview
├── my-computer/                      # Analyst VM iframe
├── my-notes/                         # Draggable notes editor
├── documentation/                    # Scenario documentation
├── guide-pages/                      # Help guides
├── landing-sections/
│   ├── threat-hunting-home/          # Landing page
│   ├── threat-hunting-scenarios/     # Scenario listing
│   ├── threat-hunting-stats/         # User statistics
│   └── threat-hunting-leaderboard/   # Leaderboard
└── components/
    ├── navigation/                   # Top navigation bar
    ├── notes-editor/                 # Draggable note-taking
    ├── vms-frame/                    # VM iframe wrapper
    ├── timeline-card/                # Timeline stage cards
    ├── tactic-select/                # MITRE tactic picker
    ├── mitre-accuracy/               # MITRE accuracy visualization
    └── machine-expiring-notification/# VM expiry warnings

4. Data Model

ThreatHuntingRun (threat_hunting_runs)

Timeline Stage (user-created, embedded in run)

CMS Timeline Stage (ground truth, snapshotted into meta)

Report (embedded in run)

5. Run Lifecycle

Key Difference from SOC-Sim

Threat Hunting has an explicit investigation start step (hasInvestigationStarted). After VMs are ready, the user reviews the scenario overview and threat intelligence before clicking "Start Investigation." In SOC-Sim, investigation begins immediately when the run is ready.

6. VM Creation & Deployment

Shared Infrastructure with SOC-Sim

Threat Hunting reuses the SOC-Sim VM infrastructure directly:

threat-hunting/vms.ts imports from soc-sim/vms.ts:
  ├── deployVMsWithRetry()        # Core EC2 deployment + retry logic
  ├── insertSOCSimInstance        # (aliased as insertThreatHuntingInstance)
  ├── insertMockInstance          # Local dev mock
  ├── extendVmTime                # 1-hour extension logic
  └── validateSimulationStarted   # Prevent duplicate VMs

The critical difference: Threat Hunting VMs come with pre-loaded logs on the SIEM. There is no EventBridge log scheduling or alert insertion. The user proactively hunts through existing data rather than reacting to incoming alerts.

7. The Timeline: Attack Chain Building

Timeline API Endpoints

The frontend uses @dnd-kit for drag-and-drop reordering of timeline stages.

8. AI Integration (GPT-4o)

AI Function 1: Adversary Step Description Evaluation

When: Called for each timeline stage during submitFindings() Purpose: Evaluates the quality of the user's free-text adversary step description against CMS-defined criteria Model: GPT-4o via OpenAIService.chatGptCompletionAPI()

System prompt (summarized):

"You are a Senior Threat Hunter. Evaluate the adversary step description strictly based on the given 'Criteria'. Calculate a totalScore. Write feedback as a senior SOC analyst speaking to a Junior Threat Hunter. Do not reveal evaluation criteria. Limit to 1000 characters."

Input:

• User's adversary step description
• Selected user & asset
• Evaluation criteria from CMS (defines what to look for and max scores)

Output (Zod-validated):

{ summary: string, totalScore: number }

AI Function 2: Threat Report Generation

When: User explicitly triggers via "Generate Threat Report" button Purpose: Creates a markdown threat case report from the full attack chain

System prompt (summarized):

"You are a Senior Threat Hunter. Generate a precise threat case report in markdown from attack chain stages. Validate data completeness. Reject single-stage chains. Limit to 15 sentences."

Input: All timeline stages with title, description, timestamp, tactic/technique names, user, asset, IOC, SIEM URL

Output: Markdown-formatted threat report

AI Function 3: Overall Run Evaluation

When: After scoring completes (async) Purpose: Generates manager-style overall performance feedback

System prompt (summarized):

"You are a Senior Threat Hunter. Provide overall feedback on the user's performance based on individual stage evaluations. 2-3 sentences focusing on improvement areas. Address user as 'Junior Threat Hunter'."

Input: All stage evaluations (AI-generated summaries from Function 1)

Output: 2-3 sentence overall feedback

AI Configuration

const openAiOptions = {
  temperature: 0.7,
  top_p: 0.95,
  n: 1,
  stream: false,
  max_tokens: 1000,
  presence_penalty: 0,
  frequency_penalty: 0,
};

All three functions use structured output (Zod schema validation) via OpenAI's chatGptCompletionAPI, which includes:

• Up to 3 retries with exponential backoff
• Policy violation detection
• Rate limit / quota handling
• Error capture via Sentry

9. Scoring & Evaluation

Scoring Categories (per stage)

IOC Matching Detail

const calculateIocPoints = (stageIoc: string, cmsIocs: Array<{ value: string; points: number }>) =>
  cmsIocs.reduce((acc, ioc) => {
    const normalizedStageIoc = stageIoc.replace(/\\/g, '');
    const normalizedValue = ioc.value.replace(/\\/g, '');
    if (normalizedStageIoc.toLowerCase().includes(normalizedValue.toLowerCase()))
      acc.iocPointsAwarded += ioc.points;
    acc.totalIocPoints += ioc.points;
    return acc;
  }, { iocPointsAwarded: 0, totalIocPoints: 0 });

IOCs are matched via case-insensitive substring inclusion -- the user's IOC field must contain the expected IOC value somewhere in the text.

Final Score Calculation

totalScenarioPoints = sum(all CMS stage points) + sum(all IOC points) + hypothesisOutcomePoints
totalAwarded = sum(all awarded stage points) + hypothesisOutcomePointsAwarded
success = (totalAwarded / totalScenarioPoints) * 100 >= 80

Report Breakdown Generation

After scoring, the system generates detailed breakdowns stored in run.report:

10. Frontend User Flow

11. API Endpoints Summary

Run Management

Timeline

Content & Reference

VMs

Company / Management Dashboard

12. SOC-Sim vs Threat Hunting: Architecture Comparison

13. Feature Flags (GrowthBook)

14. Infrastructure Summary

Key characteristics:

• No EventBridge / Lambda -- unlike SOC-Sim, logs are pre-loaded on VMs
• No Azure Sentinel path -- simplified; uses same constant but Sentinel VMs work differently
• VM lifetime: 3 hours default, extendable by 1 hour (same constraints as SOC-Sim)
• Health check polling: Every 3 seconds via Agenda.js until VMs respond or 30-minute timeout
• 25-second post-deployment wait (vs 18s in SOC-Sim) for VM readiness before marking READY