THM Sytem Design

2022-01-25

TryHackMe Infrastructure & System Design

A comprehensive technical guide to the AWS infrastructure, system design patterns, and cloud architecture that powers the TryHackMe platform.

Table of Contents

1. Infrastructure Overview
2. AWS Multi-Region Architecture
3. Infrastructure as Code (SST v3)
4. VPC & Networking
5. VM Deployment Pipeline
6. Guacamole Remote Desktop System
7. Networks v2 (Pulumi Dynamic IaC)
8. AWS Services Map
9. Data Layer
10. Real-Time Communication
11. Job Scheduling & Background Processing
12. Authentication & Session Management
13. Observability Stack
14. Request Lifecycle
15. Configuration & Secrets
16. Key File Reference

1. Infrastructure Overview

2. AWS Multi-Region Architecture

Why multi-region? VM interactions are latency-sensitive -- typing in a remote desktop session with 200ms latency is unusable. By deploying Guacamole clusters and VM instances in 5 regions, users connect to the nearest one. The API itself stays in eu-west-1 (single-region) because it only handles REST/WebSocket traffic where 100-200ms latency is acceptable.

TryHackMe operates across 5 AWS regions to minimize latency for global users deploying VMs.

Region Responsibilities

Cross-Region Connectivity

Transit Gateway peering connects all VM regions for:

• VPN client pool routing (10.8.0.0/24, 10.9.0.0/24, 10.13.0.0/24)
• Network CIDR routing between regions
• External BADR account peering (account 439472160432)

3. Infrastructure as Code (SST v3)

Why SST v3? SST v3 wraps Pulumi and Terraform under a TypeScript-first API, so infrastructure is defined in the same language as the application. Unlike SST v2 (which was CDK/CloudFormation), v3 deploys faster, supports multi-provider (AWS + Cloudflare in one config), and avoids CloudFormation's 500-resource stack limits. The Link system lets Lambda functions access secrets and resource ARNs without manual env var wiring.

All infrastructure is managed via SST v3 (not v2/CDK). The root config is at sst.config.ts.

SST Apps

Shared Infrastructure (infrastructure/shared.ts)

Defines resources shared across all SST apps:

Deployment

# Deploy all apps to staging
npx sst deploy --stage staging
 
# Deploy specific app
APPS=core npx sst deploy --stage production
 
# Selective deployment via APPS env var
APPS=guacamole,nginx npx sst deploy --stage staging

4. VPC & Networking

VPC Layout (Production, eu-west-1)

Security Groups

us-east-1 (Modern Region, Dynamic VPC)

Unlike legacy regions, us-east-1 creates its VPC dynamically in SST:

VPC: 10.201.0.0/16
├── Vulnerable VMs Subnet: 10.201.0.0/17
├── Remote Access Subnet:  10.201.128.0/18
├── LB Subnet:             10.201.192.0/24
├── Internet Gateway (dynamic)
├── Transit Gateway Attachment (tgw-01c8741ec73fb2f5c)
└── S3 Gateway Endpoint (private S3 access)

5. VM Deployment Pipeline

This is the core of TryHackMe's infrastructure -- deploying EC2 instances for users to interact with.

Deployment Flow

Key Concepts

Upload (VM Template): An Upload defines a VM image -- its AMI, instance type, OS, remote connection type, credentials, and boot time. Uploads are stored in MongoDB and reference AMIs in each region.

Instance: A running VM instance. Tracks AWS instance ID, connection details, expiry time, and state. Stored in MongoDB.

Instance Types & Selection:

Retry Strategy:

• Spot instances: up to AWS_SPOT_MAX_RETRIES retries (default 4)
• On-demand: up to AWS_ON_DEMAND_MAX_RETRIES retries (default 8)
• Backoff delay: AWS_BACKOFF_DELAY_MS (default 100ms)

Instance Lifecycle:

• Default expiry: varies by context (1-3 hours)
• Extension: +1 hour (once, only in last hour, max 6 hours total)
• Termination: User-initiated, auto-expiry, or spot interruption

Cost Center Tagging:

Every EC2 instance is tagged for billing attribution:

6. Guacamole Remote Desktop System

Why Guacamole? Users need browser-based access to Windows (RDP) and Linux (VNC/SSH) VMs without installing any client software. Guacamole is the only mature open-source solution that renders RDP/VNC/SSH in a browser via HTML5 Canvas. The guacd daemon handles protocol translation, while the web app serves the UI and manages connections. Running it on ECS Fargate per-region keeps latency low for users connecting to their nearest VM.

Apache Guacamole provides browser-based access to VMs via RDP, VNC, and SSH.

Connection Types

Guac-A-Worker (Cloudflare)

The Guac-A-Worker is a Cloudflare Worker that routes VM requests:

1. User requests *.guacaworker.{domain}
2. Worker extracts task ID from subdomain
3. Looks up public DNS name in Cloudflare KV
4. Routes request to correct Guacamole instance

Resources created by SST:

• KV Namespace for task-to-DNS mapping
• Cloudflare DNS records (wildcard + base)
• Advanced Certificate Manager (Google CA)
• Worker route: *guacaworker.{domain}/*
• IAM user with ECS/EC2 describe permissions

Nginx Reverse Proxy

For VNC connections, an Nginx reverse proxy runs per-region:

# IP-based subdomain routing
# 10-x-x-x.reverse-proxy-{region}.{domain} -> 10.x.x.x
server {
    location / {
        proxy_pass http://$target_ip:$target_port;
    }
    location /websockify/ {
        proxy_pass http://$target_ip:$target_port;
        # WebSocket upgrade for VNC
    }
}

Scaling:

• Production: 10-50 ARM64 instances (0.25 vCPU, 0.5GB each)
• Staging: 2-4 instances

7. Networks v2 (Pulumi Dynamic IaC)

Why Pulumi inside Lambda (and not SST/Terraform)? Each user's network lab needs its own VPC, subnets, VPN server, and multiple VMs -- created on-demand and destroyed hours later. SST/Terraform are for long-lived infrastructure. Pulumi's Automation API lets you run pulumi up and pulumi destroy programmatically from code, making it ideal for ephemeral, per-user infrastructure. Lambda provides the compute, SQS provides the queue, and EFS persists the Pulumi binary and state between invocations.

Networks v2 creates isolated network environments for users using Pulumi Automation API inside a Lambda function.

Lambda Configuration

What Pulumi Creates Per Network

Network Expiry

A cron Lambda runs every minute to check for expired networks:

1. Queries MongoDB for expired network instances
2. Sends SQS message to destroy the Pulumi stack
3. Lambda destroys all AWS resources for that network

8. AWS Services Map

Service Usage Summary -- What & Why

9. Data Layer

MongoDB

Why MongoDB? The data is deeply nested and schema-diverse -- a VM instance document looks nothing like a user document or a SOC-Sim run. MongoDB's flexible schema handles this naturally. Mongoose adds just enough structure (schemas, validation, indexes) without the rigidity of a relational DB. The platform also needs fast reads on varied query patterns (by user, by room, by instance ID, by status), which compound indexes on a document store handle well.

Connection settings:

• Pool: min 2, max 50
• Read preference: primary
• Read concern: local
• Write concern: majority
• Retry reads and writes enabled

Key collections:

• instances -- Running VM instances
• uploads -- VM templates/images
• users -- User accounts
• soc_sim_runs / soc_sim_run_alerts -- SOC Sim data
• threat_hunting_runs -- Threat Hunting data
• scheduler_jobs -- Agenda.js job queue
• sessions -- Express sessions
• Network templates, paths, modules, rooms, companies, etc.

Redis

Why Redis? Sub-millisecond reads for rate limiting (every single API request checks Redis), Socket.IO pub/sub across multiple API instances, and Bull queue backing. Two instances isolate the latency-critical main workload from the Tutor feature's higher-volume writes.

Two separate Redis instances serve different purposes:

Kafka

Used for event streaming between the API and analytics consumers:

Why Kafka? High-throughput, ordered event streaming that decouples producers from consumers. Analytics events, tutor interactions, and MongoDB change events generate a firehose of data -- Kafka handles this without back-pressure on the API. Consumer groups allow multiple independent consumers (analytics pipeline, tutor service) to process the same events at their own pace.

Topics include MongoDB change events and analytics events. A Vercel-to-ECS HTTP bridge exists for the Next.js frontend to publish events.

10. Real-Time Communication

Socket.IO Architecture

Why Socket.IO over raw WebSockets? Socket.IO provides automatic reconnection, room-based broadcasting (critical for SOC-Sim multiplayer), binary streaming, and -- most importantly -- the Redis adapter for multi-instance scaling. Raw WebSockets would require building all of this from scratch.

The Redis adapter (@socket.io/redis-adapter) ensures events published on one API instance reach clients connected to other instances. A Redis emitter (@socket.io/redis-emitter) bridges Vercel serverless functions to ECS-hosted Socket.IO.

Socket Handlers:

11. Job Scheduling & Background Processing

Three Scheduling Systems

12. Authentication & Session Management

Why Passport.js? Strategy-based architecture means adding a new auth provider (e.g., WorkOS for enterprise SSO) is just adding a new strategy file -- no changes to the core auth flow. The ecosystem has battle-tested strategies for every provider TryHackMe uses.

Why connect-mongo for sessions? Sessions must survive API restarts and be shared across multiple API instances. MongoDB is already in the stack, so connect-mongo avoids adding another dependency. Redis sessions would be faster but add operational complexity for marginal gain (sessions are read once per request, not per millisecond).

Why WorkOS? Enterprise B2B customers need SAML/OIDC SSO with their identity provider (Okta, Azure AD, etc.). WorkOS provides a unified API for all SSO protocols, directory sync, and admin portal -- building this in-house would take months.

• Sessions stored in MongoDB via connect-mongo
• Session cookie max age: 7 days
• Passport serializes user ID into session
• Socket.IO connections share the same Passport session middleware

13. Observability Stack

Why Pino over Winston? Pino is the fastest Node.js logger by a wide margin -- it uses worker threads for serialization and avoids blocking the event loop. On a platform that handles thousands of concurrent VM deployments, logging overhead matters. Winston would add measurable latency to every request.

Why Sentry? Stack traces, breadcrumbs, release tracking, and performance profiling in one tool. The @sentry/profiling-node integration provides continuous profiling in production to catch slow functions. Sentry's Express integration automatically captures unhandled errors with full request context.

Why Segment? Single API for routing analytics events to multiple destinations (Mixpanel, Amplitude, data warehouse, etc.). Changing analytics providers means updating Segment config, not rewriting event calls across the codebase.

Why Customer.io over SES directly? Templated transactional emails with visual builders, drip campaigns, and user segmentation. SES is used under the hood (via Nodemailer) only for custom emails like calendar invitations where Customer.io templates don't apply.

Log levels: trace, debug, info, warn, error, fatal (configurable via LOG_LEVEL)

Redaction: Cookies and sensitive headers are redacted from HTTP logs.

14. Request Lifecycle

Rate Limiting

20+ endpoint-specific rate limiters backed by Redis:

Redis Store (rate-limit-redis)
  └── Fallback: In-memory store

Why Redis-backed rate limiting? Rate limits must be shared across all API instances -- if instance A sees 15 requests from a user, instance B must know about them. Redis provides the shared counter. The rate-limit-redis store handles atomic increments. In-memory fallback ensures the API doesn't crash if Redis is temporarily unavailable (it just rate-limits per-instance instead of globally).

Why express-rate-limit over a WAF? Fine-grained, per-endpoint control. The "start VM" endpoint has a different limit than "get user profile." WAF rules can't express "5 VM deploys per hour per user but 100 profile reads per minute." 20+ distinct limiter configs in the codebase reflect this granularity.

Rate limiters are warmed up on server start. Socket.IO has separate per-minute and per-day rate limits via rate-limiter-flexible.

15. Configuration & Secrets

Why Cloudflare in front of everything? DDoS protection, global CDN, DNS management, and the Workers edge compute platform -- all in one. The Guac-A-Worker runs at Cloudflare's edge (300+ PoPs worldwide), routing VM desktop requests to the nearest Guacamole cluster with near-zero latency. Cloudflare also provides the wildcard SSL certificates that make *.guacaworker.{domain} subdomain routing possible.

Why GrowthBook for feature flags? Open-source, self-hostable, and provides server-side SDK for Node.js. Feature flags gate every major feature (SOC-Sim VM deploy, curriculum mapping, threat hunting SIEM tool modal). GrowthBook's targeting rules allow gradual rollouts by user segment, company, or percentage -- critical for safely shipping infrastructure changes like new VM connection types.

Why Sanity CMS? Scenario content (SOC-Sim alerts, Threat Hunting timelines, evaluation criteria) changes frequently and is authored by non-engineers. Sanity provides a structured content authoring UI, real-time collaboration, and a GROQ query API. Content is fetched at runtime, not baked into deploys, so scenario updates ship instantly without redeployment.

Config Architecture

AWS Credential Profiles

S3 Bucket Map

16. Key File Reference

Infrastructure as Code

AWS Service Wrappers

VM & Connection Services

System Design Components

All src/ paths above are relative to apps/api/app/api/v2/.