Multiplayer-Online-Strategy-Game-Backend

Stars
6
Language
C#
Updated
View on GitHub

Multiplayer Online Strategy Game Backend

A massively scalable, real-time multiplayer online strategy game backend built with C# and .NET, designed to support 100,000+ concurrent players and 10,000+ simultaneous matches with sub-100ms latency.

Architecture Overview

High-Level System Diagram (Massive Scale)

[Global Load Balancer (AWS ALB/GCP GLB)]
    |
    +--- [Regional Gateways (API Gateway + SignalR)]
    |       |
    |       +--- [Auth Service Cluster] (Stateless, JWT)
    |       +--- [Matchmaking Service Cluster] (Redis-backed queues)
    |       +--- [Game Session Workers] (Distributed, assignable)
    |       +--- [Player State Service] (Redis cache-first)
    |       +--- [Ranking Service] (Event-driven updates)
    |       +--- [Anti-Cheat Service] (Distributed validation)
    |       +--- [Notification Service] (Async queues)
    |
[Redis Cluster] <--- State/Cache/PubSub/Backplane
    |
[PostgreSQL Cluster] <--- Persistence (Read Replicas)
    |
[Message Queue (Kafka/RabbitMQ)] <--- Events (Match Results, Rankings, Analytics)
    |
[Monitoring Stack] (Prometheus + Grafana + Jaeger)

Key Scaling Principles

  • Stateless by Default: All services are stateless; state lives in Redis
  • Horizontal Scaling: Auto-scale based on CPU/memory/connections
  • Event-Driven: Async processing for non-real-time operations
  • Regional Distribution: Players routed to nearest region
  • Zero SPOF: Multi-region, multi-AZ deployment

Scalability Requirements Breakdown

1. Stateless-by-Default Services

Why Statelessness Matters:

  • Enables unlimited horizontal scaling without session affinity
  • Allows instant failover and rolling updates
  • Prevents memory leaks and state corruption
  • Simplifies load balancing and auto-scaling

How Horizontal Scaling is Achieved:

  • All REST APIs use JWT for authentication (no server-side sessions)
  • Game state stored in Redis with TTL
  • Match-to-worker assignments stored in Redis
  • No in-memory caches that need synchronization

2. Horizontal Scaling Strategy

API Servers: Auto-scale on CPU >70% or connections >10k SignalR Hubs: Scale on WebSocket connections >5k per instance Matchmaking Service: Scale on queue depth >1000 Game Simulation Workers: Scale on active matches >500 per worker

Load Balancer Strategy:

  • Layer 7 routing based on path (/auth, /matchmaking, etc.)
  • Least connections for SignalR to distribute WebSockets
  • Health checks every 10s

Sticky Session Handling:

  • SignalR uses Redis backplane for cross-server messaging
  • No sticky sessions required; connections can migrate
  • Connection state stored in Redis with 30s TTL

3. Distributed Game Session Model

Match-to-Worker Allocation:

  • Workers register in Redis set with capacity
  • Matchmaking service assigns worker via Redis sorted set (by load)
  • Assignment stored as Redis key: match:{matchId}:worker:{workerId}

Heartbeat Mechanism:

  • Workers send heartbeat every 5s to Redis
  • If heartbeat missing >15s, worker marked unhealthy
  • Orchestrator reassigns matches to healthy workers

Failover Handling:

  • On worker failure, orchestrator detects via heartbeat
  • Reassigns match to new worker
  • New worker loads state from Redis
  • Clients reconnect via SignalR backplane

Worker Lifecycle Diagram:

Worker Startup → Register in Redis → Accept Assignments
     ↓
Heartbeat Loop (5s) → Update Load Metrics
     ↓
Failure Detected → Reassign Matches → Cleanup State

Rebalancing Strategy:

  • Monitor worker load every 30s
  • If imbalance >20%, migrate low-load matches
  • Use Redis pub/sub to notify workers of reassignments

4. Redis at Scale

Key Structure:

  • player:{playerId}:state - Player live state (TTL: 30m)
  • match:{matchId}:state - Match state (TTL: 24h)
  • queue:{region}:{mode} - Matchmaking queue (TTL: 5m)
  • leaderboard:{season}:{region} - Cached rankings (TTL: 1h)

TTL Policies:

  • Live state: 30m (auto-cleanup abandoned sessions)
  • Match state: 24h (post-match analysis)
  • Cache: 1h (refresh on demand)

Sharding Strategy:

  • Hash-based sharding by key prefix
  • 3+ shards per region for HA
  • Redis Cluster for automatic failover

Hot-Key Prevention:

  • Distribute leaderboard reads across replicas
  • Use Redis pipelines for batch operations
  • Implement local caching with invalidation

5. Event-Driven Architecture

Domain Events:

  • MatchCompletedEvent
  • PlayerRankedEvent
  • AchievementUnlockedEvent

Asynchronous Processing:

  • Match results → Queue → Ranking update → Leaderboard refresh
  • Player actions → Queue → Analytics → Anti-cheat analysis

Why Async Matters at Scale:

  • Decouples services, prevents cascading failures
  • Allows burst handling without blocking real-time paths
  • Enables exactly-once processing with idempotency keys

Exactly-Once vs At-Least-Once:

  • Use at-least-once for performance (match results)
  • Exactly-once for financial operations (purchases)
  • Idempotency via event deduplication

6. SignalR Scaling (CRITICAL)

Why Naive SignalR Fails:

  • Single server can't handle 100k connections
  • No cross-server messaging
  • Memory usage scales with connections

Backplane Solution:

  • Redis pub/sub for message distribution
  • All hubs subscribe to match groups
  • Messages routed via Redis channels

Message Ordering Guarantees:

  • Per-match ordering via sequence numbers
  • Redis streams for ordered delivery
  • Client-side reordering for out-of-order messages

7. Latency Optimization

Region-Based Routing:

  • Geo-DNS routes to nearest region
  • Players matched within region first
  • Cross-region only if queue >5m wait

Ping-Based Region Detection:

  • Client measures ping to regional endpoints
  • Reports preferred region on login
  • Server validates and may override

Cross-Region Play Handling:

  • Increased latency tolerance for international matches
  • State sync every 100ms vs 50ms
  • Deny if ping >200ms for competitive modes

8. Database Scaling

Read Replicas:

  • 3+ replicas per region
  • Leaderboards read from replicas
  • Automatic failover to new master

Indexing Strategy:

  • Composite indexes on (season, region, rating)
  • Partial indexes for active players
  • BRIN indexes for time-series data

Query Optimization:

  • Use CTEs for complex rankings
  • Paginate all leaderboard queries
  • Cache frequent queries in Redis

9. Fault Tolerance & Recovery

Circuit Breakers:

  • Open after 5 failures in 10s
  • Half-open test every 30s
  • Close after 3 successful calls

Match Recovery:

  • State snapshots every 30s in Redis
  • Event log for replay
  • Clients buffer actions during reconnect

Graceful Degradation:

  • Disable non-critical features (achievements) under load
  • Queue matchmaking requests
  • Serve cached data when DB down

10. Observability at Scale

Metrics:

  • QPS per endpoint
  • P95 latency
  • WebSocket connection count
  • Redis hit rate

Detecting Issues:

  • Laggy matches: Monitor state sync delays
  • Cheating clusters: Anomaly detection on action patterns
  • Desyncs: Compare client/server state hashes

11. Load & Stress Testing

Tools:

  • k6 for WebSocket load testing
  • Locust for API stress
  • Custom C# clients for realistic traffic

Pass Criteria:

  • <100ms P95 latency under 50k connections
  • <1% packet loss
  • Auto-scale within 2 minutes

12. Cost-Aware Scaling

Scale-to-Zero:

  • Notification service scales to zero off-peak
  • Use spot instances for non-critical workloads

Cost Control:

  • Monitor Redis memory usage (target <70%)
  • Right-size instances based on load patterns
  • Use reserved instances for baseline capacity

Technologies

  • ASP.NET Core 8: Stateless APIs and SignalR
  • Redis Cluster: Distributed state and messaging
  • PostgreSQL: Sharded persistence
  • RabbitMQ/Kafka: Event streaming
  • Kubernetes: Orchestration and scaling
  • Prometheus/Grafana: Monitoring

Setup

  1. Deploy to Kubernetes cluster
  2. Configure Redis Cluster and PostgreSQL
  3. Set environment variables for regions
  4. Run kubectl apply -f k8s/

API Documentation

  • Swagger available at /swagger on each service
  • WebSocket events documented in /docs/websocket.md

Development

  • Use .NET 8 SDK
  • Run dotnet build for all services
  • Use docker-compose for local development
  • Run k6 run load-test.js for stress testing

Deployment

  • Multi-region Kubernetes clusters
  • CI/CD with GitHub Actions
  • Blue-green deployments for zero-downtime updates
  • Automated scaling based on custom metrics