Real-Time-Social-Signal-Intelligence-Platform

Stars
6
Language
TypeScript
Updated
View on GitHub

Real-Time Social Signal Intelligence Platform (SSIP)

Meta Interview Excellence: Production-grade system design with advanced features for senior engineering interviews.

A production-grade system that processes real-time social interaction data at scale, similar to feed-ranking and analytics systems used in large social platforms.

🏗️ Architecture Overview

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Event Ingestion   │ -> │  Signal Processing  │ -> │   APIs & Cache    │
│   (Rate Limited)    │    │  (MongoDB Agg)     │    │   (Redis)        │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                        │                        │
         ▼                        ▼                        ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   MongoDB       │    │   Change Streams │    │   Monitoring    │
│   (Sharded)     │    │   (Real-time)    │    │   (Prometheus)  │
└─────────────────┘    └──────────────────┘    └─────────────────┘

🚀 Features

  • High-throughput event ingestion (10k–100k events/sec simulated)
  • Advanced feed ranking algorithm with time decay, creator momentum, personalization
  • Sliding window aggregations (5min, 10min, 1hr, 24hr) with incremental updates
  • MongoDB sharding simulation with optimized shard keys
  • Backpressure protection with rate limiting and load shedding
  • Failure recovery with retry logic, dead letter queues, circuit breakers
  • Comprehensive observability with health/ready endpoints and detailed metrics
  • Real-time signal processing using MongoDB aggregation pipelines and change streams
  • Low-latency APIs for feed, analytics, and recommendations
  • MongoDB-first architecture with optimized data models

📊 System Design Walkthrough

Core Design Decisions

1. MongoDB-First Architecture

  • Why: Time-series data, complex aggregations, horizontal scaling
  • Tradeoff: Eventual consistency vs immediate consistency
  • Mitigation: Change streams for real-time updates, retry logic for failures

2. Sliding Window Aggregations

  • Why: Real-time signals need fresh data without full recomputation
  • Implementation: Incremental updates using $merge with upsert semantics
  • Performance: O(1) updates per event vs O(n) full aggregation

3. Feed Ranking Algorithm

score = (engagement × 0.4) + (time_decay × 0.3) + (creator_momentum × 0.2) + (personalization × 0.1)
  • Time Decay: e^(-λ × hours) prevents old content from dominating
  • Creator Momentum: Growth velocity + trend score for rising creators
  • Personalization: Interest overlap using user vectors

4. Sharding Strategy

  • interaction_events: {contentId: 1, timestamp: -1} - Groups related events, enables time queries
  • content_engagement_stats: {contentId: 1, window: 1} - Co-locates engagement data
  • user_interest_vectors: {userId: 1} - User-centric queries
  • creator_growth_metrics: {creatorId: 1} - Creator analytics

5. Backpressure & Reliability

  • Rate Limiting: Token bucket algorithm with Redis
  • Load Shedding: Drop non-critical requests under high load
  • Circuit Breakers: Prevent cascade failures
  • Dead Letter Queues: Handle persistent failures

Scaling Limits & Tradeoffs

Current Limits:

  • Ingestion: ~50k events/sec (single node, can shard horizontally)
  • Query Latency: P95 < 100ms (with Redis cache)
  • Data Freshness: < 5 seconds (change streams + periodic aggregation)
  • Storage: Scales with MongoDB sharding

10× Scale Considerations:

  • Sharding: Add more MongoDB shards, update shard keys if needed
  • Caching: Redis cluster, cache warming strategies
  • Processing: Stream processing (Flink/Spark) for heavy aggregations
  • APIs: API Gateway with regional deployment
  • Monitoring: Distributed tracing, anomaly detection

📋 Prerequisites

  • Node.js 18+
  • MongoDB 7.0+
  • Redis 7+
  • Docker & Docker Compose

🛠️ Installation

  1. Clone the repository

    git clone <repository-url>
cd social-signal-intelligence-platform

  2. Install dependencies

    npm install

  3. Start infrastructure

    docker compose up -d

  4. Configure environment

    cp .env.example .env
# Edit .env with your MongoDB and Redis credentials

  5. Start the application

    npm start

📖 API Documentation

Event Ingestion

POST /ingest/events

Ingest a single interaction event with rate limiting.

Rate Limit: 1000 requests/minute per user

POST /ingest/events/batch

Ingest multiple events with rate limiting.

Feed APIs

GET /feed/trending?window=10min

Get trending content with advanced ranking.

Rate Limit: 5000 requests/minute per user

GET /feed/recommended?userId=123

Get personalized recommendations using ranking algorithm.

Analytics APIs

GET /analytics/content/{contentId}

Get detailed analytics with freshness metrics.

GET /analytics/creator/{creatorId}

Get creator growth metrics and trends.

GET /analytics/trends?window=1hr&limit=50

Get trending analytics across creators.

Health & Monitoring

GET /health

Basic health check.

GET /ready

Readiness check (MongoDB + Redis + Signal Processor).

GET /metrics

Prometheus metrics including:

  • Event processing lag
  • Aggregation freshness
  • Cache hit ratios
  • Rate limit hits
  • Dead letter queue sizes
  • Circuit breaker states

🔧 Configuration

Environment Variables

  • MONGODB_URI - MongoDB connection string
  • REDIS_HOST - Redis host
  • REDIS_PORT - Redis port
  • PORT - Application port (default: 3000)
  • RANKING_* - Feed ranking algorithm weights
  • RATE_LIMIT_* - Rate limiting configuration

Ranking Configuration

{
  weights: {
    engagement: 0.4,      // Raw engagement score weight
    recency: 0.3,         // Time decay weight
    creator: 0.2,         // Creator momentum weight
    personalization: 0.1  // User interest overlap weight
  },
  timeDecayLambda: 0.1,   // Decay rate per hour
  cacheTTL: 300          // Cache duration in seconds
}

📊 Monitoring & Observability

Key Metrics

  • ssip_aggregation_lag_seconds: How fresh aggregations are
  • ssip_signal_freshness_seconds: Age of cached signals
  • ssip_ranking_latency_seconds: Time to rank content
  • ssip_rate_limit_hits_total: Rate limit enforcement
  • ssip_dead_letter_queue_size: Failed operation backlog
  • ssip_circuit_breaker_state: Service health status

Alerting Thresholds

  • Aggregation lag > 30 seconds
  • Cache hit ratio < 80%
  • Dead letter queue size > 100
  • Circuit breaker open > 5 minutes

🧪 Testing

Load Testing

# Run basic API tests
npm run load-test

# Run heavy load test (500 RPS for 30 seconds)
npm run load-test:heavy

Unit Tests

npm test

🚀 Deployment

Production Considerations

  1. MongoDB Sharding:

    sh.enableSharding("ssip")
sh.shardCollection("ssip.interaction_events", {contentId: 1, timestamp: -1})

  2. Redis Cluster: For cache horizontal scaling

  3. API Gateway: For rate limiting, authentication, regional routing

  4. Monitoring Stack: Prometheus + Grafana + AlertManager

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ssip
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: ssip
        image: ssip:latest
        env:
        - name: MONGODB_URI
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: uri
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000

🔒 Security & Reliability

Security Measures

  • Input validation and sanitization
  • Rate limiting on all endpoints
  • CORS configuration
  • Secure MongoDB and Redis connections

Failure Scenarios & Recovery

MongoDB Node Failure:

  • Automatic reconnection with exponential backoff
  • Circuit breaker prevents cascade failures
  • Failed operations go to dead letter queue
  • Manual replay capability

Redis Outage:

  • Graceful degradation (serve stale data)
  • Circuit breaker activation
  • Automatic reconnection

Signal Processor Crash:

  • Stateless design allows immediate restart
  • Checkpoint-based recovery
  • Idempotent operations prevent duplicates

High Load:

  • Load shedding for non-critical endpoints
  • Rate limiting with backpressure
  • Horizontal scaling of ingestion workers

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests
  5. Submit a pull request

📄 License

MIT License - see LICENSE file for details.

🎯 Interview Preparation

This system demonstrates:

  • System Design: Scalable architecture decisions
  • Data Modeling: MongoDB optimization for time-series
  • Performance: Caching, aggregation, indexing strategies
  • Reliability: Failure handling, monitoring, recovery
  • Scalability: Sharding, load balancing, backpressure

Key Interview Talking Points:

  • Why MongoDB over traditional RDBMS?
  • How sliding windows avoid expensive recomputation?
  • Ranking algorithm tradeoffs and alternatives?
  • Failure recovery patterns and their importance?
  • Observability for production systems?

Built for Meta-scale interviews and production systems 🚀 /Users/manavaryasingh/Real-Time Social Signal Intelligence Platform/README.md