Case Study: TahminApp — 500K Users, 99.9% Uptime, 6 Months in Production

# Case Study: TahminApp — 500K Users, 99.9% Uptime, 6 Months in Production

Problem: On a sports prediction platform, instantly push real-time match events (goals, cards, halftime) to 500K+ users while updating live scores in the UI.

Result: 6 months in production, 500K MAU, p99 latency 180ms, 99.94% uptime, 0 incidents (>30min downtime).

Pro Tip: The bottleneck is not where you expect it. For us it wasn't Firestore writes — it was push delivery routing. Every big scaling story starts the same way.

1. Architecture Overview

3 tiers:

• Edge layer (CF Workers): Webhook receiver, dedup, fan-out trigger
• Backend (Firestore + Cloud Functions): Persistent state, business logic
• Mobile (iOS native + Flutter Android): Realtime listener + push notification handler

1[Sports Data Feed]
2        ↓ webhook
3[CF Worker — dedup + transform]
4        ↓ event
5[Cloud Function — match state update]
6        ↓ write
7[Firestore — predictions, users, scores]
8        ↓ realtime listener
9[Mobile Apps — UI update]
10        ↓
11[FCM/APNs — push notification]

2. The Bottleneck Story

First hypothesis: Firestore writes don't scale. Didn't hold up.

Second hypothesis: Cloud Functions cold starts are the problem. Didn't hold up either (warm instance pool).

The real bottleneck: APNs delivery latency. Apple's production token routing was delivering 500K pushes in 4–6 minutes instead of 30 seconds. During big-match events, "late notification" complaints spiked to 12%.

Fix: Priority routing — match-event pushes use APNs apns-priority 10 plus parallel batch sends from Cloudflare Workers. 500K pushes now deliver in 25–35 seconds. Complaints dropped from 12% to 0.3%.

3. Firestore Design

500K users × 50 predictions/user × 100 matches/season = 2.5M documents. A naive /predictions/{id} design would have collapsed.

The right pattern — sharded collection:

1/predictions/shard-{0-19}/items/{predictionID}

Shard count is 20, assigned from a hash of the prediction ID. Each shard has its own index and its own write limit. Total write throughput: 20x.

Realtime listener: Mobile listens to /predictions/shard-{userShard} — only its own shard. Network savings: 95%.

4. Cloud Functions Patterns

40+ Cloud Functions in total. The 3 most critical:

1. `onMatchEvent`: webhook trigger, 30-second dedup window, state update + push trigger
2. `onPredictionWrite`: leaderboard recalculation (sharded counter pattern)
3. `onUserWrite`: profile cache invalidation (Cloudflare KV purge)

Every function:

• Memory: 256MB (most), 1GB (leaderboard)
• Timeout: 60s
• Max instances: 1000 (auto-scale)
• Retry: 3x exponential backoff
• Idempotency: Firestore set-based dedup on eventID

5. iOS App Architecture

iOS native (SwiftUI, iOS 16+):

• TCA (The Composable Architecture) for state management
• Firestore Snapshot Publisher — Combine pattern
• APNS Notification Service Extension — payload decryption + smart actions
• Local cache (SwiftData) — offline read-only mode

Memory baseline: 95MB. Peak (live match-time UI): 145MB.

Crash-free session rate: 99.84% (Crashlytics, 6-month average).

6. Cost Profile

Monthly infrastructure cost (at 500K MAU peak):

Per-user cost: $0.00127/user/month. Outstanding unit economics for an indie project.

7. Lessons Learned

1. The predicted bottleneck is rarely the real bottleneck. We only realized APNs delivery was the problem 4 weeks after launch.

2. Pay for sharding up front. Adding sharding after you've already hit 500K users is 4 months of technical debt.

3. Idempotency via event ID is critical — webhook retries were silently creating duplicates.

4. The Notification Service Extension is gold for content modification and alert prioritization.

5. Doing dedup + transform at the edge layer (CF Workers) cut Cloud Functions cost by 4x.

6. Run Crashlytics and custom telemetry together. Crashlytics gives you aggregation, custom telemetry gives you decision-making data.

7. In App Store reviews, average notification latency during big match moments is what users comment on the most. Optimizing latency took the 5-star rate from 62% to 78%.

Bottom line: Scaling to 500K production users is usually not a "Firestore problem" — it's a "delivery infrastructure" problem. Design the architecture as multi-tier from day one, then measure and refactor.