TL;DR — Going from 1,000 to 20,000 users did not require new architecture. It required fixing four things the database was quietly absorbing at small scale: missing composite indexes on the real access patterns (800ms → under 50ms), no pagination on list endpoints, a new connection per request instead of a pool, and synchronous side work (notifications, analytics) running inside the request cycle. The one genuinely structural change was pre-computing meal quotas so a checkout does an O(1) cache read instead of an O(N) scan — because 80% of a day’s orders arrive inside a 90-minute lunch window, at roughly 15× the daily average.

When I took over the backend of a food-tech SaaS, the platform had around 1,000 registered users across a handful of client sites. A year later, it’s at 20,000+ users processing 50,000 meals per month. This is what that scaling journey actually looked like - not the clean version, but the real one.

The Early Cracks

At 1,000 users, most things “worked.” But as we onboarded more institutional clients, cracks started showing.

API response times crept up. Database queries that ran fine with 10,000 rows started timing out at 200,000. Concurrent meal orders during our intense lunch rush (11:30 AM – 1:00 PM) caused request queuing and database locks that slowed the entire site to a crawl.

1. Database-First Optimization

Almost every scaling problem traced back to the database. When I audited the system, I focused on three immediate interventions:

  • Indexing Strategy Overhaul: I audited every query using EXPLAIN ANALYZE and added composite indexes for the most common access patterns (such as combining venue_id with order_status and delivery_time). This alone dropped average query time from 800ms to under 50ms for the order listing endpoints.
  • Schema Normalization: Some tables had been designed with flexibility in mind (utilizing JSON columns for dynamically defined fields). I migrated these to proper relational columns with appropriate types and foreign key constraints, which dramatically cut down CPU evaluation costs in query filters.
  • Connection Pooling: We were creating new database connections per request, creating heavy TCP handshake overhead. Implementing proper connection pooling using pg-pool reduced connection overhead and kept connection saturation safely below thresholds. Worth knowing where the ceiling is: every Postgres connection is a backend process, and max_connections is a hard server-side limit that an in-process pool per app instance can still collectively exceed. Once you run several instances, an external pooler like PgBouncer is what keeps the total bounded.

2. API Architecture Refinement

The original API had grown organically - lots of endpoints doing too many things. I refactored our core router with a focus on:

  • Pagination Everywhere: Enforced strict query pagination using limit/offset and cursors so that list views never try to render thousands of rows in a single DOM draw.
  • Response Shaping: Different clients needed different data structures. Instead of sending full database models, I implemented lightweight view DTOs to shrink our network payload size.
  • Caching Hot Data: Venue menus change once a week, not once a second. Adding a cache layer in front of dynamic menus eliminated thousands of database reads per hour.

3. The Lunch Rush Problem

Institutional cafeterias have extremely predictable and highly congested usage patterns. Unlike standard food delivery, 80% of all meal orders come in during a tightly bounded 90-minute lunch window. This meant our servers needed to handle peak loads that were 15x the daily average.

To survive these spikes without scaling our cloud costs infinitely:

  1. Decoupling Non-Critical Work: I offloaded tasks like push notifications, admin email alerts, and analytic log compilations into a background queue processed after peak hours.
  2. Pre-Computing Availability: Instead of scanning ordering records to determine inventory limits on every checkout, we pre-computed meal item quotas every hour, converting an O(N) scan into an O(1) cache read.

Monitoring That Actually Matters

I set up monitoring dashboards that tracked metrics predictive of failures rather than just CPU alerts:

  • API p95 Latency: Warns us when p95 responses exceed 200ms.
  • Connection Pool Saturation: Alerts us when available database pools drop below 15%.
  • Job Queue Depth: Helps us spot background processing lag.

What I’d Do Differently Next Time

  • Plan the Indexing Early: Invest in database indexes from day one rather than retrofitting them under pressure.
  • Bake in Caching First: Integrate cache and pagination middlewares into the router framework rather than adding them as custom interventions.
  • Realistic Load Testing: Build synthetic load tests modeling the concentrated lunch rush early in staging.

FAQ

Does scaling from 1,000 to 20,000 users require new architecture? Usually not. In this case it required fixing what the database was absorbing at small scale — missing composite indexes, unpaginated endpoints, a connection per request, and side work running inside the request cycle. Re-architecting before exhausting those is solving the wrong problem.

Why do queries that were fine suddenly time out? Because the query never changed but the row count did, and a sequential scan that was imperceptible over ten thousand rows is not over two hundred thousand. The assumption that current performance holds as data grows is the actual bug.

What does connection pooling actually fix? It removes the TCP and authentication handshake from every request and bounds how many backend processes the database has to maintain. Without it, connection setup becomes a per-request tax and saturation becomes an outage during peak.

How do you handle a predictable traffic spike? Pre-compute what the spike will ask for. If most of a day’s orders arrive in a ninety-minute window, the work to do is turning an expensive per-checkout calculation into a cheap cache read before the window opens, rather than scaling compute to brute-force it.

Should notifications and analytics run inside the request? No. They make your response time depend on systems you do not control, and they fail the user for work the user did not ask for. Move them to a background job and return as soon as the durable write is done.

What should you monitor when scaling a SaaS backend? The metrics that predict failure rather than confirm it — query duration, p95 latency, connection-pool saturation and error rate. CPU tells you something is already on fire; pool saturation tells you it is about to be.

The one idea to take away

Scaling problems arrive as a step function, not a slope. Nothing on this list was a bad decision at 1,000 users — an unpaginated list endpoint over 800 rows is fine, and a sequential scan over 10,000 rows is fast. They all failed at roughly the same moment, because they were all being carried by the same thing: a dataset small enough to hide them. The work of scaling is mostly finding out which of your current choices are being subsidised by your current size.

If you want the specifics rather than the overview, the database-side detail is here, the full list of what I got wrong is here, and the deployment side is here.