Load testing: finding the limits before a crowd does

The scalability reports (AWS) found their bottlenecks by reading code and pointing ab at one URL at a time. That was enough to rank them, but nobody can repeat it. It doesn’t catch the next bottleneck a pull request adds, and it never tried a real mix of learners. This page proposes a repeatable way to spike-test the app ahead of time. It is a proposal: none of the files named below exist yet.

Short version. Use k6 (one brew install, nothing added to the app). Write one scenario file that plays a learner the way the reports describe one: sign up, open a course, read, answer every ~6 s, leave after ~20 min. Run it in three places, each closer to production than the last:

1. The Mac, against ./.dev-start --release. Ten minutes. Finds code bottlenecks.
2. A Docker container on the Colima runners, with the AWS unit’s CPU and memory limits. Thirty minutes. Gives Linux memory and small-box numbers.
3. A throwaway copy of the AWS stack, before a launch or press. An hour, under a dollar. Gives real TLS, Caddy, CPU credits and network.

Each run passes or fails on numbers written in the scenario file, so it answers “did this get worse?” without anyone reading graphs. Setup takes about two days.

1. Why not just ab again

The alternatives to k6:

• oha: keep it for single-URL micro-benchmarks. It is ab with better output.
• Locust: needs Python and a worker process per core.
• Artillery: adds an npm tree.
• Playwright (already in e2e/): one browser per learner, so a laptop manages a few dozen, not 1,500. It stays the tool for correctness.

k6 is one static binary, scenarios are plain JavaScript, and it can drive a few thousand virtual users from one laptop.

2. What to add to the repo

All of it is dev-only. Nothing is compiled into the app or shipped.

One change to the app, needed only for the staging run. The rate limits in security.rs are constants that only dev mode turns off. Dev mode needs a loopback bind (src/dev.rs), so it can’t serve a load generator on another machine. Every request from that generator then shares one address and hits REGISTER_PER_IP after 20 sign-ups. Add SKRONGLE_RATE_LIMITS=off. It logs a warning at startup, and deploy-aws.sh refuses to write it unless AWS_STACK_NAME contains load. The change is about 20 lines.

3. The virtual learner

The traffic model comes from the first report’s section 1, so test numbers and report numbers mean the same thing.

A learner (75 % of arrivals) is one k6 virtual user each:

1. GET / (landing), GET /register, then submit the form with res.submitForm() and the address load-<run>-<vu>-<iter>@example.test. This exercises argon2, the outbox row and the session. Follow the Terms step if the app asks.
2. POST /courses/open/<id> for a random built-in course. 20 % of learners also pick a theme, spread evenly over all of them.
3. Repeat for ~20 minutes: GET a lesson or /practice every ~30 s. Read the question ids from data-qid and POST /api/answer with {question_id, answer} every ~6 s (wrong answers cost the same). Every ~40 answers, finish a round.
4. Some learners come back: 30 % of them sign in again with POST /login (the second argon2 path).

A visitor (25 % of arrivals) loads three pages (/, a static asset, /register) and leaves.

One setting scales the load: -e LEARNERS=500 is the reports’ peak hour. Sleeps are randomised (±50 %), so 500 learners don’t answer in lockstep.

4. The profiles

Each profile targets bottlenecks the reports found, and fails if the bottleneck is still there. The thresholds are a starting point. Tune them after the first green run on each target.

smoke, peak, spike and signups cover the viral day. The rest target specific findings and run when the code they test changes.

Seeding. The history profile needs learners with weeks of answers. Writing 14,000 answers each through the API would take an hour, so the script seeds them straight into SQLite before the run (copy quiz_history rows from a template learner with shifted timestamps). It must stop the app first and start it with SKRONGLE_DEV_FORCE=1, because dev mode refuses a database with other accounts (#388).

5. What to watch on the server

The client sees only latency and status codes. While k6 runs, the script also samples the server every 5 s, and it checks a few facts at the end.

Reading a failure. Each bottleneck has a recognisable shape:

6. Where to run it

1. The Mac: every time a hot path changes

scripts/load-test.sh --where mac --profile peak

./.dev-start --release --reset --background with the simulated LLM, then k6 on the same machine. Dev mode turns rate limits off and keeps all mail in the dev mailbox. Dev mode also has no hourly mail ceiling unless one is set, so the script sets SKRONGLE_MAIL_PER_HOUR to the production default. Without it, signups would pass falsely.

The numbers are flattering (a fast laptop, no TLS). What counts is the comparison with load/baseline.json, so a change that makes the home page 3× slower fails here, before review. Run it when a PR touches src/db.rs, src/auth.rs, routes/auth.rs, routes/pages.rs, routes/quiz.rs, src/mail.rs, src/courses.rs or src/ai.rs. The QA playbook (docs/qa.md) can name it for those paths.

2. Docker on Colima: monthly, and before a release that changes limits

scripts/load-test.sh --where docker --size aws --profile spike

The runners already run arm64 Linux in Colima, the same architecture and OS family as the Graviton instance. The script starts the release binary in a container sized like the systemd unit:

Dev mode answers only requests addressed to localhost, so k6 runs in a second container that shares the app’s network namespace (docker run --network container:skrongle-load grafana/k6 …). This run gives the Linux memory numbers both reports list as not measured.

Run it outside CI hours, one at a time. The runners share the Mac’s disk, and a full disk takes them all offline.

3. A throwaway AWS stack: before an expected spike

AWS_STACK_NAME=skrongle-load AWS_INSTANCE_TYPE=t4g.nano scripts/deploy-aws.sh
scripts/load-test.sh --where https://<ip>.sslip.io --profile spike
scripts/deploy-aws.sh --destroy      # with AWS_STACK_NAME=skrongle-load

This uses the same template, Caddy and unit as production, with SKRONGLE_RATE_LIMITS=off and no mail provider (the log chain, so no real e-mail is sent and the ceiling still applies). Run k6 from a second instance in the same region, a c7g.large for an hour (~$0.07), so the laptop’s uplink doesn’t become the bottleneck. Never point it at production. Nothing stops the run there, and 1,500 fake accounts would stay in the real database.

Only this run sees real TLS cost, Caddy’s compression, CPU credits being spent, and the instance’s memory with the OS and Caddy in it. Watch CPUCreditBalance in CloudWatch during spike. Run peak once on t4g.nano and once on t4g.small to decide whether the event needs the bigger box. A run costs well under a dollar (instance-hours plus a few GB of traffic within the free allowance).

7. Why not on every pull request

CI runs on eight Docker runners on one laptop, alongside the e2e shards. Load numbers taken there move with whatever else is running, so a threshold either flaps or is too loose to catch anything. Instead:

• PRs that touch a hot path get the Mac run (an author’s two-line note in the PR body: profile, p99 before and after).
• A weekly scheduled peak + signups in Docker, at night when the runners are idle, posts its result file to an issue and fails loudly against the baseline.
• The AWS run is on the checklist for any planned launch.

8. Getting started, in order

1. brew install k6. Write load/learner.js with the smoke and peak profiles, and run them against ./.dev-start --release. (~½ day)
2. scripts/load-test.sh for the Mac: start, run, sample RSS, grep the journal, count outbox rows, write the result file, compare with the baseline. (~½ day)
3. Add spike and signups. signups should fail today on the mail ceiling, and that failure proves the harness works. (~2 h)
4. Docker mode with --size, and the first Linux memory numbers. (~½ day)
5. SKRONGLE_RATE_LIMITS=off, the guard in deploy-aws.sh, and one staging run before the next launch. (~½ day)
6. memory, history, ai and soak as the matching bottlenecks get fixed. Each one turns a fix into a regression test.