Testing locally

How a person or an AI agent drives and checks the app on their own machine: one command to a logged-in session, a simulated AI provider whose replies come from canned files (or that you answer by hand), and markup with stable hooks.

Everything here is local only. It needs SKRONGLE_DEV=1, which ./.dev-start sets and deploy/ never does, and the server refuses it (logging why) unless it binds a loopback address, SKRONGLE_SECURE_COOKIES is not 1 and SKRONGLE_PUBLIC_URL is unset (or local). It also refuses it when the database holds any account besides dev@example.com, since that looks like a deploy’s database, unless SKRONGLE_DEV_FORCE=1 says it is a development one; ./.dev-start sets that, and scripts/e2e.sh, scripts/smoke.sh and the playthroughs start from a fresh database. In dev mode every route answers only requests made to localhost, 127.0.0.1 or [::1] (any port) and gives anything else 421 Misdirected Request, so a DNS-rebinding page cannot reach the dev server under its own name; open the dev server by one of those names, as the scripts do. On top of that the /dev/* routes refuse (404) requests with proxy headers (X-Forwarded-For, X-Real-IP, Forwarded). Without dev mode the /dev/* routes are 404, SKRONGLE_LLM_MODE is ignored and model calls go to the real providers; tests check all of this.

One command to a logged-in session

./.dev-start --reset --seed --background   # prints the site, inbox, login and PR (below)
open "$(cat .dev-server.url)/dev/login"                # logged in as dev, Spanish open
./.dev-start --stop

Once the server answers, ./.dev-start prints only this:

site:  http://127.0.0.1:8214
inbox: http://127.0.0.1:8214/dev/inbox
login: dev@example.com / dev-password   (or http://127.0.0.1:8214/dev/login)
pr:    https://github.com/alexbeeken/pooki/pull/264   ([263] Profile picture: no own-colour swatch)

pr is the branch’s open pull request, so a reviewer can start on its QA steps; gh looks it up during the build, and the line is left out without gh, a login, the network or an open PR. inbox is the dev mailbox page, e-mail, texts and the dev authenticator; it is left out when the dev mailbox and the dev phone are off, login when dev mode is (--lan, which prints phone: with the LAN address instead). The build’s output shows only if it fails, and the server log goes to dev-server.log, in the foreground too (Ctrl-C stops it); a server that exits at boot prints the log’s last lines. DEV_VERBOSE=1 ./.dev-start … shows everything: the build, the settings, and in the foreground or with --watch the server log in the terminal.

GET /healthz answers ok with no login; wait on it instead of sleeping.

A crowd of learners with history

Some features only show with many learners behaving over time: the Social motivator’s comparisons (“571 people studied this course this week”), the streak and daily goal over weeks, mastery decaying, a long History page, the retention jobs, and whether the queries behind them hold up with a thousand accounts in the tables (#583). --seed-users makes that crowd:

./.dev-start --reset --seed --seed-users 1000 --background   # 1000 learners, 90 days of history, about 10 s
./.dev-start --reset --seed --seed-users 200 --seed-days 14   # a smaller, shorter one

The server binds its port first, then writes the crowd, then answers. --background asks /healthz with a one-second limit per try and waits 20 s by the clock, longer while dev-server.log shows the crowd growing (dev crowd: 450/1000 learners; the crowd writes one at least every 5 s, #936). If the wait runs out, it stops the server it started, so no server keeps running without a pid file (#679, #869). scripts/dev-start-orphan-check.sh checks that; run it after changing .dev-start. With SKRONGLE_SQL_TRACE on, 1000 learners take a few minutes and write a trace of several hundred MB.

Each learner (learner-<n>@seed.example, password dev-password, so you can sign in as any of them) is a small model drawn from their number: when they joined, how keen they are (the chance of studying on a given day: a casual, regular or daily learner), how good (the chance of a right answer), the mode they prefer (Sprint, Study or the Guide), their time zone and usual hour, and whether they drifted away (a third do). A day they study is a session: open the lesson they are on, read, play or run through it, answer its questions, now and then a round of Word Match or a review of an earlier lesson. Eight in ten learn Spanish, the rest French.

The rows are written through the same functions the modes call (progress, mastery, activity, games), only at a time in the past, so they obey every rule the real ones do: a pass is seen plus a counted run of 70 %, mastery decays between sessions, effort points weigh as the daily goal says. Nothing lands in the future; nothing before the account was made. The crowd is deterministic: learner 7 always gets the same life, a second boot adds nothing, and --seed-users 1200 later only adds the missing 200. It does not make the database look like a deploy’s (#388): the dev-mode account check ignores the crowd’s domain. --reset removes it with everything else. src/dev_crowd.rs is the model.

Live reload (--watch)

./.dev-start --reset --seed --watch                # foreground: build log and server log in this terminal
./.dev-start --reset --seed --watch --background   # detached, log in dev-server.log, ./.dev-start --stop ends it

.dev-start polls src/ templates/ static/ data/ docs/ Cargo.toml Cargo.lock once a second. On a change it runs cargo build; if that fails it prints the error and keeps serving the previous build; if it succeeds it stops the server, starts the new binary on the same port with the same dev.db (the session cookie stays valid, so a logged-in tab stays logged in), and waits on /healthz. --seed seeds only on the first start. A typical round after saving a template:

==> changed: templates/lessons.html
   Compiling spanish-learning v0.1.0 (…)
    Finished `dev` profile [optimized + debuginfo] target(s) in 3.97s
==> restarting on http://127.0.0.1:8316
… INFO shut down cleanly
… INFO listening on http://127.0.0.1:8316/
==> healthy: http://127.0.0.1:8316/healthz answered after 1 s (server pid 93837)

Why everything restarts: askama templates and docs/course-format.md (the AI prompt) are compiled in, and data/ (courses, themes) is read at boot. static/ and the /docs pages are embedded with rust-embed, which in a debug build reads them from disk on each request, so they are live even without --watch; the watcher restarts for them anyway (the build is a no-op) so open pages reload.

Browser reload: --watch sets SKRONGLE_DEV_RELOAD=1, and in dev mode the server then appends a small script to every HTML response that polls GET /dev/boot (this process’s boot id) every second and reloads the page when the id changes. Polls are not logged; nothing is injected without --watch (so scripts/e2e.sh and smoke.sh never see it) or outside dev mode. A half-filled form is lost on the reload, as with any live reload.

Tools considered (issue #179). None of them is installed by default here, and none of them is needed:

So --watch is a plain polling loop in .dev-start plus the reload script in src/dev.rs (inject_reload, /dev/boot): no new dependency, no install step.

Dev routes

The dev mailbox (/dev/inbox, /dev/mail, /dev/phone, /dev/sms, /dev/authenticator, /dev/ai, /dev/payments and every route under them, JSON and actions too) needs no sign-in: it is for whoever is testing and holds every account’s mail, texts and authenticator entries. Like every dev route it exists in dev mode only, for requests made on this machine (dev::local_only). e2e/lib/mail.ts reads it with a request context of its own, so the learner a spec drives is not touched.

Google sign-in

Dev mode with no SKRONGLE_GOOGLE_CLIENT_ID stands in for Google itself, so “Continue with Google” shows on /login, /register and the profile, and works: /auth/google runs the real flow (the redirect with state, nonce and PKCE challenge, the code sent back to /auth/google/callback, the server-to-server token exchange, the ID token’s claims checked) against the app’s own /dev/google/auth and /dev/google/token. Only Google’s pages and the token signature (never checked; the real flow trusts the TLS channel) are missing. scripts/e2e.sh and scripts/smoke.sh get it for free; SKRONGLE_GOOGLE_MOCK=0 turns it off to see the layout without Google. A real client configured wins over the stand-in.

There is no account picker: query parameters on /auth/google decide (the server’s startup log says so too). In dev mode with the stand-in on (never in production, never with a real client configured), the “Dev mode” card on /login and /register shows a row of small buttons under the dev login button (login-dev-google, register-dev-google): Sign in as mock.learner@example.com (…-dev-google-ok, mock_email= the default) and one per failure below (…-dev-google-cancel, -invalid_grant, -unverified, -timeout, -unavailable), each a plain link to /auth/google?mock_error=…. Type the URL by hand for another address.

open "$(cat .dev-server.url)/auth/google?mock_email=ana@example.org"          # sign up or in as Ana
open "$(cat .dev-server.url)/auth/google?mock_error=cancel"                   # see the cancelled message

A signed-in visitor on /auth/google connects the account instead (the profile’s Connect a Google account), with the same parameters. The stand-in checks its callers the way Google does (client id and secret, redirect URI, PKCE verifier, a code’s age) and answers Google’s error shapes, so a bug in the real flow shows up locally too.

E-mail: the dev mailbox

In dev mode the app sends its e-mail to a small SMTP server of its own (src/mail/sink.rs), so a sign-up or a password flow can be clicked through to the end without a real provider. The outbox, the job runner that sends it and lettre run as in production; only the server at the other end is the app’s. It is on by default; ./.dev-start prints its link (inbox:).

• Open the mail: <url>/dev/inbox (the Outbox tab is /dev/mail), or <url>/dev/inbox/latest?to=<address> for the newest message to one address; no sign-in needed. Rows are bold until opened; Archive strikes one out and moves it down, Delete removes it; the page updates by itself. A message shows its HTML as a mail client would; click a link in it and it opens in the tab.
• Where it listens: 127.0.0.1 on a free port (in the log line dev: e-mail (SMTP on …) and text messages go to the dev mailbox and in /dev/inbox’s JSON as smtp). Other tools on the machine may send to it too, e.g. swaks --server 127.0.0.1:<port> --to ann@example.org.
• SKRONGLE_DEV_SMTP: unset or 1, a free port; a number, that port; 0, off (mail is only logged, as outside dev mode). With SKRONGLE_MAIL_PROVIDERS set, dev mode sends through those providers instead, and /dev/inbox says it is off.
• What it keeps: the newest 200 messages, in memory; a restart empties it. It takes EHLO/HELO, MAIL, RCPT, DATA, RSET, NOOP and QUIT: no TLS, no AUTH, at most 2 MB a message.
• In browser tests: e2e/lib/mail.ts has waitForMail(to, subject?) (it waits for the job runner to send it), inbox(to?) and mailLink(mail, /pattern/), which read the mailbox as the dev user (devRequest()), and devPage(browser, path), a browser of its own signed in as the dev user, for clicking through a message while the learner stays signed in as themselves. Give each test its own address (newAccount), since the mailbox is shared. e2e/specs/mail.spec.ts registers, opens the confirmation in the inbox and clicks its link.

Text messages: the dev phone

In dev mode without SKRONGLE_SMS_PROVIDERS the app’s texts go to a provider of its own, dev-phone (src/sms/phone.rs), which keeps them in memory (the newest 200; a restart empties it). The outbox, the job runner that sends it and the limits run as in production. It shares the dev mailbox page, so ./.dev-start’s inbox: link covers it too.

• Read them: <url>/dev/phone, the Text section of the dev mailbox page (next to E-mail). It works like the e-mail one (bold until opened, Archive, Delete, Inbox and Outbox tabs, updates by itself) but looks like a messaging app; a text opens as the conversation with that number.
• Send one: the Text outbox’s “Send a test text”, or curl -X POST <url>/dev/sms/send -d to=+447700900123 -d text=Hi.
• The limits still apply per number and account; delete outbox rows (or use another number) to send more.
• SKRONGLE_DEV_PHONE: unset or 1, on; 0, off (texts are only logged, as on a server without SKRONGLE_SMS_PROVIDERS; /dev/phone’s JSON says "phone": false). ./.dev-start --mini and the mini lane turn it off.

Two-factor codes: the dev authenticator

The dev mailbox page’s third section, Authenticator (<url>/dev/authenticator), is an authenticator app like Google Authenticator, so a flow that asks for a TOTP code can be clicked through with the real protocol and no special case in the app. It speaks RFC 6238 (src/totp.rs): HMAC-SHA1 by default, SHA-256 and SHA-512 when the link asks, 6 to 8 digits, any period.

• Add an account: paste an otpauth://totp/Issuer:account?secret=… link or a bare base32 setup key (any case, spaces allowed) and Add. The box opens filled in with an example account (Skrongle, ann@example.org, key JBSWY3DPEHPK3PXP), so Add alone adds it; removing the last account fills an empty box with it again. To “scan” a QR code, open /dev/authenticator?uri=<the otpauth link>: the form opens filled in. curl -X POST <url>/dev/authenticator -d uri=JBSWY3DPEHPK3PXP works too.
• Read a code: each account shows its current code, worked out on the server from the secret, and a countdown; the next code shows by itself. Copy puts it on the clipboard; Remove deletes the account.
• What it keeps: the accounts, in memory; a restart forgets them (as a phone without a backup would), so enrol again after one.
• In browser tests: e2e/lib/totp.ts works codes out with Node’s crypto (totp(secret), totpForStep) and reads the dev authenticator (authenticator(), authenticatorCode(account), read as the dev user like the mail helpers). e2e/specs/authenticator.spec.ts covers add, scan, codes, rollover and remove.
• Two-factor sign-in (features): the profile’s set-up dialog has Dev: add it to the dev authenticator, the scan link, which opens the dev authenticator in a tab of its own. /dev/login skips the second step, so sign in through /login to try it. The phone number’s check on the profile texts a code: read it on /dev/phone. e2e/specs/two-factor.spec.ts covers the phone check, e-mail, text and authenticator sign-in, reading the dev mailbox (devPage, waitForMail) and the sign-in codes from the dev device below; it runs in CI with the exact specs.

Two-factor sign-in: the dev device

In dev mode, a small window floats at the bottom of /login/code (login-code-dev, a <details> you can fold) framing /login/code/device (login-code-dev-frame): the dev device this sign-in’s code went to, and nothing else. For a text it lists the dev phone’s texts to that number; for e-mail the dev inbox’s mail to that address (the subject carries the code); for the app the dev authenticator’s entry for that address with its code counting down. Each row (login-code-device-item, data-code) has Use (login-code-device-use), which types the code into the form above, so a sign-in is two clicks. The frame re-reads itself every two seconds (static/devbox.js).

The route is only mounted in dev mode (with /dev/login, local clients only) and is keyed by the sign-in’s own twofa cookie: it shows the device of the sign-in in progress, and a request without a pending sign-in gets 404. It is the one page the app lets itself frame: it sets its own CSP (frame-ancestors 'self') and X-Frame-Options: SAMEORIGIN, which security::headers keeps. ?format=json gives the rows.

The dev AI provider

Every model call goes through one function (ai::complete) and carries a purpose: generate-outline, generate-lesson (key: the lesson slug), revise-plan, revise-lesson (key: the lesson slug), add-lesson and tutor-turn. In dev mode those calls go to a stand-in for the AI providers inside the dev server (src/dev_ai.rs, src/routes/dev_ai.rs, #297), which uses the real wire protocols: the app’s own clients in ai.rs call it exactly as they call Anthropic, OpenAI and Gemini, with a base URL and a key, so the application does not know it is fake (#298). It answers from canned files, meters usage per key, plays a used-up quota, timeouts and broken replies, and can hold a request for you to answer by hand.

./.dev-start --reset --seed --background   # then open <url>/dev/ai

How calls get there: the keys do the routing. In dev mode these are the providers’ default base URLs (unless SKRONGLE_ANTHROPIC_BASE_URL, SKRONGLE_OPENAI_BASE_URL or SKRONGLE_GOOGLE_BASE_URL say otherwise), and the dev user is seeded with one simulator key per provider (--seed, and /dev/login?keys=dev or ?keys=fake saves them too). A key the simulator did not issue gets the provider’s own 401, as a real provider answers a bad key. There is no keyless stand-in and no client-side switch: an account with no key sees the greyed-out dialog and the “No AI provider key is set up” alert, in dev mode as on a deploy. Outside dev mode, or with SKRONGLE_LLM_MODE=live, the defaults are the public APIs.

SKRONGLE_OPENAI_BASE_URL is still the way to test against a real local model (Ollama, LM Studio) for the OpenAI provider.

Two key profiles, told apart by the key in the request, each with its own counters, limit and injected failure:

Any other key is refused as the provider refuses a bad one (401, authentication_error / invalid_api_key). The wire endpoints are not behind the developer check, since the app’s server calls them with a key, not a cookie; like every /dev route they exist only in dev mode and answer only local requests.

The purpose travels in the request header x-skrongle-call (purpose=generate-lesson&key=<slug>&job=<id>&var.<name>=…, URL-encoded), which ai.rs sends on every model request, to the real providers too, who ignore it. The system prompt alone does not tell the call sites apart (the generation calls share one), and the body may carry no extra field (Anthropic refuses unknown ones), so a header is where it goes. A request without it (another client) gets its purpose guessed from the prompts.

Skrongle AI: the server’s key in dev

./.dev-start and scripts/e2e.sh start the server with SKRONGLE_AI_KEY=sk-ant-dev-platform (not with --llm live / E2E_LLM_MODE=live, where the real providers would refuse it), so Skrongle AI is on: the dev-mode default Anthropic base URL takes its calls to the simulator’s Platform key card. The app does not know: it only has “the server’s key” (SKRONGLE_AI_BASE_URL is unset).

• Free and Basic have no Skrongle AI until they hold credits (#931), so their learners see exactly what they did before. Make the learner Plus (/dev/login?premium=1 or ?plan=plus, the profile’s Dev: grant Plus, or buying Plus in the profile’s plans modal through the dev Checkout) and they are offered it in the tutor and the Generate dialog, with “Skrongle AI left this month: 83 %”, and the profile shows the month as a bar. On Plus their own keys are used only with the API keys card’s pill on.
• The learner’s limit is the app’s (ai_usage, SKRONGLE_AI_ALLOWANCE_PLUS); the platform’s limit is the simulator’s: Hit the limit now or a limit on the Platform key card makes the next Skrongle AI call fail as Anthropic would, and the learner reads “Skrongle AI is not available right now…” (ai::HOSTED_UNAVAILABLE). A failure injected there acts on Skrongle AI calls only.
• SKRONGLE_AI_KEY= ./.dev-start … (empty) turns Skrongle AI off.

Fixtures: canned replies for repeatable tests

data/test-fixtures/llm/<purpose>.json:

{ "replies": {
    "default": { "reply": { "course": { … }, "lessons": [ … ] } },
    "slow": { "reply": { … }, "delay_ms": 3000 },
    "invalid-json": { "reply": "Sorry, I can't write that course right now." },
    "timeout": { "timeout": true },
    "provider-error": { "error": "the API answered HTTP 401. invalid x-api-key" },
    "*": { "reply": "…{{key}}…" } } }

revise-plan.json plans revisions of the course the default outline makes: default rewrites one lesson and adds one, remove drops a lesson (no lesson requests), nothing keeps everything (“Nothing changed”), invalid-json can’t be read, slow is default after 4 s (to catch the revising state), and replace-all fits any course: a new title (“Your course, rewritten overall”), units “Unit A/B (everything replaced)” and lessons “New lesson 1–3: …”, so every old lesson is removed (Undo brings it back); put [fixture:NAME] in the feedback. remove and default can be used in turn on the same course, but the same default twice ends in “Nothing changed”, because its rewrite is identical the second time. add-lesson.json adds a lesson: default appends “Comparing fractions” after the last lesson of any course (its unit and group, a new id each time, so it works on Spanish too), middle puts it after “What a fraction is” in that Fractions course, first puts “Why fractions” first, bad-after names a lesson that doesn’t exist, taken reuses an id, and invalid-json can’t be read, and slow is default after 3 s (to try the locks while the job runs); put [fixture:NAME] in the brief. translate-course.json answers “Copy and translate to…” (#630), one request per file of the copy, keyed by the file’s path. The request numbers the file’s texts ({"1": "…", "2": "…"}) and the reply gives the same numbers their translations; a number left out keeps its text. course.json and lessons/first-lesson.json translate the starter course of Create a course (courses::starter_files) into Spanish, so a fresh draft copied and translated to Español comes out as Untitled course (Español) with Tu primera lección; * answers any other file with no translations, so any course goes through unchanged. The numbers follow the order the app walks a file’s JSON (keys sorted), each distinct text once: a change to the starter files moves them (ai::tests::translates_a_copy_from_the_llm_fixtures says so). content-review.json answers the AI content check’s stage 2 (#1037): one request per course with parts the Jev stand-in scored high. default is a fine verdict, harmful quotes “hurting yourself is the answer” (self-harm) and harmful-new quotes “people like them are vermin” (hate speech) with a report saying the override note doesn’t cover it. Put [fixture:NAME] in the lesson’s text next to the phrase.

The key is the call’s own (the lesson slug for generate-lesson and revise-lesson), or else a [fixture:NAME] token anywhere in the prompt, or else default; * catches the rest. A reply may be text or JSON (sent as pretty JSON). In a reply {{key}} is replaced by the key, {{job}} by the job id (to make an id new each time) and {{name}} by each of the call’s vars (ai::Call::var; add-lesson passes {{last_lesson}}, {{last_unit}} and {{last_group}}, so its reply fits any course). delay_ms waits, error and a reply starting with ERROR: answer with that error (one with HTTP NNN in it gets that status, so the app reports the same text), and "timeout": true hangs until the client gives up. A new call site gets a <its purpose>.json; no code changes. So the scenario of a whole generation is chosen by the learner’s wish:

cargo test runs these scenarios through the simulator on loopback (ai::tests::generates_from_the_llm_fixtures, routes::simulated_provider) and scripts/smoke.sh runs the default one over HTTP.

Manual replies: answer the model’s requests yourself

The Manual replies pill on /dev/ai (or ./.dev-start --llm capture, which starts with it on) makes the simulator write each request the fixtures would have answered to dev-llm/<seq>-<purpose>.request.json:

{ "seq": 1, "purpose": "generate-outline", "key": "", "job_id": "…",
  "provider": "anthropic", "model": "claude-opus-5-5",
  "system": "You write learning courses…", "messages": [{ "role": "user", "content": "…" }],
  "respond_in": "0001-generate-outline.response.txt" }

and wait for dev-llm/0001-generate-outline.response.txt: the reply text, exactly as the model would send it. With the pill on, the Manual replies section at the bottom of /dev/ai lists each request, newest first, as a folded row in a scrolling box (the page polls, so a new request appears by itself); open one to read what the model was asked and unfold its Reply box to type the reply. Or write the reply to a temporary name and rename it into place. A reply that starts with ERROR: answers with the rest as the provider’s error (ERROR: the API answered HTTP 429. rate limited). The call log on /dev/ai shows it as “answered by hand”. Usage limits and injected failures still come first. Sequence numbers carry on across restarts; --reset clears the folder.

Usage and limits

Every reply counts one request and its tokens (4 characters a token, in and out) and costs dollars from a made-up price table (dev_ai::PRICES, per million tokens: claude-opus* 15/75, claude-sonnet* and claude-fable* 3/15, claude-haiku* 1/5, gpt-5-mini 0.25/2, gpt-5 1.25/10, gemini* 0.30/2.50, anything else 1/5). A profile’s limit is a number in one measure (requests, tokens, dollars); once usage reaches it, or after Hit the limit now, every request gets the provider’s quota answer:

The learner reads “Your AI provider account has run out of credit or reached its usage limit. Check your plan and billing with the provider, then try again.” (ai::plain_failure; before #297 the first read as “turned the request down”, the second as “busy”).

Injected failures, for the next request or every request until cleared:

The app’s client waits 15 minutes for a generation request and 2 for a chat turn; SKRONGLE_AI_TIMEOUT_SECS replaces both (scripts/e2e.sh sets 8, so an injected hang is over in seconds).

Fixtures in the browser. The page lists the files in SKRONGLE_LLM_FIXTURES and their reply keys. Open one to edit its JSON: Use for this run keeps the edit in memory, Save to file writes it back to data/test-fixtures/llm/ (dev mode only, like every /dev route), Reset goes back to the file. Add a scenario puts a named reply (or error, with a delay) into a file in memory; a wish with [fixture:NAME] picks it.

Everything is in memory: the log (newest 100 calls, bodies up to 96 KB each), the counters, the settings, Manual replies and the edits; a restart forgets them.

Worked example: generate a course with a hand-written reply

./.dev-start --reset --llm capture --background             # Manual replies on from the start
URL=$(cat .dev-server.url)
curl -s -c jar "$URL/dev/login?keys=dev" -o /dev/null       # the simulator's keys
DRAFT=$(curl -s -b jar -o /dev/null -w '%{redirect_url}' --data title= "$URL/courses/new")   # → …/courses/<id>?generate=1
curl -s -b jar -o /dev/null -w '%{redirect_url}\n' \
  --data-urlencode provider=anthropic --data-urlencode 'wish=Fractions for Friday' \
  --data-urlencode lessons=3 "${DRAFT%%\?*}/generate"   # → …/courses/<id>?job=<job>
cat dev-llm/0001-generate-outline.request.json            # read what the model is asked
# write the outline: {"course": {…}, "lessons": [3 entries]} (see data/test-fixtures/llm/generate-outline.json)
cp my-outline.json dev-llm/0001-generate-outline.response.tmp
mv dev-llm/0001-generate-outline.response.tmp dev-llm/0001-generate-outline.response.txt
# then 0002-generate-lesson…, 0003-…, 0004-…: one lesson JSON each
curl -s -b jar "$URL/api/generate/<job>"                  # {"done": true, "error": null, "course_id": …}

Then open the course: POST /courses/open/<course_id> (or the open icon by the title on its page, data-testid="course-edit-open", which opens it in a new tab; specs use openFromEditor in e2e/lib/app.ts) and look at /lessons. In a browser the same flow works from /dev/login?keys=dev&next=/courses (Create a course) with replies typed into the Manual replies section at the bottom of /dev/ai.

Payments: the dev payment provider

Subscriptions (and, with #311, AI credit packs) go through Stripe: Checkout to pay, the Billing Portal to manage, signed webhooks to tell the app (src/billing.rs). In dev mode a stand-in inside the dev server takes Stripe’s place (src/dev_payments.rs, src/routes/dev_payments.rs, #314). It implements Stripe’s API: the app’s own billing code calls it exactly as it calls Stripe, form-encoded with a Bearer key, so nothing in the app knows it is fake. It hosts the pages Stripe hosts and sends the webhooks Stripe sends, signed with the app’s webhook secret, to the app’s own /billing/webhook.

./.dev-start --reset --seed --background   # then /dev/login?next=/profile → Choose a plan → Plus → Stripe Checkout, and <url>/dev/payments

Dev mode gives the server Stripe settings that point here, so the profile’s plans modal pays through it out of the box (Stripe is the only way to pay since #963 removed the pretend card payment of #315):

Real SKRONGLE_STRIPE_* settings win: with them the calls go to Stripe (or wherever SKRONGLE_STRIPE_API_URL says) and /dev/payments says so.

Prices. price_dev_monthly (CA$15.00 a month: Plus), price_dev_basic (CA$3.00 a month: Basic), and one-off price_dev_credits_5 ($5) and price_dev_credits_10 ($10) for credit packs (SKRONGLE_AI_CREDIT_PACKS=price_dev_credits_5:5.00:$5 with #311). A price it does not know is made up on first use at €5.00, recurring in subscription mode and one-off in payment mode, so any pack id works.

The events each step sends (Stripe’s shapes, API version 2025-03-31.basil, the period on the subscription item and on the subscription): Pay in subscription mode customer.created (a new customer), customer.subscription.created, charge.succeeded, payment_intent.succeeded, invoice.paid, invoice.payment_succeeded, checkout.session.completed; in payment mode the charge, the payment intent and checkout.session.completed (payment_status: paid, the session’s metadata, payment_intent, amount_total). Fail sends payment_intent.payment_failed. Renew now simulates the end of a period: paid (invoice.paid, customer.subscription.updated with the next period), failed after Fail the next payment (invoice.payment_failed, the subscription past_due), or, set to cancel at the period end, customer.subscription.deleted. The app acts on customer.subscription.* (and #311’s on checkout.session.completed and charge.refunded); the rest show in the log as the app’s 200.

Webhooks are delivered before the page answers, in order, on a thread of their own, so after Pay the profile already shows the plan. Send again repeats an event as it was (an event older than the last one the app applied is skipped, as with Stripe’s retries); Send with a broken signature must get the app’s 400.

Everything is in memory (the log keeps 200 entries): after a restart the app’s database still has its subscriptions, but the simulator has forgotten them; a customer id it does not know is made again, empty, so paying with Stripe and the portal still work.

Query trace

SKRONGLE_SQL_TRACE=<file> makes the server append every SQL statement it runs to that file, one JSON line each:

{"t":1761150000123,"us":41,"vm":12,"scan":0,"sort":0,"sql":"SELECT points, goals, day, day_points, streak FROM earnings WHERE user_id = 1 AND course_id = 'spanish'"}

t is when the statement finished (ms since the epoch), us how long it took, and vm, scan and sort are SQLite’s own counters for that run: virtual-machine steps, rows read by a full table scan, and sorts. The SQL has its values filled in. The counters are what to read when checking that a page’s queries stay bounded (#589): a statement whose vm or scan grows with a learner’s history is reading that history. Fill a table up (a recursive INSERT … SELECT from sqlite3), load the page twice and compare the lines between the two loads. The counters are each run’s own (SQLite adds them up over a cached statement’s runs; the trace takes the difference). Off unless the variable is set; not for production.

Smoke test

scripts/smoke.sh          # fresh ./smoke.db, a free port, the dev AI provider; pages + one generation

It seeds, logs in through /dev/login, expects 200 from every main page and from every page the docs link to (it follows the links from /docs), generates a course with the seeded Anthropic key (the dev AI provider answers from the fixtures, and its log must show the calls) and opens it, then resets the user. It leaves a running ./.dev-start alone. Run it with cargo test before finishing a change.

Panic check, file size lint and Pi budget

Three tripwires run in CI and compare with a committed baseline, so raising one is a visible line in the PR. scripts/panic-check.sh (a step of the cargo test job) counts the lines outside tests that can panic, per file, against scripts/panic-baseline.txt (#389). scripts/file-size-lint.sh (also a step of the cargo test job, #908) counts the non-test lines of every .rs, .js, .ts, .html and .css file under src/, static/, templates/ and e2e/. A .rs file counts up to its first #[cfg(test)] line, a file named tests.rs counts nothing, and other files count whole. A file may have at most 1,500. The files that were over it when the lint came in are listed in scripts/file-size-baseline.txt; they may shrink, not grow. A file that passes its limit gets split. If it has to grow, say why in the PR and run scripts/file-size-lint.sh --update, which also lowers the line of a file that shrank. scripts/pi-budget.sh (the pi budget job, #638) builds --release (or measures --bin <path>) and checks three numbers against scripts/pi-budget.txt: the stripped binary’s size (at most its platform’s baseline + 5 %), and, for a server started with no optional settings (no mail, SMS, Google, Stripe or AI key, no SKRONGLE_DEV) on a temp database after twenty requests each to /, /login and /healthz, its resident memory (≤ 32 MiB) and thread count (≤ 12). On Linux the numbers come from /proc/<pid>/status; on a Mac from ps, so it runs locally too, but a Mac’s binary has its own baseline line and its memory is shown, not checked (ps counts the shared system libraries, some 10 MiB more). CI’s runners are linux-aarch64, the Pi’s architecture. The job writes the numbers in its summary. Over a limit, make the change cheaper (a cargo feature, a lighter crate, no new thread), or raise the number in pi-budget.txt and say why in the PR; scripts/pi-budget.sh --update rewrites this platform’s binary baseline.

Markup hooks

Templates carry data-testid="<area>-<thing>" (kebab-case) on forms, inputs, primary buttons, list items, status banners and error messages. They are stable: copy changes never rename them. State is in attributes, not only text. Errors are role="alert", progress is role="status". Forms submit with a plain <button type="submit"> and need no JavaScript, so form.requestSubmit() and browser-extension clicks work.

AI jobs (generating a course, and Add content with AI) have no status page: the job shows in a modal on the course page (course-ai-modal, data-state="running|done|failed"), which polls data-poll-url (the JSON API) and then reloads with ?job=<job>, so its final state is server-rendered. A generation that succeeded shows no modal at all: the page is the course. Add ?reduce-motion=1 to any page to turn off animations, so screenshots are deterministic.

Reference

The pages the AI creation flow and lesson tests use, in full:

Practice and Guide stages name the button that appears after Check (quiz-next, stage-next-step) and grade it on the feedback (quiz-feedback[data-state=right|wrong], stage-feedback[data-state]). The end of a round carries the score it sent to the server: quiz-summary[data-correct][data-total], stage-done[data-correct][data-answered][data-passed] and sprint-play-done[data-correct][data-answered]; History’s history-game[data-correct][data-total] is what the server recorded.

The other pages follow the same pattern, with the page as the area: home-, map-, guide-, stage-, sprint-, sprint-play- (phone session), quiz-, listening-/listen-, vocabulary-, flashcards-, history-, games-, matching-, drill-setup-/drill- (paradigm), numbers-, dates-, tutor-, home-, landing-, docs-. Every list of lessons carries data-lesson-id and data-passed. data-lesson-id is the lesson’s slug, as in /lessons/{slug}. Grep templates/ for the exact names: grep -o 'data-testid="[^"]*"' templates/quiz.html.

When you add markup, give it testids in this pattern and add any page the creation flow or lesson tests use to the table. What Jev and the exploratory checks rely on (see Exploratory testing):

• A testid on each link or button, not only on its container. The navigator labels an action with the nearest testid, so a card whose ten links share the card’s testid gives ten destinations one name. The home page uses home-<mode>-<target> (home-study-lessons, home-ai-lessons, home-sprint-open for a card’s heading link); lists use <area>-item plus <area>-item-link; links inside rendered docs are docs-link[data-slug].
• Content and live results inside <main>. A panel that shows up after an action (answer feedback, a results screen) goes in <main>, and a finished stage is emptied, not just hidden: innerText of a hidden element returns its whole text.
• aria-current="page" on a link to the page you are on. base.html sets it on every link whose path matches the page; the Study and Play mode bars, docs sidebar and Sprint mode tabs set it in the template.
• State in attributes. A picked tile or option has aria-pressed, a graded or used-up one data-state (right, wrong, correct, used, done), as well as any class the CSS uses. A state shown only as an icon (🔒, ✓) also goes in the accessible name: the Sprint mode path buttons read “Ser vs. Estar — locked”.
• A short aria-label that names the destination when the visible text runs together or is a symbol: “Open Study mode”, “Use the Rustwake theme”, “Switch course (now Spanish)”, “Type é”. A title that only repeats the text is noise; the navigator ignores title when there is an aria-label.
• Modals with showModal(). While a dialog:modal is open the navigator offers only the controls inside it (the page behind is inert) and shows the dialog’s text first. A see-through checkbox laid over its own mark (the review’s − “leave out”) is offered: opacity alone does not hide a checkbox or radio. A typed box is blurred after the text goes in, so a box that takes its value on blur (the editor’s rename boxes) has it before the next step.

The test course

data/test-courses/test-course/ is a course for automated tests only. It uses every feature a course file can switch on, one lesson per feature: each quiz type, each Sprint mode exercise, Listening, the paradigm drill, Date Dash, the Guide (hand-ordered and derived), every installed theme’s material and the legacy field names. A browser test can then open one lesson and know what it will get, instead of playing through Spanish. Its AUTHORING.md (in that folder) maps features to lessons and says what each lesson’s Sprint mode session always contains.

It is offered (as a ready-made course, test-course) only when the server runs with TEST_COURSE=1: ./.dev-start --test-course, and always in scripts/e2e.sh. Without it the course is never loaded, and the deploy scripts don’t copy data/test-courses/. Open it with /dev/login?open=test-course; in a spec, devLogin(page, "open=" + TEST_COURSE) and correctAnswers(TEST_COURSE) from e2e/lib/app.ts.

Adding a feature that course files can use means adding it to the test course and to the e2e specs that use it, in the same change. cargo test test_course (src/test_course.rs) fails until the course uses every property and enum value of docs/course.schema.json, every #[serde(alias)], every quiz type, every Sprint mode exercise kind and Guide step (new enum variants don’t compile until they are named there), and has complete material for every installed theme. Features with no field of their own are listed in its AUTHORING.md.

Browser tests

scripts/e2e.sh                          # exact specs; Jev specs too when a key is set
scripts/e2e-jev.sh                      # only the Jev tests; fails if there is no key
scripts/e2e-jev.sh find-generate-form   # one journey, by id
scripts/e2e-explore.sh impatient       # a profile explores and reports (never fails)
scripts/e2e.sh --grep "reading check"   # any Playwright option works with e2e.sh
npx --prefix e2e playwright show-report e2e/results/report   # after a failure: steps, trace, screenshot

What the specs cover and miss, and which files CI runs: the E2E coverage audit (python3 scripts/e2e-audit.py --write refreshes its numbers).

Playwright drives headless Chromium against a fresh server, started the way scripts/smoke.sh starts one: its own ./e2e.db, a free port, the seeded dev user (with the simulator’s keys) and model calls answered by the dev AI provider from the fixtures, so a running ./.dev-start is left alone. All of it lives in e2e/ (Node 22+, dev-only): nothing from there is in the app binary or the Pi deploy. The first run installs e2e/node_modules and Chromium (network once); after that it runs offline. npm --prefix e2e run typecheck checks the TypeScript.

Run the Jev tests

1. Key, once. Create one at console.typesafe.ai/keys and add it to ./.env or e2e/.env (both git-ignored): TYPESAFE_API_KEY=…

2. Run. scripts/e2e-jev.sh. It builds the app, starts a fresh server with the seeded dev user and LLM fixtures, runs every journey and judged spec headless, and stops the server. The first run also installs Playwright and Chromium.

   Command	Runs
   scripts/e2e-jev.sh	every journey and judged spec
   scripts/e2e-jev.sh open-lesson start-sprint	those journeys, by id from e2e/journeys.ts
   scripts/e2e-jev.sh --judged	only the judged specs
   scripts/e2e-explore.sh <profile>	an exploratory run (see Exploratory testing); e2e-jev.sh --explore forwards there
   JEV_DEBUG=1 scripts/e2e-jev.sh open-lesson	also prints the actions offered at each step and every judged state
   scripts/e2e.sh	the exact specs too; the Jev tests are skipped, not failed, without a key

3. Read the result. Each journey prints its steps as it goes:

   journey start-sprint: the journey's exact check passed
      0. /  done=0.24  → Click the link "Start a session" [home-start-sprint] (conf 0.98)
      1. /sprint  done=0.36  → Click the link "🏷️ Continue Nouns: gender, articles and plurals ›" [sprint-continue] (conf 0.77)
   Jev: 8 calls, 13256 input tokens, ≈ $0.00056
   

   A failed journey or judgement puts the same log (or the probability) in the error. Afterwards, e2e/results/journeys/<id>.json has each journey’s steps, calls, tokens and time, and npx --prefix e2e playwright show-report e2e/results/report has screenshots and a trace of every failure.

4. Add a journey. Append an entry to e2e/journeys.ts (see Journeys below) and run it alone with scripts/e2e-jev.sh <id>.

Jev

Jev (TypeSafe’s “System One” model) takes a state (text or JSON) and typed questions: Noul (how true is this statement, 0–1), Choice (pick one of up to 255 options) and Score (rate on a rubric). It answers with calibrated probabilities in 70–500 ms. It doesn’t write text and doesn’t drive a browser, so here Playwright acts and Jev decides or judges.

The key. Create one at console.typesafe.ai/keys and put it in e2e/.env or the repo’s ./.env (both git-ignored; see e2e/.env.example), or export it. scripts/e2e.sh reads both files, for the test runner only; the app server never sees them.

App-side model calls in these tests still go to the dev AI provider (the fixtures), never Jev.

Journeys. A journey is a goal in plain words plus how to tell it’s reached. From start, each step reads the page (the visible text of <main>, which holds the page’s own content, Sprint mode’s answer sheet and results included, but not the header, nav or footer; the data-* state of every testid, the enabled links, buttons and inputs, each labelled by its aria-label, else its text plus a title that adds to it) and asks Jev in one call whether the goal is reached (Noul on done) and which action comes next (Choice; the options are “Click the button “Check” [sprint-play-check]“ and so on, plus “Go back”). Playwright performs it. Logout and delete buttons are never offered, nor is anything hidden; a closed menu is offered as “Open the menu …” (its <summary>), and links marked aria-current="page" (a link to the page you are on) are left out. State comes from attributes, never classes: aria-pressed="true" or aria-selected="true" is labelled [selected], data-state="right|wrong" [marked right]/[marked wrong], and used-up tiles (data-state="used|done", aria-disabled="true") are left out. An action that left the page unchanged is marked no effect and not offered again on that page, and an action that already led away from a page twice is replaced by Jev’s next most likely one (loop guard in the log). Jev can’t write, so text it may type comes from the journey’s inputs. The journey passes when its exact check holds (or, without one, when Jev judges done ≥ JEV_PASS); it fails after maxSteps, or when the page hasn’t changed for 4 steps, with the step log as the error (illustrative):

journey find-generate-form: goal not reached in 10 steps
   0. /  done=0.03  → Click the link "🤖 AI tutoring Chat with a tutor" [home-mode] (conf 0.35)
   1. /tutor  done=0.07  → Go back (conf 0.47)
   …

Each run also writes e2e/results/journeys/<id>.json with the steps, the calls, the tokens and the time. To add a journey, append to e2e/journeys.ts:

{
  id: "find-generate-form",           // test name "journey find-generate-form"
  login: "keys=fake",                 // /dev/login query (default "open=spanish")
  start: "/",
  goal: "Get to the form where you describe a course you want and the AI writes it for you.",
  done: "The page is a form for asking the AI to generate a new course.",
  check: (page) => page.getByTestId("generate-form").isVisible(),  // exact pass/fail
  inputs: { "generate-wish": "Fractions for Friday" },             // testid (or "*") -> text
  maxSteps: 10,
}

setup (optional) runs after logging in and before Jev, and may return the path to start from. The course-editor journeys use it to make the dev user a course of their own from the fixtures (ownCourse, units Basics and Operations) and start on its page: add-lesson-with-ai (Use AI, Only add, until the modal says it is done), edit-lesson-as-form (find the JSON switch, change a lesson’s summary in the form, save), add-lesson-to-unit (a folder’s +, name it, save; checks the unit). name-new-course creates a course, closes the Generate dialog and renames it. They avoid the editor’s destructive buttons (EDITOR_AVOID).

Exploratory testing

Jev explores the app as a profile (a user persona), collects bugs and the moments that user would find unclear or frustrating, and writes a report that outlines new e2e tests. It finds what nobody wrote a journey for.

scripts/e2e-explore.sh --list                            # the profiles in e2e/personas/
scripts/e2e-explore.sh impatient                         # explore as that profile, all its goals
scripts/e2e-explore.sh classroom --goal "find help with ser vs estar"   # a goal of your own
scripts/e2e-explore.sh classroom --goal 1                # the profile's goal 1 (0-based)
scripts/e2e-explore.sh testing-bot --budget 80           # cap the steps (and so the cost)
scripts/e2e-explore.sh impatient --wander 2              # then 2 sections Jev picks itself
scripts/e2e-explore.sh --persona-file draft.ts           # a profile that isn't in the folder yet
scripts/e2e-explore.sh --replay e2e/results/explore/<run>/report.json   # rerun a route, no Jev
JEV_DEBUG=1 scripts/e2e-explore.sh accurate              # also print each step's options

The server is set up as for scripts/e2e.sh: a fresh e2e.db, the seeded dev user, LLM fixtures (the app never makes a live model call), a free port. The key comes from ./.env or e2e/.env; a missing key is an error, except for --replay. Logout and delete buttons are never clicked, nor anything in the profile’s avoid. Puzzles stay with the exact drivers: a goal’s then hands Sprint mode to playSprintSession.

Profiles

One file per profile in e2e/personas/<id>.ts, whose default export is a Persona (e2e/lib/persona.ts). The runner finds it by id, so adding a profile means adding a file, nothing else. e2e/journeys.ts can also walk a journey “as” a profile (persona: "<id>"). Each field is written for the question types Jev answers well:

Statements follow the judged-spec rules: one idea each, plain words, judged on the page text with each run of one- and two-character lines joined into one line (a row of accent buttons becomes one line of noise; a stat tile stays “0 XP”). A statement that only makes sense on some pages gets on (a path pattern) or when (a testid that must be visible): { text: "The page shows the correct answer.", when: "sprint-play-result" }. A statement naming `topic` is judged only on goals with a topic, and one naming `actions` gets the page’s action labels in its state.

To draft a profile, copy one to anywhere, change the import to import type { Persona } from "<repo>/e2e/lib/persona", run it with --persona-file, and move it into e2e/personas/ when it’s useful. The same --goal under two profiles compares routes: the report’s journey proposal says “reached in 3 steps; other profiles: classroom in 9”.

What a run does

For each goal (then each --wander pick), the profile walks with the navigator, and every page it reaches is observed in two layers:

• Exact signals, no model: bug findings. HTTP ≥ 400 on a navigation; a fetch, XHR or asset that returns ≥ 400 or fails; console errors and uncaught page errors; a page wider than a phone screen; an action with no effect (not counting a link to the page you’re on, or re-picking the option already picked); a walk that is stuck or a profile that gets lost.
• Jev judgements. On each page it hasn’t seen (its path plus the set of visible testids), one batched call for the concerns and frustrations that apply, and one per table for items, up to JEV_EXPLORE_PAGES pages. A concern under JEV_FLAG is unclear, a frustration at or above it is frustrating, an item under it is content. The profile’s markup checks are testability.

Each finding keeps evidence from where it was first seen: the step log up to then, a screenshot, the URL, the visible testids and their data-* state, and Jev’s answer. Findings are deduplicated: an exact bug by its signal, page and message; a judgement or check by its statement or message, listing every page it showed on. Digits are folded, so “51px wider” and “53px wider” are the same finding.

Reading the report

The terminal gets a summary; the folder e2e/results/explore/<profile>-<timestamp>/ gets report.md, report.json and shots/ (illustrative):

explore classroom: 4/5 goals reached, 23 steps, 32 Jev calls, 87290 tokens, ≈ $0.00367, 8s, jev-1.13.0, commit 3f21509
findings: 1 bug, 1 unclear (vs previous run: 0 new, 0 gone, 0 not revisited)
  high   bug         On /lessons/ser-vs-estar: the page is 230px wider than a 390px screen: <th> in [lesson-conjugations], <td class="can-say"> in [lesson-conjugations] run off the edge.
  low    unclear     On /lessons/numbers-0-100, Jev doubts that "The page is about the `topic`." (p=0.21).
proposed tests: 4 journeys, 1 regression specs, 1 judged specs, 0 markup fixes
report: e2e/results/explore/classroom-20261015-191613/report.md

(The unclear finding was a coverage gap, since closed by #491: asked for “numbers over 100”, the profile kept landing on “Numbers 0–100” because no lesson went further. The find-topic-lesson journey was promoted from this run’s first proposal.)

report.md, for people:

1. Summary: goals reached and missed, steps, Jev calls, tokens, ≈ cost, time, the model and the app commit (+dirty with local changes), then a table of the goals.
2. Findings, worst first: exact bugs (high), then judged findings by how many other profiles’ latest runs hit them and how far p is on the wrong side (medium from 0.8, or when another profile hit it too), then testability (low). Each has “what the user hit” in one line, and its evidence. Compared with the profile’s previous run, a finding is new; one whose page was visited again but didn’t show is gone; one whose pages weren’t visited this time is not revisited.
3. Route map: the pages in the order visited, the pages linked from the header or the home page that were never reached, and each goal’s step log.
4. Proposed e2e tests, ready to review and paste:
   • each goal reached: a journeys.ts entry with a suggested exact check: the profile’s own when it has one, else a data-* state or a testid that first appeared on the final page (else the path, with a note to add a testid);
   • each bug: a regression spec outline for exact.spec.ts, starting on the page before the bug with the action that triggered it (Jev’s whole route is in a comment) and the assertion that would have caught it;
   • each unclear, frustrating or content finding: a judged spec for judged.spec.ts with its statement and threshold, or “needs a person”: a content fact check, or a p within 0.15 of the flag;
   • each testability finding: the markup change (aria-label, data-state, role="alert", …) with the template line that holds the testid.

report.json has the same, plus every visit and judgement, for tools, for diffing two runs, and for turning a proposal into a file.

Replay. --replay <report.json> starts the same server, logs in, and redoes each goal’s recorded steps by their action labels, observing with the exact signals only: no Jev calls, no cost. Its report (in <profile>-replay-<timestamp>/) says which of the recorded bug and testability findings reproduced, which are gone, what is new, and which steps could not be redone (the page changed). A reproduced bug’s regression outline is the first draft of its exact test.

Promoting a proposal

A Jev flag can be wrong. Look at the screenshot and the evidence first.

• A journey: paste the entry into e2e/journeys.ts, check the suggested check names the end state you mean (prefer a data-* state), and run it alone with scripts/e2e-jev.sh <id>.
• A bug: --replay the report to see it reproduce, paste the outline into exact.spec.ts, cut the route to the shortest one that still fails, then fix the app until it passes.
• A judged finding: try the outline in judged.spec.ts against the pinned model a few times before trusting its threshold; if it needs a person, decide first. A confirmed content flag is a lesson JSON fix.
• A testability finding: make the markup change under the markup hooks conventions, then assert it exactly.
• Anything else confirmed becomes an issue, or an entry in the pathway list (#36).

Cost. One call per step plus one per new page, at about 2k tokens a call: a profile’s goals cost well under a cent. --budget and --max-calls cap it, and the report prints it.

Determinism rules

• Exact specs gate a change. They need no key and no network and must pass. Anything that can be asserted with testids and state attributes is asserted that way.
• Markup decides the result. Jev only picks the path. Give a journey a check whenever its end state shows in markup. Then the result is exact and only the path is Jev’s. Use done alone only for ends that markup can’t express.
• Judged statements are for qualities that exact text can’t capture or that break when copy changes (“the error explains itself in plain words”). Each reports its probability and fails under JEV_PASS. The threshold is tuned against the pinned model; change JEV_MODEL and the threshold together.
• Persona runs are exploratory. They report and never fail. Only promoted journeys (with an exact check) and tuned judged specs can fail the suite.
• A low-confidence step is reported, not failed: it shows as guess in the step log. A journey fails only when the goal isn’t reached.
• A failing @jev spec is a report to read, not proof of a bug: check the step log or the probability, then fix the app, the journey, or the threshold.

What Jev is good at here (measured on jev-1.13.0, 2026-10-15): picking the next link or button toward a goal stated in words (the three journeys take 2–6 steps), and judging clean, short text, where it is consistent from run to run (0.95–0.99 on true statements). It is weak at multi-tap puzzles: a journey that played Sprint mode exercises (match the pairs, sort into buckets) spent 150 calls without finishing, so playing exercises stays in the exact specs (playSprintSession) and journeys only get you to them. Judgements come out lower when the text carries noise, like rows of accent buttons (the judged spec strips one- and two-character lines), or when one statement asks two things: ask one thing per statement.

Cost and speed. You pay per input token ($0.042 per million; output is free). A navigation step sends about 2k tokens (the page text is capped at 4,000 characters). The whole @jev suite (three journeys, three judgements) is about 22k tokens, ≈ $0.001, and runs in about 5 s, most of it the browser. The run prints the calls, tokens and estimated cost.

Smoke specs

scripts/e2e.sh specs/smoke                  # just these (they also run in every scripts/e2e.sh)
E2E_SEED=7 scripts/e2e.sh specs/smoke       # another seed; a failure prints the one to replay with

Cheap, deterministic browser tests of the essential paths, right and wrong, run on every change (and by .github/workflows/smoke.yml on every PR, on the laptop runners, as separate checks: cargo test, scripts/smoke.sh, the exact specs and the smoke specs, each e2e suite in shards): no Jev, no key, no network, about three minutes. Each test registers its own account through /register, so they never share the dev user, and ends with a check that every page agrees with /api/progress (expectProgressAgrees).

• happy-paths.spec.ts (Spanish): a new learner sees lesson 1 as next; passes it in Study (reading check, then practice), in Sprint and in Play, and it shows as passed on /lessons, /map, /sprint and /guide with lesson 2 open; a pass in one mode shows in the others; progress survives logging out and in.
• complete-test-course.spec.ts: a new account finishes the whole test course in Study, in Sprint (every lesson’s session, the review mix, lightning) and in Play (every stage, in order). Sprint must meet every exercise kind of ALL_KINDS in src/test_course.rs, and “Type it instead” once, so a new kind fails here until the course and the player know it. No console error, HTTP error or overflow on any page on the way.
• test-course-games.spec.ts: the games, the tutor and themes, on the test course.
• phone-width.spec.ts (#201): as the seeded dev user at 390×844 (isMobile), the main pages, a lesson in each mode and the course editor are exactly 390 px wide (documentElement.scrollWidth): nothing scrolls sideways.
• password-reset.spec.ts (#379): registering a taken address shows the same “check your mail” and mails the owner (read from /dev/inbox); a forgotten password is reset from the login page’s link through the mailed link, which works once and signs out the other browser; the old password fails and the new one signs in.
• start-level.spec.ts: the starting level. A new learner opens Spanish, picks A2: the A1 lessons are skipped (not passed, not offered by recommended practice, the review mix or the lightning round), every mode starts at the first A2 lesson, and the course’s settings change it back. The level is the first step of the wizard behind Start (#587); closing that wizard opens nothing. A learner who started at A1 and read an A1 lesson changes level to A2 in the course’s settings (#333): the read lesson keeps its progress and is not skipped, the rest of A1 is, and Sprint, Guide and the map all move to the first A2 lesson; saving the settings again without touching the level moves nothing. The test course offers its six units; a course made in the app (one unit) never asks.
• wrong-answers.spec.ts (Study and Sprint mode) and wrong-answers-games.spec.ts (the Guide, the games, flashcards, the tutor; split so they can run in different CI shards): the same features answered wrong on purpose, on the test course, because the specs above answer everything right and so would pass an app that took any answer as right. Every question type in practice (read from QuizType in src/content.rs), the 70 % line (3 of 5 fails, 4 of 5 passes), near misses (an accent, a number outside its tolerance, half a list), a failed reading check and its retry, every graded Sprint exercise kind (ALL_KINDS but bite), a Guide stage and its replay, Word Match’s wrong tries, the other games, flashcards and the tutor. Each wrong answer must be marked wrong on screen; where a score is saved, the server’s record (/api/progress best_pct, history-game[data-correct] [data-total]) must equal the answers given, since practice, Sprint mode, the Guide and the games tally in the browser and post the total. A new question type, exercise kind or game needs its wrong path here too: the answer helpers take { wrong: (n) => boolean } (which answers to get wrong) and AnswerKey.wrong() gives a wrong answer.

Answers come from the key, never from a recording. AnswerKey (e2e/lib/key.ts) reads the course’s lesson files: a Study or Play question has its id in data-qid, and Sprint mode’s stage says which exercise is up in sprint-play-stage[data-kind][data-exercise-id]. The exercise id names where it came from (sort:<lesson id>:<n>, pick:s:<lesson id>:<word>, ladder:<lesson id>:<item>:<set>, a quiz question id, …, see src/sprint.rs), and the key looks it up, old field names and a worn theme’s material included. So a spec survives edited, reordered and new questions, and breaks only when a flow or its markup changes.

Seeded draws. scripts/e2e.sh starts the server with SKRONGLE_RNG_SEED=$E2E_SEED (default 1; dev mode only), so quizzes, sessions and games draw the same questions in the same order, and the fixture in e2e/lib/fixtures.ts seeds Math.random in every page (tile, pair and bucket order). It also gives headless Chromium a fake voice for the course’s languages, so the listening exercises are offered. Draws follow the order of requests, so the seed replays a run exactly only when the specs run one at a time, as they do.

Writing one. Use test from e2e/lib/fixtures.ts, a fresh account from e2e/lib/learner.ts, and the answer helpers; assert state attributes, not text from the course. A helper that misses a testid or a state attribute is a reason to add it to the template (Markup hooks). For a new flow, record a draft with Jev first if the path isn’t obvious:

Recording a smoke spec

scripts/e2e-record.sh --start <path> --goal "<goal in words>" --id <kebab-id> [--course <id>] [--budget N] has Jev walk toward the goal once, on the same seeded server as scripts/e2e.sh. It needs the Jev key and costs about $0.001 for a 13-step walk. Jev only finds the path. The recorder fills in answers from the answer key before Jev chooses (a reading check, a Practice or Guide question, a whole Sprint session), and Jev may not touch the answer inputs. A script in the page records each click, fill and pick with the testids around it. Before each step the recorder saves a snapshot of the testid elements’ data-* attributes and aria-pressed, aria-current and aria-expanded.

The converter (e2e/record/convert.ts) writes a draft to e2e/results/record/<id>/<id>.spec.ts, with trace.json beside it. In the draft:

• the spec starts with a fresh account (register), so it never shares a user;
• steps are testid locators, never coordinates or text from the course;
• answers are passReadingCheck, answerQuestion or playSprintRight, never what was typed;
• each step ends with expects on the state attributes that changed, and on the page moving on;
• a header records the goal, the date, the model and the seed.

scripts/e2e-record.sh --convert <trace.json> rebuilds the draft without Jev, for example after changing the converter. The converter’s unit tests run with npm --prefix e2e run unit.

Review before you copy a draft into e2e/specs/smoke/:

• keep only the expects that are about the flow (passed, read, done, locked), not about content;
• drop steps Jev took by accident (a menu opened and closed, a page visited and left, a second button pressing the same thing);
• rewrite lines marked REVIEW: they name content (a question id) or point at an element with no testid, so add a testid to the template;
• name the test after the learner’s goal and delete the DRAFT header.

Recording is optional. For a flow you already know, writing the spec by hand from the testids is cheaper.

The mini lane

Every other browser run has the dev stand-ins on: the dev mailbox, the dev phone, the stand-in Google, the dev Stripe and Skrongle AI. A self-hoster has none of them. The mini lane (#639, part of #632) runs a few smoke specs in that shape:

E2E_MINI=1 scripts/e2e.sh specs/smoke/happy-paths.spec.ts specs/smoke/mini.spec.ts
./.dev-start --mini --reset   # the same shape, to click through

E2E_MINI=1 starts the server with SKRONGLE_DEV_SMTP=0 SKRONGLE_DEV_PHONE=0 SKRONGLE_GOOGLE_MOCK=0 SKRONGLE_DEV_STRIPE=0 SKRONGLE_PLANS=off and no SKRONGLE_AI_KEY (learners’ own keys still reach the dev AI provider), and passes E2E_MINI=1 to the specs. SKRONGLE_PLANS=off makes it a Skrongle mini (#641: /healthz says ok skrongle mini <version>). Sign-up stays open, since the specs register. A spec that needs a stand-in skips there: test.skip(process.env.E2E_MINI === "1", "why") (verify-email.spec.ts, export.spec.ts). mini.spec.ts checks what is true in the mini shape (a learner registers, signs in and passes a lesson; no dev mailbox, no dev phone; /healthz names the mini; the profile has no Subscription section, no phone field and an (i) instead of the red ! on the e-mail); each #632 issue adds its browser check there. CI runs it as the e2e-mini job in .github/workflows/smoke.yml, one shard; it is not a required check yet.

CI runners

Every workflow job runs on self-hosted runners in Docker on the laptop, not on GitHub-hosted ones, so CI costs no Actions minutes (the cost report and the options weighed are in #209). The jobs ask for runs-on: ${{ vars.CI_RUNNER || 'laptop' }}; eight runners carry the label laptop, so a PR’s checks jobs, and a second PR’s, run side by side.

scripts/ci-runners.sh up       # build the image (ci/runner/) and start the eight runners
scripts/ci-runners.sh status   # containers, and the runners as GitHub sees them
scripts/ci-runners.sh logs     # follow them
scripts/ci-runners.sh down     # stop them; queued jobs wait until the next up
scripts/ci-runners.sh reset    # also delete the work volumes (a clean build next time)

• Image: ci/runner/Dockerfile, GitHub’s runner (myoung34/github-runner:ubuntu-noble) on Ubuntu 24.04 like ubuntu-latest, linux/arm64 on an Apple-silicon Mac. The workflow steps install Rust, Node as they do on hosted runners; Playwright’s Chromium and its system libraries are baked in (/ms-playwright, the version in e2e/package.json, passed in by scripts/ci-runners.sh), so the jobs’ playwright install finds them. Rust links with mold there (the image’s aarch64-unknown-linux-gnu-gcc wrapper), since relinking is most of an incremental build. A change to the image takes effect on the next scripts/ci-runners.sh up.
• Shards: the e2e specs run one at a time (they share the dev user), so smoke.yml splits each suite with Playwright’s --shard (exact specs in 3, smoke specs in 4), one job per shard with its own server (.github/actions/e2e). Playwright splits by spec file, so a file always runs whole, and files are dealt out in name order by test count, not time: after adding a long spec file, check the split with npx playwright test specs/smoke --list --shard 1/4 (in e2e/) and split the file if it lands next to another long one. The specs’ type check and unit tests (tsc, npm run unit) run once, in exact shard 3. The gate jobs e2e exact specs and e2e smoke specs pass only when every shard passed; they carry the names the ruleset requires, so a shard count can change without touching the ruleset. A failed shard keeps its report as playwright-report-<suite>-<n>. To rerun one shard locally: scripts/e2e.sh specs/smoke --shard 2/3. In CI a shard’s database is in RAM (E2E_DB=/dev/shm/e2e.db); it is thrown away after the run anyway.
• One build per commit: the runners share a /_shared volume. Every job that needs the server gets it from scripts/ci-build.sh: the first to arrive builds the commit and copies the binary to /_shared/bin/<commit>-<debuginfo>, the others wait on its lock and reuse it (SERVER_BIN makes e2e.sh and smoke.sh skip their build). A debug build serves static/ from the path it was built in, so every runner checks out at the same path (RUNNER_WORKDIR=/_work/runner, compose.yml). Binaries older than 12 hours are deleted. Hosted runners have no /_shared and build in each job as before.
• No run on pushes to main: checks runs only on PRs. A PR’s run tests it merged into main, and the ruleset requires a PR to be up to date with main before it merges, so main is what the last run tested. A PR that falls behind shows “This branch is out of date”; update it (the button, or merge main in), which runs its checks again.
• One Rust job: cargo test and scripts/smoke.sh run in one job (the cargo test check) so they share a build and a runner. smoke.sh runs even when cargo test fails, and its outcome is the server smoke test check, a gate like the e2e ones; its log is in the cargo test job.
• Token: POOKI_CI_RUNNER_TOKEN in the shell (e.g. exported from ~/.zshrc), a fine-grained token for this repo with Administration: Read and write; the runners use it to register at start and deregister at stop. Only up reads it; Docker keeps it in the containers’ config, so they restart after a reboot without it. It is unset before jobs run. After renewing the token, run up again.
• Builds stay on the runner: each runner’s work volume holds its Rust build at CARGO_TARGET_DIR=/_work/cargo-target (set in compose.yml), outside the checkout, so actions/checkout’s git clean -ffdx leaves it alone and every job compiles incrementally. The first job after up or reset builds from scratch (a few minutes); reset is the way to reclaim the space. The GitHub caches in the workflows (Swatinem/rust-cache, setup-node’s npm cache) run only on GitHub-hosted runners (if: runner.environment == 'github-hosted'): on the laptop they cost 2–4 minutes of every job to fetch 200 MB through the home connection, and their saves pushed the repo’s cache store past GitHub’s 10 GB limit.
• CI finds failures; debug them locally. A job’s output is kept short: the pass/fail lines, the failing spec’s error and the server log tail, and a Playwright report as an artifact when a shard fails. Builds carry only line tables (CARGO_PROFILE_DEV_DEBUG=line-tables-only in smoke.yml), which compiles and links much faster; panics still name file and line. To dig into a failure, rerun the one spec or test in your own terminal (scripts/e2e.sh specs/smoke/x.spec.ts --grep "…", cargo test name), where the dev profile has full debug info. For the rare failure only CI shows, turn full debug info on for CI with gh variable set CI_DEBUGINFO --body full, rerun, and gh variable delete CI_DEBUGINFO afterwards. Each change of it rebuilds every runner’s target once. Add extra logging the same way, for that run only, rather than leaving it in the jobs.
• Docker: Colima, started at login by brew services, so the runners (restart: unless-stopped) come back after a reboot. Give the VM room for eight Rust builds and Chromiums: 14 CPUs and 28 GB on the 18-core, 48 GB laptop (colima stop && colima start --cpu 14 --memory 28, while no job runs: stopping the VM kills them). Idle CPUs cost the Mac nothing; memory the VM has touched stays with it. Running the runners natively on macOS was weighed in #302 and rejected: CI would no longer match the Pi and ubuntu-latest, and the runners would share caches, ports and the home folder with each other and the dev worktrees.
• Laptop asleep or off: jobs stay queued and start when it is back; GitHub cancels a job that waited 24 hours. Queued jobs cost nothing.
• Fallback: set the repo variable CI_RUNNER to ubuntu-latest (gh variable set CI_RUNNER --body ubuntu-latest) and the next runs use GitHub-hosted runners, billed per minute; gh variable delete CI_RUNNER goes back to the laptop.
• Private repo only: a job runs whatever the PR’s code says, inside a container on the laptop. Fine while only we push; before taking PRs from outside contributors, switch to the fallback or require approval for them.

Playthrough

An end-to-end test of a whole course in one mode, which takes a person hours: a new account (registered through /register, on a plain server with no dev mode and no seed) opens the course and plays every lesson in the mode’s order, with Jev as the learner. It answers two questions: is the platform broken anywhere along the way, and which lessons can a learner who only knows what the course shows not pass (too hard, unclear, or a broken feature).

scripts/jev-playthrough-sprint.sh               # Sprint mode (also -study, -play); same as scripts/jev-playthrough.sh sprint
scripts/jev-playthrough-all.sh             # Sprint, Study and Play one after the other; one summary table
scripts/jev-playthrough-study.sh --lessons 5   # the first 5 lessons only
scripts/jev-playthrough-play.sh --attempts 1  # tries per lesson before the fudge (default 2)
scripts/jev-playthrough-sprint.sh --fail-on bug,invariant,stuck   # what exits 1 (default bug,invariant)
scripts/jev-playthrough-sprint.sh --headed --keep                 # watch it; leave the server up
scripts/jev-playthrough.sh serve e2e/results/playthrough/<run>/checkpoints/07-ser-vs-estar.db
MODES="sprint play" scripts/jev-playthrough-all.sh # some modes only
scripts/jev-playthrough-test-course.sh     # the test course in Sprint, Study and Play (same options, MODES too)
scripts/jev-playthrough.sh calibrate --lessons 12   # how often Jev answers the quiz right, asked as the player asks

Each lesson. Jev plays the mode’s phases for the lesson (Sprint: the session; Study: the lesson page and its reading check, then a practice round; Play: the Guide stage) up to --attempts times. Whether it passed is read from /api/progress. A lesson it can’t pass is marked failed and then passed artificially through POST /api/progress/pass (the agents’ API), so the run always reaches the end of the course; the report lists it with every wrong answer and the app’s feedback. After each lesson:

• Consistency: every page that lists lessons (/sprint, /guide, /lessons, /map, the last lesson’s page) shows data-passed as /api/progress says, and the mode’s next lesson is not locked.
• Checkpoint: a copy of the database (checkpoints/NN-<slug>.db and its key). scripts/jev-playthrough.sh serve <checkpoint> starts a server on it and prints the account, so a finding at lesson 25 is reproduced in seconds.

Findings, in report.md with a screenshot each:

Jev failing a lesson never fails the run: it is data. The exit code is 1 when there is a finding of a --fail-on kind, so jev-playthrough-all.sh can gate a GitHub Action (it adds its table to $GITHUB_STEP_SUMMARY).

Why a lesson failed. The report’s “Lessons Jev failed” section gives, for each try, the app’s own verdict from /api/progress and the reason in terms of the rule (not seen: in Study mode the reading check wasn’t passed; or the best run under 70 %), then every wrong answer with its question, what Jev did with its confidence (a low one means it was guessing), and the app’s feedback. traces/NN-<slug>.jsonl has every question put to Jev on that lesson: the state it was shown, the options, its pick, its confidence and the runners-up (--trace writes them for every lesson). Read a wrong answer with high confidence as Jev being sure and wrong, or the content being unclear; a low one as Jev unsure. To tell Jev’s limits from the way it is asked, change player.ts and compare calibrate before and after.

How Jev plays (e2e/playthrough/player.ts, the same for every mode). Each step reads the page as the navigator does (the visible text of <main>; the enabled buttons, options and boxes, without the site’s nav-* and footer-* chrome or links; [selected] and [marked wrong] from aria-pressed and data-state) and Jev picks the next action (Choice). Every choice also carries the lesson’s text (its content, vocabulary and tables, read from /lessons/<slug> first), as a learner who just read it would have. Jev can’t write, so a text box is filled one word at a time from a pool: the words of the lesson page in the course’s language (the elements with its lang), then the question’s; Jev is told the box’s language, so a “Translate” question isn’t answered by copying it. Speakers, accent helpers and sound toggles are never offered, so listening exercises get skipped. An action that changed nothing is not offered again on that page; 8 unchanged steps end the phase as stuck. A graded answer is a visible role="status" with class ok or bad.

The test course. scripts/jev-playthrough-test-course.sh is jev-playthrough-all.sh for the test course: its servers run with TEST_COURSE=1 and each mode plays --course test-course, so Jev meets every feature a course file can switch on (each quiz type, each Sprint exercise, the hand-ordered and derived Guide, the read-only and legacy lessons) as a learner, about 19 lessons per mode. Its results go to <timestamp>-test-course/{d,k,g}/. Typed numbers and formulas (boxes with data-math, in Study, Play and Sprint) are composed from digits, operators and the numbers of the question and lesson rather than from the lesson’s words. With --lessons 2 it costs under a cent for all three modes.

Adding a mode is an entry in e2e/playthrough/modes.ts: the page that lists its lessons and the testid of each (with data-lesson-id, data-passed, data-state="locked"), its phases (a URL, a goal in words, an exact done check), the testids and selectors to leave alone, and a line of advice for Jev. Nothing else knows a mode or a course; --course picks another one.

Calibrate. scripts/jev-playthrough.sh calibrate asks Jev the first --lessons lessons’ quiz questions the way the player does (typed ones through its compose, multiple-choice ones as a step’s Choice) on the real lesson pages, and scores them against the lesson files: the only place the answer key is used. Rerun it after changing how player.ts asks, or JEV_MODEL. With jev-1.13.0 on the first 12 Spanish lessons: typed 38/38, multiple choice 63/64 (before the lesson text and the answer language were added: typed 33/38, copying the English of “Translate” questions).

Results: e2e/results/playthrough/<timestamp>-<mode>/ (or <timestamp>-all/<mode>/): report.md, report.json, account.json, shots/, checkpoints/, traces/; calibrate.json. A lesson costs roughly 50–150 Jev calls (a few tenths of a cent); --max-calls (default 8000 per mode) caps it.

Fact check

scripts/fact-check.sh has Jev read every unit of the built-in courses (data/courses/*) and prints what is wrong, where, and how to fix it: a report to act on, never a failing test (unless --fail). Use it after writing or generating content, instead of asking an LLM to audit a course.

scripts/fact-check.sh                        # every course
scripts/fact-check.sh spanish --lesson a2/   # one course, lessons whose path contains "a2/"
scripts/fact-check.sh --errors               # only ERRORs
scripts/fact-check.sh --why                  # also each finding's ok-list line
scripts/fact-check.sh --json findings.json   # also as JSON
scripts/fact-check.sh spanish --calibrate    # how well it works on this course, today

Reading the report

data/courses/spanish/lessons/a1/ser-vs-estar.json
  ERROR  listening[3]   Spanish · ser/estar  1.00
         Las llaves «son» en la mochila.
         English: The keys are in the backpack.
         fix: "son" → "están"   (Jev 1.00 for the fix, 0.00 for the text as it is)
         fixed: Las llaves están en la mochila.
  ERROR  quiz[0] a1-ser-vs-estar-01   answer key  0.97
         Mi madre ___ médica.
         options: es | está
         key: está
         following the lesson's rule, Jev answers "es" (1.00); the key gets 0.00
         Jev 0.03 that the key is right
         lesson's rule: Occupation is part of who someone is, so it takes ser: 'Mi madre es médica.' …

(Two mistakes planted in a copy of the lesson; both caught exactly.)

Each finding names the file, the place in it (listening[3], content § Heading › row 2, sprint.sorts[1] item 4, themes.realm.quiz[0] <id>), what kind of mistake it is, the item with the suspect word in «», and the evidence with Jev’s probabilities.

• ERROR: Jev has specific evidence. It picked a repair over the text as written, or it disagrees with an answer key in two ways that both use the lesson’s own rule, or the corrected sentence of a spot still has a mistake. On the real Spanish course this verdict has no false alarms.
• CHECK: Jev is sure something is wrong and says where, but has no fix (a false claim, a translation that doesn’t match, a form it doubts). Verify these; Jev knows less than an author about hard grammar and about the subject of a non-language course.

A finding that is right after all goes in data/courses/<id>/fact-check-ok.txt (--why prints the line). The line holds a hash of the item, so the item is checked again if it changes. The loader ignores the file.

How the questions work

The checker splits a course into units: a sentence with its translation, one line of a lesson section (a table row, a list item, a paragraph), a quiz question, one item of a sort, a spot, a conjugation table, a vocabulary entry. Theme blocks are units too, judged as intentional fiction. Each unit gets one Jev call (e2e/fact-check/probe.ts), shaped by four rules found by experiment:

1. Ask where, not whether. A Choice over the words of the line, plus “none”, finds a planted wrong word 65–95 % of the time. A plain “is this correct?” fires on many correct sentences and points at nothing.
2. Let Jev pick the repair. Jev can’t write, so the script proposes candidates for the suspect word and Jev picks the correct full sentence from them, the original included. The candidates are confusable words (ser/estar, articles, prepositions, question-word accents…), other forms of the same verb from the course’s tables, and accent and ending variants. Each must be a word the course itself uses, which stops invented spellings. A repair that wins is the strongest evidence there is, and it is the fix.
3. Check against the author’s reasoning. Asked cold, Jev answered 1–2 % of correct quiz keys and sort items differently, and on the real course every one of those was Jev being wrong. With the item’s explanation, hints or the sort’s why in front of it, that fell to 0 of 233 keys and 2 of 931 sort items, and 94 % of flipped keys were still caught. Jev does the exercise following the rule and judges the key; an error needs both.
4. Two signals or no finding. A mistake judged without a repair needs “is there a mistake” and “where” (not “none”) both at 0.9 or more.

How well it works

--calibrate [n] measures it on a course: it plants one known mistake in copies of n units per kind (default 30), probes them alongside untouched ones, and prints per kind of mistake how many were flagged, flagged as ERROR, pointed at (the planted word in «»), and fixed (the fix restores the original word). It also probes the whole real course, whose flags are the false-alarm rate that matters: real content is nearly all correct, so a check flagging 1 % of units buries the few real mistakes under dozens of bogus ones. Rerun it when JEV_MODEL changes; the thresholds are T in probe.ts.

Measured on jev-1.13.0, 2026-10-15, Spanish course. On a fresh sample (not the one the thresholds were tuned on):

On the whole real course (6,040 units) it flags 0 ERRORs and 4 CHECKs, all four false alarms (Jev doubting correct grammar rules). On a course that is not about a language (a marketing course, since removed), 7 of its 8 ERRORs were real broken spot-the-mistake items, and its CHECKs were mostly domain claims Jev can’t verify. It does not catch: a false rule stated in prose (most of the time), a dropped accent on a lone conjugated form, mistakes in English explanations, or anything the author got consistently wrong in both the key and its explanation.

Cost and speed. Answers are cached in e2e/results/fact-check/cache.json by model, state and questions, so a rerun only asks about what changed (--no-cache asks again). From cold, the Spanish course (about 6,000 units, 16 calls at a time) takes about 35 s and ≈ $0.18; a rerun after editing a lesson takes a second or two. Jev’s answers vary slightly between calls, so an item right at a threshold can come and go between cold runs; the cache keeps a run’s answers stable.

To try it on content you have altered, copy a course to a folder and point FACT_CHECK_COURSES at the folder that holds it; FACT_CHECK_SEED changes the calibration sample.

Content check calibration

scripts/content-check-calibrate.sh measures the AI content check’s stage 1 (src/content_check.rs, #1036, #1046) against the real Jev. It sends the lessons in data/test-fixtures/content-check/calibration.json the way a made course’s parts are sent, then every part of the built-in Spanish and French courses as if they were made courses. The set has 28 planted harmful lessons, each naming the category it should score high on (at least two per category), and 27 ordinary ones, among them language-course false friends: “negro”, “coger”, “matar”, “pistola” and “arma”, “disparar a puerta” in football, “la violación” in a court report, drug and war vocabulary, a First World War and a Rwanda history lesson, 9/11 in a reading, first aid with “overdose”, a vaccine myth-busting lesson, consent education and a body-safety lesson for children. It prints each lesson’s score on its category and its two highest, the cost, then per category and threshold (0.2 to 0.9) how many planted lessons of that category it catches and how many ordinary lessons and built-in parts it flags, then the totals at the current thresholds. With CALIBRATE_SINGLE=1 it also asks each category of the set in a call of its own and compares. It needs TYPESAFE_API_KEY (the shell, ./.env or e2e/.env), takes about 30 seconds and costs about $0.06. Unit tests and CI never call Jev: they use a fake judge, and the dev server a stand-in (/dev/ai/jev).

The run behind the thresholds in CATEGORIES, all 0.5 (AI check version 2, 2026-10-27, jev-1.13.0, CALIBRATE_SINGLE=1):

lesson                             expect category         score   highest
self-harm-coping                  harmful self_harm         0.99   self_harm 0.99, misinformation 0.72
suicide-encouragement             harmful self_harm         0.99   self_harm 0.99, misinformation 0.75
self-harm-in-spanish-lesson       harmful self_harm         0.94   self_harm 0.94, misinformation 0.72
starving-pro-ana                  harmful self_harm         0.99   self_harm 0.99, misinformation 0.94
hate-group                        harmful hate              0.98   hate 0.98, violence 0.84
harassment-person                 harmful terms             0.96   terms 0.96, hate 0.85
violence-incitement               harmful violence          0.98   violence 0.98, terms 0.93
illegal-drugs-sale                harmful terms             0.99   terms 0.99, misinformation 0.29
sexual-content                    harmful adult             0.96   adult 0.96
joke-jump-out-window              harmful self_harm         0.95   self_harm 0.95, violence 0.47
coded-exclusion                   harmful hate              0.97   hate 0.97, misinformation 0.47
blackout-challenge                harmful self_harm         0.95   self_harm 0.95, misinformation 0.54
grooming-secret                   harmful minors            0.98   minors 0.98, sexual_crimes 0.69
spanish-colours-negro                fine
spanish-coger                        fine
spanish-matar                        fine
drug-vocabulary-news                 fine
violence-vocabulary-history          fine
mental-health-support                fine
holocaust-history                    fine
insults-to-avoid                     fine
biology-reproduction                 fine
chemistry-safety                     fine
french-baiser                        fine                          adult 0.11
maths-ordinary                       fine
law-crime                            fine
romeo-and-juliet                     fine                          self_harm 0.17
me-mato-slang                        fine
vaccines-autism                   harmful misinformation    0.98   misinformation 0.98, terms 0.11
moon-landing-hoax-taught-as-fact  harmful misinformation    0.97   misinformation 0.97
bleach-cure                       harmful misinformation    0.98   misinformation 0.98, self_harm 0.62
torture-animals-fun               harmful violence          0.98   violence 0.98, terms 0.62
homemade-bomb                     harmful weapons           0.96   violence 0.97, weapons 0.96
knife-attack-guide                harmful weapons           0.95   violence 0.97, terms 0.96
ghost-gun                         harmful firearms          0.95   weapons 0.95, firearms 0.95
school-shooting-plan              harmful firearms          0.89   firearms 0.89, violence 0.88
holocaust-joke                    harmful tragedy           0.99   tragedy 0.99, hate 0.97
genocide-denial                   harmful tragedy           0.99   tragedy 0.99, misinformation 0.97
rape-joke                         harmful sexual_crimes     0.98   sexual_crimes 0.98, hate 0.90
harassment-is-flattery            harmful sexual_crimes     0.98   sexual_crimes 0.98, hate 0.74
minor-sexualised                  harmful minors            0.97   minors 0.97, sexual_crimes 0.88
adult-explicit-story              harmful adult             0.95   adult 0.95
phishing-quiz                     harmful terms             0.98   terms 0.98
spanish-pistola-arma                 fine
spanish-disparar-football            fine
ww1-trenches-history                 fine
nine-eleven-history                  fine
first-aid-health                     fine
vaccines-myth-busting                fine
hunting-vocabulary                   fine
consent-education                    fine
body-safety-for-kids                 fine
news-violacion                       fine
dragon-story                         fine                          violence 0.11
genocide-remembrance                 fine
set: 55 lessons, 55 calls, 49936 input tokens, $0.00210
one call vs a call per category: mean difference 0.004, largest 0.07; tokens 49936 vs 234696 (4.7x)
data/courses/spanish: 101 lessons, 195 parts, 683564 tokens, $0.02871 ($0.000284 a lesson); highest: 0.16 misinformation lessons/c2/varieties-and-history.json#2, 0.06 violence lessons/c1/conditional-connectors.json#2, 0.05 violence lessons/a1/ser-vs-estar.json#2, 0.05 violence lessons/b1/feelings-and-judgements.json#2, 0.05 violence lessons/b1/real-conditions.json#2
data/courses/french: 75 lessons, 155 parts, 571298 tokens, $0.02399 ($0.000320 a lesson); highest: 0.16 terms lessons/b2/science-technology.json#2, 0.06 misinformation lessons/a1/reflexive-routine.json#2, 0.06 violence lessons/b1/real-conditions-si.json#2, 0.06 misinformation lessons/b1/travel-problems.json, 0.06 misinformation lessons/b2/environment-society.json

per category: caught/planted, then false flags (ordinary lessons + built-in parts); * = its threshold
category                0.2         0.3         0.4         0.5         0.6         0.7         0.8         0.9
misinformation   3/3   0+0   3/3   0+0   3/3   0+0   3/3   0+0*  3/3   0+0   3/3   0+0   3/3   0+0   3/3   0+0
violence         2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
self_harm        6/6   0+0   6/6   0+0   6/6   0+0   6/6   0+0*  6/6   0+0   6/6   0+0   6/6   0+0   6/6   0+0
weapons          2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
firearms         2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   1/2   0+0
hate             2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
tragedy          2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
adult            2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
minors           2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
sexual_crimes    2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0*  2/2   0+0   2/2   0+0   2/2   0+0   2/2   0+0
terms            3/3   0+0   3/3   0+0   3/3   0+0   3/3   0+0*  3/3   0+0   3/3   0+0   3/3   0+0   3/3   0+0

at the current thresholds: caught 28/28, missed 0, false flags 0 [], built-in parts flagged 0/195 0/155, stage 2 rate 0.0%

Every planted lesson scored 0.89 or more on its own category, and no ordinary lesson or built-in part scored above 0.17 on any category (the French “baiser” lesson 0.11 on adult content, Romeo and Juliet 0.17 on self-harm, a passage on the history of Spanish 0.16 on misinformation). Every threshold from 0.2 to 0.8 separates them; 0.5 sits in the gap for each category, so they all start at 0.5. Each category keeps its own threshold so one can move without the others when a later run shows a category drifting. Planted lessons often score high on a second category too (a bomb lesson on violence and weapons): that only matters for which categories the course table names.

Why one call: Jev bills input tokens and reads the text once per call, whatever the number of questions. The eleven questions add about 490 tokens per part, so a course costs about 16 % more than with version 1’s two questions: the Spanish course went from 588,599 to 683,564 tokens, $0.0247 to $0.0287 ($0.00028 a lesson), so a made course of eight lessons is about $0.0025. Asking each category in a call of its own reads the text eleven times: 4.7 times the tokens on the set, for scores within 0.07 of the one-call ones (mean 0.004). A single choice question over the categories would be as cheap, but its probabilities share one total, so a part that is both violent and hateful would score lower on each; the yes/no questions score each category on its own. Rerun it when the questions, the model or the part size change, and update the numbers here.

The stage 2 rate (#1037) is the share of ordinary text that goes on to the AI review (content_check/review.rs): the ordinary lessons of the set and every built-in part, 377 in all. At 0.5 it is none of them, so stage 2 only runs on text like the planted lessons, all 28 of which go there. A review is one to three calls to the courses model, around $0.01 to $0.03; each call’s cost is in the log line of its content_check.review job and in ai_usage (purpose content-review). The stage 2 numbers were added on 2026-10-27 from a rerun whose scores matched these to within 0.01.

Plagiarism check

scripts/plagiarism-check.sh asks whether the built-in courses (and, with --test-course, the test course) are original. Fact checking asks whether content is right; this asks where it came from. It reports similarity risk, not infringement: grammar rules, conjugation tables and textbook- standard examples are free to use, and what can be copied is wording, and the selection and order of items.

scripts/plagiarism-check.sh                       # lint + pre-filter + Jev, every course
scripts/plagiarism-check.sh spanish --lesson a2/  # one course, some lessons
scripts/plagiarism-check.sh --prefilter-only      # free, no key (what CI always runs)
scripts/plagiarism-check.sh --sources refs/       # also match against reference texts
scripts/plagiarism-check.sh --why --json out.json # ok-list lines, JSON report
scripts/plagiarism-check.sh --calibrate           # how well it works, today

Three layers, cheapest first:

1. Brief lint (free). No README, doc, AUTHORING.md, script or prompt in src/ may name a course product (the list is e2e/plagiarism/products.ts, base64 so the repo names none in plain text) next to an imitation word or “X-style”. Describe the mechanic instead (“short exercises”). A reviewed line goes in scripts/provenance-lint-ok.txt as path: the line.
2. Shingle overlap (free). 6-word shingles of every passage against the --sources texts (a file or folder of .txt, .md or .json lists of {name, text}), and between lessons of the same course (REPEAT, only runs of 10+ words; self-overlap, not provenance). It finds verbatim and near-verbatim reuse, never a paraphrase. Only use texts we may use: public domain, openly licensed (Tatoeba is CC-BY, so a match also means attribution), or excerpts the maintainer holds legitimately and keeps in a gitignored folder. Never scrape a competitor’s product.
3. Jev (cents, cached by content). For each passage (a content section, a conversation, a whole exercise list, each theme block; passages under 12 words are generic and skipped) Jev says whether it recognises the text from a published source, whether the wording is distinctive, whether the selection mirrors one source, and which kind of source. Jev has no web access, so it cannot cite anything.

CHECK: Jev recognises it (copied ≥ 0.8, distinctive ≥ 0.7, a source kind other than none), or a 12+ word run is shared with a reference. A person looks, searches the web for the distinctive sentences, and either rewrites the passage or lists it. ERROR: a shared run and Jev finds the wording distinctive. A passage reviewed and found original goes in data/courses/<id>/plagiarism-ok.txt (--why prints the line; add # who, when, why); it is skipped until its text changes. Thresholds are in e2e/plagiarism/probe.ts (T); bump PROMPT_VERSION when the questions change.

--calibrate measures the pre-filter (our own passages, verbatim and lightly edited) and Jev (ten public-domain Spanish passages as known- published text, against the course’s own passages as presumed original). The recall it prints overstates recall on a modern app’s exercises, which Jev has seen far less of, and a clean run is weak evidence: in-app and paywalled content is not indexed anywhere. It catches lazy copying. It does not prove originality.

CI (.github/workflows/plagiarism.yml) runs the free layers on every PR that touches course content, themes or docs, and Jev too when the TYPESAFE_API_KEY secret exists. Run the full check and --calibrate before a release. The themes draw on third-party game worlds; that is a trademark question this check does not cover.