AWS setup

You do these steps once, in order, before scripts/deploy-all.sh aws works. Budget about 30 minutes, mostly waiting for AWS.

1. An AWS account

1. Go to https://aws.amazon.com/ → Create an AWS account. You need an e-mail address, a phone number for the verification call/SMS and a payment card. Choose the Basic (free) support plan.
2. New accounts are offered a free plan (credits, up to six months, no charges beyond them) or a paid plan (pay as you go, with some free-tier credits). Either runs this app. The paid plan keeps working after the credits are used. Accounts created before July 2025 have the older 12-month free tier instead, which covers the instance and the IPv4 address.
3. Sign in as the root user, open the account menu (top right) → Security credentials → assign an MFA device to the root user. Then stop using the root user for day-to-day work.

2. A spending alarm (recommended)

Billing console → Budgets → Create budget → Use a template → Monthly cost budget: amount 10 USD, your e-mail. AWS e-mails you if the month is heading past $10. The stack this app creates costs about $7.40/month; anything above that means something else is running.

3. A deployment user and its access key

The deploy script needs an identity that may create EC2 instances and CloudFormation stacks, the instance’s IAM role, and the Parameter Store entries the app’s secrets live in. Do not use the root user’s keys for this.

1. Console → IAM → Users → Create user.

   • User name: learnmap-deploy.
   • Leave Provide user access to the AWS Management Console unticked.

2. Set permissions → Attach policies directly → search and tick:

   • AmazonEC2FullAccess
   • AWSCloudFormationFullAccess
   • AmazonSSMReadOnlyAccess (the stack looks up the current Amazon Linux image through an SSM parameter) Next → Create user.

3. Open the user → Permissions → Add permissions → Create inline policy → JSON, paste this, name it learnmap-deploy-extra. It lets the stack create the instance’s role, the backup bucket and the snapshot policy, lets scripts/aws-env.sh manage the settings under /skrongle/, and lets the deploy script read backups (it can’t delete them):

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "iam:CreateRole", "iam:DeleteRole", "iam:GetRole", "iam:TagRole",
           "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:GetRolePolicy",
           "iam:PassRole",
           "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile",
           "iam:GetInstanceProfile", "iam:AddRoleToInstanceProfile",
           "iam:RemoveRoleFromInstanceProfile",
           "iam:AttachRolePolicy", "iam:DetachRolePolicy"
         ],
         "Resource": [
           "arn:aws:iam::*:role/learnmap-*",
           "arn:aws:iam::*:instance-profile/learnmap-*"
         ]
       },
       {
         "Effect": "Allow",
         "Action": [
           "ssm:PutParameter", "ssm:DeleteParameter", "ssm:GetParameter",
           "ssm:GetParameters", "ssm:GetParametersByPath", "ssm:AddTagsToResource"
         ],
         "Resource": "arn:aws:ssm:*:*:parameter/skrongle/*"
       },
       {
         "Effect": "Allow",
         "Action": ["kms:Encrypt", "kms:Decrypt"],
         "Resource": "*",
         "Condition": { "StringLike": { "kms:ViaService": "ssm.*.amazonaws.com" } }
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:CreateBucket", "s3:ListBucket", "s3:ListBucketVersions",
           "s3:GetBucket*", "s3:PutBucket*", "s3:DeleteBucketPolicy",
           "s3:GetLifecycleConfiguration", "s3:PutLifecycleConfiguration",
           "s3:GetEncryptionConfiguration", "s3:PutEncryptionConfiguration",
           "s3:GetObject", "s3:GetObjectVersion"
         ],
         "Resource": ["arn:aws:s3:::learnmap-*", "arn:aws:s3:::learnmap-*/*"]
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListAllMyBuckets",
           "dlm:CreateLifecyclePolicy", "dlm:GetLifecyclePolicy", "dlm:GetLifecyclePolicies",
           "dlm:UpdateLifecyclePolicy", "dlm:DeleteLifecyclePolicy",
           "dlm:TagResource", "dlm:UntagResource"
         ],
         "Resource": "*"
       }
     ]
   }
   

   (learnmap-* is the stack name; change every one if you set AWS_STACK_NAME.)

4. Open the user → Security credentials tab → Access keys → Create access key → use case Command Line Interface (CLI) → tick the confirmation → Next → Create access key.

5. Copy the Access key ID and the Secret access key now; the secret is shown once. They go into deploy/deploy.env as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. (Alternatively run aws configure --profile learnmap and set AWS_PROFILE=learnmap instead.)

If a key ever leaks, come back here, Deactivate then Delete it, and create a new one; nothing else changes.

4. A region

Pick one region and keep it; the stack lives there. eu-west-1 (Ireland) or us-east-1 (N. Virginia) are good defaults; a region near you lowers latency. Prices for t4g.nano are within cents of each other everywhere. Put it in deploy.env as AWS_REGION.

Every region comes with a default VPC; the stack uses it. If someone deleted it (console → VPC → Your VPCs is empty), restore it with Actions → Create default VPC.

5. Tools on your Mac

brew install awscli zig          # the AWS CLI and the cross-compiler zig provides
cargo install cargo-zigbuild     # cross-compiles the Rust binary for ARM
rustup target add aarch64-unknown-linux-gnu
aws --version                    # should print aws-cli/2.x

You do not need aws configure when the keys are in deploy.env.

6. An ssh key

The public half is placed on the instance when it is created; the deploy script uses the private half to ssh in as ec2-user. Any existing key works (~/.ssh/id_ed25519); to make a dedicated one:

ssh-keygen -t ed25519 -f ~/.ssh/learnmap -N "" -C learnmap

and set AWS_SSH_KEY=~/.ssh/learnmap. The private key stays on your Mac.

7. Fill in deploy/deploy.env

cp deploy/deploy.env.example deploy/deploy.env
$EDITOR deploy/deploy.env

Minimum for AWS:

DEPLOY_TARGETS="aws"             # or "pi aws" once the Pi is configured too
AWS_ACCESS_KEY_ID=AKIA…
AWS_SECRET_ACCESS_KEY=…
AWS_REGION=eu-west-1
AWS_SSH_KEY=~/.ssh/learnmap
AWS_DOMAIN=                      # empty: use the free <ip>.sslip.io name

No secrets of the app go in this file; those are the next step.

Check it without touching AWS:

scripts/deploy-all.sh aws --dry-run

8. The app’s secrets, in Parameter Store

This is a hosted server, so it needs every provider or it refuses to start and names what is missing: e-mail (SKRONGLE_MAIL_*), text messages (SKRONGLE_SMS_*), Google sign-in (SKRONGLE_GOOGLE_*, step 10) and Stripe (SKRONGLE_STRIPE_*). Only a Skrongle mini runs without them.

They live in AWS Systems Manager → Parameter Store, one encrypted SecureString per setting, named like the environment variable it becomes, under /skrongle/<stack name>/ (AWS_SSM_PATH). Nothing secret is on your Mac or in the deploy: the instance has an IAM role that may read that path and nothing else, and reads the settings right before every start into tmpfs (/run/learnmap/env). Standard-tier parameters are free.

scripts/aws-env.sh init                          # one TODO placeholder per setting the server needs
openssl rand -hex 32 | scripts/aws-env.sh set SKRONGLE_SECRET_KEY   # once, never again (see below)
scripts/aws-env.sh set SKRONGLE_MAIL_FROM        # asks for the value, hidden
scripts/aws-env.sh set SKRONGLE_MAIL_PROVIDERS resend
scripts/aws-env.sh set SKRONGLE_MAIL_RESEND_KIND resend
scripts/aws-env.sh set SKRONGLE_MAIL_RESEND_API_KEY
…
scripts/aws-env.sh list                          # which are set, which still TODO
scripts/aws-env.sh check                         # what the deploy will refuse on

Or fill them in the console: Systems Manager → Parameter Store → the parameter → Edit. A value of TODO is a placeholder: the instance skips it, so the app sees the setting as unset. Each setting’s description says what goes in it; which settings a provider needs is in configuration. Change a value, then scripts/deploy-aws.sh --skip-build (or sudo systemctl restart learnmap on the box) to make the app read it.

SKRONGLE_SECRET_KEY encrypts the AI keys learners save. Set it once, before the first deploy, and never change it: a new key makes every saved AI key unreadable. Keep a copy in your password manager too. Because it lives here and not on the instance’s disk, a backup of the database is all a restore needs (Backups). The deploy refuses to go ahead without it. On a stack that already ran without it, copy the instance’s existing key up instead of making a new one (Moving the key off the disk).

Settings stored before the rename to SKRONGLE_ (#584) keep working for one release, with a warning in the log. scripts/aws-env.sh migrate moves each to its new name, and every deploy runs it first.

9. First deploy

scripts/deploy-all.sh aws

What happens, and how long it takes:

The last line printed is the site URL, e.g. https://54-72-10-20.sslip.io. Open it, register, pick a course.

If the public health check says anything but 200 on the first run, wait a minute and run scripts/deploy-aws.sh --status; the certificate is usually the only thing still in progress.

10. Google sign-in

In the Google Cloud console, open your OAuth client (APIs & Services → Credentials) and add an Authorized redirect URI:

https://<your AWS host>/auth/google/callback

exactly as printed at the end of the deploy. The client id and secret are the SKRONGLE_GOOGLE_CLIENT_ID and SKRONGLE_GOOGLE_CLIENT_SECRET settings of step 8.

11. Your own domain (optional)

Set AWS_DOMAIN=learn.example.com, deploy, then create an A record for that name pointing at the IP from scripts/deploy-aws.sh --status. Caddy notices when the name resolves and fetches a certificate; no restart needed. Add the new redirect URI to the Google client if you use it.

Day to day

scripts/deploy-all.sh aws                   # release a new version (database untouched)
scripts/deploy-aws.sh --status              # URL, IP, ssh command
scripts/aws-env.sh list                     # the app's settings in Parameter Store (set NAME to change one)
scripts/deploy-aws.sh --backup              # back up the database to S3 now
ssh -i ~/.ssh/learnmap ec2-user@<ip>        # look around
  sudo journalctl -u learnmap -f            #   app log
  sudo journalctl -u caddy -n 50            #   proxy / certificate log
  systemctl list-timers learnmap-backup     #   when the next backup runs
scripts/deploy-aws.sh --destroy             # back up, then delete everything (asks for confirmation)

The ssh rule in the security group is rewritten to your current public IP on every deploy (AWS_ADMIN_CIDR=auto), so after changing networks just deploy again or run scripts/deploy-aws.sh --skip-build.

Backups

The stack keeps two copies of the database, both in the stack’s region:

Nothing is kept longer than 31 days, the limit the privacy notice gives for data of a deleted account. The bucket blocks public access, encrypts every object, refuses plain HTTP and keeps old versions for a day, so a bad copy doesn’t replace a good one at once. The instance may add backups but can’t read or delete them. Your deploy user can read them but can’t delete them.

Deleting the stack (--destroy) takes a backup first and refuses to delete if that backup fails. The bucket outlives the stack. It empties itself as the copies expire, and is then an empty bucket you can delete. The disk snapshots outlive it too, but nothing prunes them any more; delete them in the EC2 console (Snapshots, filter on the tag) once you no longer need them.

The backup is the database alone. The key that encrypts saved AI keys is SKRONGLE_SECRET_KEY in Parameter Store (step 8), which a new stack reads the same way. A disk snapshot holds the database and, on a stack from before the move, the old key file as well.

Costs: the S3 copies are a few cents a month; the snapshots store only what changed from day to day, well under $1 a month for the 8 GB disk.

To check that backups run: scripts/deploy-aws.sh --status names the latest one. On the instance, systemctl list-timers learnmap-backup shows the next run and journalctl -u learnmap-backup the last ones. A failed run shows in systemctl --failed.

Moving the key off the disk

A stack deployed before #964 made its own key file, /var/lib/learnmap/learn.db.key. Copy that value into Parameter Store, without making a new one:

scripts/deploy-aws.sh --status                       # the ssh command
ssh … ec2-user@<ip> 'sudo cat /var/lib/learnmap/learn.db.key' | scripts/aws-env.sh set SKRONGLE_SECRET_KEY
scripts/deploy-aws.sh --skip-build

Save the same value in your password manager. The deploy compares the key in Parameter Store with the file on the instance and stops, changing nothing, if they differ. Once a deploy has gone through, the app reads the key from Parameter Store; the file can stay where it is.

Restore a backup onto the running stack

For a database that went wrong (a bad migration, a mistake in the admin pages):

scripts/deploy-aws.sh --restore              # the latest backup
scripts/deploy-aws.sh --restore 2026-10-27   # the backup of that day (UTC)

It asks for the stack name, then the instance downloads the backup through a link that works for 15 minutes, checks it, stops the app, puts the current database aside as learn.db.before-restore-<time>, starts the app and checks its health. Sign in and look at your account. Delete the before-restore file once you are happy (it holds personal data too).

Restore onto a new stack

When the instance or the whole stack is gone:

1. Find the bucket of the old stack: aws s3 ls | grep learnmap lists learnmap-backupbucket-… buckets; the old one has the copies in it (aws s3 ls s3://<bucket>/daily/).
2. Deploy as usual: scripts/deploy-all.sh aws. It makes a new stack with a new, empty bucket and an empty database. SKRONGLE_SECRET_KEY and the other settings are still in Parameter Store.
3. Restore from the old bucket: scripts/deploy-aws.sh --restore s3://<old bucket>/daily/<date>.db.gz.
4. Sign in and check your account. If the domain pointed at the old IP, point it at the new one (--status).

Restore from a disk snapshot

When the S3 copies are no help (say a backup ran after the damage was done), the nightly disk snapshots go back 14 days:

1. EC2 console → Snapshots → filter on the tag learnmap-snapshot-of=<stack> → pick one → Create volume from snapshot, in the instance’s Availability Zone.
2. Volumes → the new volume → Attach volume to the instance (it shows up as /dev/nvme1n1).
3. On the instance:

   sudo mkdir -p /mnt/old && sudo mount -o ro,nouuid /dev/nvme1n1p1 /mnt/old
   sudo systemctl stop learnmap
   sudo mv /var/lib/learnmap/learn.db /var/lib/learnmap/learn.db.before-restore
   sudo rm -f /var/lib/learnmap/learn.db-wal /var/lib/learnmap/learn.db-shm
   sudo cp -a /mnt/old/var/lib/learnmap/learn.db* /var/lib/learnmap/
   sudo systemctl start learnmap
   sudo umount /mnt/old
   

   The snapshot was taken from a running disk, so copy the -wal file with the database (the learn.db* above does); SQLite replays it on start.
4. Detach and delete the extra volume.

Restore test

Do one full restore on a throwaway stack before launch, and again after a change to backups. Use the live settings, so the test stack reads the same key:

export AWS_STACK_NAME=learnmap-restoretest AWS_SSM_PATH=/skrongle/learnmap AWS_DOMAIN=
scripts/deploy-aws.sh
scripts/deploy-aws.sh --restore s3://<live bucket>/daily/<date>.db.gz

Sign in with a real account and check its courses and plan. Then scripts/deploy-aws.sh --destroy the test stack (same exports), delete its bucket and its snapshots, and write the date here. The test server sends e-mails and texts like the live one, so test soon after a backup, when none are waiting to go out.

Last full restore test: not done yet.

Data exports bucket

A learner’s Download my data file goes to a second private bucket, the stack’s FilesBucket, under exports/ (#871). The deploy writes its name to /etc/learnmap/env as SKRONGLE_S3_BUCKET; the server refuses to start without it. The instance’s role may put, read and delete objects under exports/ only. A download is a pre-signed link that works for 60 seconds; the app deletes the file five minutes after it is downloaded, and the bucket’s lifecycle rule deletes anything under exports/ two days after upload. Like the backup bucket, it is kept when the stack is deleted.

The content check in its own process

The AI content check (Jev and the AI review of what scored high) can run in a process of its own, so its work and its keys stay apart from the web server. Set SKRONGLE_CONTENT_CHECK=external in Parameter Store; the server then only queues the checks. Start the binary’s content-check command on the same box, with the server’s settings (the same SKRONGLE_DB, and TYPESAFE_API_KEY and SKRONGLE_AI_KEY from Parameter Store):

sudo systemd-run --unit=learnmap-check --uid=learnmap --gid=learnmap \
  -p EnvironmentFile=/etc/learnmap/env -p EnvironmentFile=/run/learnmap/env \
  -E SKRONGLE_DB=/var/lib/learnmap/learn.db -E SKRONGLE_DATA_DIR=/opt/learnmap/data \
  /opt/learnmap/learnmap content-check
sudo journalctl -u learnmap-check -f        # its log
sudo systemctl stop learnmap-check          # stop it

The course table on /superuser shows when the worker last checked in and how many checks wait; a red ! means it has been silent for a day. Stop it any time: a check it was in the middle of runs again after 10 minutes. It doesn’t come back after a reboot; start it again.

It has to run on the server’s computer: the database is one SQLite file. Running it on your own computer waits for the shared database (#581).

If something goes wrong

If personal data may have leaked (someone got into the box, a key turned up in public), stop here and follow the incident response runbook, ops/incident-response.md in the repo.

• An error occurred (AccessDenied): the IAM user is missing one of the policies in step 3, or the keys were pasted with a stray space.
• “the hosted server refuses to start without these”: a required setting in Parameter Store is missing or still TODO; the lines under it are the commands to set them (step 8).
• The service keeps restarting, the log says learnmap-env: the instance could not read Parameter Store. sudo journalctl -u learnmap -n 20 shows the AWS error; usually the stack was created before the role existed (deploy again) or AWS_SSM_PATH in deploy.env is not where the settings are (scripts/aws-env.sh list).
• Stack … is in ROLLBACK_COMPLETE state (a first creation failed): delete it in the console (CloudFormation → the stack → Delete) or with scripts/deploy-aws.sh --destroy, read the Events tab for the reason, then deploy again. The usual causes are no default VPC (step 4) or a region where t4g.nano is not offered (change AWS_INSTANCE_TYPE or region).
• ssh never connects: the security group only allows your IP at deploy time; on a VPN or a network that changes IPs set AWS_ADMIN_CIDR to your network’s range, or 0.0.0.0/0 to allow all (key-only login is still required).
• “could not pin the instance’s ssh host key”: the boot console showed no host key. Open the instance with EC2 Instance Connect, run ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub, put the SHA256:… value in AWS_SSH_HOST_KEY_FINGERPRINT and deploy again.
• Host key verification failed: the instance no longer has the key pinned in deploy/aws/known_hosts.<instance-id>. If you know why (you rebuilt it by hand), delete that file and deploy again; otherwise stop.
• “SKRONGLE_SECRET_KEY in Parameter Store is not the key in … on the instance”: the stack ran before the key moved to Parameter Store, and the value there is a different key. Copy the instance’s key up (Moving the key off the disk).
• --destroy says “no backup, so no delete”: the backup failed; the lines above it are the backup’s log. A stack from before #964 has no bucket yet: deploy once, then destroy.
• Site shows “waiting for the first deploy”: Caddy is up but the rsync step did not finish; run the deploy again.