# AWS setup

You do these steps once, in order, before `scripts/deploy-all.sh aws`
works. Budget about 30 minutes, mostly waiting for AWS.

## 1. An AWS account

1. Go to <https://aws.amazon.com/> → *Create an AWS account*. You need an
   e-mail address, a phone number for the verification call/SMS and a payment
   card. Choose the **Basic (free) support plan**.
2. New accounts are offered a **free plan** (credits, up to six months, no
   charges beyond them) or a **paid plan** (pay as you go, with some free-tier
   credits). Either runs this app. The paid plan keeps working after the
   credits are used. Accounts created before July 2025 have the older 12-month
   free tier instead, which covers the instance and the IPv4 address.
3. Sign in as the root user, open the account menu (top right) → *Security
   credentials* → **assign an MFA device** to the root user. Then stop using
   the root user for day-to-day work.

## 2. A spending alarm (recommended)

Billing console → *Budgets* → *Create budget* → *Use a template* → **Monthly
cost budget**: amount `10` USD, your e-mail. AWS e-mails you if the month is
heading past $10. The stack this app creates costs about $7.40/month; anything
above that means something else is running.

## 3. A deployment user and its access key

The deploy script needs an identity that may create EC2 instances and
CloudFormation stacks, the instance's IAM role, and the Parameter Store
entries the app's secrets live in. Do not use the root user's keys for this.

1. Console → **IAM** → *Users* → *Create user*.
   - User name: `learnmap-deploy`.
   - Leave *Provide user access to the AWS Management Console* unticked.
2. *Set permissions* → **Attach policies directly** → search and tick:
   - `AmazonEC2FullAccess`
   - `AWSCloudFormationFullAccess`
   - `AmazonSSMReadOnlyAccess` (the stack looks up the current Amazon Linux
     image through an SSM parameter)
   Next → *Create user*.
3. Open the user → *Permissions* → *Add permissions* → **Create inline
   policy** → *JSON*, paste this, name it `learnmap-deploy-extra`. It lets
   the stack create the instance's role, the backup bucket and the snapshot
   policy, lets `scripts/aws-env.sh` manage the settings under `/skrongle/`,
   and lets the deploy script read backups (it can't delete them):

   ```json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "iam:CreateRole", "iam:DeleteRole", "iam:GetRole", "iam:TagRole",
           "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:GetRolePolicy",
           "iam:PassRole",
           "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile",
           "iam:GetInstanceProfile", "iam:AddRoleToInstanceProfile",
           "iam:RemoveRoleFromInstanceProfile",
           "iam:AttachRolePolicy", "iam:DetachRolePolicy"
         ],
         "Resource": [
           "arn:aws:iam::*:role/learnmap-*",
           "arn:aws:iam::*:instance-profile/learnmap-*"
         ]
       },
       {
         "Effect": "Allow",
         "Action": [
           "ssm:PutParameter", "ssm:DeleteParameter", "ssm:GetParameter",
           "ssm:GetParameters", "ssm:GetParametersByPath", "ssm:AddTagsToResource"
         ],
         "Resource": "arn:aws:ssm:*:*:parameter/skrongle/*"
       },
       {
         "Effect": "Allow",
         "Action": ["kms:Encrypt", "kms:Decrypt"],
         "Resource": "*",
         "Condition": { "StringLike": { "kms:ViaService": "ssm.*.amazonaws.com" } }
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:CreateBucket", "s3:ListBucket", "s3:ListBucketVersions",
           "s3:GetBucket*", "s3:PutBucket*", "s3:DeleteBucketPolicy",
           "s3:GetLifecycleConfiguration", "s3:PutLifecycleConfiguration",
           "s3:GetEncryptionConfiguration", "s3:PutEncryptionConfiguration",
           "s3:GetObject", "s3:GetObjectVersion"
         ],
         "Resource": ["arn:aws:s3:::learnmap-*", "arn:aws:s3:::learnmap-*/*"]
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListAllMyBuckets",
           "dlm:CreateLifecyclePolicy", "dlm:GetLifecyclePolicy", "dlm:GetLifecyclePolicies",
           "dlm:UpdateLifecyclePolicy", "dlm:DeleteLifecyclePolicy",
           "dlm:TagResource", "dlm:UntagResource"
         ],
         "Resource": "*"
       }
     ]
   }
   ```

   (`learnmap-*` is the stack name; change every one if you set `AWS_STACK_NAME`.)
4. Open the user → *Security credentials* tab → *Access keys* → **Create
   access key** → use case *Command Line Interface (CLI)* → tick the
   confirmation → *Next* → *Create access key*.
5. Copy the **Access key ID** and the **Secret access key** now; the secret
   is shown once. They go into `deploy/deploy.env` as `AWS_ACCESS_KEY_ID` and
   `AWS_SECRET_ACCESS_KEY`. (Alternatively run `aws configure --profile
   learnmap` and set `AWS_PROFILE=learnmap` instead.)

If a key ever leaks, come back here, *Deactivate* then *Delete* it, and create
a new one; nothing else changes.

## 4. A region

Pick one region and keep it; the stack lives there. `eu-west-1` (Ireland) or
`us-east-1` (N. Virginia) are good defaults; a region near you lowers latency.
Prices for `t4g.nano` are within cents of each other everywhere. Put it in
`deploy.env` as `AWS_REGION`.

Every region comes with a **default VPC**; the stack uses it. If someone
deleted it (console → VPC → *Your VPCs* is empty), restore it with
*Actions* → *Create default VPC*.

## 5. Tools on your Mac

```sh
brew install awscli zig          # the AWS CLI and the cross-compiler zig provides
cargo install cargo-zigbuild     # cross-compiles the Rust binary for ARM
rustup target add aarch64-unknown-linux-gnu
aws --version                    # should print aws-cli/2.x
```

You do not need `aws configure` when the keys are in `deploy.env`.

## 6. An ssh key

The public half is placed on the instance when it is created; the deploy
script uses the private half to ssh in as `ec2-user`. Any existing key works
(`~/.ssh/id_ed25519`); to make a dedicated one:

```sh
ssh-keygen -t ed25519 -f ~/.ssh/learnmap -N "" -C learnmap
```

and set `AWS_SSH_KEY=~/.ssh/learnmap`. The private key stays on your Mac.

## 7. Fill in `deploy/deploy.env`

```sh
cp deploy/deploy.env.example deploy/deploy.env
$EDITOR deploy/deploy.env
```

Minimum for AWS:

```sh
DEPLOY_TARGETS="aws"             # or "pi aws" once the Pi is configured too
AWS_ACCESS_KEY_ID=AKIA…
AWS_SECRET_ACCESS_KEY=…
AWS_REGION=eu-west-1
AWS_SSH_KEY=~/.ssh/learnmap
AWS_DOMAIN=                      # empty: use the free <ip>.sslip.io name
```

No secrets of the app go in this file; those are the next step.

Check it without touching AWS:

```sh
scripts/deploy-all.sh aws --dry-run
```

## 8. The app's secrets, in Parameter Store

This is a hosted server, so it needs every provider or it refuses to start
and names what is missing: e-mail (`SKRONGLE_MAIL_*`), text messages
(`SKRONGLE_SMS_*`), Google sign-in (`SKRONGLE_GOOGLE_*`, step 10) and Stripe
(`SKRONGLE_STRIPE_*`). Only a Skrongle mini runs without them.

They live in **AWS Systems Manager → Parameter Store**, one encrypted
`SecureString` per setting, named like the environment variable it becomes,
under `/skrongle/<stack name>/` (`AWS_SSM_PATH`). Nothing secret is on your
Mac or in the deploy: the instance has an IAM role that may read that path
and nothing else, and reads the settings right before every start into
tmpfs (`/run/learnmap/env`). Standard-tier parameters are free.

```sh
scripts/aws-env.sh init                          # one TODO placeholder per setting the server needs
openssl rand -hex 32 | scripts/aws-env.sh set SKRONGLE_SECRET_KEY   # once, never again (see below)
scripts/aws-env.sh set SKRONGLE_MAIL_FROM        # asks for the value, hidden
scripts/aws-env.sh set SKRONGLE_MAIL_PROVIDERS resend
scripts/aws-env.sh set SKRONGLE_MAIL_RESEND_KIND resend
scripts/aws-env.sh set SKRONGLE_MAIL_RESEND_API_KEY
…
scripts/aws-env.sh list                          # which are set, which still TODO
scripts/aws-env.sh check                         # what the deploy will refuse on
```

Or fill them in the console: Systems Manager → Parameter Store → the
parameter → *Edit*. A value of `TODO` is a placeholder: the instance skips
it, so the app sees the setting as unset. Each setting's description says
what goes in it; which settings a provider needs is in
[configuration](architecture#configuration). Change a value, then
`scripts/deploy-aws.sh --skip-build` (or `sudo systemctl restart learnmap`
on the box) to make the app read it.

`SKRONGLE_SECRET_KEY` encrypts the AI keys learners save. Set it once, before
the first deploy, and never change it: a new key makes every saved AI key
unreadable. Keep a copy in your password manager too. Because it lives here
and not on the instance's disk, a backup of the database is all a restore
needs ([Backups](#backups)). The deploy refuses to go ahead without it. On a
stack that already ran without it, copy the instance's existing key up
instead of making a new one ([Moving the key off the disk](#moving-the-key-off-the-disk)).

Settings stored before the rename to `SKRONGLE_` (#584) keep working for
one release, with a warning in the log. `scripts/aws-env.sh migrate` moves
each to its new name, and every deploy runs it first.

## 9. First deploy

```sh
scripts/deploy-all.sh aws
```

What happens, and how long it takes:

| Step | Time |
|---|---|
| tests + cross-compile (first time; later builds are incremental) | 2–5 min |
| the required settings are checked in Parameter Store (`scripts/aws-env.sh check`) | seconds |
| CloudFormation creates the security group, the instance and its read-only role, and the static IP | ~2 min |
| first boot installs rsync and a pinned, checksum-verified Caddy (the script waits for it) | ~2 min |
| the script pins the instance's ssh host key from its boot console (AWS API) in `deploy/aws/known_hosts.<instance-id>`; every later ssh checks it | with the wait above |
| rsync; the nightly backup timer is installed; at start the instance reads its settings from Parameter Store; health check | seconds |
| Caddy obtains the HTTPS certificate for `<ip>.sslip.io` | 10–60 s |

The last line printed is the site URL, e.g.
`https://54-72-10-20.sslip.io`. Open it, register, pick a course.

If the public health check says anything but 200 on the first run, wait a
minute and run `scripts/deploy-aws.sh --status`; the certificate is usually
the only thing still in progress.

## 10. Google sign-in

In the Google Cloud console, open your OAuth client (APIs & Services →
Credentials) and add an **Authorized redirect URI**:

```
https://<your AWS host>/auth/google/callback
```

exactly as printed at the end of the deploy. The client id and secret are
the `SKRONGLE_GOOGLE_CLIENT_ID` and `SKRONGLE_GOOGLE_CLIENT_SECRET` settings of
step 8.

## 11. Your own domain (optional)

Set `AWS_DOMAIN=learn.example.com`, deploy, then create an **A record** for
that name pointing at the IP from `scripts/deploy-aws.sh --status`. Caddy
notices when the name resolves and fetches a certificate; no restart needed.
Add the new redirect URI to the Google client if you use it.

## Day to day

```sh
scripts/deploy-all.sh aws                   # release a new version (database untouched)
scripts/deploy-aws.sh --status              # URL, IP, ssh command
scripts/aws-env.sh list                     # the app's settings in Parameter Store (set NAME to change one)
scripts/deploy-aws.sh --backup              # back up the database to S3 now
ssh -i ~/.ssh/learnmap ec2-user@<ip>        # look around
  sudo journalctl -u learnmap -f            #   app log
  sudo journalctl -u caddy -n 50            #   proxy / certificate log
  systemctl list-timers learnmap-backup     #   when the next backup runs
scripts/deploy-aws.sh --destroy             # back up, then delete everything (asks for confirmation)
```

The ssh rule in the security group is rewritten to your current public IP on
every deploy (`AWS_ADMIN_CIDR=auto`), so after changing networks just deploy
again or run `scripts/deploy-aws.sh --skip-build`.

## Backups

The stack keeps two copies of the database, both in the stack's region:

| What | When | Kept | Where |
|---|---|---|---|
| The database, copied with SQLite's online backup, checked and gzipped | every night at 03:30 UTC, and before `--destroy` | 30 days (a replaced copy one more day) | the stack's private S3 bucket, `daily/YYYY-MM-DD.db.gz` |
| A snapshot of the whole disk | every night at 04:30 UTC | 14 days | EBS snapshots tagged `learnmap-snapshot-of=<stack>` |

Nothing is kept longer than 31 days, the limit the privacy notice gives for
data of a deleted account. The bucket blocks public access, encrypts every
object, refuses plain HTTP and keeps old versions for a day, so a bad copy
doesn't replace a good one at once. The instance may add backups but can't
read or delete them. Your deploy user can read them but can't delete them.

Deleting the stack (`--destroy`) takes a backup first and refuses to delete
if that backup fails. The bucket outlives the stack. It empties itself as
the copies expire, and is then an empty bucket you can delete. The disk
snapshots outlive it too, but nothing prunes them any more; delete them in
the EC2 console (*Snapshots*, filter on the tag) once you no longer need them.

The backup is the database alone. The key that encrypts saved AI keys is
`SKRONGLE_SECRET_KEY` in Parameter Store (step 8), which a new stack reads
the same way. A disk snapshot holds the database and, on a stack from
before the move, the old key file as well.

Costs: the S3 copies are a few cents a month; the snapshots store only what
changed from day to day, well under $1 a month for the 8 GB disk.

To check that backups run: `scripts/deploy-aws.sh --status` names the
latest one. On the instance, `systemctl list-timers learnmap-backup` shows
the next run and `journalctl -u learnmap-backup` the last ones. A failed
run shows in `systemctl --failed`.

### Moving the key off the disk

A stack deployed before #964 made its own key file,
`/var/lib/learnmap/learn.db.key`. Copy that value into Parameter Store,
without making a new one:

```sh
scripts/deploy-aws.sh --status                       # the ssh command
ssh … ec2-user@<ip> 'sudo cat /var/lib/learnmap/learn.db.key' | scripts/aws-env.sh set SKRONGLE_SECRET_KEY
scripts/deploy-aws.sh --skip-build
```

Save the same value in your password manager. The deploy compares the key
in Parameter Store with the file on the instance and stops, changing
nothing, if they differ. Once a deploy has gone through, the app reads the
key from Parameter Store; the file can stay where it is.

### Restore a backup onto the running stack

For a database that went wrong (a bad migration, a mistake in the admin
pages):

```sh
scripts/deploy-aws.sh --restore              # the latest backup
scripts/deploy-aws.sh --restore 2026-10-27   # the backup of that day (UTC)
```

It asks for the stack name, then the instance downloads the backup through
a link that works for 15 minutes, checks it, stops the app, puts the
current database aside as `learn.db.before-restore-<time>`, starts the app
and checks its health. Sign in and look at your account. Delete the
`before-restore` file once you are happy (it holds personal data too).

### Restore onto a new stack

When the instance or the whole stack is gone:

1. Find the bucket of the old stack: `aws s3 ls | grep learnmap` lists
   `learnmap-backupbucket-…` buckets; the old one has the copies in it
   (`aws s3 ls s3://<bucket>/daily/`).
2. Deploy as usual: `scripts/deploy-all.sh aws`. It makes a new stack with
   a new, empty bucket and an empty database. `SKRONGLE_SECRET_KEY` and the
   other settings are still in Parameter Store.
3. Restore from the old bucket:
   `scripts/deploy-aws.sh --restore s3://<old bucket>/daily/<date>.db.gz`.
4. Sign in and check your account. If the domain pointed at the old IP,
   point it at the new one (`--status`).

### Restore from a disk snapshot

When the S3 copies are no help (say a backup ran after the damage was
done), the nightly disk snapshots go back 14 days:

1. EC2 console → *Snapshots* → filter on the tag
   `learnmap-snapshot-of=<stack>` → pick one → *Create volume from
   snapshot*, in the instance's Availability Zone.
2. *Volumes* → the new volume → *Attach volume* to the instance (it shows up
   as `/dev/nvme1n1`).
3. On the instance:
   ```sh
   sudo mkdir -p /mnt/old && sudo mount -o ro,nouuid /dev/nvme1n1p1 /mnt/old
   sudo systemctl stop learnmap
   sudo mv /var/lib/learnmap/learn.db /var/lib/learnmap/learn.db.before-restore
   sudo rm -f /var/lib/learnmap/learn.db-wal /var/lib/learnmap/learn.db-shm
   sudo cp -a /mnt/old/var/lib/learnmap/learn.db* /var/lib/learnmap/
   sudo systemctl start learnmap
   sudo umount /mnt/old
   ```
   The snapshot was taken from a running disk, so copy the `-wal` file with
   the database (the `learn.db*` above does); SQLite replays it on start.
4. Detach and delete the extra volume.

### Restore test

Do one full restore on a throwaway stack before launch, and again after a
change to backups. Use the live settings, so the test stack reads the same
key:

```sh
export AWS_STACK_NAME=learnmap-restoretest AWS_SSM_PATH=/skrongle/learnmap AWS_DOMAIN=
scripts/deploy-aws.sh
scripts/deploy-aws.sh --restore s3://<live bucket>/daily/<date>.db.gz
```

Sign in with a real account and check its courses and plan. Then
`scripts/deploy-aws.sh --destroy` the test stack (same exports), delete its
bucket and its snapshots, and write the date here. The test server sends
e-mails and texts like the live one, so test soon after a backup, when
none are waiting to go out.

Last full restore test: not done yet.

## Data exports bucket

A learner's *Download my data* file goes to a second private bucket, the
stack's `FilesBucket`, under `exports/` (#871). The deploy writes its name
to `/etc/learnmap/env` as `SKRONGLE_S3_BUCKET`; the server refuses to start
without it. The instance's role may put, read and delete objects under
`exports/` only. A download is a pre-signed link that works for 60 seconds;
the app deletes the file five minutes after it is downloaded, and the
bucket's lifecycle rule deletes anything under `exports/` two days after
upload. Like the backup bucket, it is kept when the stack is deleted.

## The content check in its own process

The AI content check (Jev and the AI review of what scored high) can run
in a process of its own, so its work and its keys stay apart from the
web server. Set `SKRONGLE_CONTENT_CHECK=external` in Parameter Store; the
server then only queues the checks. Start the binary's `content-check`
command on the same box, with the server's settings (the same
`SKRONGLE_DB`, and `TYPESAFE_API_KEY` and `SKRONGLE_AI_KEY` from Parameter
Store):

```sh
sudo systemd-run --unit=learnmap-check --uid=learnmap --gid=learnmap \
  -p EnvironmentFile=/etc/learnmap/env -p EnvironmentFile=/run/learnmap/env \
  -E SKRONGLE_DB=/var/lib/learnmap/learn.db -E SKRONGLE_DATA_DIR=/opt/learnmap/data \
  /opt/learnmap/learnmap content-check
sudo journalctl -u learnmap-check -f        # its log
sudo systemctl stop learnmap-check          # stop it
```

The course table on /superuser shows when the worker last checked in and
how many checks wait; a red ! means it has been silent for a day. Stop it
any time: a check it was in the middle of runs again after 10 minutes.
It doesn't come back after a reboot; start it again.

It has to run on the server's computer: the database is one SQLite file.
Running it on your own computer waits for the shared database (#581).

## If something goes wrong

If personal data may have leaked (someone got into the box, a key turned
up in public), stop here and follow the incident response runbook,
`ops/incident-response.md` in the repo.

- **`An error occurred (AccessDenied)`**: the IAM user is missing one of the
  policies in step 3, or the keys were pasted with a stray space.
- **"the hosted server refuses to start without these"**: a required setting
  in Parameter Store is missing or still `TODO`; the lines under it are the
  commands to set them (step 8).
- **The service keeps restarting, the log says `learnmap-env`**: the instance
  could not read Parameter Store. `sudo journalctl -u learnmap -n 20` shows
  the AWS error; usually the stack was created before the role existed (deploy
  again) or `AWS_SSM_PATH` in `deploy.env` is not where the settings are
  (`scripts/aws-env.sh list`).
- **`Stack … is in ROLLBACK_COMPLETE state`** (a first creation failed):
  delete it in the console (CloudFormation → the stack → *Delete*) or with
  `scripts/deploy-aws.sh --destroy`, read the *Events* tab for the reason,
  then deploy again. The usual causes are no default VPC (step 4) or a
  region where `t4g.nano` is not offered (change `AWS_INSTANCE_TYPE` or region).
- **ssh never connects**: the security group only allows your IP at deploy
  time; on a VPN or a network that changes IPs set `AWS_ADMIN_CIDR` to your
  network's range, or `0.0.0.0/0` to allow all (key-only login is still required).
- **"could not pin the instance's ssh host key"**: the boot console showed
  no host key. Open the instance with EC2 Instance Connect, run
  `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`, put the `SHA256:…`
  value in `AWS_SSH_HOST_KEY_FINGERPRINT` and deploy again.
- **`Host key verification failed`**: the instance no longer has the key
  pinned in `deploy/aws/known_hosts.<instance-id>`. If you know why (you
  rebuilt it by hand), delete that file and deploy again; otherwise stop.
- **"SKRONGLE_SECRET_KEY in Parameter Store is not the key in … on the
  instance"**: the stack ran before the key moved to Parameter Store, and
  the value there is a different key. Copy the instance's key up
  ([Moving the key off the disk](#moving-the-key-off-the-disk)).
- **`--destroy` says "no backup, so no delete"**: the backup failed; the
  lines above it are the backup's log. A stack from before #964 has no
  bucket yet: deploy once, then destroy.
- **Site shows "waiting for the first deploy"**: Caddy is up but the rsync
  step did not finish; run the deploy again.