TYPESCRIPT · ASSESSMENT PREP

AI Evals Study Lab

Review a Docker environment, close gaps in a verifier, and draft a backend task you can defend on video.

Your mission path

The assessment lists Go, TypeScript, Python, and Rust. Use TypeScript as your assessment language. Bring your C# experience with typed interfaces, regression tests, and CI.

Translate what you know from C#
  • Directory.Packages.props centralizes version declarations. package-lock.json records npm’s resolved tree. Compare the lockfile role to packages.lock.json in NuGet.
  • Think of tsc --noEmit as a type check. TypeScript types disappear at runtime, so validate JSON from requests and files.
  • Use Vitest like xUnit for assertions and test setup. Exercise the CLI or API and check the durable effect.

  1. Write three findings. Name the failing step, cause, fix, and check.

  2. List the required inputs. Explain how to check the build and first real request without network access.

  3. Pick three drills. Name one broken implementation each test accepts, then add a check that rejects it.

  4. Pick a pinned repository. Save a prompt under 400 words, one plausible partial fix, and a verifier.

  5. Record five to seven minutes with your screen and webcam. Play it back once.

  6. Complete one timed pass. Write down the first step where you lost time.

Environment review

Trace the inputs from base image to the first real request. Name the failing step, the cause, the fix, and the check.

Practice · Payments API

The author builds on a Mac. The target is Linux. The API must work without a network after input preparation.

FROM node:22-slim
WORKDIR /app
COPY . .
RUN npm install
RUN npm run build
CMD ["node", "dist/index.js"]
Reveal the review
  1. Pin the base image by digest and specify the target platform. Stage the image before the offline build.
  2. Commit a lockfile and use npm ci. Keep project npm settings that affect the dependency tree with the lockfile.
  3. Exclude node_modules and dist from the build context. Install dependencies for the target platform and rebuild from source.
  4. Check install scripts and the first real request for downloads. A passing build or health check can miss those paths.

TypeScript offline build drills

Answer each drill before you reveal the check.

The lockfile exists. Why does the build need a network?

The lockfile identifies artifacts. A clean checkout still needs package data. Start with an empty cache and prepare it under the target Node, npm, and platform. Carry that cache into the offline build and try npm ci --offline --cache /npm-cache.

Why can npm still fail after you prepare the cache?

A lifecycle script can fetch its own binary outside npm. A native dependency can also need a compiler or a target-specific binary. Read the scripts and stage their inputs explicitly. Check the complete build with outbound access disabled.

The app builds on your Mac. Why does the Linux container fail?

A copied node_modules can contain macOS binaries. A copied dist can hide a broken compile step. Exclude both directories and build from a clean checkout under the target runtime.

What does docker build --network=none prove?

It applies to build instructions that run commands. A cached layer can hide an install step, and the builder can still need to resolve a base image or frontend. Use staged inputs and a builder with outbound access blocked. Disable the build cache for the check. Run the container without a network and exercise a real request.

npm ci reference · Docker build network reference

Verifier review

For each requirement, name an incomplete implementation that passes the current test. Add the smallest check that rejects it.

Practice · Payment retries

The verifier sends two sequential requests with the same key and checks only that both return HTTP 200. The contract requires one durable payment per key and the original payment ID on retry.

Check durable state

Two successful responses can hide two writes. Compare the payment IDs and inspect persistent storage for exactly one payment. Submit a distinct key and check that it creates a second payment.

Check concurrency and restart

Sequential requests can hide a race. Start two requests at a controlled overlap and inspect the stored effect. Restart the service before a retry to catch an in-memory cache.

Check verifier integrity

Keep fixtures and assertions under grader control. Vary data across cases. Run the candidate through the documented entry point and score observed behavior. Check that the baseline fails and a reference fix passes.

Project example · Codesurface checks that a later index pass preserves the same language selection as the first pass. Regression tests

Verifier design drills

For each drill, name the incomplete implementation that passes the current check. Then open the reveal and compare your answer.

Idempotency key

Current check: send the same key twice and assert both responses are HTTP 200.

POST /payments key=A -> 200
POST /payments key=A -> 200
Reveal worked answer

Weakness: two 200 responses can hide two writes. The second call might create a second payment and still return success.

  1. Assert both responses carry the same payment ID.
  2. Inspect persistent storage for exactly one payment for key A.
  3. Restart the service, retry with key A, and assert the same payment ID.
  4. Submit a distinct key B and assert a second payment is created.

Concurrency

Current check: two sequential requests with the same key return the same payment ID.

Reveal worked answer

Weakness: sequential calls can hide a check-then-act race. Two overlapping calls can both pass the check and both write.

  1. Start two requests at a controlled overlap with the same key.
  2. Wait for both to finish.
  3. Assert exactly one payment exists in storage.
  4. Assert both responses carry that same payment ID.

Partial failure

Current check: after one failed update, the API still returns the previous result.

Reveal worked answer

Weakness: the service might return stale data while storage is half written. The failure leaves partial state that breaks the next attempt.

  1. Seed a complete index in a temporary directory.
  2. Change several files and inject a failure after the first change.
  3. Query and assert the old results still appear.
  4. Restart, retry, and assert additions, edits, and deletions applied once with no duplicates.

In-memory cache

Current check: retry after a write returns the new value.

Reveal worked answer

Weakness: an in-memory cache can serve the new value without durable storage. A restart loses it.

  1. Write a value and assert the read returns it.
  2. Restart the process.
  3. Read again and assert the value survived.
  4. Inspect storage directly to confirm the write reached disk.

Ordering and persistence

Current check: the final list is sorted.

Reveal worked answer

Weakness: a test with distinct sort keys misses unstable order when two records tie.

  1. Insert records with equal primary sort keys in reverse tie-break order.
  2. Query and assert sorted output.
  3. Restart and query again.
  4. Assert the same documented tie-break order. Repeat with a different insertion order.

Grader integrity

Current check: the test file ships with the repo and the candidate can read it.

Reveal worked answer

Weakness: visible tests let the candidate hard-code outputs. Scores stop measuring behavior.

  1. Keep fixtures and assertions under grader control.
  2. Vary data across cases.
  3. Run the candidate through the documented entry point.
  4. Score observed behavior, not internal calls.
  5. Check that the baseline fails and a reference fix passes.

Long-horizon task

Choose a real backend or infrastructure repo with an allowed license. Pin a starting commit. State one outcome and the existing behavior that must hold.

See a worked TypeScript example

Repo Drev at cd39e30, MIT. Treat this as a practice task.

Prompt Add an offline option to drev resume that reads the local store and skips remote synchronization. Retain path rewriting and session launch. If the session is missing locally, report the error without launch or transcript creation. Preserve the current default behavior.

Model difficulty A patch can skip the pull and still contact the remote through another path. It can also skip part of the shared resume flow and lose path rewriting.

Verifier Use a temporary local store and a test launcher. Block network access. Resume a local fixture and check the rewritten transcript and launch arguments. Try a missing session and check that no transcript or launch occurs. Test the default path against a local test remote.

Drafts save in this browser.

Task authoring drills

Each drill gives a weak task. Open the reveal for the rewrite that makes it gradable.

Vague outcome

Task: "Make the index faster and cleaner."

Reveal worked answer

Rewrite with one outcome and one constraint.

Make updates to the SQLite index atomic.
If indexing fails midway, queries return the last complete index.
A later retry applies additions, edits, and deletions without duplicates.
Preserve the CLI and API response shapes.

Why it works: one outcome, explicit failure behavior, explicit retry behavior, explicit compatibility.

No pinned starting point

Task: "Fix the bug in Codesurface."

Reveal worked answer
  1. Pick a real repo with an allowed license.
  2. Pin a starting commit hash.
  3. State the expected behavior before the fix.
  4. State the behavior that must still hold.

Example: Codesurface at commit 3845446, MIT. Treat it as practice only.

Scope creep

Task: "Refactor the storage layer and add caching and new endpoints."

Reveal worked answer

Split into separate tasks. One task, one outcome. Constraints stay explicit.

  • Task 1: make index writes atomic.
  • Task 2: add a cache behind the existing query path.
  • Task 3: add one endpoint with a defined response shape.

Each task needs its own verifier and its own partial-fix story.

Missing partial fix

Task: state the fix and the check, but no plausible wrong fix.

Reveal worked answer

Add one plausible partial fix and the check that rejects it.

Partial fix: wrap each row write in a transaction.
Still wrong: the saved file state advances even on failure.
Check: inject a failure after the first change, then assert the file state marker did not advance and retry applies all changes once.

Unmeasurable acceptance

Task: "Improve reliability."

Reveal worked answer

Replace with observable checks.

  • Seed an index in a temporary directory.
  • Change several files and inject a failure after the first change.
  • Query after failure and assert old results.
  • Restart, retry, and assert exact new results with no duplicates.

License and reproducibility

Task uses a repo with unclear license and no pinned commit.

Reveal worked answer
  1. Confirm the license allows the use you intend.
  2. Pin a commit hash, not a branch.
  3. Record the setup commands.
  4. Verify a fresh checkout builds and tests pass.

90-minute mock assessment

90:00
  1. 15 min: Find three environment failures. For each one, state when it fails and how to check the fix.
  2. 20 min: Find three verifier gaps. Write the expected result and the state to inspect.
  3. 35 min: Choose a repo and draft the task. Explain one partial fix and the check that catches it.
  4. 20 min: Record the walkthrough and play it back.

Video checklist

Target five to seven minutes. Show the repo and starting point, explain the task, and trace one verifier case.

Video drills

Target five to seven minutes. Show the repo and starting point, explain the task, and trace one verifier case.

Structure

Plan: talk for seven minutes without a plan.

Reveal worked answer
  1. 0:00-0:45: repo, commit, and starting behavior.
  2. 0:45-1:30: the task in one sentence.
  3. 1:30-3:00: trace one verifier case step by step.
  4. 3:00-4:30: show a plausible partial fix and how the verifier rejects it.
  5. 4:30-5:30: show the reference fix passing.
  6. 5:30-6:30: playback check and closing note.

Readable screen

Plan: record at default zoom and font size.

Reveal worked answer
  • Increase terminal font size before recording.
  • Zoom the editor to at least 125 percent.
  • Keep code lines under 90 characters.
  • Show one file at a time.
  • Pause on each command result.

Narration

Plan: describe the code while scrolling.

Reveal worked answer
  • State what you are about to check before running it.
  • State the expected result in one sentence.
  • State the actual result and whether it matches.
  • Name the partial fix and the exact check that rejects it.

Audio and playback

Plan: record once and submit.

Reveal worked answer
  1. Record a short test clip first.
  2. Check levels and background noise.
  3. Play back the full video at 1x speed.
  4. Confirm text is readable on a phone screen.
  5. Re-record any section with unclear audio.

Partial fix demonstration

Plan: describe the partial fix in words only.

Reveal worked answer
  • Show the partial fix diff on screen.
  • Run the verifier and show the failure.
  • Point at the exact assertion that failed.
  • Explain why the fix looked correct but was not.

Self-check before submit

  • Screen and webcam are visible.
  • The walkthrough explains a plausible partial fix and shows how the verifier rejects it.
  • Playback confirms clear audio and readable text.

Optional Python reference

Use these drills to compare the same environment concepts across package managers.

Open Python environment and offline drills

Environment review drills

Five new scenarios. Each one passes on the author's machine and fails somewhere else. Write your answer in four parts before you open it: failing step, cause, fix, check.

Jump to: Offline build drills · Reference build · Quick fire · Timed write-up

E1 · The file that git ignores

The package reads its version from payments_api/_version.py. A build tool writes that file on the author's machine, and .gitignore excludes it. The author builds from their working copy.

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install --no-index --find-links wheels -r requirements.lock
CMD ["python", "-m", "payments_api"]
Hint

Compare the build context on the author's machine with a fresh clone.

Worked answer
  1. Failing step: container start, on a clean checkout. The build passes because nothing imports the package during the build.
  2. Cause: COPY . . copies a generated file that exists only in the working copy. A fresh clone does not have it, so the import fails with ModuleNotFoundError.
  3. Fix: generate the file in the preparation step, or remove the dependency on it. Copy explicit paths (src/, pyproject.toml) and add a .dockerignore for .venv, .git, .env and __pycache__.
  4. Check: git clone into an empty directory, build, then run python -c "import payments_api" in the container.

E2 · Lock from a newer Python

The author ran pip-compile on Python 3.12. The image uses Python 3.10 to match production. The lock pins anyio. The offline install fails, but only in the image.

ERROR: Could not find a version that satisfies the requirement exceptiongroup
ERROR: No matching distribution found for exceptiongroup
Hint

Some dependencies apply only below a Python version. Find the marker.

Worked answer
  1. Failing step: the offline pip install in the image.
  2. Cause: anyio requires exceptiongroup only when python_version < "3.11". The resolver on 3.12 dropped it from the lock and from the wheelhouse. pip still resolves the dependencies of each locked package, so on 3.10 it asks for exceptiongroup and the wheelhouse does not have it.
  3. Fix: compile the lock with the target interpreter. Run pip-compile inside the pinned base image, then download wheels for that Python version.
  4. Check: install offline in the target image and run pip check. Compare pip freeze with the lock.

E3 · The health check passes

The service loads a tokenizer file on the first request to /score. The library downloads the file and caches it. The author verified the image with docker run --network none and a request to /health.

Hint

Which code path ran during the check? Which code path needs the file?

Worked answer
  1. Failing step: the first real request, at runtime, without network access.
  2. Cause: a first-run download. /health does not touch the tokenizer, so the check did not cover the download.
  3. Fix: stage the file with the other inputs and copy it into the image. Point the library at the local path, and set its offline switch if it has one (for example HF_HUB_OFFLINE=1). Load the file at startup so that a missing file fails immediately.
  4. Check: start the container with --network none and send a real /score request. A health endpoint is not evidence for a code path that it does not run.

E4 · A compiler in the image

The project depends on psycopg2. The Dockerfile installs build tools first.

RUN apt-get update && apt-get install -y gcc libpq-dev
RUN pip install --no-index --find-links /wheels -r requirements.lock
Worked answer
  1. Failing step: apt-get update. It needs the Debian mirrors.
  2. Cause: the build compiles from source, so it needs system packages from the network. The wheelhouse also holds an sdist, not a wheel.
  3. Fix: use a binary wheel (psycopg2-binary or psycopg[binary]) for the target platform. If the project must compile, build the wheel in the preparation step and stage the result. Remove the apt-get step.
  4. Check: ls wheels/*.tar.gz returns nothing, and the build passes with the network off.

E5 · A smaller base image

A reviewer changes python:3.12-slim to python:3.12-alpine to reduce the image size. The wheelhouse does not change. The install fails for pydantic-core and orjson, but not for requests.

Worked answer
  1. Failing step: the offline pip install.
  2. Cause: Alpine uses musl libc. The staged wheels have manylinux tags, which need glibc. Pure-Python wheels (py3-none-any) still install, so only compiled packages fail.
  3. Fix: keep the slim image, or download wheels with a musllinux platform tag. A change to the base image is a change to the inputs. Prepare the inputs again.
  4. Check: list the platform tags in the wheelhouse and compare them with the base image.

Offline build drills

Each drill gives a preparation step or an install step. Find the gap between "it built" and "it builds offline from staged inputs".

O1 · Wheels from the wrong machine

The author works on an Apple Silicon Mac. The image targets linux/amd64.

# preparation, on the Mac
pip download -r requirements.lock -d wheels

# Dockerfile
RUN pip install --no-index --find-links /wheels -r requirements.lock
Hint

Open wheels/ and read the end of each filename.

Worked answer

pip downloaded wheels for the Mac: macosx_11_0_arm64. Pure-Python wheels still work, so the failure appears only for compiled packages. Download for the target platform and interpreter:

pip download -r requirements.lock -d wheels \
  --only-binary=:all: \
  --platform manylinux2014_x86_64 \
  --python-version 3.12 --implementation cp --abi cp312

pip requires --only-binary=:all: (or --no-deps) when you set --platform. It cannot build an sdist for another platform. Check: ls wheels | grep -v -e none-any -e manylinux prints nothing.

O2 · The sdist that needs setuptools

One dependency publishes only an sdist. --only-binary=:all: failed on it, so the author removed that flag. The offline install then fails:

ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0
Worked answer

pip builds an sdist in an isolated environment. It installs the build requirements from [build-system] requires, and with --no-index it looks for them in the wheelhouse. The wheelhouse does not have them. The build can also need a compiler and headers.

Fix: build the wheel once during preparation, inside the target image, and stage the wheel:

docker run --rm --platform linux/amd64 -v "$PWD/wheels:/out" \
  python:3.12-slim@sha256:<digest> \
  pip wheel --no-deps -w /out "legacy-pkg==2.3.1"

Adding the build backend wheels to the wheelhouse also works, but only if the build needs no system packages. Check: no .tar.gz or .zip files in the wheelhouse.

O3 · It passes online

RUN pip install --find-links /wheels -r requirements.lock

The build passes in CI. The wheelhouse is missing two packages.

Worked answer

Without --no-index, --find-links adds a location. It does not replace PyPI. pip gets the missing packages from the index, so the build passes online and fails offline. A host or CI PIP_EXTRA_INDEX_URL can hide the gap the same way.

Fix: add --no-index. Check: the build log has no Downloading https:// lines, and the build passes with the network off.

O4 · Hashes for some packages

The author added --hash to the direct dependencies by hand. The install now fails:

ERROR: Hashes are required in --require-hashes mode, but they are missing from some requirements.
Worked answer

One --hash in the file turns on hash mode for the whole install. In hash mode, every requirement, transitive ones included, must be pinned with == and must have a hash. Generate the full lock with hashes instead of editing it by hand:

pip-compile --generate-hashes -o requirements.lock pyproject.toml

pip-tools records the hashes of all files for each pinned version, so the lock also covers the Linux wheels. Check: install with --require-hashes from the wheelhouse.

O5 · The project itself

All dependencies install offline. The last step fails:

RUN pip install --no-index --find-links /wheels .
# ERROR: ... No matching distribution found for hatchling
Worked answer

Installing . builds the project. That build needs the backend from pyproject.toml, here hatchling, and pip looks for it in the wheelhouse. Two fixes:

  • Build the project wheel during preparation with pip wheel --no-deps -w dist ., then install that file with --no-deps.
  • Stage the backend wheels in the wheelhouse.

The first fix is simpler. The image then gets a finished artifact and needs no build tools.

O6 · "I tested it offline"

The author ran docker build --network=none . and it passed. Name three ways this check can pass while the offline build is still broken.

Worked answer
  1. Build cache. Cached layers do not run again, so a step that downloads can be skipped. Add --no-cache.
  2. Scope of the flag. --network applies to RUN steps. The builder can still contact a registry to resolve FROM. A # syntax= line can also pull a frontend image. For a full test, pull the base image first, then disconnect the host or build on a machine with no outbound access.
  3. Build only. The build passes, but the runtime downloads at first use (see E3). Run the container with --network none and exercise a real request.

Reference build

Illustrative files for a small API. Replace the base digest and use the target platform for preparation.

prepare.sh · runs online, once
set -euo pipefail
BASE=python:3.12-slim@sha256:<digest>

docker pull --platform linux/amd64 "$BASE"

# Run these commands in the same target Linux environment.
# Install the pinned version of pip-tools before this step.
pip-compile --generate-hashes -o requirements.lock pyproject.toml
pip download --require-hashes --only-binary=:all: \
  -r requirements.lock -d wheels
pip wheel --no-deps -w dist .
Dockerfile · needs no network
FROM --platform=linux/amd64 python:3.12-slim@sha256:<digest>
WORKDIR /app
COPY requirements.lock .
COPY wheels/ /wheels/
RUN --network=none pip install --no-index --find-links /wheels \
    --require-hashes -r requirements.lock
COPY dist/ /dist/
RUN --network=none pip install --no-index --no-deps /dist/*.whl
COPY assets/ /app/assets/
ENV TOKENIZER_PATH=/app/assets/tokenizer.json
CMD ["python", "-m", "payments_api"]

There is no # syntax= line, because it pulls a frontend image. Recent Docker versions support RUN --network=none without it.

verify.sh · runs with the network off
set -euo pipefail
docker build --no-cache --network=none -t api:check .
docker run --rm --network none api:check pip check
docker run -d --name api-check --network none api:check
trap 'docker rm -f api-check' EXIT
# Wait for the service to start before the request.
docker exec api-check python -c "import urllib.request as u; \
  print(u.urlopen(u.Request('http://127.0.0.1:8000/score', \
  data=b'{\"text\":\"hello\"}', \
  headers={'Content-Type':'application/json'})).status)"
Why each line is there
  • Digest and platform fix the base image (E5, O1).
  • Lock in the base image fixes the interpreter version (E2).
  • --only-binary=:all: keeps sdists out (O2, E4).
  • --no-index stops silent downloads (O3).
  • Generated hashes cover the full graph (O4).
  • Project wheel removes the build backend from the image (O5).
  • Explicit COPY keeps host files out (E1).
  • Staged asset and a real request cover the runtime (E3, O6).

Quick fire

Answer each one in one sentence, then open it.

1. Does a tag such as python:3.12-slim give the same image next month?

No. Tags move when the maintainers publish a new build. A digest does not move.

2. You pin an index digest. What else do you set for a multi-platform image?

The platform, for example --platform=linux/amd64. An index digest points to several images, and the builder picks one for the host by default.

3. Does pip install -r requirements.lock skip dependency resolution?

No. pip still resolves the dependencies of each listed package. That is why a lock with a missing package fails offline (E2).

4. What does pip check catch, and what does it miss?

It catches installed packages with missing or incompatible dependencies. It does not catch a wrong version that still satisfies the declared ranges, and it does not run your code.

5. A wheel is named orjson-3.10.7-cp312-cp312-musllinux_1_2_x86_64.whl. Does it install on python:3.12-slim?

No. The slim image uses glibc. You need a manylinux wheel.

6. Why is py3-none-any a weak test of the wheelhouse?

Pure-Python wheels install on every platform. They pass when the platform tags of the compiled wheels are wrong.

7. The build reads PIP_INDEX_URL from a CI variable. Why does this matter?

The environment of the build changes what pip downloads. Use explicit flags in the Dockerfile, so the CI settings cannot change the result.

8. The image builds offline. What is the next check?

Run it with --network none and exercise the main code path. The build proves only the build.

Timed write-up · 15 minutes

Pick two drills that you did not open. Write one review note for each in the format below. Then compare your notes with the worked answers.

Failing step:  <build step or runtime path, and the environment>
Cause:         <the mechanism, in one or two sentences>
Fix:           <the smallest change that removes the cause>
Check:         <a command or request that fails before the fix and passes after>
Sample note · E2

Failing step: offline pip install in the Python 3.10 image.

Cause: the lock came from Python 3.12, so it omits exceptiongroup, which anyio needs below 3.11.

Fix: compile the lock inside the pinned base image and prepare the wheelhouse again.

Check: the offline build passes, pip check passes, and pip freeze matches the lock.

Rubric
  • The failing step names when it fails: build, start, or a specific request.
  • The cause is a mechanism, not a symptom. "pip failed" is a symptom.
  • The fix changes the inputs or the process, not only the one command.
  • The check fails before the fix. If it also passes on the broken version, it is not a check.