Model Mechanism
Learning state
updated 2026-06-29T20:22
sessions 8 · events 45
built 2026-06-29 20:23
sessions 8 · events 45
built 2026-06-29 20:23
Exit tests — the deliverable
2/8
Depth where studied
68%
12 of 82 concepts engaged
Reasoning bugs resolved
8
1 open
01
The inference loop
The spine. Everything hangs off this.
Tokenization (BPE)
72%
letter-counting failures · JSON corruption · multilingual cost asymmetry
- Initially explained non-English token bloat as 'longer tokens' + 'less efficient algorithm' + 'we need more information' (precision-drift flip + confabulated cause)
1 resolved
Autoregressive loopreview due
70%
sequential latency · streaming · can't revise emitted tokens
1 resolved
Logits → softmax → samplingreview due
70%
nondeterminism · creativity · repetition loops
3 resolved
02
Attention & the transformer
Conceptual, no QKV math.
Self-attention (intuition)review due
69%
O(n²) context cost · order sensitivity · in-context learning
Layers & residual stream
70%
composition with depth
1 resolved
Positional encoding / RoPE
64%
lost in the middle · position bias · context extension
Context window / KV cache
75%
context ≠ memory · cost scaling · context rot
03
Why it behaves that way
The insight layer. Most important.
Hallucination is structuralreview due
69%
confident wrong citations · confidence ≠ correctness
1 resolved
In-context learning
0%
few-shot with no weight updates · induction
no data yet
04
Training, load-bearing parts only
Causal story, no math.
Pretraining
0%
knowledge cutoff · parametric vs retrieved knowledge
no data yet
Post-training (SFT → RLHF/DPO)
0%
assistant persona · refusals · sycophancy
no data yet
05
Variants & edge cases
Practical.
Reasoning / test-time compute
0%
scratchpad tokens · different cost/latency class
no data yet
Mixture of Experts
0%
fast for its size · routing nondeterminism
no data yet
Quantizationreview due
60%
smaller/faster/measurably dumber
Multimodal
0%
images → patch embeddings → tokens
no data yet
Why temp=0 isn't reproducible
0%
floating point · batching · MoE routing
no data yet
06
P1 — Python for a Swift engineer
The deltas that bite a value-types mind — enough to ship the builds.
Object model & reference semanticsreview due
64%
no value types — everything is a reference · = binds a name, never copies · mutable-aliasing & shared-default-argument bugs · is vs == ; copy/deepcopy when you need a struct
Dynamic typing, hints & Nonereview due
72%
runtime typing — type errors surface when hit, not at compile · hints are optional and unenforced (mypy/pyright to enforce) · None and X | None instead of Optional; no optional chaining · duck typing — shape over declared conformance
Collections, comprehensions & slicing
0%
list/dict/tuple/set vs Array/Dictionary/Set · comprehensions instead of map/filter chains · slicing & negative indices · unpacking, *args / **kwargs
no data yet
Iterators, generators & lazy pipelines
0%
generators (yield) for memory-bounded streams · the iterator protocol · lazy vs eager — when each bites · itertools over hand-rolled loops
no data yet
Exceptions & control flow
66%
unchecked exceptions, not typed throws / Result · try / except / else / finally · EAFP (ask forgiveness) over guard-let LBYL · raising & defining exception types
1 resolved
Classes, dunders & dataclasses
0%
__init__ / __repr__ / __eq__ and friends · @dataclass as the struct stand-in (still a reference) · duck typing & Protocol vs ABC · no access control — convention, not enforcement
no data yet
Decorators & context managers
0%
reading @decorator (every AI/web lib uses them) · with-blocks for deterministic cleanup (RAII-like) · __enter__ / __exit__ · writing a simple decorator / context manager
no data yet
async/await, the event loop & the GIL
0%
asyncio event loop vs Swift structured concurrency · async def / await / gather for concurrent API calls · the GIL — threads don't parallelize CPU; no actors · a blocking call stalls the whole loop
no data yet
Packaging, venvs & the toolchain
0%
venv / uv & pip vs SPM · pyproject.toml, imports & package layout · ruff / black / mypy in the loop · running scripts vs modules (-m)
no data yet
Pydantic & validation at the edges
0%
models as the typed I/O boundary · validate untrusted input where it enters · structured LLM output → validated JSON (B1) · settings / config via BaseSettings
no data yet
07
L2 — Context & prompt engineering
Get the most from the window without touching weights.
Prompt structure & roles
0%
system/user/assistant roles · instruction placement · delimiters & formatting
no data yet
Few-shot / in-context examples
0%
example selection · format consistency · when examples beat instructions
no data yet
Chain-of-thought & decomposition
0%
scratchpad reasoning · task decomposition · when CoT helps vs hurts
no data yet
Structured output
0%
function calling / JSON mode · schema-constrained decoding · validity failure modes
no data yet
Context engineering
0%
what to put in the window · ordering vs lost-in-the-middle · retrieval vs stuffing
no data yet
Prompt injection (intro)
0%
untrusted input in the prompt · instruction-override risk
no data yet
Automated prompt optimization
0%
DSPy-style optimization · eval-driven prompt search · stop hand-tuning
no data yet
08
L3 — Retrieval / RAG
Knowledge from the corpus, not the weights.
Embeddings & vector space
0%
semantic similarity as distance · embedding models · dimensionality
no data yet
Chunking strategies
0%
size & overlap · semantic vs fixed · chunk boundaries
no data yet
Vector stores & ANN
0%
approximate nearest neighbour · index types · recall vs latency tradeoff
no data yet
Hybrid search
0%
BM25 + dense · when lexical beats semantic
no data yet
Reranking
0%
cross-encoders · two-stage retrieve-then-rerank · why a second stage
no data yet
Retrieval evaluation
0%
recall@k · MRR · context relevance
no data yet
RAG failure modes
0%
irrelevant retrieval · stale index · context dilution
no data yet
Document ingestion & parsing
0%
PDF/HTML/code parsing · OCR & tables · cleaning: garbage-in, garbage-out
no data yet
Query transformation
0%
HyDE · multi-query & decomposition · query rewriting
no data yet
Metadata filtering
0%
structured filters + semantic search · access control in retrieval
no data yet
Advanced retrieval patterns
0%
contextual retrieval · GraphRAG · parent-doc / small-to-big
no data yet
09
L5 — Evaluation
The differentiator. A lens, not a final topic — introduced early, applied everywhere.
The eval mindset
0%
offline vs online · measure before/after · no eyeballing
no data yet
Golden / eval-set construction
0%
coverage · edge cases · labeling quality
no data yet
LLM-as-judge
0%
rubric prompting · judge biases (position/verbosity/self-preference) · when judges fail
no data yet
Task metrics
0%
exact vs semantic match · rubric scoring · pass@k
no data yet
Regression testing / CI gates
0%
prompt/chain regression suites · CI gates · catching silent drift
no data yet
Online eval & monitoring
0%
production monitoring · drift detection · A/B testing
no data yet
Evaluating RAG & agents
0%
component vs end-to-end · trajectory evaluation · beyond single completions
no data yet
Build an eval harness
0%
30+ case harness · error bars · before/after deltas
no data yet
Synthetic data generation
0%
generating eval cases · augmentation & edge-case mining · data for tuning
no data yet
The eval data flywheel
0%
mine production traces into tests · failure-driven test growth · continuous eval-set expansion
no data yet
10
L4 — Agents & orchestration
Tools, loops, memory — and when a single call is better.
Tool use / function calling
0%
tools as agency · tool schemas · tool selection
no data yet
ReAct & plan-then-execute
0%
reason-act loops · plan then execute · when to stop
no data yet
State & memory
0%
working vs persistent memory · context is not memory at the app layer · when each is needed
no data yet
Multi-agent patterns
0%
decomposition across agents · when one agent is better · coordination cost
no data yet
Orchestration & MCP
0%
graphs / state machines (LangGraph) · MCP tool/context protocol · deterministic vs model-driven control
no data yet
Agent failure modes
0%
loops · runaway cost · error propagation · silent wrong-tool
no data yet
Human-in-the-loop & approvals
0%
approval gates before actions · confidence-based escalation · review queues
no data yet
11
L6 — Inference ops / production
Bounded latency & cost, graceful degradation, fully traced.
Latency budgeting & token accounting
0%
prefill vs decode · time-to-first-token vs total · token budgets
no data yet
Cost modeling
0%
per-request/user/at-scale · input vs output token cost · context cost scaling
no data yet
Streaming
0%
token streaming UX · partial parsing
no data yet
Caching
0%
prompt caching · semantic caching · when each applies
no data yet
Reliability
0%
retries/timeouts/fallbacks · circuit breaking · structured-output reliability at scale
no data yet
Rate limits & batching
0%
rate-limit handling · batching · throughput
no data yet
Observability / tracing
0%
spans · token + cost per span · tracing chains & agents
no data yet
Model selection & routing
0%
cascades: cheap-first, escalate · fallback chains · which model per call
no data yet
Serving open-weight models
0%
vLLM / TGI · self-host vs API tradeoff · throughput basics
no data yet
Deployment & CI/CD for AI
0%
prompt/chain versioning · staged rollout · gating deploys on evals
no data yet
12
L8 — Safety & guardrails
Adversarial input is the default, not the exception.
Prompt injection & jailbreaks (defense)
0%
direct vs indirect injection · defense patterns · isolating untrusted content
no data yet
Output validation & refusals
0%
schema enforcement · refusal handling · fail-closed
no data yet
PII & data governance
0%
PII detection/redaction · retention policy · logging hygiene
no data yet
Content moderation
0%
moderation layers · policy enforcement
no data yet
Adversarial robustness basics
0%
attack-surface mapping · red-teaming mindset
no data yet
Hallucination mitigation & grounding
0%
forced citations · abstention / I-don't-know · verification passes
no data yet
13
L7 — Adaptation / fine-tuning
Lowest priority for the app layer; knowing when NOT to is the skill.
Fine-tune vs RAG vs prompt
0%
the decision framework · cost/benefit framing · when each wins
no data yet
SFT and LoRA / PEFT (conceptual)
0%
what SFT changes · LoRA/PEFT idea (no math) · adapter swapping
no data yet
Preference tuning / DPO (conceptual)
0%
preference data · DPO vs RLHF idea (no math)
no data yet
Distillation
0%
teacher to student · why distill
no data yet
Data curation for tuning
0%
dataset quality · data beats technique
no data yet
Exit test — complete at 8/8
✓Explain mechanistically why a model produces a confident, wrong citation1 attempt
◇ a cold, no-notes verbal explanation that locates the flat/near-uniform distribution over content tokens, states nothing wires that uncertainty to a hedge, cites the post-training penalty on 'I don't know', and concludes confidence != correctness
Hallucination is structuralLogits → softmax → samplingPretraining
et1 marginal pass. Cold first pass (6.5): had form-vs-content (peaked vs flat) and no-truth-only-probabilities, but dropped the frozen-weights/corpus-gone root and gave the shallow 'incentivized to be confident' for the confidence crux. On one redirect, produced the missing-wire decoupling himself cleanly (flat distribution not connected to the hedge decision; 'no process expresses that the token was picked from a flat distribution') AND added an unprompted insight: a guessed token gets appended and baked in as truth, conditioning the rest of the answer. Substance is all there; the supplied piece (frozen weights) he's independently nailed cold earlier this session. Pass = test-time completeness lesson, not a knowledge gap.
✓Given two prompts, predict which costs more and why1 attempt
◇ a correct cold prediction with the mechanism — quadratic attention cost in token count (one 1000^2 vs ten 100^2), input-vs-output token pricing, and context/window growth
Tokenization (BPE)Self-attention (intuition)Context window / KV cache
passed et2 (cost prediction): B, ~200x tokens; reconciled linear per-token bill vs super-linear attention compute (latency/GPU), plus the sharp insight that per-token pricing is a deliberate linear abstraction over quadratic compute. came out in pieces under prompting - worth a cold re-confirm later; underlying n^2 mechanism already solid from M2
○Explain why the same prompt at temp=0 returned two different answers
◇ a cold explanation that temp=0 is greedy, not deterministic — float non-associativity under batching/parallelism (and MoE routing) flips the argmax between near-tied logits
Logits → softmax → samplingWhy temp=0 isn't reproducible
2 concepts to go○Explain why last week's fact isn't in the model but works once pasted into context
◇ a cold explanation that the fact isn't in the frozen pretrained weights (knowledge cutoff) but works once pasted because attention over the in-context tokens conditions the answer (in-context learning), with no weight change
PretrainingContext window / KV cacheIn-context learning
3 concepts to go○B1 (build): turn messy dev artifacts (logs, stack traces, API docs) into validated JSON at ~100%, no fine-tuning
◇ a CLI/service that turns messy dev artifacts (logs, stack traces, API docs) into schema-validated JSON, with a small eval showing ~100% validity on a held-out set and no fine-tuning — committed repo + eval output
Structured outputContext engineeringObject model & reference semanticsExceptions & control flowPydantic & validation at the edges
5 concepts to go○B2 (build): docs/code RAG over a real repo with a retrieval eval harness that proves a measured improvement
◇ a docs/code RAG over a real repo plus a retrieval-eval harness (recall@k on a labeled set) that proves a measured before/after improvement from one change — committed repo + eval numbers
RAG failure modesRetrieval evaluationAdvanced retrieval patternsThe eval mindset
4 concepts to go○B3 (build): a dev-tools agent (PR review / log triage) with human approval gates that recovers from tool failure
◇ a dev-tools agent (PR review / log triage) with human approval gates before actions and recovery from an injected tool failure — committed repo + a trace showing the gate firing and a recovered run
Agent failure modesTool use / function callingHuman-in-the-loop & approvalsPrompt injection & jailbreaks (defense)
4 concepts to go○B4 CAPSTONE: ship a cloud service from B2/B3 - model routing, caching, full tracing, a 30+ case eval gating CI, and a documented before/after metric
◇ a deployed cloud service (model routing + caching + full tracing) with a 30+ case eval gating CI and a documented before/after metric — live endpoint + CI config + the metric writeup
Build an eval harnessModel selection & routingDeployment & CI/CD for AIObservability / tracingOnline eval & monitoring
5 concepts to goBug patterns (tutor's read)
depth-as-procrastinationdormanttreats 'I could go deeper here' as a reason to stay on an already-mastered concept; optimization bias operating on the syllabus instead of the next real gap
completion-seekingdormantwants full coverage of everything before moving on; control preference. Redirect to the exit test / deliverable, not coverage
premature-convergence×2closes options / commits to one explanation before testing alternatives; efficiency over exploration