Gürkan Özkan
Domain

AI

Models, systems and the practice of bringing AI into real work.

Map of the domain: AIAI — Models, systems and the practice of bringing AI into real work.AIFormal verification — Establishing with a mechanical checker, such as the kernel of a proof assistant like Lean, that a statement or a program meets a precise mathematical specification.Formal verificationMachine learning — The field that studies how computers improve their performance on a task by learning patterns from data rather than being explicitly programmed.Machine learningLarge language models — Neural network models, typically with billions of parameters, trained on very large text corpora to predict the next token.Large language modelsHybrid architectures — Language model designs that interleave attention layers with layers of another kind, typically state-space or linear-attention layers, to lower the cost of long contexts.Hybrid architecturesInference serving — Running a trained model as a service that accepts requests and returns outputs, scheduling computation and memory so that many requests can be answered with acceptable latency and throughput.Inference servingBatching — Processing several requests together in one pass of a model, so that each read of the weights from memory serves more than one sequence, raising total throughput at some cost to the latency of each request.BatchingCold start — The period after a service or model server starts, before its caches are filled and its kernels compiled or loaded, during which the first requests run slower or behave differently from later ones.Cold startKV cache — The attention keys and values of the tokens a transformer has already processed, kept in memory so that they are not recomputed for every new token it generates.KV cachePrefix caching — Reusing the key-value cache already computed for a prompt prefix, such as a shared system prompt or the earlier turns of a conversation, for later requests that begin with the same tokens.Prefix cachingSpeculative decoding — A decoding method in which a cheaper drafter proposes several tokens ahead and the target model checks them in one forward pass, accepting a prefix of them without changing the target model's output distribution.Speculative decodingLLM agents — Systems in which a language model works in a loop, choosing actions such as tool calls, observing their results and deciding the next step towards a goal.LLM agentsMulti-agent systems — Arrangements of several agents, usually separate instances of one or more language models, that divide, share or coordinate the work on a task.Multi-agent systemsMixture of experts — A model design in which a router sends each token to a small subset of specialised sub-networks, the experts, so that only a fraction of the total parameters is active for any one token.Mixture of expertsRepeated sampling — Drawing many independent attempts from a model for the same problem and keeping any that a verifier accepts, so that the share of problems solved grows with the number of attempts.Repeated samplingSampling temperature — A parameter that rescales a model's output probabilities before a token is drawn; lower values make generation more deterministic, and a temperature of zero always takes the most likely token (greedy decoding).Sampling temperatureState-space models — Sequence models, such as Mamba, that carry a fixed-size recurrent state through a sequence instead of attending to every earlier token, so that their memory per sequence does not grow with context length.State-space modelsModel evaluation — Methods and metrics used to estimate how well a model will perform on data it has not seen.Model evaluationBenchmark saturation — The state of a benchmark on which current models score so close to the maximum that it no longer separates them.Benchmark saturationData leakage — Information that would not be available in real use reaching the model during training or evaluation, making performance look better than it is.Data leakageEvaluation harness — The software that runs a model on a benchmark: it builds the prompts, calls the model, executes or scores the outputs and aggregates the results.Evaluation harnessQuantisation — Storing a model's weights, and sometimes its activations or cache, with fewer bits per number, such as 8 or 4 instead of 16, to cut memory use and speed up computation at some possible cost in accuracy.QuantisationMemory bandwidth — The rate at which data can be moved between memory and the processor, usually given in gigabytes per second; generating text one token at a time is often limited by it rather than by arithmetic.Memory bandwidthUnified memory — A memory architecture in which the CPU and the GPU share one physical pool of memory instead of having separate system and device memory.Unified memoryStatistical power — The probability that a test detects an effect of a given size when the effect is really there; it grows with the sample size and the size of the effect.Statistical powerFifteen ways my coding agent and I mis-measured two models on one DGX Spark — Before you compare models on your own hardware, fix the sampling, the token budget, the harness and the number of runs: on one DGX Spark, three of the four moved a score by more than the gap between the models.Fifteen ways my coding agent and I…Small agent teams win by trying more, not by coordinating — A six-worker GLM-5.3 team proved more Lean statements than one agent of the same model, but four copies that could not talk to each other matched it. Here there is no evidence that coordination adds: try independent attempts with a verifier first.Small agent teams win by trying more, not…Speculative decoding on GB10: a wider draft speeds up one stream and slows sixteen — Choose the speculative-decoding draft width for the number of users you serve: in darkdatter's sweep on a DGX Spark, 16 draft tokens instead of 8 made one stream 28% faster and sixteen streams 10% slower.Speculative decoding on GB10What 128 GB of unified memory buys you: serving LLMs on a DGX Spark — On a GB10 the GPU memory fraction is a share of the whole machine: size the KV cache from the workload, and expect bandwidth and prefill, not capacity, to set the limits.What 128 GB of unified memory buys youThe 1,802-second hang: serving a hybrid linear-attention MoE under agent load — In July I ran coding-agent evaluations on SWE-bench Verified against Qwen3.6-35B-A3B, served by vLLM 0.21 ( ) on one DGX Spark, four requests at a time. The…The 1,802-second hang
Map of the domain

Concepts

AI

  1. Formal verification1 essay
  2. Machine learning
  3. Large language models1 essay
  4. Hybrid architectures1 essay
  5. Inference serving2 essays
  6. Batching2 essays
  7. Cold start
  8. KV cache2 essays
  9. Prefix caching1 essay
  10. Speculative decoding1 essay
  11. LLM agents1 essay
  12. Multi-agent systems1 essay
  13. Mixture of experts1 essay
  14. Repeated sampling1 essay
  15. Sampling temperature1 essay
  16. State-space models1 essay
  17. Model evaluation3 essays
  18. Benchmark saturation1 essay
  19. Data leakage1 essay
  20. Evaluation harness1 essay
  21. Quantisation1 essay
  22. Memory bandwidth2 essays
  23. Unified memory1 essay

Essays

Notes