Running a trained model as a service that accepts requests and returns outputs, scheduling computation and memory so that many requests can be answered with acceptable latency and throughput.
BatchingProcessing several requests together in one pass of a model, so that each read of the weights from memory serves more than one sequence, raising total throughput at some cost to the latency of each request.
Cold startThe period after a service or model server starts, before its caches are filled and its kernels compiled or loaded, during which the first requests run slower or behave differently from later ones.
KV cacheThe attention keys and values of the tokens a transformer has already processed, kept in memory so that they are not recomputed for every new token it generates.
Prefix cachingReusing the key-value cache already computed for a prompt prefix, such as a shared system prompt or the earlier turns of a conversation, for later requests that begin with the same tokens.
Speculative decodingA decoding method in which a cheaper drafter proposes several tokens ahead and the target model checks them in one forward pass, accepting a prefix of them without changing the target model's output distribution.