Concept

Batching

Processing several requests together in one pass of a model, so that each read of the weights from memory serves more than one sequence, raising total throughput at some cost to the latency of each request.