When choosing an LLM for long conversations, the token limit is only part of the capacity question. Keeping conversation history also consumes memory, and growing per-conversation storage can affect how many conversations fit at once.

The AWS Laplace 3B case study published on October 20, 2026 offers a useful way to examine that relationship. Developed by i14, the model keeps recurrent state at 29.5 MB per sequence regardless of context length and uses native PyTorch for training and serving on AWS Trainium2.

If that state stays fixed, what should guide the design of a long-conversation service? Understanding what the model stores helps separate its memory advantage from the performance properties that still need measurement.

KV caches and recurrent state represent history differently

Full-history attention stores keys and values from earlier tokens in a KV cache for reuse during generation. More history means more stored entries, although mechanisms such as sliding windows limit the retained range. The Transformers cache documentation describes these distinctions.

Linear attention can instead be expressed as updates to, and queries against, a matrix state. The GLA paper, first submitted on December 11, 2023, explores hardware-efficient training in this family. It provides background on how parallel training and recurrent execution can represent the computation.

Laplace 3B’s fixed state concerns the representation of history. By comparison, Transformers’ StaticCache reserves a maximum allocation for keys and values in advance. A reserved KV capacity and a recurrent representation serve different purposes, even though both can be described as fixed-size storage. The StaticCache specification makes that distinction useful when comparing designs.

The 29.5 MB figure budgets state per sequence

The 29.5 MB figure does not describe the whole serving process, including model weights and temporary working memory. It describes one sequence’s recurrent state.

Ignoring duplication and management overhead, simple addition of the reported figure gives 29.5 MB × 100 = 2,950 MB for 100 sequences. At 200 sequences, that component doubles. Removing dependence on context length does not remove dependence on concurrency.

A useful capacity model therefore separates the number of sequence states from the space required for weights and execution. This arithmetic illustrates state storage only; it is not a benchmark demonstrating that 100 sequences can run at a particular speed.

Keeping history in bounded space is also a different property from retrieving earlier details accurately. One possible evaluation places a condition near the beginning of a conversation, inserts unrelated turns, and asks about the condition later. This is a suggested adoption test, not a reported Laplace 3B result.

Native PyTorch still leaves kernel and compilation decisions

The execution stack connects the model’s state design to an actual service. The Laplace 3B port replaced GPU-specific Triton kernels with standard PyTorch operations. Native support does not imply that every custom kernel transfers unchanged.

For current platform context, TorchNeuron provides a PyTorch device backend for Trainium. The official PyTorch introduction announced public beta availability on October 20, 2026. As checked on October 21, 2026, the AWS Neuron documentation also identifies the Neuron 2.33 release as public beta. That describes current availability; it does not establish which SDK version produced the case study’s measurements.

The documentation covers eager execution and graph compilation through torch.compile. Changes to shapes or data types can trigger recompilation. A fixed recurrent state therefore does not guarantee an unchanged execution graph.

Requests arriving and completing change the number of sequences processed together, and that batch width is part of the shape. In the Laplace 3B case study, a previously uncompiled width incurred roughly 11 seconds on first use; padding to fixed widths removed that overhead. These are results from the implementation and conditions in the AWS serving study.

Evaluate aggregate throughput separately from user latency

The reported serving result on one trn2.48xlarge is 11,544 output tokens per second across 1,280 concurrent sequences. This is instance-wide throughput, not the generation rate experienced by one user. Keep that distinction when reading the AWS measurements.

For adoption testing, vary context length while holding concurrency fixed, then vary request arrival intervals while keeping context lengths comparable. This helps separate state-capacity behavior from serving behavior. Compare time to first token, delays during generation, and aggregate output with and without initial compilation.

Peak throughput alone cannot establish a cost advantage. Relate operating cost to delivered work under conditions that meet the required response quality and latency.

For long conversations, fixed recurrent state changes the constraint that history storage must expand with context. The remaining questions are whether the model retrieves the information the application needs and whether the service meets its latency targets under expected arrivals. Evaluating capacity, quality, and latency separately turns the fixed-state property into a decision about an actual workload.

Source

• Title: Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch
• URL: https://aws.amazon.com/blogs/machine-learning/beyond-the-kv-cache-training-and-serving-i14s-laplace-3b-linear-attention-model-on-aws-trainium2-with-native-pytorch/