How does the Mistral transformer architecture differ from traditional models?

25 Haz 12:28

Mistral’s main trick is to replace the vanilla self‑attention’s O(N²) cost with a combination of sparse/local windows and a low‑rank global token. In practice that means most tokens only look at a handful of neighbors (like a sliding‑window or block‑sparse pattern), while a small set of “summary” tokens aggregates information across the whole sequence. Classic transformers, by contrast, let every token attend to every other token, which blows up quickly as the sequence grows. The result is a hybrid that keeps the quadratic term down to roughly O(N·k) + O(N·r) (k ≈ window size, r ≈ rank of the global projection), so you get near‑linear scaling without completely giving up long‑range context. The trade‑off is that you lose some flexibility in modeling arbitrary token‑to‑token interactions—especially for patterns that don’t line up with the predefined windows. To mitigate that, Mistral often adds a few cross‑attention layers or rotary embeddings to preserve relative positioning, which helps retain performance on tasks that need global reasoning. When fine‑tuning on limited data, the usual tricks apply: start with a lower learning rate, freeze the early sparse‑attention layers, and use adapters or LoRA modules to inject task‑specific capacity without over‑fitting. Data augmentation (e.g., random cropping of long inputs) and curriculum training—where you first train on shorter sequences and gradually increase length—also help the model make the most of its lightweight architecture.