Large language model

A large language model is a machine learning model trained on very large amounts of text to predict what comes next in a sequence. That single ability — continuing text plausibly — turns out to cover summarising, translating, answering questions, extracting structured data from documents and writing code.

An LLM has no database of facts inside it and does not look anything up unless it is given a tool to do so. It has a statistical sense of how language about a subject tends to go, which is why it is fluent about everything and reliable about nothing in particular. Grounding it in your own data through retrieval-augmented generation is what turns it from an impressive demo into a usable feature.

Cost and latency both scale with the amount of text going in and out, measured in tokens, so an LLM feature that reads an entire document on every keystroke will be both slow and expensive. That constraint shapes the design more than the choice of model does.