Large Language Model (LLM)
Definition. A neural network trained on very large text corpora to predict the next token in a sequence, which after instruction tuning and alignment behaves as a general-purpose text transformation engine. Architecturally almost all production LLMs are decoder-only transformers with billions of parameters, exposed either as a hosted API or as locally served weights. The model has no persistent memory, no built-in factual database, and no ability to act on the world except through whatever text it emits.
How it works. You integrate an LLM by sending a request containing a system prompt, a message history, and generation parameters (max tokens, temperature, stop sequences) to an inference endpoint, then consuming the streamed or batched response. Production integration adds a token budget calculation before the call, a timeout, a retry policy for 429 and 5xx responses, and a validation layer on the output. For deterministic business logic you constrain the output with a JSON schema rather than parsing free text.
Example. A support platform receives 40,000 tickets a month. Each ticket is sent to a mid-tier model with a 900-token system prompt defining twelve categories and a strict JSON schema. The response is parsed into {category, urgency, suggested_reply}. At roughly 1,100 input tokens and 180 output tokens per ticket, and a blended price of $0.60 per million input and $2.40 per million output tokens, the monthly cost is about $30 for input and $17 for output, well under the cost of the two agents previously triaging by hand.
Use case. Use an LLM wherever the task is unstructured language in and structured or semi-structured language out: summarisation, classification, extraction, rewriting, translation, drafting, and conversational interfaces. It is the correct tool when the rule set is too large or too fuzzy to express as code, and the wrong tool when a deterministic function or a SQL query would answer exactly and cheaply.
Warnings. Never place an LLM in a path where a wrong answer is silently accepted. Treat every output as untrusted input: validate types, ranges and enumerations before it reaches a database, a payment call, or a customer. Do not send personal, regulated or secret data to a third-party endpoint without a data processing agreement and a documented retention policy.
Limitations. LLMs hallucinate, are sensitive to prompt phrasing, drift when the provider silently updates the model behind an alias, and cannot reliably do exact arithmetic or count characters. Context is finite, cost scales with tokens, and latency is dominated by output length. They are also non-deterministic even at temperature zero, so any workflow requiring byte-identical output needs caching or a deterministic post-processor.
| Pros | Cons |
|---|---|
| Converts varied language inputs into structured output. | Requires output validation and has variable token cost and latency. |