The Secret Life of Words

How machines learned to speak our language.

Behind the apparent intelligence of ChatGPT and its rivals lies a surprisingly simple idea: language can be broken into numbers, patterns can be learned, and the next word can be predicted. Understanding how this works reveals both the extraordinary power and the stubborn limitations of modern artificial intelligence.

Imagine sitting across from someone who has read an extraordinary proportion of the world's books, studied millions of conversations, absorbed vast amounts of computer code and can discuss almost anything at a moment's notice. Ask about Shakespeare, and you get literary criticism. Ask for a computer program, and one appears. Pose a complicated business question, and an apparently thoughtful explanation follows.

It is tempting to conclude that such a conversational partner must understand the world much as humans do.

Yet the technology behind modern large language models (LLMs) operates in a fundamentally different way.

Beneath the fluent sentences and seemingly effortless reasoning lies a mathematical system that processes numbers, calculates relationships and predicts what should come next. It has not acquired language through childhood experience, conversation or human senses. It has learned statistical and structural patterns from enormous collections of data.

The remarkable discovery is that this process can produce something that looks strikingly like intelligence.

To understand why, we need to begin with an unglamorous question: what happens to a sentence when it enters a machine?

Before machines could talk, they had to learn to count

For decades, researchers have pursued the ambition of making computers work with human language. The discipline is called natural language processing, or NLP.

Its early approaches were relatively modest. A computer might classify an email as spam, recognise a person's name in a sentence or determine whether a product review sounded positive or negative.

Some systems relied on dictionaries and carefully written rules. Others used statistics to discover patterns.

Consider two sentences:

"The dog bites the man."

"The man bites the dog."

A simple method known as bag-of-words would treat both sentences as essentially the same collection of words. It could count the vocabulary but would lose the ordering that gives the sentences their very different meanings.

More sophisticated statistical techniques examined sequences of neighbouring words, known as n-grams. By looking at combinations of two or three words, machines could make elementary predictions about what was likely to follow.

These approaches were useful, but language proved too complicated to capture with local patterns alone.

Human meaning depends on context, sometimes separated by many sentences. The word "bank" may refer to a financial institution or the side of a river. A joke can mean the opposite of its literal wording. A pronoun may refer to a person mentioned paragraphs earlier.

The challenge was no longer simply recognising words. It was discovering relationships between them.

The hidden currency of artificial intelligence

Before a language model can analyse a sentence, it must translate that sentence into a representation a computer can manipulate.

This process begins with tokenisation.

A token is a small unit of text. Sometimes it is a complete word. Sometimes it is part of a word, a punctuation mark or even a unit derived from individual bytes.

Take the sentence:

"The company is restructuring."

A tokeniser might divide it into pieces resembling:

"The" | " company" | " is" | " restructur" | "ing" | "."

The exact division depends on the tokeniser. Different models may split the same sentence differently.

Each piece is then assigned a numerical identifier, known as a token ID. These identifiers act rather like catalogue numbers in a vast library. They enable the system to locate the learned numerical representation associated with each token.

Crucially, the ID itself carries no mathematical meaning about the word. Token 2,000 is not twice as important as token 1,000.

The meaning emerges later, when the model processes the token's representation in context.

This apparently mundane step has far-reaching consequences.

A common English word may fit into a single token. A technical database column name, an unusual surname or a phrase in another language might require several.

That affects how much information a model can process, how much computation it requires and, in many commercial applications, how much it costs to use.

The unit of economic activity in many AI systems is therefore not the word, the sentence or even the question. It is the token.

How machines learn to cut words into pieces

There is more than one way to tokenise language.

One approach would be to assign a separate identifier to every word. That sounds straightforward until the machine encounters unfamiliar vocabulary, spelling variations or new technical terms.

Another possibility is to treat every character separately. This avoids much of the unfamiliar-word problem but makes sequences considerably longer.

Modern systems often prefer a compromise called subword tokenisation: frequently occurring text patterns can remain together, while unfamiliar words can be assembled from smaller pieces.

One influential method is Byte Pair Encoding, or BPE. It starts with small units and repeatedly combines pairs that appear frequently in training data. Over time, recurring patterns become entries in a learned vocabulary.

Other approaches, such as WordPiece and Unigram, solve a similar problem through different algorithms. Libraries such as SentencePiece provide frameworks for applying tokenisation methods, including BPE and Unigram.

The broader principle is simple. Rather than teaching a machine every possible word, engineers give it a flexible alphabet of reusable fragments.

It is a bit like assembling language from a box of Lego bricks. Familiar constructions are quick to build. Rarer ones require more pieces.

From a dictionary of numbers to a map of meaning

Having converted text into token IDs, a language model faces a deeper problem.

Numbers alone do not reveal that "Paris" is associated with France, that a loan involves finance or that the same word can carry different meanings.

To make such relationships computable, neural networks use embeddings.

An embedding represents a token as a vector: a collection of numerical values learned during training.

Think of it as assigning each token a position in a space with hundreds or thousands of dimensions. The individual coordinates are not easily interpretable, but together they provide a foundation for recognising patterns.

What makes modern models powerful is that these representations do not remain fixed as a sentence is processed.

Consider:

"The bank approved the mortgage."

"The children played beside the bank."

The word "bank" may start with the same token embedding in both sentences. But the surrounding context changes its representation as the information passes through the model.

The first sentence points towards finance. The second towards geography.

This ability to construct contextual representations marks an important departure from earlier methods that relied heavily on fixed word associations.

It is also worth distinguishing these internal representations from the embeddings used in search systems. A retrieval embedding is typically designed to represent an entire question, sentence or passage so that similar pieces of information can be found. It serves a different purpose from the representation of an individual token inside a language model.

That distinction becomes particularly important when companies try to connect AI models to their own documents and databases.

The invention that changed everything

The major architectural breakthrough behind many modern LLMs was the Transformer.

Its central mechanism, known as attention, allows the model to calculate how information at different positions in a sequence should influence one another.

Imagine reading a long paragraph about a company. It mentions a chief executive, several financial results and a strategic decision. Towards the end comes the sentence:

"She decided to change course."

To interpret "she", a reader must connect the pronoun to the relevant person earlier in the text.

Attention helps a model establish relationships of this kind.

In simplified terms, the mechanism constructs three numerical representations for each position:

• A query, indicating what information a position is seeking.
• A key, indicating how another position might match that query.
• A value, representing information that position can contribute.

The model calculates how strongly queries match keys, converts those scores into weights and uses them to combine information from the corresponding values.

These are mathematical operations, not little agents asking questions inside the computer. But the analogy provides a useful intuition.

Transformers typically use multiple attention heads, allowing different learned relationships to contribute to the resulting representation. They also employ feed-forward neural networks, which further transform the information, and mechanisms that encode the positions of tokens.

By stacking many such layers, a model can develop increasingly elaborate representations of the text it receives.

Not all Transformers work in precisely the same way. Encoder-oriented models such as BERT have often been used to analyse text, while GPT-style decoder models are designed for generating it. Encoder–decoder models, such as T5, combine both patterns for tasks including translation.

The result has been a substantial expansion in what a single model architecture can do.

Instead of building one specialised system to identify names, another to summarise documents and a third to answer questions, developers can now use broadly capable models across many language tasks.

The next-token bet

For all this architectural sophistication, the central training objective of many generative language models is surprisingly straightforward.

Given the text so far, predict the next token.

Take the beginning of a sentence:

"The central bank raised interest…"

A trained model may assign a high probability to a continuation such as "rates".

It does not simply retrieve a stored sentence. It calculates a distribution of probabilities across possible next tokens.

A decoding procedure selects one of them. The selected token is appended to the text, and the process repeats.

One token becomes a sentence. Sentences become paragraphs. Paragraphs become an article, a software program or a lengthy answer.

During pretraining, the system repeatedly encounters sequences of tokens and adjusts its internal parameters to make the observed continuations more probable.

Its mathematical objective is commonly expressed through loss: a measure of how poorly the model predicted the target tokens. Training aims to reduce that loss.

What makes this approach remarkable is the amount of structure that language contains.

To predict the next piece of text well, a model benefits from learning grammar, relationships between ideas, factual associations, conventions of argument and patterns of reasoning.

When trained on code, it can also learn the structures of programming languages. When exposed to scientific writing, it can learn how technical arguments and explanations are constructed.

An apparently narrow prediction task therefore creates pressure to learn a much broader collection of useful capabilities.

But there is a catch.

The ability to produce a plausible statement is not the same as the ability to establish that it is true.

A language model can generate an elegant explanation that contains a factual error. It may produce a convincing but nonexistent reference. It may answer confidently when the available evidence is incomplete.

The training objective rewards accurate prediction of data patterns. It does not, by itself, guarantee external verification.

This is among the most consequential distinctions in modern AI.

Why the same machine can sound cautious or creative

A language model does not always have to select the most probable next token.

When generating text, the system can use different decoding strategies.

One option, known as greedy decoding, repeatedly selects the highest-scoring continuation. Other methods sample from the probability distribution.

A setting called temperature changes the shape of that distribution.

A lower positive temperature makes high-probability tokens relatively more dominant. A higher temperature spreads probability more widely among alternatives.

This helps explain why the same model may produce relatively predictable wording in one setting and more varied language in another.

It also reveals why apparent confidence should not be confused with reliability.

Reducing randomness can make an answer more consistent. It cannot turn an unsupported statement into a verified fact.

Behind the scenes, generation also has practical engineering costs. The model first processes the prompt, a stage called prefill, before generating new tokens during decoding.

To avoid repeating unnecessary calculations, many systems retain temporary attention information in what is known as a key-value cache, or KV cache.

This can improve efficiency, but it uses memory. It is not a long-term knowledge store, nor does it mean the model has permanently learned a conversation.

Such details may sound remote from the experience of chatting with an AI assistant. Yet they help determine response speed, computing requirements and the economics of running a service at scale.

Why companies cannot simply ask an AI to know everything

The limitations of next-token prediction become especially important when artificial intelligence moves from demonstrations into business operations.

Imagine an employee asking a company chatbot:

"What were our most profitable products last quarter?"

A language model may understand the question perfectly well. It may even generate a convincing answer.

But unless it has access to the relevant financial information, that answer has no reliable foundation.

This has led to growing interest in retrieval-augmented generation, commonly shortened to RAG.

The idea is to retrieve relevant information from an external source and supply it to the language model as part of the context for its answer.

A company might search its internal reports for relevant passages, provide those passages to the model and ask it to produce a grounded summary.

This differs from fine-tuning.

Fine-tuning changes the model's learned parameters through additional training. Supplying retrieved documents changes the information available to the model during a particular request.

The distinction is similar to that between teaching an employee a new professional skill and handing that employee the latest financial report.

Both may help the employee perform a task, but they solve different problems.

RAG is particularly useful where information changes frequently. Fine-tuning can be useful when the goal is to adapt recurring behaviour or improve performance on a specialised task.

The approaches can also be combined.

Neither removes the need for evaluation. A retrieval system may fetch irrelevant material, while a model may misinterpret a perfectly accurate document.

When ordinary English becomes a database query

Another increasingly important application involves translating natural-language questions into structured database queries, often called NL2SQL.

A business user might ask:

"Which customers increased their spending most over the past six months?"

The system must understand the question, identify appropriate database tables, determine how they relate and produce a valid SQL query.

That is substantially harder than simply writing fluent English.

Consider a database column called customer_lifetime_value.

To a database engineer, this is a single, meaningful concept. To a tokeniser, it may be several separate text fragments, depending on the vocabulary.

The model must reconstruct the relevance of those fragments while also understanding the surrounding database schema.

And even when it produces SQL that runs successfully, the query may still be wrong.

It might choose an inappropriate date field, join tables incorrectly or misunderstand what the company means by "spending".

The important difference is between syntactic correctness and business correctness.

A query can be valid SQL and still return a misleading answer.

This is why production AI systems require more than a powerful language model. They need accurate schema information, carefully constructed context, validation and tests that compare results against known correct answers.

The model provides a powerful language interface. The surrounding engineering determines whether that interface can be trusted.

The real test of intelligence is not eloquence

The popularity of conversational AI has created an unusual problem for evaluating technology.

A fluent answer is immediately impressive. A correct answer is often harder to recognise.

Researchers may measure language-model performance through prediction loss or perplexity, which reflects how well a model predicts sequences of tokens.

These measures are useful for studying language modelling, but they do not establish that a business application is dependable.

Nor is perplexity necessarily directly comparable across models that use different tokenisation systems.

A financial assistant might need to be evaluated on factual accuracy and appropriate evidence. An NL2SQL system should be judged on whether its queries execute and return correct results. A retrieval system must be assessed on whether it finds the information needed.

There are operational questions too. How quickly does the system respond? How much computing capacity does it consume? Does important context get truncated? What happens when a user asks an ambiguous question?

No single metric answers all of these.

The wider lesson is that a better language model is not automatically a better AI application.

Reliability emerges from the combination of model capability, relevant information, software architecture and careful evaluation.

A new interface to knowledge, with an old problem underneath

The development of language models represents a remarkable change in how people interact with computers.

For much of computing history, humans have had to adapt their intentions to the machine. They learned commands, database languages, programming syntax and rigid interfaces.

LLMs suggest a different arrangement: people can increasingly express intentions in ordinary language, while software translates those intentions into operations.

A manager may ask a question rather than construct a spreadsheet formula. A developer may describe a function rather than type every line. A researcher may explore a document collection through conversation rather than search terms alone.

Yet this shift introduces a new form of dependence.

When an interface is conversational, it becomes easy to confuse the quality of its language with the quality of its knowledge.

The model may sound like an expert even when it lacks the necessary evidence. The challenge for AI engineering is therefore no longer just to produce convincing language. It is to connect language generation to trustworthy information, measurable outcomes and appropriate safeguards.

That requires understanding the entire chain, from tokenisation and embeddings to attention, prediction, retrieval and evaluation.

The individual mechanisms are mathematical. Their consequences are increasingly economic and organisational.

Companies will need to decide when to retrieve information, when to adapt a model, how to manage computational costs and where independent verification matters most.

The most useful AI systems may not be those that appear the most human. They may be those whose designers understand precisely where machine-generated language ends and reliable knowledge begins.

And that is the paradox at the heart of the LLM revolution.

Machines have become astonishingly good at producing human language by learning to predict its smallest pieces. The next challenge is to ensure that those beautifully assembled pieces correspond to something true.

Further reading

• Hugging Face — LLM Course: NLP, Transformer architectures and modern language models.
• Hugging Face — Tokenizer Summary: BPE, WordPiece, Unigram and tokenisation frameworks.
• IBM — Large Language Models: Language-model fundamentals and generation.
• Hugging Face — KV Cache: The engineering behind efficient token generation.
• Hugging Face — Model Training: Fine-tuning and adaptation.