Back to blog Back to Home

What Happens When You Talk to an AI? Tokens, Transformers, and Temperature

9 min read
AI LLM Transformers Web Development

The rapid evolution of Generative AI has created two clear roles in tech: Model Creators and AI Application Engineers. Knowing the difference between them helps any developer understand how modern AI systems are built.

We can understand these roles using a simple analogy — an electrical power network:

The Power Station Side (Model Creation)

Machine Learning (ML), NLP, and Computer Vision (CV) engineers work on the supply side. Their job is data, raw computing power, and training huge models from scratch (like GPT-4 or Gemini) — just like engineers who build a power plant.

The Home Side (Application Engineering)

Full-stack and frontend developers work on the other side — the side that uses this power. A developer doesn’t need to know how a generator works inside; they just need a working socket. As application engineers, our job is to build the “appliances” — apps, UIs, and features — that turn this raw AI power into something useful for people.

The wall socket connecting these two sides is the API (Application Programming Interface).

So as application engineers, we don’t need to build the model ourselves — but it helps to know roughly how it works behind the scenes. That’s what the rest of this blog covers.

1. What is a Large Language Model (LLM)?

  • Definition: An LLM is a deep learning model trained on huge amounts of data to understand, generate, and predict text.

  • Why it’s useful: Normal programs follow strict rules (if/else). They can’t handle messy human language well. LLMs solve this by understanding natural language at scale.

  • Examples: GPT-4 (OpenAI), Gemini (Google), Claude (Anthropic), and Llama (Meta, open-source).

  • Where we use them daily: Code completion tools like GitHub Copilot, chatbots, autocomplete, and document writing tools.

Breaking Down “GPT”

  1. Generative: It doesn’t copy answers from a database — it creates a new answer every time, word by word.

  2. Pre-trained: It learns language patterns from huge amounts of text before it’s ever used.

  3. Transformer: The architecture (design) that powers how it reads and understands text.

2. What Happens When You Send a Message to ChatGPT?

When you type something and hit send, a few steps happen instantly behind the scenes:

Step 1 — Typing the Prompt

You type your message in the chat box and hit send. The app packages this text and sends it over the internet.

Step 2 — Processing the Message

Before the model can “think,” your message goes through a translation process:

  • Tokenization: Your text is broken into small pieces (tokens) and each piece is given a number. For example, the word “I” might become the number 343.

  • Vector Embeddings: These numbers are placed on a big multi-dimensional map. Words with similar meanings sit closer together on this map.

  • Positional Encoding: The model also tracks word order, so it knows the difference between “dog bites man” and “man bites dog.”

Step 3 — Generating the Response

Now the model predicts the next word, one at a time, in a loop. It looks at everything written so far, picks the most likely next word, adds it, and repeats — until it reaches a stop signal that ends the answer.

Step 4 — Why It’s Not Just Copy-Pasting

A common myth is that ChatGPT searches the internet and copies an answer. That’s not true — it doesn’t pull from a live database.

Think of it like a person who has read millions of books. It has learned patterns in language, and when you ask something, it predicts the answer word by word based on what it learned — not by retrieving a saved copy of anything.

3. Why Computers Don’t Understand Human Language

  • Text vs Numbers: Computers only understand 0s and 1s. They can’t directly understand a word like “spoon” — it has no meaning to a chip on its own.

  • Why we need numbers: Computers are just calculators — they can only do math. So before any text can be processed, it must be turned into numbers. That’s how a sentence becomes something a computer can actually work with.

4. Tokenization

Before text reaches the model, it needs to be broken down into small pieces. This step is called Tokenization.

  • What tokens are: The smallest building blocks of text that a model understands — like pieces of a vocabulary list.

  • Why we need it: A model can’t process a full sentence directly. Breaking it into tokens with number IDs is what makes it possible to do math on language.

  • Words vs Tokens: Common words like “spoon” are usually one token. Rare words, names, or code often get split into smaller pieces.

Testing Tokenization in JavaScript

In class, we tested this using OpenAI’s tiktoken library:

npm init -y
npm install tiktoken

import { getEncoding } from "tiktoken";

const encoder = getEncoding("gpt2");
const sampleText = "Hello, I am Piyush Garg";

const tokenArray = encoder.encode(sampleText);
console.log("Encoded Tokens:", tokenArray); 
// Output: [15496, 11, 314, 716, 21915, ...]

const compiledText = encoder.decode(tokenArray);
console.log("Decoded Output:", compiledText);

5. Transformers

Once text becomes numbers, it enters the Transformer — the core design behind most modern AI models. It came from a 2017 paper called “Attention Is All You Need.”

What a Transformer Is

A network design built to process full sequences of text at once, instead of word by word.

Why It Changed AI

  • Old way: Models read text one word at a time, left to right. On long sentences, they’d often forget how it started.

  • New way: Transformers use Self-Attention — they look at the entire sentence at once, instead of one word after another.

How It Understands Meaning

Take the word “BANK”:

  • “I deposited money at the ICICI BANK” (a well-known Indian bank).

  • “We sat by the RIVER BANK.”

A Transformer looks at nearby words (“money” or “river”) to figure out which meaning fits — all at once, not one word at a time.

Feed Forward Network

Once self-attention figures out which words relate to which, that information passes through another small network called the Feed Forward Network. Think of self-attention as gathering context, and the feed forward network as refining it — taking another pass over the information before the model makes its final prediction.

Softmax — Turning Scores into a Choice

At the very end, the model has to pick the next token. But it doesn’t just have one guess — it has a probability score for every possible word in its vocabulary. Softmax is the math step that turns these raw scores into clean probabilities that all add up to 100%.

For example, after the sentence “The sky is…”, the scores might look like:

  • Blue → 92%

  • Pink → 5%

  • Green → 2%

  • Purple → 1%

The model then picks from these probabilities — and this is exactly where temperature (covered below) comes in: it decides whether the model almost always picks the safest option (Blue) or occasionally takes a riskier pick.

Why Almost Every LLM Uses Transformers

  • They process everything in parallel, so GPUs can train them very fast.

  • They’re great at remembering context over long text.

  • GPT, Claude, and Gemini are all built on this design.

6. Controlling the Engine: Context Window and Temperature

Developers can control how the model behaves using two settings:

Context Window (Short-Term Memory)

This is the model’s short-term memory — the max number of tokens it can “see” at once. Once a conversation gets too long, older messages get pushed out and the model forgets them.

Temperature

This setting (between 0.0 and 1.0) controls how predictable or creative the model’s answers are.

  • Low temperature (e.g. 0.2): Picks the safest, most likely words. Good for code or math.

  • High temperature (e.g. 0.9): Takes more risks with word choice. Good for stories or brainstorming.

7. Training vs Inference

Every LLM goes through two very different phases in its life.

Training

This is when the model actually learns. It’s shown a piece of text, asked to predict the next token, and then checked against the real answer. When it gets it wrong, it slightly adjusts its internal settings (this adjustment process is called backpropagation). This cycle repeats billions of times, over huge amounts of text, until the model gets good at predicting language. This phase happens once, before the model is ever released to the public, and takes a massive amount of computing power.

Inference

This is what happens every time we use ChatGPT, Claude, or Gemini. The model is no longer learning or adjusting anything — it’s simply using everything it already learned during training to predict the next token, again and again, until your answer is complete. No backpropagation, no learning — just prediction, using a brain that’s already fully trained.

Conclusion

Generative AI is moving fast, and it’s a great time to be an application developer. By understanding this basic pipeline — text becoming tokens, tokens becoming numbers on a map, and numbers flowing through a Transformer — we can build on top of these models with more confidence.

Going back to our power station analogy: we live on the home side, not the power plant side. But knowing how the electricity is made — tokenization, embeddings, self-attention, softmax, and the difference between training and inference — helps us build better appliances on top of it.