Skip to content
The transformer architecture: an encoder and a decoder built from multi-head attention, feed forward and add and norm blocks, leading to a linear layer, a softmax and output probabilities.
Based on Figure 1 of Vaswani et al., Attention Is All You Need (2017).

Premium

Build an LLM from Scratch

Build a small GPT-style language model step by step, from the maths and tokens to a model that follows instructions.

Last updated Estimated reading time: 52minutes

Log in to enrol

Enrolling is free and keeps track of your progress.

Some lessons need Premium

One lesson is free to start now. The remaining 18 lessons are part of Premium membership. Log in if you’re a member, or see what membership includes.

See membershipLog in

About this course

Large language models can feel like magic. This course takes the magic apart by building one yourself, one piece at a time, so you understand what happens between the text you type and the answer you get back.

You start with the maths you need, vectors and matrices, and see how models handled sequences before transformers. Then you turn text into tokens and embeddings, build attention from scratch, and assemble the blocks into a complete GPT-style transformer. Finally, you pretrain the model, generate text with it and fine-tune it to follow instructions.

Who it's for: developers who know some Python and want to understand how LLMs work from the inside, not just how to call them.

What you’ll learn

  • Explain how a large language model turns text into predictions
  • Build a byte-pair encoding tokenizer
  • Implement self-attention and multi-head attention in PyTorch
  • Assemble a GPT-style transformer model
  • Pretrain the model and generate text
  • Fine-tune the model to follow instructions

Course outline

7 sections · 19 lessons

  1. Section 1

    Foundations

    1. 1What you expect in this courseFree preview
    2. 2Math & Statistics NotesPremium, locked
    3. 3Set up Python and PyTorchPremium, locked
    4. 4Intro to NLP & Before transformersPremium, locked
    5. 5Data set discoveryPremium, locked

  2. Section 2

    Intro to transformers

    1. 6Intro to TransformersPremium, locked
    2. 7Inside the encoderPremium, locked
    3. 8Inside the decoderPremium, locked
    4. 9Training and inferencePremium, locked

  3. Section 3

    Text and tokens

    1. 10TokenizationPremium, locked
    2. 11Embeddings and positionsPremium, locked

  4. Section 4

    Attention

    1. 12Self-attention from scratchPremium, locked
    2. 13Causal and multi-head attentionPremium, locked

  5. Section 5

    The transformer

    1. 14The transformer blockPremium, locked
    2. 15Assemble a GPT modelPremium, locked

  6. Section 6

    Pretraining

    1. 16Training data and the training loopPremium, locked
    2. 17Sampling and loading pretrained weightsPremium, locked

  7. Section 7

    Fine-tuning

    1. 18Instruction fine-tuningPremium, locked
    2. 19Evaluate and go furtherPremium, locked