12 Open-Source GitHub Repositories to Learn AI and LLM Engineering From Scratch (Part 1)

You know how to build software. You can work with APIs, debug production issues, design components, and ship features. Then you decide to learn AI engineering, open GitHub, and discover hundreds of repositories covering machine learning, neural networks, transformers, LLMs, fine-tuning, and research papers.

Where do you start?

You could spend weeks watching tutorials and collecting bookmarks without building anything you understand. Or you could pick a random LLM repository, encounter unfamiliar mathematics and PyTorch code, and decide that AI is harder than expected.

The problem is usually the learning sequence.

Learning AI through random GitHub repositories is like learning full-stack development by jumping between Kubernetes, React Server Components, database internals, and compiler design. Every topic might be useful, but the missing prerequisites make progress unnecessarily difficult.

This article gives you a practical path through 12 open-source GitHub repositories, starting with mathematics and AI fundamentals and progressing toward language models, training, and deep learning research.

The goal is to help you understand what each repository teaches, what to study before opening it, and what to build afterward.

This is Part 1 of a two-part AI engineering learning series. We will focus on the foundations here. Part 2 will move toward real-world AI applications and production systems.

You do not need to complete every lesson in every repository. Follow the sequence, work through the relevant sections, and move forward when you can explain what you learned and demonstrate it in code.

Repository: mml-book/mml-book.github.io

Difficulty: Beginner to Intermediate

What it is: A companion repository for the book Mathematics for Machine Learning. It covers mathematical foundations and connects them to machine learning algorithms through examples and notebooks.

Why start here?

Models operate on numbers, vectors, matrices, and probability distributions. Understanding how these pieces work makes later concepts such as embeddings, attention, and gradient descent much easier to follow.

You do not need a mathematics degree. You need enough intuition to understand what the code is calculating.

Prerequisites: Basic algebra and a willingness to work through equations. Previous AI experience is unnecessary.

Focus on: Linear algebra, matrix operations, vector calculus, probability, optimization, and the connection between mathematical concepts and algorithms such as linear regression and principal component analysis (PCA).

Mini-project: Implement linear regression using NumPy. Calculate predictions, measure the error, and implement gradient descent to improve the model.

After completing this stage: You should understand the mathematical operations behind a basic ML model and be able to recognize them in Python code.

Do not attempt to memorize the entire book before moving on. Learn the concepts you need, then revisit them when a model introduces something unfamiliar.

Repository: microsoft/AI-For-Beginners

Difficulty: Beginner

What it is: A beginner-friendly AI curriculum containing 24 lessons, practical labs, and quizzes. It introduces several areas of AI, including neural networks and computer vision.

Why this stage?

After getting some mathematical intuition, you need a broader picture of AI and machine learning. This curriculum helps you distinguish the major approaches before specializing in generative AI.

Prerequisites: Basic programming knowledge. Review the mathematics repository when concepts such as vectors or gradients feel unfamiliar.

Focus on: AI fundamentals, perceptrons, multilayer neural networks, overfitting, deep learning frameworks, and introductory computer vision and NLP topics.

Mini-project: Train a simple image classifier using a framework covered in the course. Evaluate its predictions on examples that were not used for training.

After completing this stage: You should understand how a basic ML workflow works, how neural networks learn from data, and why training accuracy alone does not prove that a model generalizes.

Keep the project small. Your objective is to understand the training process rather than build the most accurate model.

Repository: microsoft/generative-ai-for-beginners

Difficulty: Beginner

What it is: A 21-lesson curriculum that combines explanations of generative AI concepts with practical application-building exercises. Examples include Python and TypeScript implementations where available.

Why now?

You have encountered traditional AI and ML concepts. Now you can investigate generative AI and see how applications use language models to produce responses.

This repository also helps connect AI theory to familiar software development work.

Prerequisites: Basic Python or TypeScript, plus a general understanding of AI and LLMs. You can work through the introductory lessons even if your ML knowledge is limited.

Focus on: LLM fundamentals, model selection, responsible AI, prompt engineering, text generation, open models, fine-tuning, and introductory RAG and AI agent implementations.

Some exercises require API access or a compatible model provider, so check the setup instructions before choosing a lesson.

Mini-project: Build a text-generation application that accepts a user instruction, calls a model, and displays the response. Add handling for invalid input and API failures.

After completing this stage: You should be able to explain how a generative AI application interacts with a model and build a small working application around that interaction.

Repository: dair-ai/Prompt-Engineering-Guide

Difficulty: Beginner to Intermediate

What it is: A collection of guides, papers, lessons, notebooks, and resources covering prompt engineering and related topics.

Why this stage?

Before building complex systems around an LLM, learn how instructions and input structure affect its responses. Developers often start by writing one large prompt and judging the result informally. That makes it difficult to identify why a change improved or damaged the output.

Prompt engineering gives you a more systematic way to investigate model behavior.

Prerequisites: Familiarity with LLM-powered applications from the previous repository.

Focus on: Clear instructions, examples in prompts, structured outputs, reasoning-oriented prompting techniques, and evaluating results across different inputs.

Mini-project: Build a prompt experiment for classifying customer feedback. Compare a simple instruction with a prompt containing examples. Test both on the same dataset and record accuracy, incorrect predictions, and output consistency.

After completing this stage: You should be able to design controlled prompt experiments and explain why you chose a particular prompt structure.

Remember that prompting cannot guarantee factual accuracy or eliminate a model’s limitations. Your experiments should measure results instead of assuming a more elaborate prompt is automatically better.

Repository: mlabonne/llm-course

Difficulty: Intermediate

What it is: A free course for learning large language models through learning roadmaps, notebooks, and practical resources.

The course is organized around LLM Fundamentals, The LLM Scientist, and The LLM Engineer. This makes it useful as a map of the skills involved in understanding and working with LLMs.

Why now?

You have worked with generative AI and experimented with prompts. This is the stage to build a more structured understanding of how language models are created, adapted, and used.

Prerequisites: Basic Python and some familiarity with neural networks. Use the optional fundamentals material to fill gaps.

Focus on: Tokenization, Transformer architecture, attention, model training, fine-tuning, model evaluation, and LLM application development.

Mini-project: Choose an open model and document its tokenizer, context window, intended use, and basic inference behavior. Run a small inference experiment and record what changes when you vary the input or generation settings.

After completing this stage: You should have a clearer mental model of the LLM ecosystem and know which topics require deeper implementation work.

Treat this repository as a learning map. You do not need to complete every notebook before progressing to the next step.

Repository: rasbt/LLMs-from-scratch

Difficulty: Intermediate to Advanced

What it is: The official code repository for Sebastian Raschka’s book Build a Large Language Model (From Scratch). It walks through implementing and training a GPT-like model and working with pretrained models.

Why now?

Using an LLM API hides much of what happens inside the model. This repository gives you a chance to examine those internals directly.

Prerequisites: Comfortable Python programming, basic PyTorch, and an introductory understanding of neural networks. Review matrix operations and gradient descent if necessary.

Focus on: Tokenization, token embeddings, positional information, self-attention, Transformer blocks, next-token prediction, training loops, evaluation, and fine-tuning.

Mini-project: Implement a small GPT-style model using a limited text dataset. Inspect the training loss, generate sample text, and compare outputs before and after training.

After completing this stage: You should be able to trace how text becomes token IDs, passes through a Transformer, and produces probabilities for the next token.

Your model will probably generate poor text at first. That is useful: you can examine the effects of dataset size, training duration, model capacity, and sampling settings.

Repository: FareedKhan-dev/train-llm-from-scratch

Difficulty: Intermediate to Advanced

What it is: A PyTorch implementation of a language-model training pipeline, extending from raw text and tokenization through pretraining, post-training, evaluation, and chat.

Why follow the previous repository with this one?

The previous project teaches the model’s building blocks. This repository gives you another route through the training process, including stages that turn a pretrained model into one better suited to instruction following.

Prerequisites: Python, PyTorch, basic neural network knowledge, and familiarity with tokenization and Transformers.

Focus on: Training data preparation, next-token prediction, loss functions, optimization, supervised fine-tuning (SFT), and post-training methods such as preference optimization.

Mini-project: Start with the repository’s smallest training configuration. Train a tiny model, generate text, and change one training parameter. Compare the loss and output before and after the change.

After completing this stage: You should understand the main steps in training a language model and how post-training differs from pretraining.

Small configurations are useful for learning on limited hardware. Larger experiments can require a capable GPU, so check the repository’s hardware and setup guidance before attempting them.

Repository: karpathy/nanochat

Difficulty: Advanced

What it is: Andrej Karpathy’s compact experimental system for training and working with LLMs. Its pipeline covers tokenization, pretraining, fine-tuning, evaluation, and inference, with a simple chat interface.

Why this stage?

You have seen the individual components. NanoChat gives you the opportunity to study how they fit together in a more complete, end-to-end language-model project.

Prerequisites: Comfortable Python and PyTorch, Transformer fundamentals, and an understanding of model training. Familiarity with GPU training will help.

Focus on: The complete training pipeline, model scaling, evaluation metrics, inference, checkpoints, and the trade-offs between model quality, compute cost, and training time.

Mini-project: Trace a request through the chat interface into model inference. Then run a small supported configuration and record how model size and hardware affect performance.

After completing this stage: You should understand how a compact ChatGPT-like system fits together and which components are needed to train and serve a conversational model.

Do not interpret “small” as “effortless.” NanoChat is an advanced project, and training larger configurations can still require substantial GPU resources. Its CPU and Apple Silicon paths shrink the workload for experimentation, with correspondingly limited results.

Repository: huggingface/ml-intern

Difficulty: Advanced

What it is: An open-source ML engineering agent designed to research topics and work with code across the Hugging Face ecosystem, including model-related documentation, papers, datasets, and compute.

Important status note: The repository is archived, and its hosted application and CLI have been retired. Treat it as a historical case study, not a maintained tool to depend on for current workflows.

Why include it here?

Once you understand models and training, it is useful to examine how an automated tool can coordinate research and ML development tasks.

Prerequisites: Python, Git, basic model concepts, and familiarity with the Hugging Face ecosystem.

Focus on: How the project organizes research and coding tasks, interacts with model infrastructure, and uses tools to perform ML engineering work.

Mini-project: Read the code and trace how a research task moves through the agent. Identify where documentation, model inference, dataset access, and code execution enter the workflow.

After completing this stage: You should be able to discuss the architecture of an ML engineering agent and recognize the operational challenges in automating ML work.

Because the project is retired, move on once you understand its design. Use currently maintained documentation for any implementation you plan to ship.

Repository: harvard-edge/cs249r_book

Difficulty: Intermediate to Advanced

What it is: A textbook and learning resource focused on the engineering principles behind machine learning systems, from individual machines to distributed infrastructure.

Why now?

A model that trains successfully is only one part of an AI system. Model size, memory use, inference latency, throughput, and deployment constraints influence what you can do with it.

This repository helps connect model knowledge to systems engineering.

Prerequisites: Basic ML and neural network knowledge. Familiarity with GPUs and deployment is useful, but you can learn those topics as you go.

Focus on: Training and inference systems, hardware constraints, quantization, memory hierarchies, KV-cache behavior, GPU utilization, and scaling.

Mini-project: Design a simple inference service on paper. Estimate its model memory requirements, identify likely latency bottlenecks, and explain what you would measure before optimizing it.

After completing this stage: You should be able to reason about the costs and constraints that appear when ML models operate as software systems.

Repository: labmlai/annotated_deep_learning_paper_implementations

Difficulty: Advanced

What it is: A collection of documented PyTorch implementations of deep learning papers, including Transformer variants, attention mechanisms, optimizers, generative models, and reinforcement learning algorithms.

Why now?

You have built models and studied training. Now you can compare the concepts in research papers with working implementations.

Prerequisites: PyTorch, neural network fundamentals, and familiarity with attention and Transformers.

Focus on: Translating equations into tensor operations, understanding architecture choices, tracing forward passes, and examining how different methods change a model.

Mini-project: Pick one implementation related to attention or positional encoding. Read the paper’s relevant section, trace the corresponding code, and create a small diagram showing how the tensors flow through the implementation.

After completing this stage: You should be more comfortable moving between research papers and practical code.

Avoid reading every implementation sequentially. Select a paper connected to something you have already built, then use the implementation to resolve specific questions.

Repository: rohitg00/ai-engineering-from-scratch

Difficulty: Beginner to Advanced, depending on the learning path

What it is: A broad, structured curriculum spanning mathematics, machine learning, deep learning, language models, AI engineering, agents, and production topics. It includes lessons and executable code across several programming languages.

Why finish here?

This curriculum can help you connect the topics from the previous repositories and identify gaps in your understanding. It also provides a continuing path toward the application engineering topics covered in Part 2.

Prerequisites: Basic programming. Choose the starting point according to your experience instead of attempting the entire curriculum at once.

Focus on: Revisiting mathematical foundations, ML workflows, Transformers, LLM internals, and the connections between models and larger AI systems.

Mini-project: Rebuild a small language-model experiment from your earlier work. Document the data pipeline, model architecture, training setup, evaluation method, and known limitations.

After completing this stage: You should have a clearer picture of your strengths, the concepts you still need to study, and the next practical project to tackle.

Use the curriculum selectively. It is a substantial resource, so follow the relevant phases rather than treating every lesson as mandatory.

The sequence matters, but your learning method matters just as much.

Use this cycle throughout the roadmap:

Learn → Build → Experiment → Read → Repeat

Read the lesson or documentation for one concept. Focus on understanding the problem it solves and the basic mechanism behind it.

For example, when studying attention, start by understanding why a model needs to weigh relationships between tokens. Then look at the underlying operations.

Write or run a small implementation. Avoid copying code without understanding it.

Change a variable, inspect a tensor’s shape, or add a print statement to follow the computation. Debugging is part of learning the model.

Change one variable at a time and observe the result. Compare training loss, generated text, evaluation results, or inference latency as appropriate.

Record what you expected, what happened, and what you learned. This turns a tutorial into an experiment you can explain later.

Return to the relevant documentation or research paper once you have a concrete question. You will understand the paper differently after seeing the concept in code.

Move to the next concept and revisit earlier ones when a new implementation exposes a gap.

A useful rule is to keep one small project alive throughout your learning journey. Extend it as you learn more rather than starting twelve unrelated projects.

No. You need enough mathematics to understand the task in front of you, and you can develop that understanding alongside practical work.

For example, you can call a hosted LLM API and build a basic application without deriving the Transformer equations. You need a deeper understanding of tokenization, embeddings, attention, and training when you begin inspecting or implementing model internals.

Use two parallel tracks:

As you progress to Repositories #6, #7, and #8, increase the depth of your theory work because you will be implementing the model itself.

“You don’t need to learn everything about AI before building your first AI application. You need to learn the right thing at the right time.”

Use this as your bookmarkable checklist.

By the end of Part 1, you should have a working understanding of fundamental ML concepts, generative AI, prompt engineering, tokenization, Transformer architecture, and the model training process. You should also have practical experience experimenting with small models and reading implementations.

That foundation prepares you for the next stage, where you will learn how to build useful AI-powered software.

Part 2 will focus on:

You do not need to finish all twelve repositories before moving forward. Complete the exercises that match your current level, build something you can demonstrate, and keep notes on what you still need to learn.

Start with Repository #1 today. Work through one concept, implement one small exercise, and move on when you can explain what your code is doing. That will take you further than collecting another hundred AI bookmarks.

I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.