12 Open-Source GitHub Repositories to Become an AI Engineer: Agents, RAG, MCP & System Design (Part 2)

You can build a chatbot in an afternoon. You can connect an LLM to an API, create a chat interface with React, and deploy the application before dinner.

Then someone asks you to make it reliable.

How will it answer questions using private company documents? What happens when the model calls the wrong tool? How do you prevent unauthorized data access? What happens when 1,000 users send requests simultaneously? How will you know whether a new prompt or model version has made the application worse?

Suddenly, prompting is the smallest part of the problem.

In Part 1 of this series, we explored the foundations of AI engineering, from mathematics and machine learning to Transformers, LLMs, and training. Those foundations help you understand what happens inside a model.

Part 2 takes the next step: learning how to build useful software around models.

I’ve organized these 12 open-source GitHub repositories into a learning path that moves from API integration to RAG, tool calling, AI agents, application development, and production system design.

You don’t need to master every repository. You need to understand the concepts, build projects that use them, and learn how to make those projects dependable.

The order is intentional. Learn how to make a model call first. Add retrieval and tools next. Introduce agents only when a fixed workflow is insufficient. Then study the architecture required to operate your application reliably.

Repository: openai/openai-cookbook

Difficulty: Beginner to Intermediate

The OpenAI Cookbook contains code examples and guides for working with the OpenAI API. It covers practical implementation patterns that you can adapt to your own applications, with much of the example code written in Python.

What to learn?

Start with API requests, conversation handling, embeddings, structured outputs, tool calling, and the examples related to agents and retrieval.

Pay particular attention to error handling, input preparation, and how the application consumes a model’s response.

Prerequisites: Basic programming, HTTP APIs, JSON, and environment variables. You will need API access for examples that call hosted models.

Build: Create a small AI support assistant that classifies a user request and returns a validated JSON object containing its category, priority, and suggested next action.

Afterward, you should be able to integrate an LLM into a backend service and handle its output programmatically.

Why start here? It introduces the boundary between your application code and the model. That boundary will remain important when you add retrieval, tools, and agents.

Repository: anthropics/claude-cookbooks

Difficulty: Beginner to Intermediate

Claude Cookbooks contains notebooks and recipes demonstrating ways to use Claude. Its examples cover capabilities such as classification, summarization, retrieval-augmented generation, and tool use. Much of the code uses Python.

What to learn?

Explore the same application problem you solved with the previous repository, then study how a different model and API represent inputs, outputs, and tool interactions.

Prerequisites: Basic API integration and JSON handling.

Build: Implement a document summarizer with explicit output requirements. Test it on several documents and compare its results with your first implementation.

Afterward, you should understand which parts of an AI application belong to your application architecture and which depend on the model provider.

This is also a good place to practice separating model-specific code from your business logic. A provider adapter makes it easier to test a second model without rewriting the entire application.

Repository: Shubhamsaboo/awesome-llm-apps

Difficulty: Beginner to Advanced, depending on the example

This collection contains open-source LLM applications, RAG implementations, AI agents, multi-agent projects, and MCP integrations. It is particularly useful when you want to see how individual techniques fit into working applications.

What to learn?

Start with a basic RAG application that answers questions about documents.

A typical RAG pipeline works like this:

Then investigate where retrieval fails. A model cannot give a grounded answer if the retriever returns irrelevant passages or misses the information needed to answer the question.

Prerequisites: LLM API integration, basic Python, and a general understanding of databases.

Build: Create a chat application that answers questions about your own technical documentation and links each answer to its source.

Afterward, you should understand the main components of a RAG application and be able to trace a question through ingestion, retrieval, and generation.

Important: Evaluate retrieval quality separately from answer quality. That distinction makes debugging much easier.

Repository: huggingface/agents-course

Difficulty: Intermediate

The Hugging Face Agents Course is a structured course that progresses from agent fundamentals to agent frameworks and practical assignments. Its material covers tools and frameworks including smolagents, LangGraph, and LlamaIndex, with relevant material on integrating MCP tools.

What to learn?

Focus on the relationship between an LLM, the tools it can use, and the application controlling its execution.

For example, a customer-support agent might need to retrieve an order, check its delivery status, and draft a response. The model can choose a tool, but your application should validate its arguments and enforce permissions before performing the action.

Prerequisites: API integration, Python, and basic RAG concepts.

Build: Create a support agent with two read-only tools: search documentation and retrieve an order’s status.

Afterward, you should understand tool calling, agent execution, framework abstractions, and the importance of controlling what an agent can do.

This is also a good starting point for understanding MCP. MCP standardizes how compatible clients and servers expose tools and other context. It does not automatically make tool calls safe or give an agent permission to access everything.

Repository: NirDiamant/GenAI_Agents

Difficulty: Intermediate to Advanced

This repository contains tutorials and implementations covering generative AI agent techniques, from conversational bots to more complex multi-agent systems.

What to learn?

Study how agents manage state, choose tools, coordinate steps, and recover when an action fails. Then investigate multi-agent designs in which different agents handle specialized tasks.

Consider a research assistant that gathers information, extracts evidence, and prepares a report. You could implement it as a single agent first, then compare it with a design that delegates research and review to separate agents.

Prerequisites: Tool calling, Python, and a basic understanding of agent loops.

Build: Create a research workflow with a retrieval step, a drafting step, and a verification step. Compare a fixed workflow with an agent that can decide when to retrieve more information.

Afterward, you should be able to compare agent architectures and identify when orchestration adds value.

Don’t assume multiple agents will automatically improve quality. They can introduce extra model calls, latency, coordination failures, and more complicated debugging.

Repository: KalyanM45/AI-Project-Gallery

Difficulty: Beginner to Intermediate

This repository catalogs projects across machine learning, deep learning, and generative AI. Its examples include prediction systems, chatbots, document-related tools, and agentic workflows. It works best as an exploration and project-selection resource. Not every catalog entry necessarily provides a complete implementation.

What to learn?

Look at how AI techniques map to application problems. Compare a conventional prediction project with a chatbot or agentic workflow, and identify the data, model, and user-interface requirements each one needs.

Prerequisites: Basic programming and familiarity with API-backed AI applications.

Build: Choose a project related to your interests and implement a small, working version. Document its inputs, outputs, failure cases, and evaluation method.

Afterward, you should be better at turning an AI concept into a defined software project.

Repository: asgeirtj/system_prompts_leaks

Difficulty: Intermediate

This repository collects documented system prompts attributed to various AI assistants and coding tools. Treat it as a resource for examining publicly accessible examples of instruction design and considering their security implications.

What to learn?

Study how instructions define an assistant’s role, describe available tools, constrain responses, and establish task-specific behavior. Also examine the risks of prompt injection, instruction disclosure, and overly broad tool permissions.

Prerequisites: Basic prompting and an understanding of how system instructions influence an AI application.

Build: Create a security test suite for your own assistant. Include attempts to override its instructions, request unauthorized records, and misuse available tools.

Afterward, you should be able to identify common prompt-level weaknesses and design defensive tests.

Public prompt collections are not proof that every prompt is authentic or current. Do not attempt to bypass access controls or obtain private prompts. In your own system, treat prompts as one layer of defense, not as a substitute for authorization enforced by application code.

Repository: priyanshop754/vibe-coding-playbook

Difficulty: Beginner to Intermediate

The Vibe Coding Playbook focuses on structured AI-assisted development, including prompting methods, step-by-step implementation, quality-assurance checklists, and team workflows.

What to learn?

Use AI coding tools to help implement features while retaining responsibility for architecture, tests, and code review.

Prerequisites: Basic software development and familiarity with Git.

Build: Ask a coding assistant to implement one feature in your RAG application. Require a written plan, small implementation steps, tests, and a review of the resulting diff.

Afterward, you should have a more repeatable way to use AI-assisted development without accepting generated code blindly.

Vibe coding can speed up implementation. You still need to verify security, understand the resulting code, and test behavior outside the happy path.

Repository: ombharatiya/ai-system-design-guide

Difficulty: Intermediate to Advanced

This guide focuses on AI system design, including RAG architectures, agents, model selection, evaluation, and production engineering. It also provides material for AI engineering system-design interviews.

What to learn?

Study how to choose models, structure inference and retrieval pipelines, measure quality, and manage latency and cost. Pay attention to security boundaries and observability.

Prerequisites: A working understanding of LLM applications and basic backend architecture.

Build: Design a production architecture for your document assistant. Include authentication, document ingestion, retrieval, inference, evaluation, logging, and failure handling.

Afterward, you should be able to discuss architectural choices with explicit trade-offs instead of choosing a framework first and fitting the problem around it.

Repository: donnemartin/system-design-primer

Difficulty: Intermediate to Advanced

The System Design Primer covers general distributed-system concepts, including databases, caching, load balancing, asynchronous processing, replication, and scalability. These concepts apply directly to AI applications.

What to learn?

Consider what happens when the document assistant you designed receives more traffic.

Prerequisites: Basic backend development and familiarity with APIs and databases.

Build: Draw a system diagram for 1,000 concurrent users. Identify bottlenecks, define timeout and retry policies, and decide which operations should run asynchronously.

Afterward, you should understand how traditional distributed-system design supports reliable AI services.

A model’s latency, token limits, and inference cost add constraints to familiar system-design trade-offs.

Repository: florinpop17/app-ideas

Difficulty: Beginner to Intermediate

App Ideas organizes software projects into tiers based on the experience and techniques needed, from user-facing applications to API-backed projects and applications with backend and database requirements.

What to learn?

Use the tiers to choose a manageable project and extend it as your skills improve.

Prerequisites: Basic frontend or backend development.

Build: Take a simple idea, such as a task manager, and add one AI capability: natural-language task creation, semantic search, or automatic categorization.

Afterward, you should be able to add AI to an existing product idea without making the model responsible for every operation.

Repository: The-Cool-Coders/Project-Ideas-And-Resources

Difficulty: Beginner to Advanced, depending on the project

This repository provides project ideas with objectives, user stories, and development resources. That structure helps you move from an interesting idea to a concrete implementation plan.

What to learn?

Define the problem, break it into user stories, choose a minimal scope, and plan how you will validate the result.

Prerequisites: Basic programming. For an AI project, use the API, RAG, agent, and system-design concepts from earlier stages.

Build: Deliver one complete AI application. Include a working UI, backend APIs, persistent storage where required, input validation, tests, evaluation examples, and deployment documentation.

Afterward, you should have a project you can demonstrate and explain: what problem it solves, how the model is used, how failures are handled, and where the architecture could improve.

The concepts in these repositories build on one another, but they solve different problems.

The progression looks like this:

LLM API → RAG → Tool calling → Agent → Multi-agent workflow → Production architecture

Each step introduces new engineering concerns.

A RAG system requires reliable ingestion and retrieval. An agent introduces action permissions and execution limits. A multi-agent system adds coordination and additional failure modes. Production architecture brings all of these concerns together with conventional software infrastructure.

You do not need a multi-agent system for every problem. A fixed workflow is often easier to test, operate, and maintain. Add autonomy when the problem genuinely benefits from it.

Here is how the two parts fit together:

Mathematics → AI/ML → Generative AI → Prompt Engineering → LLMs → Transformers → Training → Deep Learning Research

LLM APIs → RAG → Tool Calling → AI Agents → MCP → Multi-Agent Systems → AI Applications → System Design → Production

Use the Learn → Build → Experiment → Evaluate cycle throughout. Each repository should help you answer a question in your current project. Keep your experiments small enough to understand, and measure results instead of judging the system from a few impressive responses.

Learning how an LLM works is one skill. Building a reliable product around that LLM is an entirely different skill.

Part 1 gives you the foundations to understand models. Part 2 helps you apply that knowledge through retrieval, tools, agents, application development, and system design.

You do not need to complete all twelve repositories before starting a project. Begin with the OpenAI Cookbook, build a small application, and add capabilities as you encounter real requirements.

Your first project does not need to be an autonomous multi-agent platform. A document assistant that retrieves the right passage, cites its source, handles errors, and protects private data is already a meaningful engineering exercise.

Choose one problem, build the smallest useful version, and make it work reliably. Then use the next repository to improve it. That is how you turn AI knowledge into AI engineering experience.

I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.