For the last few years, the AI industry has been obsessed with making models bigger.
More parameters.
More GPUs.
More context.
More compute.
But there is another, much more interesting direction emerging What if AI became dramatically smaller instead?
Imagine a language model that fits into 14MB, runs an entire session in roughly 28MB of RAM, works offline, and can execute tool calls directly on devices such as phones, wearables, robots, Raspberry Pis, smart-home devices and even some microcontrollers.
That is the idea behind Needle 2, an open-source model from Cactus Compute.
And the interesting part isn’t simply its size.
Needle is designed around a different question: what is the smallest model that can reliably turn natural language into actions?
The Problem With Putting LLMs Everywhere
Suppose you build a smart ring.
A user asks to“Turn the bedroom lights down to 20 percent.”
You don’t necessarily need a 70-billion-parameter model to understand that request.
The device already knows:
set_lights(
room: string,
brightness: number
)The actual problem is relatively constrained:
Natural language
↓
Understand intent
↓
Select function
↓
Extract arguments
↓
Execute actionThat’s fundamentally different from asking an LLM to write a 2,000-word essay or reason about a complex software architecture.
If the task is constrained, the model doesn’t need the entire universe encoded inside it.
That’s the bet behind Needle.
Meet Needle 2
Needle 2 is an open 45-million-parameter model specifically optimized for:
Tool calling
Device control
Structured extraction
Function selection
Small-device AI
The entire deployed model is packaged into a 14MB binary, while a full inference session uses approximately 28MB of RAM.
Here are the numbers that make it interesting:
The model is also designed to work without a network connection during inference. Its weights and inference engine are packaged together, creating a self-contained deployment.
That changes the economics of edge AI.
Why 14MB Matters
You might look at 14MB and think “Okay, that’s small. So what?” The answer is where you can now run it.
A cloud-based AI architecture normally looks something like:
Device
↓
Internet
↓
API Gateway
↓
LLM
↓
Tool/API
↓
Internet
↓
DeviceThat introduces latency, connectivity requirements, cloud costs and privacy concerns.
Needle enables a very different architecture:
┌───────────────────────────────┐
│ Device │
│ │
│ Voice/Text │
│ ↓ │
│ Needle 2 │
│ ↓ │
│ Tool Selection │
│ ↓ │
│ Local Function │
│ ↓ │
│ Device Action │
└───────────────────────────────┘No round trip to a cloud model is required.
For a smartwatch, robot, smart appliance or wearable, that can be a huge architectural advantage.
The best AI response isn’t always the smartest response. Sometimes it’s the response that happens immediately, privately and reliably.
Needle Isn’t Trying to Be a Tiny ChatGPT
Needle isn’t positioned as a replacement for large conversational models.
Cactus specifically frames the model around tool calling, device use and structured extraction rather than general-purpose open-ended conversation.
Consider these two requests:
Request A
“What’s the history of the Roman Empire?”
A tiny model isn’t the right tool.
Request B
“Set the thermostat to 22 degrees.”
That’s a completely different problem.
The device could expose:
{
"name": "set_temperature",
"parameters": {
"temperature": 22
}
}The model’s job becomes:
User intent
↓
set_temperature
↓
temperature = 22This dramatically reduces the capability required from the model.
The Architecture: Simple Attention Network
Under the hood, Needle 2 isn’t simply a conventional transformer squeezed until it fits.
Cactus describes it as a Simple Attention Network, combining several techniques including:
Hadamard MLPs
Grouped Query Attention (GQA)
Engram key-value memory
Multi-lane hyper-connections
Gated and normalized residual paths
Byte-level grammar-constrained decoding
The architecture is designed around efficient inference and bounded memory rather than maximizing general-purpose model capacity.
One particularly interesting component is its use of a Walsh-Hadamard transform inside the architecture.
Instead of learning a conventional dense transformation for every operation, some computation can use a fixed transform. That reduces the amount of learned data that needs to be stored.
This is exactly the kind of thinking required when your target isn’t a data center GPU, but a device with a few hundred megabytes of RAM.
The Secret Weapon: 2-Bit Quantization
The 14MB number becomes possible partly because Needle 2 uses CQ2, a 2-bit quantization format from Cactus.
Traditional model weights might use:
FP32 → 32 bits
FP16 → 16 bits
INT8 → 8 bits
CQ2 → ~2 bitsReducing precision dramatically reduces storage requirements, but there’s an important engineering trade-off:
Smaller weights are useful only if the resulting model remains good enough for the task.
That’s why Needle isn’t simply trying to compress a massive general-purpose LLM.
It is designing the model around a narrower workload where aggressive compression can still produce useful results.
Cactus reports that Needle 2 trades wins with small models such as FunctionGemma 270M, LFM2.5 230M and Apple FM on its targeted tool-calling and mobile-device benchmarks, while being substantially smaller.
Structured Output Is a Big Deal
There’s another clever piece of the system.
When an AI controls a device, producing “something approximately correct” isn’t enough.
Imagine the model needs to call:
{
"name": "set_volume",
"arguments": {
"level": 50
}
}You don’t want:
Sure! I'll set the volume to approximately 50.You need valid structured output.
Needle uses byte-level grammar constraints generated from tool schemas, restricting decoding to valid outputs. That means the model isn’t just generating JSON and hoping it’s valid.
The decoding process itself is constrained by the schema. For developers building AI agents, that’s an important architectural idea:
Use the model for semantic interpretation, and use deterministic constraints for the final interface contract.
Confidence Gating: Don’t Trust the Model Blindly
Another interesting feature is Needle’s confidence mechanism. Every response can carry a calibrated confidence score.
That enables an architecture such as:
User Request
↓
Needle
↓
┌──────────┴──────────┐
↓ ↓
Confidence High Confidence Low
↓ ↓
Execute locally Ask cloud modelThis is particularly powerful for hybrid AI systems.
A small local model handles simple requests and alarger cloud model handles ambiguous or difficult requests.
You don’t have to choose between “Everything runs locally” and “Everything goes to the cloud.”
Instead Use the smallest model that can safely solve the problem. Escalate when it can’t.
Tool Retrieval Solves Another Edge-Device Problem
Real applications can expose dozens or hundreds of tools.
Sending every tool definition into the model’s context increases memory and computation.
Needle includes a tool-retrieval mechanism that selects a smaller subset of relevant tools before constrained generation. Cactus says the system can render the top five tools for a turn and constrain generation to that subset.
Conceptually:
100 available tools
↓
Tool retrieval
↓
5 relevant tools
↓
Grammar-constrained generation
↓
Tool callThat’s a much more practical architecture for embedded agents.
💡 Enjoying this article?
Every week day, I publish practical, production-ready deep dives covering Web development, System Design, Open source projects, Tech industry trends and AI Engineering and tools.
Memory Doesn’t Grow Forever
Long conversations are problematic for small devices because context can consume memory.
Needle addresses this with a 256-token sliding window, while keeping tool information pinned as KV sinks. Cactus reports that this keeps session memory around 28MB rather than allowing it to grow indefinitely with conversation length.
This is an important principle for edge AI Bounded memory is often more valuable than unlimited context.
A tiny device doesn’t need a 1-million-token conversation history to turn on a light.
Developers Can Actually Run It
This isn’t just a research paper, The project provides a Python package and quick-start workflow.
Install it with:
pip install cactus-needleThen define a tool:
import needle
@needle.tool
def get_weather(city: str):
"""Get the current weather for a city."""
return {
"city": city,
"temp_c": 27,
"sky": "clear"
}
agent = needle.Needle(tools=[get_weather])
result = agent.run(
"What's the weather in Lagos?"
)
print(result["results"])The model determines which function to call and fills in its arguments.
It also supports LoRA fine-tuning, allowing developers to adapt the model to their own tool schemas and workflows.
Where Could Needle Be Used?
This is where the project gets genuinely interesting.
Wearables
"Start my workout."
→ start_workout()Smart Home
"Turn off everything downstairs."
→ retrieve relevant devices
→ execute multiple actionsRobots
"Move the red box to the table."
→ identify available robot tools
→ select movement/action functionsAutomotive
"Set the cabin temperature to 21."
→ set_temperature(21)Offline Assistants
No internet? The device can still perform locally supported actions.
IoT
A sensor hub with limited memory can potentially interpret natural-language commands without sending private device data to a cloud API.
What Needle Does NOT Solve
It’s important not to overhype this. A 45M-parameter model isn’t going to replace frontier models for:
Complex reasoning
Long-form writing
Large-scale coding
Broad knowledge tasks
Open-ended conversations
Complex research
And small models can be more sensitive to the quality of tool descriptions and task-specific fine-tuning. That’s not a weakness in the design, it’s the point.
Needle is optimizing for a different part of the AI capability curve.
The Bigger Idea: Tiny AI Is Coming
The most interesting thing about Needle isn’t actually the 14MB number. It’s the architectural direction.
For years, we treated AI like this:
Bigger model
+
Bigger GPU
+
Bigger context
=
Better AIEdge AI suggests another equation:
Smaller model
+
Better task definition
+
Quantization
+
Structured constraints
+
Local execution
=
Useful AI everywhereCactus estimates that the edge-device universe is dominated by inexpensive hardware, including budget phones, IoT devices, wearables, robots and microcontrollers.
That creates an enormous deployment surface for models that don’t need a GPU and that may be the real significance of Needle.
What Developers Should Take Away
If you’re building an AI-powered product, don’t automatically start by asking “Which biggest LLM should I use?” now ask “What is the smallest intelligence required to solve this particular problem?”
For a coding assistant, the answer may be a large reasoning model.
For a customer-support agent, it may be a medium model plus RAG.
For a smart ring that controls a handful of functions?
It might be 14MB. That’s the fascinating shift happening in AI right now. The future isn’t necessarily one giant model running everything.
It may be an ecosystem of specialized models — large models in the cloud, medium models on computers, and tiny models embedded directly into the products we use.
And when your AI model is smaller than many app assets, “put AI on the device” stops sounding crazy and starts sounding like an engineering decision.
Final Thought
Needle 2 is a reminder that progress in AI isn’t always about adding parameters. Sometimes progress comes from asking a better question.
Not “How big can we make the model?” but “How little intelligence do we actually need and where can we run it?”
That question could define the next generation of edge AI.
Resources
Thank You for Reading!
I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.