# Why I built stressllm?

A few weeks ago I wrote about building a multi agent system on my old personal laptop. You can read more about it [here](https://blog.ceruleancloud.ca/building-a-multi-agent-system-to-track-real-madrid-matches-using-aws-strands-and-ollama). This project failed miserably because of my hardware limitations. My hardware was theoretically sufficient, but the system was unusable in practice. I realized I was flying blind against the **KV Cache**; the "working memory" that scales with context. I needed a way to know exactly where my hardware would "choke" before I spent hours debugging a lagging agent.

So, I spent the weekend **vibe-coding** [**stressllm**](https://pypi.org/project/stressllm/): a Python CLI tool that performs context saturation tests to find your hardware's breaking point.

## The Core Bottleneck: VRAM vs. Context

Most people only look at "Static VRAM" i.e the memory needed to load the model weights. But the real killer is **Dynamic VRAM**.

As your conversation grows, the model needs to store the "Keys" and "Values" (KV Cache) of every token to avoid re-processing the entire prompt. This grows linearly. On my 6-year-old laptop with 2GB of VRAM, the moment that cache spills over into system RAM, performance falls off a cliff.

## Under the Hood: The Architecture

I built StressLLM with a clean separation of concerns: **The Watcher** and **The Worker**.

I split the tool into two main components: `probe.py` (the observer) and `engine.py` (the executor).

### `probe.py`: Hardware Telemetry & Constraints

This module is responsible for heartbeat-style monitoring of your system.

*   **NVIDIA Only:** The tool currently relies on `pynvml` (NVIDIA Management Library). It specifically looks for `nvml.dll` on Windows or the standard drivers on Linux.
    
*   **Graceful Fallback:** When the GPU isn't available, the tool automatically pivots to reporting system-wide **RAM** and **CPU** usage via `psutil`.
    
*   **Error Isolation:** Each sensor (VRAM, Temperature, CPU) is independent. If a single telemetry call fails, the rest of the stats are still yielded.
    

### 2\. `engine.py`: The Saturation Logic

This is the "stress" part of the tester. It uses a specific strategy to find the breaking point:

*   **Synthetic Pressure:** Instead of asking the model to "think," it uses a `wordpool.txt` to generate a random prompt of a specific token length. The instruction is simple: *"Read this and say 'done'."* This isolates **Prompt Evaluation** speed and **KV Cache** overhead from actual generation.
    
*   **The Generator Pattern:** The test is written as a Python generator (`yield`). This is a safety feature. If the model causes a total system hang or an Out-of-Memory (OOM) error at 128k context, the tool has already "yielded" the successful results for 2k, 8k, and 32k.
    
*   **Direct GGUF vs. Ollama:** It supports two paths. It can call the **Ollama API** (testing your local server) or load a **.gguf** file directly via `llama-cpp-python`, which allows for manual control over `n_gpu_layers`.
    
*   **The Verdict:** The code maps Tokens-Per-Second (TPS) to a status. Anything under 5 TPS is flagged as the "Cliff" meaning the KV cache has likely spilled into your slower system RAM.
    

### The Verdict: Mapping the Performance Cliff

The tool doesn't just give you numbers; it gives you a "vibe check" based on the `_verdict` logic:

*   **✅ Smooth (>15 TPS):** Native GPU speeds. Your agents will feel snappy.
    
*   **⚠️ Slowing (5-15 TPS):** You’ve likely hit the "Knee" where the KV cache is spilling into system RAM.
    
*   **💀 Cliff (<5 TPS):** Total saturation. Time to lower your `num_ctx` or buy a better GPU.
    

## Getting Started

I’ve pushed the tool to PyPI and GitHub. You can test your own local setup in seconds:

You can use the following commands to test the tool

```plaintext
#install via pip
pip install stressllm

#list available models 
stressllm models

#run test
stressllm run <modelname> --depth 2
```

Whether you are building agentic frameworks or just running local LLMs for privacy, you need to know your limits. Stop guessing and start stressing.

**Find it on PyPI:** [pypi.org/project/stressllm/](https://pypi.org/project/stressllm/)