# Coolhand — Agents That Improve Your Agents

Coolhand is more than a dashboard — it's agents that improve your agents. Coolhand observes your production AI, reads the human feedback on the outcomes, and ships you fixes as pull requests. You build the agents; Coolhand does the work of making them better. Teams typically cut LLM costs 50–70% while building a quality and data advantage competitors can't copy. With 10B+ tokens analyzed, Coolhand helps engineering teams catch silent failures in production, collect high-signal human feedback, and continuously improve prompts and tool calls over time.

## What Coolhand does

- **Catching silent agent failures at 2 a.m.** — Silent failures, cost spikes, and broken tool calls are spotted in production, not discovered in a postmortem. Hard errors get a fix proposed within minutes; deeper, feedback-derived issues take longer, and Coolhand tells you which is which. Every fix ships as a reviewable pull request in your repo — you review, you decide. Fixes are at the agentic layer (prompts, tool calls), not the model layer.
- **Continuous improvement from human feedback** — Coolhand's open-source feedback-collection skill audits your codebase and shows you where to capture high-signal feedback from work you're already doing — no annotation queues, no new pipelines. Edits, explanations, and sentiment get analyzed, pressure-tested, summarized, and sent to you as suggested improvements.
- **Know which agents are worth the token spend** — Cost per unit of work, quality trends over time, and the levers to move both — with your own value assumptions plugged in, not ours.

## Who Coolhand is for

- Teams shipping AI features who want to close the loop between user feedback and prompt quality
- Engineering teams running LLM agents in production who need to catch silent failures before customers do
- Teams that want to cut LLM costs without sacrificing quality, using model bakeoffs and cost-per-outcome tracking
- Teams that want a defensible quality and data advantage built from their own production feedback, not generic benchmarks

## How It Works

Coolhand operates as a five-stage pipeline that turns raw agentic data into actionable improvements:

1. **Agentic data** — LLM logs, tool outputs, human feedback, and outcome signals are emitted by your AI system
2. **Passively gathered** — the Coolhand API collects this data automatically, with no changes to your core application logic
3. **Expert analyzed** — five investigative agents (Cost analyst, Failure SWAT, Product analyst, AI engineer, Prompt maintainer) run in parallel to examine the data from different angles
4. **Executive reviewed** — findings are synthesized into a prioritized optimization plan
5. **Turned to results** — the plan is executed as agent fixes (pull requests, prompt updates) directly in your codebase

## Supported LLM providers

Coolhand ingests logs from: OpenAI, Anthropic, Google Gemini, Google Vertex AI, Azure OpenAI, AWS Bedrock (including Bedrock Anthropic), OpenRouter, Ollama, ElevenLabs, GitHub Copilot, Claude Code, and Claude Cowork. Cloudflare AI Gateway is also supported for its OpenAI-compatible upstreams, including Groq, Mistral, and Perplexity.

Coolhand can replay a request template's historical requests through any candidate model that's available for bakeoff, comparing its cost and quality against the model you're using today. What gets captured varies by provider — some sources supply full token and cost detail, others a narrower set of fields.

## Beyond observability

Coolhand isn't an observability tool and doesn't compete on that feature set. Observability platforms like Langfuse, Braintrust, or Datadog show you what happened; Coolhand goes further — diagnosing why, proposing the fix as a pull request, and closing the loop with feedback and ROI reporting. Many teams run Coolhand alongside their existing observability stack; some use it instead. See [Beyond Observability](https://coolhandlabs.com/beyond-observability.md) for the full comparison.

## Open source

Coolhand's SDKs, JS widget, CLI, and feedback-collection skill are open source and Apache-licensed, and the API definitions are open — you can point every tool at your own backend if you choose. Nothing is trapped. See [Open Source](https://coolhandlabs.com/open-source.md) for the full story.

## SDKs

Coolhand provides SDKs for the most common backend languages:

- **Ruby** — `gem "coolhand"`
- **Python** — `pip install coolhand`
- **Node.js** — `npm install coolhand`

Each SDK wraps your existing LLM calls with a single line of code and handles logging, feedback collection, and prompt template management.

## Getting Started

1. Sign up at [coolhandlabs.com](https://coolhandlabs.com)
2. Install the SDK for your language
3. Wrap your LLM calls with the Coolhand logger
4. Embed a feedback widget in your UI
5. Review feedback and apply suggested prompt improvements

## API

Coolhand exposes a REST API for all features. The full API reference is available at [coolhandlabs.com/docs](https://coolhandlabs.com/docs). The machine-readable OpenAPI spec is at [coolhandlabs.com/api-docs/v2/coolhand_api.yaml](https://coolhandlabs.com/api-docs/v2/coolhand_api.yaml).

Authentication uses an API key passed via the `X-API-Key` header.

## Pricing

| | Free | Pro | Power User |
|---|---|---|---|
| **Price** | $0 | $199/month | $1/million tokens |
| **Weekly token cap** | 10M tokens/week | 100M tokens/week | None |
| **Dashboard** | Cost insights dashboard | Cost & performance insights | Cost & performance insights |
| **Optimizations** | 5/week | No optimization cap | No optimization cap |
| **Direct code fixes (PRs)** | $2 each | 10/week included, then $1 each | Unlimited |
| **Data retention** | 21 days | 90 days | — |
| **HIPAA compliance** | — | — | ✓ |

Volume pricing is available for annual plans — contact us at [coolhandlabs.com](https://coolhandlabs.com).

## MCP Integration

Coolhand exposes a Model Context Protocol (MCP) server at `https://coolhandlabs.com/mcp`. This allows AI coding assistants (Claude Code, Cursor, etc.) to interact directly with your Coolhand workspace — querying logs, managing templates, and reviewing feedback without leaving the assistant.

The inference API catalog is available over REST at `GET https://coolhandlabs.com/api/v2/inference_apis` — keyless (rate-limited to 1 request/min, unlimited with an API key) and the recommended way to access it. It is also exposed through the public MCP endpoint at `https://coolhandlabs.com/mcp/public` (using your public `api_key` or `private_api_key` in the `X-API-Key` header), which serves the same catalog.

Query params: `q[source_api_eq]` (e.g. `openai`, `anthropic`), `q[provider_eq]` (e.g. `OpenAI`, `Anthropic`), `q[s]` for sorting (e.g. `model desc`), `available_for_bakeoff` (boolean — a flat param, not a Ransack predicate), and `include_deprecated` (boolean, defaults to `true` on the REST endpoint for backward compatibility — pass `include_deprecated=false` to hide deprecated models). An unrecognized filter returns `400`. `GET /api/v2/inference_apis/:slug` returns a single model's full record, including fields the list endpoint omits (review status, batch pricing).

## Key terms

- **Optimization** — a proposed improvement that Coolhand's analysis agents identify from your production data. Optimizations can be completed manually, by merging the change suggestion Coolhand generates for them, or automatically when Coolhand detects the change already shipped in your codebase.
- **Change suggestion** — the concrete code change Coolhand drafts to implement a fix, such as an updated prompt or tool call, and opens against your repository as a pull request (or GitLab merge request).
- **Workload** — a named grouping of related LLM request templates and logs that represents one category of AI task in your product, e.g. "Email Drafter."
- **Bakeoff** — an A/B evaluation that replays historical production requests through a candidate LLM model to compare its quality and cost against the model you're currently using.

## FAQ

### What is Coolhand?

Coolhand is agents that improve your agents. It observes your production AI agents, proposes fixes as pull requests, helps you collect human feedback, and reports the ROI of your agent work — so your team stays in control without babysitting what's already live.

### How is Coolhand different from observability tools?

Observability platforms like Langfuse or Datadog show you what happened. Coolhand goes further: it diagnoses why, proposes a fix as a reviewable pull request, and closes the loop with feedback and ROI reporting. Coolhand isn't a replacement for observability — many teams run it alongside their existing dashboard — and it doesn't compete on observability's own feature set.

### How do Coolhand fixes work?

When Coolhand finds a problem in a production agent, it proposes a fix at the agentic layer — prompt or tool-call changes, not model retraining or infrastructure changes — as a pull request in your repository. Hard errors typically get a proposed fix within minutes; more nuanced, feedback-derived issues can take hours or longer to diagnose properly. Nothing ships to production until your team reviews and approves it.

### How does Coolhand collect feedback without an annotation team?

Coolhand's open-source feedback-collection skill reviews your codebase and shows you where to capture high-signal feedback from work you're already doing — edits users make to AI output, the actions they take instead of accepting a suggestion — rather than asking you to build a new annotation queue. You can start with passive capture and graduate to richer prompts over time.

### What LLM providers does Coolhand support?

Coolhand ingests logs from all the major providers — OpenAI, Anthropic, Google Gemini and Vertex AI, Azure OpenAI, AWS Bedrock, OpenRouter, Ollama, and more. See the "Supported LLM providers" section above for the full list.

### What is an optimization in Coolhand?

An optimization is a proposed improvement — a cost reduction, quality fix, or best practice — that Coolhand's agents identify from your production data. It can be completed manually, by merging the change suggestion Coolhand generates for it, or automatically when Coolhand detects the change already shipped in your codebase.

### What does it take to integrate Coolhand?

You wrap your existing LLM calls with the Coolhand SDK in one line and, if you want feedback capture, embed a widget in your UI. There are no new pipelines, annotation queues, or changes to your core application logic.

### Does Coolhand replace engineers?

No. Coolhand takes on the drudge work — watching for silent failures, sifting through feedback, building ROI reports — so the engineers you already have spend their time building what's next instead of babysitting what's already live.

### Is Coolhand open source?

Coolhand's SDKs, JS widget, CLI, and feedback-collection skill are open source and Apache-licensed, and our API definitions are open. You can point every tool at your own backend if you choose — nothing is locked behind Coolhand's managed service.

## Learn more

- [About Coolhand](https://coolhandlabs.com/about.md) — founder story and background
- [Ruby Is the Language for AI](https://coolhandlabs.com/ruby-is-the-language-for-ai.md) — our RubyConf 2026 challenge to the community, free Ruby usage through year-end, and no-strings $500 donations to 5 rising Ruby + AI open source projects
- [Inference API Catalog](https://coolhandlabs.com/inference-apis.md) — pricing and specs for 100+ LLM models across OpenAI, Anthropic, Google, and more
- [Terms of Service](https://coolhandlabs.com/terms-of-service.md) and [Privacy Policy](https://coolhandlabs.com/privacy-policy.md)

## Updates & Announcements

The latest news and incident reports from Coolhand are published at [coolhandlabs.com/updates](https://coolhandlabs.com/updates). Each post is available as rendered HTML and as a raw markdown file (append `.md` to the URL).