Picking the Right LLM for Your Product

Every founder building an AI feature eventually reaches the same decision point: GPT, Claude, Gemini, or something open-source?

Most teams pick based on hype, familiarity, or whatever their developer used in a side project. That’s how you end up locked into a model that’s 4× too expensive for your usage pattern, or one that quietly fails on the exact type of content your users generate.

Choosing an LLM is an engineering decision. Here’s how to make it a deliberate one.

The Four Dimensions That Actually Matter

There’s a lot of noise around “which LLM is smartest.” Intelligence benchmarks matter less than you think for most product use cases. What actually drives the right decision is a combination of four factors.

Cost at your volume

Token pricing varies by an order of magnitude across providers and model tiers. A model costing $15 per million output tokens versus $0.15 per million output tokens is a 100× difference. If your feature generates 500 tokens per request and you expect 50,000 requests a day, that’s the difference between $375/day and $3.75/day.

Map out your expected prompt length, output length, and daily request volume before evaluating anything. Most startups are surprised when they do this math.

Latency requirements

If your AI feature is in the critical path — inside a chat UI, generating content a user is waiting to read — latency matters. Models optimised for quality (GPT-4o, Claude Opus) run at 1–4 seconds for typical requests. Smaller “lite” models (GPT-4o-mini, Claude Haiku, Gemini Flash) can return results in 300–700ms and cost a fraction of the full-size variants.

For background tasks (document processing, async summarisation, batch jobs), use the best model you can afford. For real-time interactions, use the fastest model that meets your quality bar.

Task fit

Different models have genuine strengths. This isn’t marketing — it shows up in production:

• Long documents and instruction-following: Claude handles large context windows well and tends to follow multi-step instructions more precisely. Good for document review, complex structured extraction, or workflows with detailed system prompts.
• General-purpose and tool use: GPT-4o has the broadest ecosystem support and the largest library of documented prompt patterns. If you’re unsure, it’s a safe default.
• Multimodal and Google integrations: Gemini is strong when you need vision built in or when you’re already in the Google Cloud ecosystem.
• Cost-sensitive or privacy-critical use cases: Open-source models (Llama 3, Mistral, Phi-4) can run on your own infrastructure, eliminating per-token costs and keeping data entirely in-house.

Reliability and API stability

Production systems care about uptime, rate limits, and predictable API behaviour. OpenAI and Anthropic both have mature APIs with well-documented rate limit tiers, retry semantics, and SLAs for enterprise customers. Google’s API has improved significantly in 2025–26.

If you’re self-hosting an open-source model, reliability is now your engineering team’s problem. That’s fine for teams with the capacity — not fine if you have two engineers and a shipping deadline.

A Decision Framework

Rather than picking a model once and hoping for the best, treat it as a two-stage decision.

Stage 1: Filter by constraints

Start by eliminating options that don’t meet your hard requirements:

• If you can’t send user data to a third party, open-source is the only option.
• If you need sub-500ms responses in a real-time UI, eliminate full-size frontier models.
• If your context window regularly exceeds 32K tokens, check which providers support that tier at your price point.

After this filter, you should have two or three candidates.

Stage 2: Evaluate on real data

This is the step most teams skip. Take 50–100 real examples of the inputs your product will actually process — edge cases included — and run them through each shortlisted model. Score the outputs on whatever dimensions matter (accuracy, format compliance, tone, rejection rate on edge inputs).

Don’t benchmark on hypothetical prompts you wrote for the test. Benchmark on what real users will actually send. The results often change the decision entirely.

Designing for Portability

Whichever model you start with, don’t wire it directly into your application code. Use an abstraction layer — a thin wrapper or a library like LiteLLM — that normalises the API surface. Keep your prompts in versioned files, not hardcoded strings.

This lets you swap models by changing a config value rather than refactoring a codebase. And you will swap — pricing changes, new models launch, and the “best” option in Q1 is rarely the best option by Q4.

The Part Nobody Talks About

Model selection is also about what you’re ready to support operationally. Self-hosting a 70B parameter model cuts your inference costs dramatically — but it adds GPU infrastructure, model versioning, and reliability engineering to your team’s plate.

Most Israeli startups building their first AI feature should start with a hosted API. The operational simplicity is worth the higher per-token cost until you have the usage data to justify a different approach. Our AI development service uses this approach with every new client: validate the use case on a hosted API first, then optimise infrastructure once you know the feature has product-market fit.

For teams ready to move faster on AI integrations — whether adding a single feature or building a full AI-native product — we’re happy to talk through the architecture.

Yaniv Amrami is founder of quickdev. He has helped Israeli startups integrate LLMs into production products since the early days of the GPT-3 API.