Here is the take that will get me yelled at on Twitter: for single-user inference of 7B–13B parameter models, an RTX 4090 is the rational choice over an H100. Not in every scenario — but in the scenario that most individual developers, researchers, and small teams actually face day to day, the consumer card wins on cost-effectiveness by a wide margin. Let me show you the numbers.
The Hardware Tale of the Tape
The RTX 4090 packs 24 GB of GDDR6X with roughly 1 TB/s of memory bandwidth. The H100 SXM carries 80 GB of HBM3 at 3.35 TB/s. On paper the H100 crushes the 4090 in every metric — bandwidth, VRAM capacity, FP8 tensor throughput. And for batch inference serving hundreds of concurrent users, that advantage is real and unjustifiable to ignore. But inference for a single user is a fundamentally different workload than serving a fleet.
Real Pricing: The Gap Is Enormous
| GPU | Provider | On-Demand $/hr | Spot $/hr |
|---|---|---|---|
| RTX 4090 | CloudRift | $0.39 | — |
| RTX 4090 | Vast.ai | — | $0.17 |
| H100 SXM | Cudo Compute | $1.87 | — |
| H100 SXM | Vast.ai | — | $0.73 |
At on-demand rates, the H100 costs 4.8x more than the RTX 4090. Even at spot prices, the H100 is 4.3x pricier. You can check these numbers yourself on our comparison tool — they update every six hours.
Token Throughput: Less Than You Think
Take a 7B FP16 model like Mistral-7B. It consumes roughly 14 GB of VRAM — fits comfortably on the RTX 4090's 24 GB with room for KV cache. On the 4090 you'll see approximately 40 tokens/sec for single-user generation. On the H100, that number climbs to around 80 tokens/sec. The H100 is 2x faster.
But 2x faster at 4.8x the price is a terrible deal. You're paying $2.40 more per hour for an extra 40 tokens/sec that you probably don't need — 40 tokens/sec is already faster than anyone can read. At 40 tok/s a 500-token response completes in 12.5 seconds. At 80 tok/s it completes in 6.25 seconds. Is that 6-second difference worth $1.48/hr? For most individual developers, absolutely not.
Cost Per Million Tokens
At $0.39/hr generating 40 tokens/sec, the RTX 4090 produces 144,000 tokens per hour. That's $2.71 per million tokens. The H100 at $1.87/hr generating 80 tokens/sec produces 288,000 tokens/hr — $6.49 per million tokens. The consumer GPU delivers tokens at 42% of the datacenter cost. For small models, the math is unambiguous.
Where the H100 Wins (and It Does Win)
The analysis above applies to single-user, small-model inference. The H100 pulls ahead — dramatically — in several scenarios:
- Batch inference with concurrent users. The H100's 3.35 TB/s bandwidth and 80 GB VRAM allow it to serve dozens of requests simultaneously. The 4090 chokes after 2–3 concurrent streams.
- Models larger than 24 GB. A 13B FP16 model needs ~26 GB. It doesn't fit on the 4090. End of discussion.
- FP8 quantized inference. The H100's native FP8 support gives you near-FP16 quality at half the memory. The 4090 lacks dedicated FP8 hardware.
- Production SLAs. If you're serving an API with uptime requirements, datacenter GPUs with ECC memory and enterprise support are non-negotiable.
Where the RTX 4090 Wins
- Single-user inference. You're running a local LLM for coding assistance, research, or personal projects. The 4090 delivers snappy responses at a fraction of the cost.
- Dev/testing. You're iterating on prompts or fine-tuning hyperparameters. You don't need datacenter throughput; you need a cheap GPU that won't make you wait.
- Anything that fits in 24 GB. 7B FP16, 13B GPTQ 4-bit, Stable Diffusion XL — if it fits, the 4090 runs it well at one-fifth the hourly cost.
Key insight: The RTX 4090's weakness is VRAM (24 GB), not speed. If your model fits in 24 GB, the 4090 is almost always the right call. The moment it doesn't fit, you have no choice but to step up to a 48 GB or 80 GB card. The constraint is binary — it either fits or it doesn't. Use our GPU comparison tool to filter by VRAM and find the cheapest option that clears your model's memory requirement.
The Decision Framework
Ask yourself two questions. First: does your model fit in 24 GB? If yes, rent an RTX 4090 and pocket the savings. If no, you need a bigger card — check the Vast.ai or Cudo Compute listings for H100 spot prices. Second: are you serving multiple concurrent users? If yes, the H100's batch throughput justifies the premium. If you're the only user, you're paying for headroom you'll never touch.
The GPU market has this weird dynamic where people assume "more expensive = better" without asking "better at what?" For single-user inference of models that fit in 24 GB, the RTX 4090 at $0.39/hr isn't a compromise — it's the optimal choice. The H100 is a fantastic GPU. It's just not your GPU, unless you actually need what it offers. Stop overpaying for inference.
