AWS adds prompt caching for open-weight models on Bedrock
Listen to Briefing
0:00/1:30

What happened.Amazon Web Services launched prompt caching for open-weight foundation models on Amazon Bedrock, reducing inference latency by up to 85 percent and cutting input costs by 90 percent.
Why it matters.Multi-step reasoning loops repeatedly feed system prompts, schema definitions, and conversation histories into foundation models. Applying server-side prompt caching to open-weight models allows developers to deploy complex tool-calling agents without incurring linear token fee growth.
The technical specifications:
- Latency reduction: Speeds up response times by up to 85 percent for extended conversational contexts and repetitive agent retrieval prompts.
- Cost savings: Cuts input token processing charges by 90 percent on cached prefix tokens across supported model checkpoints.
- Model support: Available immediately for Llama 3.3 70B, Mistral Large 2, and Command R+ endpoints across standard AWS commercial regions.
- Cache invalidation: Automatically manages cache eviction using a 5-minute time-to-live policy that refreshes dynamically whenever a matching prompt prefix arrives.