LLM API Outages: A Fallback Strategy for Your Store Chat

When the LLM API behind your store chat is down, the chat should fail in a controlled way. Use a short timeout, retry only a few times with delays, then fall back to a cached or scripted answer. If that is not enough, hand the customer to a human or a contact form. Never leave the customer staring at a spinner.

You cannot prevent provider outages. You can decide what customers see when one happens.

Why a fallback plan matters

Every external API fails sometimes. It can return server errors, hit rate limits, or become very slow. If your chat has no plan, one of three bad things happens:

• The widget hangs, and the customer waits.
• The chat shows a raw technical error.
• The chat retries without limit and makes the problem worse.

A fallback plan turns these into a calm, short message and a useful next step.

Step 1: Set strict timeouts

A slow response is a failure. Without a timeout, one stuck call can hold a worker and a customer for a long time.

• Set a connect timeout and a total request timeout.
• If you stream answers, also set a limit for the time to first token and a limit for silence between chunks.
• Pick numbers from what your customers will tolerate in a chat. Many people give up after a few seconds without feedback.

Show a typing indicator at once, so the wait feels active. Show a polite message if the timeout hits.

Step 2: Retry safely

Retries help with short faults. They hurt during long ones.

Rules for safe retries:

1. Retry only errors that may pass: timeouts, HTTP 429 (rate limit), and 5xx server errors.
2. Do not retry client errors such as 400 or 401. The same call will fail again.
3. Use exponential backoff with jitter. Wait longer after each attempt, and add a random delay so many clients do not retry at the same moment.
4. Respect a Retry-After header when the provider sends one.
5. Limit the number of attempts, for example two or three.
6. Keep the total time inside your overall timeout.

Make sure a retry cannot cause a double action. A chat answer is safe to repeat. A tool call that creates a refund or an order is not, unless it is idempotent.

Step 3: Use a circuit breaker

A circuit breaker stops calls to a service that keeps failing.

• Closed: calls go through as normal.
• Open: after a set number of failures in a window, calls stop at once and the fallback runs.
• Half-open: after a pause, a few test calls go through. If they succeed, the breaker closes.

This protects your server from piling up requests. It also gives the customer a fast fallback instead of a slow failure.

Step 4: Prepare fallback answers

When the model is not available, what can you still offer? Prepare layers, from best to simplest.

Cached answers

Many store questions repeat: shipping times, return rules, opening hours, size guides. Store good answers for these. If you keep a cache, set an expiry and clear entries when policies change. A stale answer about returns is worse than no answer.

Match only safe, clear questions. Do not cache answers that depend on one customer's data, such as order status.

A secondary model or provider

You can route to a second model or provider. This adds work: prompts behave differently, and you need a second key, a second bill and a second set of tests. Use it only if the chat is important enough to justify that.

Scripted menus and search

If the model is down, show buttons for the common tasks: track an order, start a return, see shipping info. Link to your help pages. A plain search box over your help content also works without a language model.

Plain failure messages

Last, show an honest message. For example: "Our assistant is not available right now. You can still shop as usual. For help, write to us and we will reply by email."

Step 5: Hand off to a human

The human hand-off is the most important fallback. Some questions cannot wait.

• Offer an email form or a contact link when the chat fails.
• If you have live support, route the conversation to an agent and pass the chat history.
• Outside working hours, say when someone will answer.
• Keep the customer's message. Never ask them to type it twice.

Make sure the form works without the LLM. It should not depend on the same API that just failed.

Step 6: Keep safety first

A fallback must not create new risk.

• Do not let a cached answer contain prices or stock levels that may be out of date.
• Do not let a fallback promise refunds, discounts or delivery dates.
• Keep the same rules for personal data in fallback flows.

Step 7: Monitor and test

A fallback you never test will fail when you need it.

• Alert when the fallback activates, so you know there is a problem.
• Track how often the fallback runs and how often customers use the hand-off.
• Test it. In staging, block the LLM API and check what the customer sees.
• Review the plan after each incident.

Use an external monitor to watch the chat endpoint. Uptrack (from the 2folk team) is one option among others for this kind of check. Also tell customers about longer outages on your status page.

A quick checklist

• Total timeout and time-to-first-token limit set.
• Retries only for transient errors, with backoff, jitter and a cap.
• A circuit breaker in front of the LLM call.
• Cached answers for common, safe questions.
• Scripted menu and help search as a backup.
• A hand-off form that does not use the LLM.
• Alerts when the fallback runs.
• A tested plan, reviewed after incidents.

Frequently asked questions

How many retries should I allow?

Two or three is a common choice for a chat. More retries make the customer wait longer and can add load to a service that is already struggling. Keep the total time inside your overall timeout.

Is a second LLM provider worth it?

Only for chats that are important to your business. It adds cost and complexity, because prompts and behavior differ between models. For many stores, cached answers and a human hand-off give more value for less work.

Can I cache LLM answers?

Yes, for stable and general questions such as shipping rules. Do not cache answers that depend on a customer, an order, a price or a stock level. Set an expiry and clear the cache when policies change.

What should the chat say during an outage?

Say plainly that the assistant is not available, that the shop still works, and how to get help. Do not show technical error text. Give one clear next step, such as an email form.