What to Monitor on an AI-Powered Store: A Practical List

On a store with AI features, monitor six areas: the storefront, the checkout, the AI chat or assistant, the LLM API it depends on, background queues and jobs, and the supporting basics such as SSL and DNS. Start with what earns money (site and checkout). Then add what is new and fragile (the AI parts).

This article gives you a list you can work through in one afternoon. For each area it says what to check and what a failure looks like.

Principles before the list

Three rules keep monitoring useful.

• Monitor outcomes, not servers. Ask "can a customer buy?" before "is the CPU low?"
• Every alert needs an owner and an action. If nobody can act on an alert, remove it.
• Fewer, better alerts. Too many alerts train people to ignore them.

1. The storefront

The storefront is the base of everything. Check the pages that bring and convert traffic.

• The home page.
• One category page.
• One product page.
• The search results page.
• The cart page.

For each page, check the status code and a piece of expected text, such as the shop name or a product title. A page that returns 200 but shows an error message is a common failure, and a status-only check misses it.

Also watch page speed. A slow store loses customers without ever being "down". Track response time and set a warning limit.

2. The checkout

Checkout is the most important flow. A failure here costs sales directly.

Monitor:

• That the checkout page loads.
• That shipping options appear for a test address.
• That the payment step shows the payment methods you expect.
• That the order confirmation page works for a test order.

Use a test mode or a test payment method for this. Never place live payments to test. The article on synthetic monitoring of checkout flows covers this in detail.

3. The AI chat or assistant

The chat widget is often loaded from a third-party script. It can fail in several ways without touching the rest of the store.

Monitor:

• That the widget script loads (no blocked or failed request).
• That your chat backend endpoint responds.
• That a simple, safe test message gets an answer.
• The time to first answer.
• The share of conversations that end in an error message.

Be careful with the test message. Use a fixed, harmless question, and tag it so it does not count as customer traffic in your analytics. Use a staging environment where you can.

4. The LLM API

Most AI features call an external language model API. You do not control it, but you must know when it fails.

Track from your own code:

• Request count.
• Error rate, split into rate limits (HTTP 429), server errors (5xx) and timeouts.
• Latency, including the slow end.
• Token use and cost per day, so a bug does not become a large bill.

Also subscribe to the provider's status page and route its notices to your team channel. Provider status is a useful second signal, but do not depend on it alone. Your own measurements show what your users experience.

5. Queues and background jobs

Many stores process work in the background: order sync, stock updates, email, and AI tasks such as product description generation or tagging.

Monitor:

• Queue depth: how many jobs wait.
• Job age: how old the oldest waiting job is. This is often more useful than depth.
• Failure count: jobs that failed or went to a dead-letter list.
• Heartbeats: a ping after each scheduled job finishes. A missing ping means the job did not run.

If your platform is Shopify, also watch webhook delivery. If it is WooCommerce, watch WP-Cron or the real cron that replaces it.

6. Supporting basics

These items rarely fail, but when they do the whole store goes down.

• SSL certificate expiry. Alert well before the date, for example 30 days and 7 days ahead.
• Domain expiry. Also alert ahead of time.
• DNS records. Check that the key records still resolve to the expected values.
• Email delivery. Order emails must arrive. Watch bounce and failure counts.

7. Third-party services

Your store depends on services you do not run. List them and decide which ones are critical.

Common examples are the payment provider, the shipping carrier API, the tax service, the search service, and the analytics or tag manager scripts. For each, find its status page, and subscribe to updates. Add your own check where a failure would block a sale.

How to set priorities

Not every item needs a page-at-3-a.m. alert. Sort them.

• Critical (wake someone): storefront down, checkout broken, payments failing.
• High (act within working hours): chat down, queue stuck, certificate expires within 7 days.
• Low (review weekly): slow trends, rising cost, small error increases.

An AI chat outage is usually not critical for the business. A checkout outage is. Reflect that difference in your alert rules.

Tools

You need a mix of tools: an external uptime monitor for the public pages, application logs and metrics for the LLM API and queues, and heartbeats for scheduled jobs. Uptrack (from the 2folk team) is one option for the external checks. Several other tools do the same work, so compare them on the features you need.

A starter checklist

• Monitors for home, category, product, cart and search pages.
• A checkout flow test in test mode.
• A chat endpoint check and a widget load check.
• LLM API error, latency and cost tracking.
• Queue depth, job age and failure alerts.
• Heartbeats for scheduled jobs.
• SSL, domain and DNS expiry alerts.
• Status page subscriptions for each critical third party.

Frequently asked questions

What is the single most important thing to monitor?

The checkout. If customers cannot pay, nothing else matters. Monitor the full path from product page to order confirmation, using a test mode.

Do I need to monitor the AI features separately?

Yes. They fail in different ways than the store. A store can run fine while the chat returns errors, and the reverse. Separate monitors tell you which part broke.

How many monitors are enough for a small store?

Start with a handful: home page, product page, cart, checkout, and the chat endpoint. Add more only when an incident shows a gap. Each monitor should have an owner who acts on its alerts.

Should I monitor my LLM provider's status page?

Subscribe to it, but do not rely on it alone. Status pages can lag behind real problems. Your own error and latency data is the fastest signal.