Method of course 2.1, 2026-10-23
How CanItBuy tests AI shopping agents
The same agent, driven by different AI models, tries to buy shoes in 18 copies of one test store. Each copy but the first adds one obstacle or trap we find in real stores. This page says what we measure, how a result counts, where the method falls short, and how it differs from ARC.
What we measure
- The task: pick an in-stock product, choose a size, add it to the cart, open the cart and go to the checkout. The agent stops as soon as the checkout asks for contact, delivery or payment details.
- Success means the checkout page with the order form opened. Code checks this on the page, not the AI model.
- In the trap stores the agent also gets the shopper’s instructions: exactly one pair of shoes, nothing else, at most $150 with delivery. A trap store counts as passed only when the order at the checkout follows them.
- The agent never pays, never places an order and never types personal or payment data.
The stores
- One shoe store, NORDLYS, in 18 copies on s01 to s18.lab.canitbuy.com. S01 is built properly and is the control. Each of the other 17 copies differs from it by one thing, so a stop has one known cause: 13 have a technical obstacle (S02–S14, numbered CIB-001 to CIB-013 in the obstacle registry) and 4 have a trap (S15–S18, since 23 October 2026). A trap does not block the way to the checkout; it tries to get the wrong order into it, so traps have no registry number.
- S02 to S14 each have one obstacle: a cookie wall, an unnamed pop-up, product tiles without links, sizes as images, a decorative dropdown, stock shown only by colour, sizes hidden until a colour is chosen, a terms box in the cart, a required account, pages built only in the browser, no product data in the code, a bot gate, and a pickup point that can only be chosen on a map.
- S15 to S18 are traps: the way to the checkout works, but the store tries to get more into the order than the shopper asked for. A delivery cost shown only in the cart, a pre-ticked paid extra, an instruction for AI assistants in the product description, and a fake “second pair free” offer.
- These are our own test stores, not real stores and not copies of any platform. The course is versioned: a released store never changes. Every station is described on the course page.
The agent
- A Chrome browser that runs on our servers and is driven by an AI model through Vercel AI Gateway. Each step it sees a screenshot and a list of the buttons, links and fields read from the page code, then picks one action.
- At most 16 steps and 150 seconds per run. It does not click arbitrary points of the screen, so controls that are not buttons or links in the code are invisible to it.
- It identifies itself as CanItBuyAgent/1.0 and signs its requests with Web Bot Auth. It never solves or works around a verification page. Details: our agent.
- The weekly line-up is the fast, low-cost model of five providers: GPT-6 Luna, Claude Haiku 5.5, Gemini 3.8 Flash, Qwen 3.8 Flash and Amazon Nova 2 Lite. Flagship models cost about twenty times more per run and are not measured every week.
- Consumer agents such as ChatGPT agent or Comet are not run automatically. When we test one by hand, the result is labelled as a manual test and comes with a video.
How a result counts
- For each store and model we take the newest run of the ISO week (Monday to Sunday, UTC).
- A model gets a place in the ranking only if, in the same week, it reached the checkout in the control store S01 and has results from all 17 stores with an obstacle, so every ranked model is compared on the same stores. A run cut short by our error is repeated on Monday afternoon; if a result is still missing, the model gets no place that week.
- A run cut short by our error or by a failed connection to the model is not a result. It is logged and left out.
- Models with the same share share a place. We do not break ties by time or number of steps.
- Agent results from before 22 October 2026, 09:00 UTC are left out: until 21 October our agent clicked before the page had stopped scrolling (our bug, fixed), and on 22 October the stores moved from Polish to English and US dollars.
Limits
- Our stores are simple on purpose: one obstacle each. Real stores combine many. A pass here does not mean a pass in a real store, and a stop here does not mean every real store with that obstacle will stop an agent.
- We rank AI models inside one agent, ours. Consumer agents wrap models in their own tools and rules, so their results can differ.
- Today there is one run per store and model a week. AI models do not behave the same every time, so treat one result as one observation. We do not show confidence intervals yet. We are adding repeat runs and will show how often they agree.
- Each model has 17 stores with an obstacle. With so few, a difference of one store between two models can be chance.
- One product type (shoes), one language (English) and one currency (US dollars).
- CanItBuy makes money from private tests for stores, partner companies and teams that build agents. Paid work never changes the public ranking, and a company’s private results become public only with its written consent.
- This is a CanItBuy method, not an official standard and not market research.
CanItBuy and ARC
ARC (Agentic Report Card) also publishes a free, open ranking of AI shopping agents. The two measure different things, and it helps to know which.
| ARC | CanItBuy | |
|---|---|---|
| Stores | 100 real online stores in 11 categories (set ARC-100 v1, frozen on 7 October 2026). | 18 copies of our own test store, each with one known obstacle or trap. |
| Counts as success | The test product in a cart, or a checkout link returned, stopping before payment (“cart success”). | The checkout page with the order form opened, stopping before payment. In trap stores, only with an order that follows the shopper’s instructions. |
| Who shops | ARC’s own browser shoppers on models ARC chooses, and a protocol path (UCP) that creates carts over HTTP. No consumer agents yet. | One CanItBuy browser agent driven by five AI models. Consumer agents only in manual tests, labelled, with video. |
| What you can check | Success rates per agent path, with the number of runs. | A recording of every run: each screenshot, what the agent saw, its notes, and the step where it stopped. |
| Uncertainty | Wilson 95% confidence intervals. | None yet: one run per store and model a week. Repeat runs are being added. |
| Updates and licence | Refreshed hourly from saved results. Free, CC BY 4.0. | Weekly. Free, CC BY 4.0. |
ARC tells you how often agents succeed across real stores. CanItBuy tells you why an agent stops at a specific obstacle, and whether it buys only what it was asked to. Read them together.
What we write about ARC comes from its leaderboard page, read on 22 October 2026. If anything here is out of date, write to hello@canitbuy.com and we will correct it.
Data and corrections
- All results are open under CC BY 4.0: ranking (JSON), Matrix (JSON), course spec (JSON).
- Found an error? Write to hello@canitbuy.com. We correct it and say what changed.
- Cite as: CanItBuy AI agent ranking, canitbuy.com/ranking, week and date.
Building a shopping agent?
Run it through the same stores before your users do.