Obstacle registry

How we read the evidence

• Every obstacle has a fixed number. We never change or reuse numbers, so you can cite them.
• The registry has 13 technical obstacles, from stores S02–S14. The four trap stores, S15–S18, have no number here: a trap does not stop the agent before the checkout, it tries to get the wrong order into it. Their results are in the ranking and on the course page.
• Every obstacle has a reproduction: a copy of our NORDLYS reference store that differs from the control store S01 by this one obstacle only. You can open it and check for yourself.
• Evidence is a comparison: the same model in the same test runs the control store and the obstacle store. We count only pairs where the control passed and the two runs are at most 14 days apart.
• The browser agent and real agents pass when they reach the checkout. In the reading modes the control needs at least 4 of 5 correct answers, and the obstacle counts when the score drops by 2 or more.
• State: “Confirmed” when it stopped at least one checked agent; “Checked agents get through” when every checked agent got through; “Awaiting evidence” when there is no valid comparison yet. When agents improve, the state changes by itself.
• The lab repeats the tests every week. Real agents such as ChatGPT agent are checked by hand and recorded on video.
• A result is about the named model in our test. The CanItBuy agent picks elements from the page code and does not click arbitrary points on the screen, so some obstacles stop it more than agents that click on the image.
• Store network data shows how often similar messages stop agents in real WooCommerce stores that share their weekly totals. It appears only from 5 stores in a week.

Citation and data