What's on the Shelf? Machine Vision, LLMs, and Modern Shelf Analytics

Walk down any grocery aisle, and you're looking at one of retail's most valuable pieces of real estate, and one of its least understood. Every shelf is a snapshot of dozens of decisions: what was ordered, what was stocked, what a supplier paid to place at eye level, and what a shopper picked up, reconsidered, and set down three feet from where it belongs. For decades, the only way to know what was actually there was to send someone to look.

Machine vision is changing that. Synaptiq has worked on shelf analytics with several companies, and the field has picked up real momentum. Autonomous robots that roll through aisles, scanning shelves as they go, are just one example. A remaining challenge is turning millions of shelf images into information a business can utilize, and most importantly, trust.

Why Shelf Analytics Matter

Shelf analytics give retailers and brands a continuous, ground-level view of their business. This value shows up in several different places.

• Catching out-of-stocks before they cost sales. An empty shelf means lost revenue, and often a lost customer who heads to a competitor. Automated detection flags missing or low products quickly, so stores can restock before the gap costs them.

• Verifying planogram compliance. Planograms, the detailed layouts that specify where every product sits, represent careful merchandising strategy and, frequently, contractual commitments to suppliers. Machine vision can confirm that what's on the shelf matches what was agreed to.

• Keeping prices accurate. Missing, outdated, or mismatched price tags frustrate shoppers and can create compliance headaches. Scanning shelves makes it possible to confirm that customers see the right price and that promotions are displayed as intended.

• Tightening inventory and supply chains. Real-time shelf data closes the gap between what the system says is in the store and what's actually available. This helps teams fine-tune inventory levels, reduce waste on perishables, and plan deliveries around customer demand.

• Gathering competitive intelligence. The same technology can track competitors' pricing, promotions, and displays across hundreds of locations, a level of market visibility that manual store audits struggle to deliver at the same speed.

Why It's Harder Than It Looks

If shelf analytics were as simple as "find the cereal box," solutions to shelf analytics challenges would have likely been solved years ago. In practice, grocery shelves are among the messiest environments a vision system can face.

Many products come in variants that look almost identical, differing only by a flavor name or a small color band on otherwise matching packaging. The same item may appear in multiple sizes that are easy to confuse at a distance or from an angle. Shoppers constantly return products to the wrong spot, so the shelf never quite matches the plan. Items are partly hidden behind other items, turned sideways, or pushed to the back. The whole problem plays out across an enormous catalog: a typical store carries tens of thousands of SKUs, and packaging changes constantly with seasonal editions, redesigns, and new launches.

Any system that depends on collecting hundreds of labeled training images for every product may struggle to keep up.

Synaptiq's Approach: Few-Shot Learning Plus Domain Knowledge

Synaptiq's solution used object detection combined with a technique called few-shot learning. Instead of requiring hundreds of annotated examples of each product, the model learns to recognize an item from very little reference material, sometimes a single manufacturer product photo. That changes the economics of the problem: adding a new SKU becomes a matter of supplying an image rather than launching a labeling project.

Just as important was building in how shelves actually behave. Similar products tend to cluster together, so the model can use neighboring items as context when a single package is ambiguous. Color works as a signal too: as you scan across a row, an unexpected shift in color is often a sign that something is out of place. Encoding knowledge like this lets the system reason about the shelf as a whole instead of treating every package as an isolated puzzle.

Even with these advances, a human in the loop remained essential. Reviewers caught and corrected the mistakes the model made, and those corrections fed back into making the system better over time. The goal was never to remove people from the process, but to point their attention towards the small fraction of cases that need it.

Can a General-Purpose LLM Do the Job?

With multimodal large language models now able to "see" images, the obvious question is whether a general-purpose model could skip specialized work. Dr. Oates put that to the test by giving ChatGPT a shelf image.

The results were impressive at first glance. The model described what was on the shelf in fluent detail and could even locate a specific product when asked. Looking closer, though, the cracks appeared. ChatGPT counted seven bags of a product when there were eight, and it struggled to correctly identify which shelf the bags were sitting on.

Those errors might sound minor, but they get to the heart of the matter. Counting and spatial placement are exactly what shelf analytics exist to measure. A system that confidently reports the wrong number of units, or puts them on the wrong shelf, produces precisely the out-of-stock and compliance errors a retailer is trying to eliminate.

This pattern shows up again and again with LLMs: the capability looks remarkable on the surface, but the mistakes are subtle, hard to spot, and even harder to fix once they're baked into a downstream process. One wrong count in a demo is a curiosity. The same error repeated across thousands of shelves and hundreds of stores quietly corrupts inventory data, restocking decisions, and supplier reporting.

Trust, but Verify

Since these errors are so easy to miss at scale, Dr. Oates recommends a trust-but-verify approach built around more deliberate prompting. Start with an initial prompt to get the model's answer. Then ask the same model to check its own work, which often surfaces mistakes it glossed over the first time. For anything higher-stakes, bring in a second, different LLM to independently corroborate the first model's answer. Agreement between models builds confidence, and disagreement tells you exactly where a human should look.

None of this makes LLMs useless for visual tasks. It means they belong inside a system designed with checks, not as a drop-in replacement for one.

Shelf analytics shows both the promise and the limits of today's AI. General-purpose models can do surprising things with a shelf photo, but accuracy drives real business decisions. 

Ready to Turn Raw Data into Trustworthy Business Intelligence?

Connect with Synaptiq’s AI experts today to build purposeful AI solutions that optimize your product placement, tracking, shelf-level execution, and more. Contact us here. 

Photo by Jakob Owens on Unsplash

About Synaptiq

Synaptiq is an AI and data science consultancy based in Portland, Oregon. We collaborate with our clients to develop human-centered products and solutions. We uphold a strong commitment to ethics and innovation. 

Contact us if you have a problem to solve, a process to refine, or a question to ask.

You can learn more about our story through our past projects, blog, or podcast.