Shelf-Vision & Planogram-Compliance AI Agent

What this agent does

Counts the shelf, never certifies it

Capture fixtures from a fixed camera, a mobile unit or an autonomous scanner across supported store estates.

Pin the planogram version a capture is scored against, and refuse to score when that version is unconfirmed.

Reason

Detect facings, gaps, set order and price-label mismatches against the pinned plan.

Attach a confidence score and a capture time to every detection that leaves the model.

Mask human figures at the edge of frame on the device, and drop the raw image once the product signal is out.

Decide

Raise an internal replenishment or label task where the detection clears the threshold.

Hold every supplier-facing report, dataset or scorecard for the category manager.

Out

Retain the capture, the planogram version, the confidence and where the report went.

Execute write actions only inside the approval boundaries agreed during implementation.

→ Product statement

The agent reports what it saw, and most of that is ordinary commercial detail; the category manager decides what reaches a supplier.

Example workflow

One fixture, capture to release

AgentHuman

1Capture receivedFixed camera, mobile capture, autonomous scanner or a store-team photograph

2Frames assembledPinned planogram version, fixture map, item master and the capture time

3Detections producedFacings, gaps, label mismatches, flagged frames and confidence

4Quality checks runVersion-pin checks, human-figure masking, retention rules and confidence threshold

No human action required

Every stage before the confidence gate runs unaided, and nothing leaves the building at any of them — the agent is reporting internally, and the merchant's lane opens at the gate.

5DecisionSplits on the detection threshold

Clear detection

Goes to the category manager to approve.

Weak detection

Adds a store-audit read first.

Merchant approval

The report is held with its captures, its flagged frames and the confidence.

Approve · Amend · Send to store audit

Approved — cleared to share▼

6Merchandising systems updatedOnly where write access and approval policy allow it

7Capture evaluatedAudit agreement, merchant amendments, disputed detections and post-release corrections

Re-captures

Every merchant amendment is counted in the evaluation.

What should not run autonomously

Human approval stays in control

Outside the boundary — human approval required8 items

Releasing a dataset, report or scorecard to a supplier.

Setting the terms compliance data is shared on.

Issuing a supplier chargeback, penalty or score adjustment.

Certifying compliance for a trade-funds or allowance payment.

Automation boundaryAgent acts unaided

✓Scan fixtures and produce internal detections carrying confidence.

✓Raise an internal replenishment task on a detection for the named owner.

✓Flag a price-label-to-system mismatch for.

✓Halt itself and ask for help on obstruction.

Any write happens inside the boundaries agreed at implementation, never ahead of the merchant's release.

Authorising an autonomous unit into a customer-occupied aisle.

Overriding an obstacle-detection or safety parameter.

Changing image retention, resolution or storage location.

Confirming the authoritative planogram when the pin fails.

Example output

One detection, annotated

Everything the agent reports is attached to the capture it was drawn from.

Detection output · single fixtureIllustrative example

Fixture

Detection

Plan state

Version of record

Confidence

Where it went

Chilled dairy bay

Front rank empty across two facings, stock standing behind the front row

Set to plan

Pinned plan, current reset

87%

Internal replenishment queue

As receivedTaken from the capture and the pinned planogram — nothing on this side is inferred by the agent.

Evidence used Pinned planogram version Fixture map Earlier capture of the bay

Why this readingIt describes the facing, not the store team's execution and a person still decides.

ActionApproveAmendSend to store audit

What the score decidesBelow the configured threshold the report picks up a store-audit read before it reaches.

Value

Where AI adds value

The same four claims, placed at the point in the workflow where each one applies.

Where the value landsValue 01 – 04

Every fixtureFrom the capture

03Detection

Score against the pinned plan

Draw on the planogram version pinned for that store, the fixture map and the item master.

01Approved path

See the gap, not the guess

Routine gap and facing counts arrive already scored and already timestamped.

02Human review

Send review to what leaves

Supplier-facing reports and low-confidence detections are flagged, so the merchant's read starts where exposure sits.

04Build an evidence trail

Each report keeps its capture, its confidence and where it was sent.

Integrations

Typical integrations

Five system groups connect to the same agent. Which of them are in scope is decided in discovery.

Shelf-scanning robotsSimbe Tally
Brain Corp BrainOS

Fixed-camera shelf visionPensa Systems
Focal Systems

Shelf image recognitionTrax Retail
ParallelDots ShelfWatch

Agent

Shelf vision and planogram compliance

Reads the plan
Scores the shelf
Holds for release

Space planning and camerasBlue Yonder Category Management
Nielsen Spaceman · Verkada

Observability & evaluationOpenTelemetry · Langfuse
Supported monitoring/evaluation sources

Integration availability depends on the client's existing systems and API access.

Agent controls

Six layers between the model and the supplier

Controls nest, and what each misses is named underneath.

L6 · Outermost — last line of defenceInward → L1 · closest to the model

L6Rollback / safe modeFall back to raw capture and internal-only reporting when evaluation or production signals degrade.Roll back

L5Version monitoringTrack model, firmware, planogram-version and threshold changes.Track

L4TraceabilityRecord the capture, the version scored against, the confidence and the recipient.Record

L3Merchant approvalHold reports for the named category manager; it governs release, not whether a released number is right.Gate

L2Policy guardrailsTest detections against the pinned plan, the masking rule and the retention rule; a failure returns the report.Restrict

L1Confidence thresholdsRoute low-confidence reports to a store-audit read before the merchant sees them.Require review

Model coreReport produced — detections, flagged frames, capture time and confidence

L1 – L2Test whether a report may stand

L3Puts the release in a merchant's hands

L4 – L5Keep the capture and its confidence together

L6Falls back to raw capture when signals degrade

How Nestack evaluates it

Evaluate the whole capture path — not only the final count.

Coverage runs the whole depth of the workflow, and every layer is cut by slice.

Surface — the number the merchant reads

Depth of coverage ▼

E1Final-output evaluationDid the reported state match what a human audit found on the fixture?

E2Step-level evaluationDid the agent score against the planogram version actually in force that day?

E3Tool evaluationDid it write to the correct store, the correct fixture and the internal queue?

E4Confidence calibrationDo low-confidence detections actually attract more merchant amendments?

E5Slice evaluationHow does performance change across specific fixture types?

E6Business outcomeHow many released reports needed a correction once a supplier had read them?

Floor — the shelf the shopper walks past

Failure modes

Where each failure originates in the agent

Seven modes across the capture lifecycle.

Agent lifecycleDirection of processing →

01 · Retrieval1 mode

GX-03

Stale plan pinned

Last season's planogram is scored against a correctly reset aisle.

Stage gathersCaptures, planogram versions, with the source each came from

02 · Reasoning2 modes

GX-04

Fronted row read as empty

A shelf full behind the front rank is reported as out of stock.

GX-06

Supply gap read as execution

A distribution shortage is attributed to the store team's set.

Stage proposesDetections, flagged frames and confidence

03 · Tool / write2 modes

GX-02

Unreviewed claim written

A compliance failure reaches the vendor system before anyone has read it.

GX-05

Identifiable frame persisted

A full frame carrying shoppers is stored where the masking rule said none would be.

Stage writesOnly where write access and approval policy allow it

04 · Output1 mode

GX-01

Confidence lost in transit

A count is presented without the score that qualified it.

Stage returnsThe report the merchant releases and the supplier

05 · Change / Version1 mode

GX-07

Threshold drifts on update

A firmware or model update moves detection thresholds across the fleet overnight.

Stage tracksModel, firmware, planogram versions and thresholds

Sev-1 · a held act performed by the agent Sev-2 · a wrong number reaches a supplier Sev-3 · capture degrades, report routes to review

See the complete Retail failure-mode catalogue →

Affected slices

The average fixture is not the hard one

Start at the all-capture baseline: across every fixture scanned the false-non-compliance rate sits low. Split the same rate by fixture class and a few cohorts carry several times.

Slice performance — reported separately, not only in aggregateIllustrative example

Slice	Failure rate	Lift	Status
Small-facing, visually similar SKUs	9.5%	3.4×	Review
Chilled, frozen and gravity-fed doors	7.0%	2.5×	Review
End-caps, dump bins and seasonal	4.2%	1.5×	Watch
Large-format staples, flat gondola	1.4%	0.5×	Normal

Bar: false-non-compliance lift vs. the all-capture baseline · scale 0–4.0× · tick marks the 2.0× review threshold 2 of 4 slices over threshold

Evidence-linked improvement

A cycle is closed by a case

The loop ends where the failure becomes a standing case the next release must clear.

Improvement cycle · five stagesSwitchback — the path turns at Improve and returns at Learn

01Detect

False non-compliance rises on one fixture class.

02Diagnose

The paired captures and the pinned planogram version sit side by side until the cause narrows to one.

03Improve

The change ships versioned, with the shelf captures that exposed it attached.

04Verify

Nothing goes live while an affected case is still failing.

05Learn

It becomes a permanent case and a change to the capture rules.

Learn → DetectThe return edge. The next detection is measured against a suite this cycle lengthened.

Typical build scope

Twelve workstreams across six weeks

The build scope read against the delivery timeline. Week structure follows the six-week plan — discovery, capture, detection workflow, evaluation, integration, then production validation and handover.

Workstream Week 1Week 2Week 3Week 4Week 5Week 6

01Store and fixture discovery, and boundary definition.

02Camera and planogram-source assessment.

03Planogram-version, fixture-map and item-master mapping.

04Capture ingestion and masking at the edge.

05Detection logic and version pinning.

06Confidence scoring and merchant routing.

07Merchant release workflow.

08Shelf-system and merchandising integration.

09Detection regression cases.

10Guardrails and capture controls.

11Capture-trail instrumentation.

12Deployment, documentation and Agent Care handover.

12 workstreams · 6 weeks · bar shows the weeks a workstream is active — several run in parallel Final scope and sequence confirmed in discovery

Engagement tiers

What each tier includes

Rows are the capabilities named in each tier's scope. Higher tiers include everything below them.

Capability✓ in scope · — not at this tier PilotOne store, one category ProductionProduction store estate AdvancedMultiple banners / formats

Introduced at Pilot

Detection against your pinned plans✓✓✓

Merchant approval✓✓✓

Detection baseline✓✓✓

Introduced at Production

Reporting by fixture—✓✓

Approval workflow in your systems—✓✓

Approved internal write-back—✓✓

Shelf-system integration—✓✓

Introduced at Advanced

Multi-banner planogram rules——✓

Multi-stage merchandising approvals——✓

High capture volume——✓

Enterprise capture controls——✓

Build price From $5,000 From $8,000 Custom quote

Final build priceConfirmed after discovery based on integrations, workflow complexity, transaction volume, approval controls and deployment requirements.

Separate from buildBuild pricing is separate from recurring Agent Care, which covers managed monitoring, evaluations, incidents and verified improvements after launch.

What we need from you

What you bring, and what we build with it

Each input maps to a piece of build scope and a week in the delivery timeline.

You bringWe build with it

01Your planogram versions and fixture maps → Planogram-version and fixture mappingWeek 1

02Representative captures already taken → Detection baseline and version pinningWeek 2

03Your retention rules and entrance signage → Retention and masking-rule mapping, and automation-boundary definitionWeek 1

04Access to relevant APIs, feeds or exports → Camera, robot and capture-source assessment, then integration setupWeek 2

05Reports the merchant did not trust → Detection cases and paired capturesWeek 4

06Where a report stops and a person looks → Detection thresholds, task routing and controlsWeek 3

07A named category manager to read reports → Merchant release workflow, then pilot and production validationWeeks 5–6

Nothing else is required Deployment, documentation and Agent Care handover are ours.

Delivery timeline

Four phases across six weeks

Each band spans only the weeks it actually runs, and the fifth is doubled for a real reason.

Phase W1W2W3W4W5W6

Discovery W1

Build W2 – W3

Evaluate W4 – W5

Pilot & Launch W5 – W6

Week focus W1Fixture discovery, retention mapping and the automation boundary W2Fixture and planogram data connected W3Detection logic, confidence scoring and release controls W4Detection testing and capture guardrails W5Merchandising integration, pilot stores and targeted corrections W6A month of capture reviewed by the merchant, then handover

Reading the bandThe overlap is real: evaluation and pilot run together in week 5.

At the end of W6Verified on live fixtures, then monitoring sits with Agent Care.

DurationSix-week plan shown · typical delivery 4–6 weeks depending on scope confirmed in discovery.

Next step · Retail AI agent

Build a shelf-vision agent around the merchant who releases the report.

Show us your fixtures, your planogram versions and where your cameras already point. Your category manager decides what reaches a supplier, so we set the version pin, the masking rule and the equal-terms record before any report leaves the building.

Discuss this agent →

Nestack Agents · Shelf vision and planogram complianceAGT-RT-11 · Agent Care available after launch