The Product Experience: a Mind the Product podcast

When you use an AI tool, how much time do you spend correcting it? And would your customers be willing to do the same?
Kendra Vant, Chief Product Officer at Tapi, joins Randy Silver to explore the “reliability layer”: the people, checks and engineering that make AI products dependable. She explains why enthusiastic AI users often provide that layer themselves, and why product teams can’t assume their customers will.
Kendra shares four questions to ask before building AI into your product, from who catches mistakes to what reliability costs at scale. They also discuss designing for failure, the technical literacy she expects from product managers, and why faster prototyping makes experienced judgement so valuable.
Chapters
00:00 Introducing Kendra Vant
02:51 The gap between an AI demo and a reliable product
06:12 Accuracy, consistency and the user as the reliability layer
13:31 Four questions to ask before building AI into your product
18:27 Sponsor: Jira Product Discovery
19:01 Planning for failure and protecting customer trust
24:06 Customisation, pricing and the cost of scaling
26:37 Do product managers need to learn to code?
31:40 How Tapi uses AI to experiment and build
33:25 Why small teams have an advantage
35:05 Product judgement and the experience gap
38:08 Using AI to maintain legacy code
39:41 Ask better questions, write down answers and test your thinking
43:11 Where to follow Kendra
Key takeaways
Find the work your users are doing for the AI. Correcting answers, refining prompts and spotting mistakes can make a tool feel more reliable than it is. Consider whether your customers have the time, motivation and expertise to do that work.
Make someone responsible for reliability. Establish who specifies, builds and operates the checks around your AI. An impressive demo can conceal the fact that nobody owns this work yet.
Price the whole product. Human review, additional models, guardrails and ongoing maintenance all affect the cost of delivery. Check whether your reliability layer remains commercially viable as usage grows.
Design for the moments when it fails. Saying a model is wrong 15% of the time prompts a different conversation from saying it is 85% accurate. Identify the consequences for customers and plan how the product will recover.
Build your technical literacy. Kendra expects product managers to be comfortable interacting with Git and learning from the codebase. Understanding how software works helps you ask better questions and collaborate with engineers.
Recognise the value of experience. Faster tools increase what teams can build, but recognising flawed suggestions still requires judgement. Kendra raises an open question: how will newer practitioners develop that judgement as the work changes?
Use writing to sharpen your thinking. Break difficult problems into smaller questions, write down your answers and explain them to a colleague. The gaps often become clearer when you have to articulate your reasoning.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 
Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. 
Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.

Matt Feczko is director of product operations at Ocado, the UK grocery technology company whose warehouse robots and delivery automation now run for retailers including Kroger in the US and Coles in Australia. He spent the first half of his career as a customer-facing product manager, starting on SharePoint and Office at Microsoft before moving to the UK to join Skype and then four different startups over six years. A LinkedIn recruiter's typo — dropping a comma from his job title — led him to Ocado three and a half years ago, where his remit has since expanded to cover both product and engineering operations. Talking to host Randy Silver backstage at MTPcon London, he explains how he replaced Ocado's roadmap planning process, rebuilt its Jira taxonomy from scratch, and built a product management skills framework that other functions have since copied without being asked.
Key takeaways
Ocado's roadmap problem wasn't a lack of roadmaps, it was too many of them, rebuilt by hand every quarter. Before Feczko joined, product teams spent two months of every three-month cycle remaking roadmap slides in Google Slides, with no shared document explaining what the company was building or why, and no way for anyone outside the planning meetings to weigh in.
He traced that chaos to a single root cause: an inconsistent Jira taxonomy. Different teams used "epics" to mean different things, some nesting parent epics inside parent epics, so there was no agreed definition of what a feature or an initiative actually was. Feczko fixed the hierarchy first, using Denise Tilles and Melissa Perri's book Product Operations as his reference, before touching the roadmap process at all — his team nicknamed the project the "Jira Revolution".
The hard part wasn't designing the new structure, it was getting roughly 3,000 developers to adopt it. Feczko frames that adoption problem as the actual job of product operations: not producing an artefact, but changing how a large organisation works, thinks and describes its own output.
Two years on, Ocado has replaced its quarterly roadmap slide decks with an internal app, built on top of Jira data, and Feczko is aiming to run the next planning cycle with zero slides. His team also worked with finance to translate operational metrics — such as items added per basket, or driver deliveries per shift — into the financial value each initiative is expected to produce, so planning conversations use the same language as the business case.
To answer a question Ocado's product managers couldn't answer for themselves — what "good" looks like in the role — Feczko built a 12-skill framework across four themes, including execution, influence and strategy. He combined an existing Reforge framework with his CPO's own view of the job and his own 15 years of experience, then spent six months negotiating the exact wording with the wider team before any of it shipped.
The skill Feczko insisted on calling "storytelling and evangelising", over his CPO's preferred "effective communication", has become the framework's anchor. His argument: the ability to tell a clear story about a product is itself evidence of every other skill, because it requires command of the data, a defined strategy and the ability to prioritise.
The framework's real test came when other functions adopted it unprompted. Engineering, roughly 2,000 people, built its own version, as did design and data — keeping Feczko's four themes but writing skill definitions specific to each craft. That took total usage from around 150 product people to roughly 3,000 across the wider organisation.
One skill, product adoption, was added specifically because Ocado's product managers didn't see driving adoption as part of their job — they considered their job done once a feature shipped, and expected the business to work out how to use it. Feczko flags this as a gap worth checking for in any company's own skills work, whatever it ends up being called: he expects most will need their own version, along the lines of "value realisation".
Feczko's rule of thumb for when a company needs a dedicated product operations function is somewhere around 30 to 40 product managers, though he says the pressure shows up earlier: even at 10 people split across teams, a cross-functional roadmap with real dependencies is where the gap first becomes visible.
Asked what to do first, his advice is to resist fixing everything at once. Product operations, he says, is the "connective tissue" of an organisation: find where there's tension, solve that one problem fully, then move to the next — because switching between half-solved problems just turns the job into whack-a-mole.
Chapters
(00:00) Slides are dead for product planning
(00:26) Backstage at MTPcon London
(00:49) From computer science nerd to Microsoft PM
(02:02) The Skype mafia
(02:58) What Ocado actually builds
(04:38) How a LinkedIn typo led to Ocado's product ops job
(07:14) The minimum viable bureaucracy problem
(08:14) The Jira Revolution
(11:46) Killing the roadmap slide deck
(12:58) Translating metrics into value
(14:43) Building a skills framework for product managers
(17:52) Do skills differ across platform and customer teams
(19:40) Storytelling and evangelising as the master skill
(20:38) Using the framework for promotion and fairness
(22:29) Scaling the framework to 3,000 people
(23:43) How universal is the framework beyond Ocado
(24:53) Vibe-coding the skills tool with AI
(26:34) When does a company need product ops
(28:24) Product ops in the age of vibe-coded apps
(29:58) Product ops as the connective tissue
(31:37) Proving the value of a thankless job
(32:46) Bringing Ocado's product community together
(34:35) Wrap-up
Referenced
Ocado — https://www.ocadogroup.com
Product Operations: How Successful Companies Build Better Products at Scale, Melissa Perri and Denise Tilles — https://www.amazon.com/Product-Operations-successful-companies-products-ebook/dp/B0CKFKX94Z
Reforge — https://www.reforge.com
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 

We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 
Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. 
Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.

Jonathan Evens is a product lead at Google DeepMind, where he has spent more than a decade applying machine learning and AI across industries — from the smart grid at AutoGrid, to detecting roads and buildings from satellite imagery at Planet, to recommender systems, Google Search's AI Overviews and AI Mode, and now live avatars. He is also an advisor to the Evens Foundation, where he is building a "digital citizenry": a democracy sandbox that uses synthetic citizens to pre-test how the public might react to a policy before it is written.
Jonathan returns to The Product Experience, where hosts Lily Smith and Randy Silver pick up the conversation they started at MTPcon London, to dig further into what actually separates an AI product manager from a product manager who simply uses AI tools, why product principles have to come before evaluations, and how synthetic users can help — and mislead — at very different scales of product.
We discuss:
1. Why "AI product manager" has become a near-meaningless label, and the two distinct roles hiding underneath it: the modelling product manager working on core model capabilities, and the AI feature product manager building AI-powered products
2. Why using an LLM as a thinking partner or a coding assistant does not make someone an AI product manager — it makes them a product manager using AI tools, full stop
3. How Google Search's North Star metrics have stayed constant even as the proxy metrics beneath them — side-by-side win rates, user ratings, RLHF signals — have had to be rebuilt from scratch
4. Why product principles, not evaluations, are the real starting point for any AI feature, and how Google Search resolved the problem of trustworthy sources disagreeing on basic facts
5. How Google Search builds trust into its AI Overviews through sourcing, citation placement and UX cues such as highlighting, so users can judge at a glance what to verify
6. Where synthetic users genuinely help — cold-start problems, privacy-sensitive research, automated regression testing — and where they fall short
7. Building the Evens Foundation's "digital citizenry", and the core technical problem behind it: AI-generated personas that are less diverse and more extreme than real people
8. How team size and structure differ between a fully resourced lab like Google DeepMind and a resource-constrained non-profit team, and why Jonathan resists a single answer for the "right" team size
9. How the product manager's job is shifting as engineers absorb more of the evaluation work themselves through prompting and iteration
10. Jonathan's advice for product managers building AI features, and his case for following the Makers Manifesto
 Key takeaways
"AI product manager" covers two distinct jobs. The modelling product manager defines and measures a model's core capabilities — factuality, reasoning, long context — and that role is concentrated almost entirely inside frontier labs. The AI feature product manager builds a product or feature on top of an existing model, and needs domain expertise and user empathy far more than technical depth. Conflating the two is why the title has become so diluted.
Using an LLM to think faster or write code faster does not make someone an AI product manager. It makes them a product manager using AI as part of their toolkit — the same as any other knowledge worker. The distinction matters because it clarifies what skills are actually being tested.
Product principles have to come before evaluations, not after. Before Jonathan starts building an eval set for a new product, he first asks what the product is meant to feel like and what values it should encode. Google Search's response to sources disagreeing on a monument's construction date, or to large language models hallucinating at scale, came from principles about trustworthiness established before any metric was built.
Trust in an AI feature is built through sourcing and interface design as much as through the model itself. Google Search's AI Overviews are constrained to draw only from ranked, trustworthy documents rather than the model's own memory, and users are given UX signals — citation placement, highlighting — that let them judge at a glance how much to verify.
Synthetic users add genuine value in cold-start scenarios, privacy-sensitive research and automated regression testing. Where they fall short is diversity: AI-generated personas tend to be less varied and more extreme than real people, which is the central technical problem behind the Evens Foundation's digital citizenry project.
There is no fixed answer to the right team size. Jonathan sees a gradient, from a senior developer working entirely alone, up to the Evens Foundation's single product manager with AI-assisted development skills, up to a fully staffed Google team — with the deciding factor being how unsolved the underlying problem is, not company size.
As engineers absorb more evaluation work themselves through prompting and iteration, roles are blending. What still sits with product management is the judgement calls that follow from product principles — deciding, for example, which technical trade-offs actually matter to the use case, rather than which are easiest to measure.
Features links
- Evens Foundation — https://evensfoundation.eu
- Makers Manifesto — https://makersmanifesto.org
- Google DeepMind — https://deepmind.google
- AutoGrid — smart grid AI company where Jonathan began applying machine learning to industry
- Planet — satellite imagery company where Jonathan worked on automated road and building detection
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 
Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. 
Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.

Kirsten Mann is a board director, product leader and, with this episode, a three-time guest on the podcast. Last time she was here, she was building Vizory, an AI tool that helps company directors get through board packs. This time she's back to explain why she isn't building it any more. Talking to Randy Silver, she walks through exiting Vizory, the friction tests she ran before writing a line of code, and why she now believes distribution, not the build, is where most AI products actually die.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 
Key takeaways
Before writing a line of code, Mann tested for force rather than sentiment: she asked directors for two confidential board packs and an audio recording of themselves reviewing one, and only started building once 10 people agreed to do it. Coffee-shop enthusiasm is cheap; handing over a company's most sensitive document is not.
Vizory's most-loved feature in demos, cross-pack search, was barely used once shipped. Directors said they wanted it and reacted strongly when they saw it, but in practice they went straight to an automated triage view instead — a clean case of stated preference diverging from revealed preference.
Board software runs on quarterly, not weekly, cycles, and trust compounds slower than value does. Mann modelled two review cycles before directors would rely on the tool; in practice it took three, meaning some customers on quarterly boards needed close to a year before the value became obvious, a timeline no 30-day trial can accommodate.
Pricing got caught in a double anchoring trap. Directors first benchmarked Vizory against the $20-a-month ChatGPT and Claude price point, so Mann positioned against the cost of a governance failure instead, then created a second anchor by pricing early pilots at $150 a seat against a $300 target, which made the later increase read as a 100% price rise.
Cutting friction is the wrong instinct early on. Mann deliberately added friction, asking for sensitive documents and charging as early as possible, to separate real demand from polite interest, arguing that most founders run tests designed to confirm they're right rather than to find a reason to stop.
Mann's main lesson for other founders is that distribution is now the product problem. AI has collapsed the cost of building software but done nothing to the cost of reaching people who trust you, which is why she pushed to add distribution, adoption and willingness to pay as a third discipline in the Makers' Manifesto, alongside building the right thing and building it right.
Multi-model products carry a new operational risk. One of Vizory's AI "judges" silently stopped being selected because the underlying model had been deprecated overnight without warning, a reminder that model drift means the QA and evals work never really stops, however fast the initial build was.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 
Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. 
Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.

April Dunford is the positioning expert behind Obviously Awesome and a former repeat vice-president of marketing across seven venture-backed startups, most of which were acquired. Over the past decade she has advised roughly 300 B2B technology companies, all with sales teams, on how to position themselves against the competition. In this episode she joins hosts Lily Smith and Randy Silver to explain why so many companies are repositioning at once, how a strong point of view on the future underpins good positioning, and why the story a company tells about itself is a decision for product and sales, not just marketing.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 

Key takeaways
Positioning is a company-wide decision, not a marketing exercise. Dunford's own experience repositioning a spreadsheet tool as an embeddable database shows how it reshapes pricing, sales motion and even the product roadmap, not just the words on a website.
Major shifts such as the move to AI force widespread repositioning because customers no longer have a stable set of alternatives to judge value or price against, and both AI-native companies and traditional SaaS businesses are wrestling with where they now sit.
Intercom's rebuild of its support product around Fin shows how staying anchored to the customer's underlying job, resolving a ticket, can guide a genuine AI-first redesign rather than a bolt-on feature.
Companies need a clear, credible point of view on the future of their market, or customers facing genuine uncertainty will simply freeze and delay buying decisions.
A compelling vision of the future has to flow from a company's real differentiated value, not be reverse-engineered to flatter its current position.
Selling too far into a hyped future risks stalling deals, so the message needs a maturity model that shows customers a credible, low-risk path from where they are today to where the market is heading.
Positioning should be revisited on a disciplined schedule rather than settled by whoever argues loudest in the room, and Dunford recommends a structured check every six months on competitors, differentiators and value themes.
Customer advisory boards, quarterly executive account calls and a company's own sales team are the most reliable early-warning systems for shifts in budget, buying process and competitive threats, long before they show up as lost deals.
Chapters
(00:00) Introduction to April Dunford 
(01:53) The story behind Obviously Awesome 
(05:33) Positioning as a cross-functional decision 
(07:23) Why everyone is repositioning right now 
(09:32) Intercom, Fin and the AI-native rebuild 
(11:53) Building a point of view of the future 
(16:52) Is the vision genuine or self-serving 
(18:00) How to develop your point of view 
(22:13) Stack Overflow, Salesforce and staying silent 
(23:45) Hyping the future without stalling sales 
(27:01) When positioning needs to change 
(28:42) Running a regular positioning check-in 
(30:29) Customer advisory boards and voice of customer 
(34:47) Predicting change after covid and AI 
(37:20) Wrap-up

We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 
Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. 
Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.