Raw Data with Rob Collie

• The Apparent Meaninglessness of AI Benchmarks, plus How to Explain AI Opportunities to Others

  Every week brings a new AI benchmark. Higher scores. Bigger claims. Louder voices insisting this changes everything. And yet, when you put AI in front of a real business problem, none of that noise seems to help. In this episode, Rob and Justin dig into why AI benchmarks often feel strangely meaningless in practice and why that disconnect is the point. Benchmarks aren't useless. They're just answering a different question than the one most businesses are asking. This isn't just random conjecture either. Rob walks through what he's learned building actual AI workflows and why a twenty percent improvement on a leaderboard rarely translates into anything you can feel on the job. They talk about why model choice usually isn't the bottleneck, why swapping models should be easy if you've built things the right way, and why the most successful AI work rarely shows up as a flashy demo. Most of the value is happening quietly, off-screen, inside systems that look a lot more like normal software than artificial intelligence. Rob and Justin also talk about why explaining AI is often harder than building it. The first demo people see tends to stick, even when it's the wrong one. Consumer AI feels magical. Business AI face plants unless it's built with intent, structure, and real context. This episode gives leaders better language for that gap, without hype or panic. If you're done chasing benchmarks and just want a way to think about AI that survives contact with reality, this episode's for you.

  --------  

  34:14

  --------

  34:14
• The Power BI Fundamentals Behind Expert Development *and* AI Simplicity, w/ Microsoft's Rui Romano

  Everyone keeps asking whether AI kills Power BI or makes it stronger. Rui Romano flips that entire question on its head. As the Microsoft PM behind PBIP, TMDL, and all the file format work that rebuilt Power BI's foundation, he explains how the platform accidentally became one of the most AI-ready systems in analytics - and it wasn't by accident, not really. His team was solving problems for real developers who were tired of unsupported workarounds and offshore relay races. They weren't training agents. But the work they did means AI now feels native instead of duct-taped on. What we learned was that the semantic model is still the highest ground in this whole space. While other tools let AI stumble through raw tables and pray the math holds up, a proper model gives AI the one thing it absolutely cannot fake: context. Relationships. Business logic that works at every level of granularity without falling apart. Rui breaks down why that matters now more than ever, why all the hardening work his team did keeps your models from exploding when an agent gets ambitious, and why the future of BI isn't about cranking out another hundred pixel-perfect dashboards. It's about fast iteration, lower friction, and answers you can trust at scale. Dashboards still matter - but only the ones people use. This conversation goes deep on architecture, not hype. Rui talks about what's changing right now, what still needs work, and why natural language will eventually beat drag-and-drop for a lot of what we do today. If you've been wondering whether to invest in real semantic modeling or just let AI figure it out from scratch every single time, this episode makes the case for why foundations always win. Always. Listen in and get ahead of the shift. And if the episode lands for you, leave us a review to help other folks find the show.

  --------  

  1:30:52

  --------

  1:30:52
• Why We Should Stop Paying Attention to the % of AI Projects which Fail (and Instead Learn Why the Others Succeed)

  This episode starts with a familiar scene. A role opens, the applications pour in, and suddenly you're staring at a mountain of resumes that deserve real attention but arrive faster than anyone can process. The mix had everything… experienced candidates, newcomers trying to break in, and a growing stack of AI-generated submissions that looked sharp until you asked a second question. That's where Haystack came in. Instead of using AI as a blunt filter, Rob and the team treated it like a collaborator. Teach it what matters. Teach it what P3 looks for in a teammate. Teach it how to separate real signal from polished noise. What came back wasn't a robot recruiter. It was clarity. And Haystack is only half the story. As the conversation unfolds, Rob and Justin zoom out into the broader pattern they're seeing across all the small, useful agents taking shape inside P3. The stuff that isn't blind hype. The stuff that quietly fixes overloaded parts of the business and makes the human decisions easier to get right. Because that's the through-line here. When AI handles the overflow, people get to spend their time on the work that actually requires judgment. Queue it up and hear what happens when AI stops pretending to be magic and starts doing real work. And if you've got a corner of the business that's begging for that kind of clarity, we can help you find the tiny build that changes everything.

  --------  

  36:59

  --------

  36:59
• Cat Negotiations, Dishonest Chatbots, AI vs AI in the Job Market, and More Real World AI Use Cases

  Rob finally cracked his years long standoff with the podcast lair cat, and the fix was hilariously simple. That small victory ends up setting the tone for the whole episode, because everything that follows has the same energy: real problems that only make sense once you shrink the solution down. As Rob talks through the cat truce, Justin brings in a different kind of chaos. A customer service bot that sounded fully in command yet never actually did the thing it said it did. Pair that with a hiring queue full of AI written applications, and the whole picture starts to come into focus. Once you see the pattern, you can't unsee it. The wins only show up when the AI job gets small. The fantasy football tool works the moment AI stops trying to scrape the entire internet and instead only writes the human part. The hiring filter works when AI is there to catch repetitive patterns, not run the whole show. Even the experiments coming out of Danielson Labs click only because the AI calls are tiny and the real work sits in regular code. Everything points in the same direction. Let AI handle the one thing only AI can do, then let normal tech take it from there.

  --------  

  44:33

  --------

  44:33
• Customizing AI for Your Business is Crucial But It Isn't Hard

  Everyone's talking about AI like it's plug-and-play. Spoiler: it's not. In this episode, Rob digs into why Big Tech's billions in AI R&D haven't yet turned into matching revenue — and what that means for the rest of us. The truth? The real business wins don't come from off-the-shelf models; they come from smart customization. Rob breaks down the "magic Lego brick" approach that separates hype from practical reality, showing how everyday tools like Power BI and Power Automate can connect to AI in surprisingly simple (and powerful) ways. He also revisits Bill Krolicki's "Vendor Bot" example to prove that you don't need to be a researcher or a billionaire to make AI deliver real results. If you've ever opened ChatGPT, asked it to "optimize operations," and gotten nowhere — this one's for you.

  --------  

  12:53

  --------

  12:53

More Business podcasts

Trending Business podcasts

About Raw Data with Rob Collie

Raw Data with Rob Collie: Podcasts in Family