From Data Mesh to Lake House: Revolutionizing Metadata with Lakekeeper
SummaryIn this episode, Viktor Kessler shares his journey and insights from his extensive experience in data management—from building risk management systems and data warehouses to working as a solutions architect at MongoDB and Dremio, and now co-founding a startup.Initially exploring data mesh concepts, Viktor explains how real-world challenges—such as the disconnect between technical data models and business needs, inconsistent definitions across departments, and the difficulty in managing actionable metadata—led him and his co-founder to pivot toward building a lake house solution. His startup is developing Lakekeeper, an open source REST catalog for Apache Iceberg, which aims to bridge the gap between decentralized data production and centralized metadata management. The conversation also delves into the evolution of data catalogs, the necessity for self-service analytics, and how creating consumption-ready data products can transform data functions from cost centers into profit centers. Finally, Viktor outlines ways for interested listeners to get involved with the Lakekeeper community through GitHub, upcoming meetups, and a dedicated Discord channel.Chapters00:00 Introduction to Viktor Kessler and His Journey04:57 Transitioning from Data Mesh to Lake House09:15 Understanding Data Mesh: Pain Points and Solutions13:47 The Role of Metadata in Data Management18:16 The Evolution of Catalogs and Metadata Management28:14 Stabilizing the Consumption Pipeline31:18 Centralizing Metadata for Decentralized Organizations37:09 Bridging the Gap: Technical and Business Perspectives43:17 Rethinking Data Products and Consumption50:45 Finding Balance: Control and Flexibility in Data Management
--------
57:25
Reinventing Stream Processing: From LinkedIn to Responsive with Apurva Mehta
SummaryIn this episode, Apurva Mehta, co-founder and CEO of Responsive, recounts his extensive journey in stream processing—from his early work at LinkedIn and Confluent to his current venture at Responsive. He explains how stream processing evolved from simple event ingestion and graph indexing to powering complex, stateful applications such as search indexing, inventory management, and trade settlement. Apurva clarifies the often-misunderstood concept of “real time,” arguing that low latency (often in the one- to two-second range) is more accurate for many applications than the instantaneous response many assume. He delves into the challenges of state management, discussing the limitations of embedded state stores like RocksDB and traditional databases (e.g., Postgres) when faced with high update rates and complex transactional requirements. The conversation also covers the trade-offs between SQL-based streaming interfaces and more flexible APIs, and how Responsive is innovating by decoupling state from compute—leveraging remote state solutions built on object stores (like S3) with specialized systems such as SlateDB—to improve elasticity, cost efficiency, and operational simplicity in mission-critical applications.Chapters00:00 Introduction to Apurva Mehta and Streaming Background08:50 Defining Real-Time in Streaming Contexts14:18 Challenges of Stateful Stream Processing19:50 Comparing Streaming Processing with Traditional Databases26:38 Product Perspectives on Streaming vs Analytical Systems31:10 Operational Rigor and Business Opportunities38:31 Developers' Needs: Beyond SQL45:53 Simplifying Infrastructure: The Cost of Complexity51:03 The Future of Streaming ApplicationsClick here to view the episode transcript.
--------
58:13
Semantic Layers: The Missing Link Between AI and Data with David Jayatillake from Cube
In this episode, we chat with David Jayatillake, VP of AI at Cube, about semantic layers and their crucial role in making AI work reliably with data. We explore how semantic layers act as a bridge between raw data and business meaning, and why they're more practical than pure knowledge graphs. David shares insights from his experience at Delphi Labs, where they achieved 100% accuracy in natural language data queries by combining semantic layers with AI, compared to just 16% accuracy with direct text-to-SQL approaches. We discuss the challenges of building and maintaining semantic layers, the importance of proper naming and documentation, and how AI can help automate their creation. Finally, we explore the future of semantic layers in the context of AI agents and enterprise data systems, and learn about Cube's upcoming AI-powered features for 2025.00:00 Introduction to AI and Semantic Layers05:09 The Evolution of Semantic Layers Before and After AI09:48 Challenges in Implementing Semantic Layers14:11 The Role of Semantic Layers in Data Access18:59 The Future of Semantic Layers with AI23:25 Comparing Text to SQL and Semantic Layer Approaches27:40 Limitations and Constraints of Semantic Layers30:08 Understanding LLMs and Semantic Errors35:03 The Importance of Naming in Semantic Layers37:07 Debugging Semantic Issues in LLMs38:07 The Future of LLMs as Agents41:53 Discovering Services for LLM Agents50:34 What's Next for Cube and AI Integration
--------
59:03
From black holes to AI in mathematics: AI Innovation in Mathematics and Health with Yaron Hadad
In this episode, we chat with Yaron Hadad, a fascinating individual who transitioned from theoretical physics to entrepreneurship. We explore his groundbreaking work on black holes and gravitational waves, and learn about the Ramanujan Machine - an algorithmic system he helped develop that discovers new mathematical formulas and democratizes mathematical research. We'll hear about the scientific community's mixed reactions to this innovative approach. The conversation then shifts to his work with Neutrino, a company he founded that uses AI and continuous monitoring devices to understand how food affects individual health. We delve into the complexities of nutrition science, the challenges of processing multiple data streams, and the future of personalized health monitoring. Throughout the episode, Yaron shares insights on bridging theoretical research with practical applications, and the role of AI in advancing both pure mathematics and healthcare.00:00 Yaron Hadad's Journey: From Physics to AI in Healthcare04:50 The Complexity of Einstein's Equations and Their Solutions10:12 AI in Mathematics: The Ramanujan Machine and Conjectures15:41 Navigating Criticism: The Scientific Community's Response to Innovation29:24 The Impact of Algorithms in Mathematics35:30 The Planck Machine: A New Approach41:15 Neutrino: A Personal Journey in Nutrition50:11 Connecting Food Complexity to Health Metrics
--------
59:24
Building a Native Search Engine in PostgreSQL: ParadeDB's Journey to Replace Elasticsearch with Philippe Noël
In this episode, we chat with Philippe Noël, founder of ParadeDB, about building an Elasticsearch alternative natively on PostgreSQL. We explore the challenges and benefits of extending PostgreSQL versus building a separate system, diving into topics like full-text search, faceted analytics, and why organizations need these capabilities. We discuss the emerging bring-your-own-cloud deployment model, the state of the PostgreSQL extension ecosystem, and what makes a truly production-ready database extension. Philippe shares insights on the future of search technology and how recent AI developments are actually increasing the demand for traditional search capabilities. The conversation also covers the misconceptions around PostgreSQL's scalability and the trade-offs between multi-tenant and single-tenant architectures in modern data infrastructure.Chapters00:00 Introduction to ParadeDB and Its Mission06:35 User-Facing Search and Analytics11:45 The Role of Postgres in Modern Data Solutions17:30 Future of Multimodal Databases31:04 The Rise of Fintech and Data Integrity36:36 Deployment Models: BYOC and Control Plane43:41 The Evolution of Cloud Infrastructure and Serverless Databases49:38 The Future of Search and Community EngagementClick here to view the episode transcript.
Join Kostas and Nitay as they speak with amazingly smart people who are building the next generation of technology, from hardware to cloud compute.
Tech on the Rocks is for people who are curious about the foundations of the tech industry.
Recorded primarily from our offices and homes, but one day we hope to record in a bar somewhere.
Cheers!
Ireland