Thanks for the sharing here as I’m also tackling similar random access improvement to Parquet for my firm’s AI use cases. Great blog.
I personally also learnt and benefited a lot from Weston Pace’s blog series[1] on how lance evolved their file format. There must be a lot of opportunities for the two communities to advance this area. Best Haocheng [1] https://www.lancedb.com/blog/file-readers-in-depth-parallelism-without-row-groups <https://www.lancedb.com/blog/file-readers-in-depth-parallelism-without-row-groups> On Fri, Jul 31, 2026 at 4:41 AM Will Edwards via dev <[email protected]> wrote: > Howdy folks :) > > At Spotify we have been exploring how to efficiently use the data lake - > something normally organised and optimised for big batch workloads - for > fast point queries too. Eg the AI agent that needs to answer your question > about what you did last summer. > > We found that if we extract metadata and put it in a fast key value store > we can know exactly which parts of which files to read without doing the > normal loading footers and searching, which dramatically changes the > performance and cost profile. And that there are some tricks you can do > when writing parquet files that helps this access patterns. It’s basically > not so different from the metadata store that speeds up analytic workloads > but by key instead. > > Here’s blog we just posted that is hopefully interesting: > > > https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries > > Happy to go into more details and share perf tips and tricks if you have > any thoughts or just wanna chat about Parquet perf! > > Best, > Will > -- Regards, Haocheng
