Hi Priyanka, In production practice, small dimensional tables are more common. If large dimensional tables are needed, it is necessary to consider starting a Lookup Service, but Lookup Services are generally not easy to start in K8S environments.
Best, Jingsong On Sat, Jul 11, 2026 at 2:48 AM Priyanka Raju <[email protected]> wrote: > > > Hi all, we’re evaluating Paimon at Stripe for a Flink lookup join use case > and would love to hear from anyone who’s run this at scale. > > Our setup: > - 8+ lookup joins in a single Flink pipeline > - ~600TB total data across the lookup tables > - Primary key tables with frequent upserts. > - Lookup is not always by the primary key > Questions: > 1. Has anyone run Paimon lookup joins at this data volume? What were the > main operational pain points? > 2. How did you size buckets — what’s the right bucket count vs. data volume > tradeoff in practice? > 3. How effective is the local RocksDB cache on TaskManagers in practice — > does it hold up under high-cardinality keys or does it thrash? > 4. How do you handle the cold start problem after a TaskManager > restart/failover — does the cache warm-up latency cause backpressure issues? > 5. Any experience with S3 as the underlying store specifically (vs. HDFS)? > Compaction behavior and small file accumulation on S3 in particular. > 6. Is there a recommended Paimon + Flink version combination that’s proven > stable for lookup joins in production? > Any war stories, configs that helped, or things you wish you knew earlier > would be hugely appreciated. > > -- > Regards, > Priyanka
