Hi all, we’re evaluating Paimon at Stripe for a Flink lookup join use case and would love to hear from anyone who’s run this at scale.
Our setup: - 8+ lookup joins in a single Flink pipeline - ~600TB total data across the lookup tables - Primary key tables with frequent upserts. - Lookup is not always by the primary key Questions: 1. Has anyone run Paimon lookup joins at this data volume? What were the main operational pain points? 2. How did you size buckets — what’s the right bucket count vs. data volume tradeoff in practice? 3. How effective is the local RocksDB cache on TaskManagers in practice — does it hold up under high-cardinality keys or does it thrash? 4. How do you handle the cold start problem after a TaskManager restart/failover — does the cache warm-up latency cause backpressure issues? 5. Any experience with S3 as the underlying store specifically (vs. HDFS)? Compaction behavior and small file accumulation on S3 in particular. 6. Is there a recommended Paimon + Flink version combination that’s proven stable for lookup joins in production? Any war stories, configs that helped, or things you wish you knew earlier would be hugely appreciated. -- Regards, Priyanka
