Hi Julian, I'm writing this to address your comment and concerns raised here: https://lists.apache.org/thread/ko7jpqj5h14n2ybzfjod2c51o9yvgn1c and https://lists.apache.org/thread/n13jtvdqggpb9wss0nzvs8fwnxqtg5do I think it's better to discuss this matter in a new thread. I understand the concern about the submission volume, and no further cases will be submitted as I have completed the investigation and work of the current particular problem.
To respond, I would like to provide some context about the work. The performance of large IN predicates has been a recurring problem in Druid since at least 2019. The history, mitigations, and current reproduction are summarized in: https://github.com/apache/druid/issues/20326 Calcite PR #5263 <https://github.com/apache/calcite/pull/5263> addressed a quadratic validation path exposed by Druid’s existing InPlanningBenchmark.queryStringFunctionInSql, originally added in druid#16388 <https://github.com/apache/druid/pull/16388>. With one million string literals, Calcite 1.42.0 did not complete a measured operation after 20 minutes; with #5263, it completed in approximately 10 seconds. After #5263, I'm thinking about whether we can push this 10 seconds further. JFR profiling still showed substantial temporary allocation in both Calcite and Druid. The new proposals came from examining those allocation paths individually. The Druid-side work is tracked in #20362 <https://github.com/apache/druid/pull/20362> and #20367 <https://github.com/apache/druid/pull/20367>. I recognize that benchmark improvements do not by themselves prove that every proposed optimization is suitable for Calcite. Before these changes were submitted, I reviewed the affected implementations carefully and added focused tests, but I may have missed broader contracts or design considerations. You mentioned that several proposals contain mistaken assumptions. Could you identify the specific assumptions you found? That would help me reassess the changes and decide which should be corrected, narrowed, consolidated, or withdrawn. In particular: - #5281 <https://github.com/apache/calcite/pull/5281> relies on BasicSqlType being immutable, consistent with its current Javadoc. - #5282 <https://github.com/apache/calcite/pull/5282> depends on workload locality and therefore deserves particular discussion. - I have not yet identified the mistaken assumption in #5284 <https://github.com/apache/calcite/pull/5284>, so specific feedback would be helpful. I hope we can separate the submission-process concern from the underlying technical problem and work toward a smaller, well-supported set of changes. Lastly, I also want to address the characterization of these contributions as “slop.” I find that description dismissive and unhelpful. The proposals may contain mistakes as I'm not an expert as you are, and I welcome specific technical criticism, but they were produced through source-code analysis, JFR profiling, repeatable benchmarks, focused tests, and CI validation. They were not generated or submitted without investigation. Thanks and regards.
