Michael Smith has uploaded a new patch set (#7) to the change originally created by Joe McDonnell. ( http://gerrit.cloudera.org:8080/23228 )
Change subject: IMPALA-13548: Schedule scan ranges oldest to newest for tuple caching ...................................................................... IMPALA-13548: Schedule scan ranges oldest to newest for tuple caching Scheduling does not sort scan ranges by modification time. When a new file is added to a table, its order in the list of scan ranges is not based on modification time. Instead, it is based on which partition it belongs to and what its filename is. A new file that is added early in the list of scan ranges can cause cascading differences in scheduling. For tuple caching, this means that multiple runtime cache keys could change due to adding a single file. To minimize that disruption, this adds the ability to sort the scan ranges by modification time and schedule scan ranges oldest to newest. This enables it for scan nodes that feed into tuple cache nodes (similar to deterministic scan range assignment). Testing: - Modified TestTupleCacheFullCluster::test_scan_range_distributed to have stricter checks about how many cache keys change after an insert (only one should change) - Modified TupleCacheTest#testDeterministicScheduling to verify that oldest to newest scheduling is also enabled. Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47 --- M be/src/scheduling/scheduler.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java M fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/ddl.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-iceberg.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-ddl-parquet.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q01.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q03.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q06.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q07.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q08.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q09.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q11.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q12.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q13.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q15.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q16.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q17.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q18.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q19.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q20.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q21.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q22.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q24b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q25.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q26.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q27.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q28.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q29.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q30.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q31.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q32.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q34.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q36.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q37.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q38.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q39b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q40.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q41.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q42.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q43.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q44.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q45.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q46.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q47.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q48.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q50.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q51.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q52.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q53.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q55.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q57.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q59.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q61.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q62.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q63.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q64.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q65.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q66.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q67.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q68.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q69.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q70.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q72.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q73.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q74.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q78.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q79.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q80.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q81.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q82.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q83.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q84.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q85.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q86.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q87.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q88.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q89.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q90.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q91.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q92.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q93.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q94.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q95.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q96.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q97.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q98.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_tuple_cache/tpcds-q99.test M tests/custom_cluster/test_tuple_cache.py 111 files changed, 2,104 insertions(+), 261 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/23228/7 -- To view, visit http://gerrit.cloudera.org:8080/23228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47 Gerrit-Change-Number: 23228 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
