[ https://issues.apache.org/jira/browse/IMPALA-4475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693818#comment-16693818 ]
Michael Ho commented on IMPALA-4475: ------------------------------------ Whatever is done here will still have benefit even after IMPALA-7467 as we will probably keep the plan and descriptor structures in Thrift for the initial implementation of IMPALA-7467. > Compress ExecPlanFragment before shipping it to worker nodes to reduce > network traffic > -------------------------------------------------------------------------------------- > > Key: IMPALA-4475 > URL: https://issues.apache.org/jira/browse/IMPALA-4475 > Project: IMPALA > Issue Type: New Feature > Components: Distributed Exec > Affects Versions: Impala 2.6.0 > Reporter: Mostafa Mokhtar > Assignee: Vuk Ercegovac > Priority: Major > Labels: ramp-up, scalability > Attachments: count_store_returns.txt.zip, > slow_query_start_250K_partitions_134nodes.txt > > > Sending the ExecPlanFragment to remote nodes dominates the query startup time > on clusters larger than 100 nodes, size of the ExecPlanFragment grows with > number of tables, blocks and partitions in the table. > On large cluster this is limits query throughput. > From TPC-DS Q11 on 1K node cluster > {code} > Query Timeline: 5m6s > - Query submitted: 75.256us (75.256us) > - Planning finished: 1s580ms (1s580ms) > - Submit for admission: 2s376ms (795.652ms) > - Completed admission: 2s377ms (1.512ms) > - Ready to start 15993 fragment instances: 2s458ms (80.378ms) > - First dynamic filter received: 2m35s (2m33s) > - All 15993 fragment instances started: 2m35s (40.934ms) > - Rows available: 4m53s (2m17s) > - First row fetched: 4m53s (176.254ms) > - Unregister query: 4m58s (4s828ms) > - ComputeScanRangeAssignmentTimer: 600.086ms > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org