[ https://issues.apache.org/jira/browse/IMPALA-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-3701. ----------------------------------- Resolution: Won't Fix We expect these to be generally incompressible, so not worth pursuing. > Evaluate compressing Runtime filters to save coordinator network bandwidth > -------------------------------------------------------------------------- > > Key: IMPALA-3701 > URL: https://issues.apache.org/jira/browse/IMPALA-3701 > Project: IMPALA > Issue Type: New Feature > Components: Distributed Exec > Affects Versions: Impala 2.5.0 > Reporter: Mostafa Mokhtar > Assignee: Henry Robinson > Priority: Major > Labels: runtime-filters, scalability > Attachments: image-2016-06-08-22-55-36-966.png, query17.sql.2.out > > > When running complex queries on large clusters with lots of runtime filters > the coordinator quickly becomes network bound due to the extra incoming and > outgoing traffic for runtime filters, once the coordinator becomes network > bound all other fragments in the cluster are negatively affected as they get > blocked on shuffling/broadcasting data to the coordinator node. > This bottleneck was identified when running large scale tests on EC2 nodes > with less than ideal network throughput. > In attached png is aggregate network throughput across the 32 nodes in the > cluster with the coordinator in red. > !image-2016-06-08-22-55-36-966.png|thumbnail! > Compression should alleviate this bottleneck but we should consider other > solutions -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org