[
https://issues.apache.org/jira/browse/IMPALA-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-682.
----------------------------------
Resolution: Won't Fix
With the local catalog improvements, the metadata topic only contains
invalidations. So no need to invest in this.
> Improve statestore network performance with large topics
> --------------------------------------------------------
>
> Key: IMPALA-682
> URL: https://issues.apache.org/jira/browse/IMPALA-682
> Project: IMPALA
> Issue Type: Improvement
> Components: Distributed Exec
> Affects Versions: Impala 1.2.2
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Priority: Minor
>
> When the statestore has a large topic to transmit (e.g. over 100MB), a lot of
> network bandwidth will be used. This is particularly acute at startup, when
> many subscribers are competing for the complete version of a single large
> topic.
> A lot of the statestore's content is textual, and therefore likely to be very
> compressible. We can use Thrift's {{TZlibTransport}} to transparently
> compress large topics, but the problem then is that we'll be doing a lot of
> redundant work to compress the same topic many times.
> Instead, maybe we will have to serialise the Topic's thrift structure to a
> byte string (much as we do for topic values), and then compress it in a
> single thread. We can do this repeatedly in the background as topics get
> updated. There'll be some double-serialisation cost to pay, since presumably
> serialising and deserialising Thrift structs in the application involves
> another copy, but it should be worth it.
> We can also mitigate the startup problem by having subscribers wait for a lot
> longer to get their first heartbeat after registration, since the first set
> of topic updates is going to be large.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
