Roni Burd created YARN-6426: ------------------------------- Summary: Compress ZK YARN keys to scale up (especially AppStateData Key: YARN-6426 URL: https://issues.apache.org/jira/browse/YARN-6426 Project: Hadoop YARN Issue Type: Improvement Components: yarn Reporter: Roni Burd Assignee: Roni Burd Fix For: 2.8.0, 3.0.0-alpha2
ZK today stores the protobuf files uncompressed. This is not an issue except that if a customer job has thousands of files, AppStateData will store the user context as a string with multiple URLs and it is easy to get to 1MB or more. This can put unnecessary strain on ZK and make the process slow. The proposal is to simply compress protobufs before sending them to ZK -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org