[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1596: -- Labels: performance (was: ) > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Rajesh Balamohan > Labels: performance > Fix For: 0.5.2 > > Attachments: Shuffle_generateHash.png, > Shuffle_generateHash_afterFix.png, TEZ-1596.2.patch, TEZ-1596.hack.patch, > shuffle-secure-drilldown.png, shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !Shuffle_generateHash.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1596: -- Attachment: TEZ-1596.2.patch Shuffle_generateHash_afterFix.png [~gopalv], [~sseth] - Can you please review? > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Rajesh Balamohan > Attachments: Shuffle_generateHash.png, > Shuffle_generateHash_afterFix.png, TEZ-1596.2.patch, TEZ-1596.hack.patch, > shuffle-secure-drilldown.png, shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !Shuffle_generateHash.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1596: - Description: Generating the hash for YARN's secure shuffle is more expensive than the actual HTTP call once keep-alive is turned on. !Shuffle_generateHash.png! was: Generating the hash for YARN's secure shuffle is more expensive than the actual HTTP call once keep-alive is turned on. !shuffle-secure.png! > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V > Attachments: Shuffle_generateHash.png, TEZ-1596.hack.patch, > shuffle-secure-drilldown.png, shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !Shuffle_generateHash.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1596: -- Attachment: (was: Screen Shot 2014-10-24 at 4.31.04 AM.png) > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V > Attachments: Shuffle_generateHash.png, TEZ-1596.hack.patch, > shuffle-secure-drilldown.png, shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !shuffle-secure.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1596: -- Attachment: Shuffle_generateHash.png > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V > Attachments: Shuffle_generateHash.png, TEZ-1596.hack.patch, > shuffle-secure-drilldown.png, shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !shuffle-secure.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1596: -- Attachment: Screen Shot 2014-10-24 at 4.31.04 AM.png Adding the profiler snapshot of yesterday's run with latest codebase. @sseth, it would be good to fix in 0.5.2 as it helps reducing the CPU usage. Might have to slightly deviate from YARN codebase for this. ShuffleHandler on YARN would have similar issue (but that can't be avoided at this time). > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V > Attachments: Screen Shot 2014-10-24 at 4.31.04 AM.png, > TEZ-1596.hack.patch, shuffle-secure-drilldown.png, shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !shuffle-secure.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1596: - Attachment: TEZ-1596.hack.patch shuffle-secure-drilldown.png [~rajesh.balamohan]: the expensive code is in hadoop SecretManager, we end up spinning 42 instances of HMACSha1. !shuffle-secure-drilldown.png! Badly written patch attached. > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V > Attachments: TEZ-1596.hack.patch, shuffle-secure-drilldown.png, > shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !shuffle-secure.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1596) Secure Shuffle utils is extremely expensive for fast queries
[ https://issues.apache.org/jira/browse/TEZ-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1596: - Attachment: shuffle-secure.png > Secure Shuffle utils is extremely expensive for fast queries > > > Key: TEZ-1596 > URL: https://issues.apache.org/jira/browse/TEZ-1596 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V > Attachments: shuffle-secure.png > > > Generating the hash for YARN's secure shuffle is more expensive than the > actual HTTP call once keep-alive is turned on. > !shuffle-secure.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
