[jira] [Updated] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3990: - Attachment: TEZ-3990.004.patch > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642016#comment-16642016 ] Kuhu Shukla commented on TEZ-3990: -- Addressed comments by [~jeagles]. Agreed on the issues mentioned with delay calculation and testability. [~jeagles], should I go ahead and create JIRAs for these issues? > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642016#comment-16642016 ] Kuhu Shukla edited comment on TEZ-3990 at 10/8/18 3:32 PM: --- Addressed comments by [~jeagles]. Agreed on the issues mentioned with delay calculation and testability. [~jeagles], should I go ahead and create JIRAs for these issues? P.S. The unordered case doesn't seem to have the concept of penalties fyi.. which is odd.. was (Author: kshukla): Addressed comments by [~jeagles]. Agreed on the issues mentioned with delay calculation and testability. [~jeagles], should I go ahead and create JIRAs for these issues? > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642115#comment-16642115 ] TezQA commented on TEZ-3990: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 28s{color} | {color:orange} tez-runtime-library: The patch generated 8 new + 707 unchanged - 1 fixed = 715 total (was 708) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 59s{color} | {color:green} tez-runtime-library in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 10s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 10m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | TEZ-3990 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12942846/TEZ-3990.004.patch | | Optional Tests | dupname asflicense javac javadoc unit findbugs checkstyle compile | | uname | Linux asf906.gq1.ygridcore.net 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus/precommit/personality/tez.sh | | git revision | master / 7d73bb2 | | maven | version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) | | Default Java | 1.8.0_172 | | findbugs | v3.0.1 | | checkstyle | https://builds.apache.org/job/PreCommit-TEZ-Build/17/artifact/out/diff-checkstyle-tez-runtime-library.txt | | Test Results | https://builds.apache.org/job/PreCommit-TEZ-Build/17/testReport/ | | modules | C: tez-runtime-library U: tez-runtime-library | | Console output | https://builds.apache.org/job/PreCommit-TEZ-Build/17/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying a
[jira] [Commented] (TEZ-3979) Redact configuration when it's being logged, sent to the timeline server or displayed in some way
[ https://issues.apache.org/jira/browse/TEZ-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642303#comment-16642303 ] Jaume M commented on TEZ-3979: -- ping [~harishjp] > Redact configuration when it's being logged, sent to the timeline server or > displayed in some way > - > > Key: TEZ-3979 > URL: https://issues.apache.org/jira/browse/TEZ-3979 > Project: Apache Tez > Issue Type: Bug > Components: UI >Affects Versions: 0.9.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: TEZ-3979.1.patch, TEZ-3979.2.patch, TEZ-3979.3.patch > > > The configuration is being logged in some places like > [here|https://github.com/apache/tez/blob/90c8195de0ddee9da3d4cd07519c8e287c85ad50/tez-dag/src/main/java/org/apache/tez/dag/history/utils/DAGUtils.java#L508] > and > [here|https://github.com/apache/tez/blob/90c8195de0ddee9da3d4cd07519c8e287c85ad50/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/MRTask.java#L177] > . We should do something like [what it's done in the hadoop code > base|https://github.com/apache/hadoop/blob/d951af22b42a22c09cfeecd3c866d5f1cd412120/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L558] > and redact the properties held in {{mapreduce.job.redacted-properties.}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642392#comment-16642392 ] Jonathan Eagles commented on TEZ-3888: -- [~ewohlstadter], [~jlowe], this is causing NoSuchMethod errors in HiveServer2. Should we limit this Jira to branch-0.10 only? Or perhaps HS2 or Tez has moved to an incompatible change > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642405#comment-16642405 ] Gopal V commented on TEZ-3888: -- bq. this is causing NoSuchMethod errors in HiveServer2. Which version is this? Since the public interface for HS2 goes over HTTP, CVE-2009-1523 needs to go back in. > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642411#comment-16642411 ] Jonathan Eagles commented on TEZ-3888: -- Not sure the HS2 version. Only the dependency. Pretty much different across the board. java.lang.NoSuchMethodError: org.eclipse.jetty.io.ssl.SslConnection.setRenegotiationLimit(I)V https://abi-laboratory.pro/index.php?view=compat_report&lang=java&l=jetty&v1=9.3.23.v20180228&v2=9.4.0.v20161208&obj=7f297&kind=bin Tez 0.9.2-SNAPSHOT: 9.3.22.v20171030 HS2: 9.3.8.v20160314 Hadoop 3.x: 9.3.19.v20170502 > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642412#comment-16642412 ] Gopal V commented on TEZ-3888: -- Sure, standardizing the versions for ABI makes sense - I think HIVE-19421 moved HS2 ahead of where you are. > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642481#comment-16642481 ] Jonathan Eagles commented on TEZ-3990: -- [~kshukla]. Couple more very minor things I noticed that I should have caught before. - please add "final" to maxPenaltyTime - please add "shuffle" to the tez runtime parameter if you think that is correct? Also, go ahead and file the subsequent jira along with the new penalties feature for unordered. > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3497) Upgrade Jetty to 9.3.X
[ https://issues.apache.org/jira/browse/TEZ-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-3497. -- Resolution: Duplicate > Upgrade Jetty to 9.3.X > -- > > Key: TEZ-3497 > URL: https://issues.apache.org/jira/browse/TEZ-3497 > Project: Apache Tez > Issue Type: Improvement >Reporter: darion yaphet >Assignee: darion yaphet >Priority: Major > Attachments: TEZ-3497.1.patch > > > Jetty 6.X have not keep in maintain and should upgrade to the current version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3961) Tez UI web.xml tries to reach out to java.sun.com for validation after moving to jetty-9
[ https://issues.apache.org/jira/browse/TEZ-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642516#comment-16642516 ] Jonathan Eagles commented on TEZ-3961: -- [~kshukla], this looks similar to YARN-8401. Can you provide a patch? > Tez UI web.xml tries to reach out to java.sun.com for validation after moving > to jetty-9 > > > Key: TEZ-3961 > URL: https://issues.apache.org/jira/browse/TEZ-3961 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > > Tez UI can throw a 503 error when hosted on a server that cannot reach public > IPs like java.sun.com which are listed as servers for DTDs in web.xml. This > behavior change comes from moving to jetty 9 (Tez and Hadoop 3.0) which > removed provided schemas that were being shipped with earlier versions. It is > suboptimal even in cases where public IPs are accessible to fetch the DTD for > a very very simple web.xml file. We can choose to either remove the DTD > validation or add dependency explicitly to org.eclipse.jetty.toolchain » > jetty-osgi-servlet-api to allow for this jetty change to not affect the > behavior of tez-ui. -- This message was sent by Atlassian JIRA (v7.6.3#76005)