[jira] [Updated] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3990: - Attachment: TEZ-3990.004.patch > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642016#comment-16642016 ] Kuhu Shukla commented on TEZ-3990: -- Addressed comments by [~jeagles]. Agreed on the issues mentioned with delay calculation and testability. [~jeagles], should I go ahead and create JIRAs for these issues? > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642016#comment-16642016 ] Kuhu Shukla edited comment on TEZ-3990 at 10/8/18 3:32 PM: --- Addressed comments by [~jeagles]. Agreed on the issues mentioned with delay calculation and testability. [~jeagles], should I go ahead and create JIRAs for these issues? P.S. The unordered case doesn't seem to have the concept of penalties fyi.. which is odd.. was (Author: kshukla): Addressed comments by [~jeagles]. Agreed on the issues mentioned with delay calculation and testability. [~jeagles], should I go ahead and create JIRAs for these issues? > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[
https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642115#comment-16642115
]
TezQA commented on TEZ-3990:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} tez-runtime-library: The patch generated 8 new +
707 unchanged - 1 fixed = 715 total (was 708) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
59s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | TEZ-3990 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942846/TEZ-3990.004.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-144-generic #193-Ubuntu SMP Thu
Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus/precommit/personality/tez.sh
|
| git revision | master / 7d73bb2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_172 |
| findbugs | v3.0.1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/17/artifact/out/diff-checkstyle-tez-runtime-library.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/17/testReport/ |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output | https://builds.apache.org/job/PreCommit-TEZ-Build/17/console
|
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> The number of shuffle penalties for a host/inputAttemptIdentifier should be
> capped
> --
>
> Key: TEZ-3990
> URL: https://issues.apache.org/jira/browse/TEZ-3990
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1, 0.10.0
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch,
> TEZ-3990.003.patch, TEZ-3990.004.patch
>
>
> In a scenario where the same mapId fetches fail, the penalty code allows
> adding the same Host/InputAttemptIdentifier over and over with revised
> penalty time that grows exponentially. It should at some point drop the
> retrying a
[jira] [Commented] (TEZ-3979) Redact configuration when it's being logged, sent to the timeline server or displayed in some way
[
https://issues.apache.org/jira/browse/TEZ-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642303#comment-16642303
]
Jaume M commented on TEZ-3979:
--
ping [~harishjp]
> Redact configuration when it's being logged, sent to the timeline server or
> displayed in some way
> -
>
> Key: TEZ-3979
> URL: https://issues.apache.org/jira/browse/TEZ-3979
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.9.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: TEZ-3979.1.patch, TEZ-3979.2.patch, TEZ-3979.3.patch
>
>
> The configuration is being logged in some places like
> [here|https://github.com/apache/tez/blob/90c8195de0ddee9da3d4cd07519c8e287c85ad50/tez-dag/src/main/java/org/apache/tez/dag/history/utils/DAGUtils.java#L508]
> and
> [here|https://github.com/apache/tez/blob/90c8195de0ddee9da3d4cd07519c8e287c85ad50/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/MRTask.java#L177]
> . We should do something like [what it's done in the hadoop code
> base|https://github.com/apache/hadoop/blob/d951af22b42a22c09cfeecd3c866d5f1cd412120/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L558]
> and redact the properties held in {{mapreduce.job.redacted-properties.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642392#comment-16642392 ] Jonathan Eagles commented on TEZ-3888: -- [~ewohlstadter], [~jlowe], this is causing NoSuchMethod errors in HiveServer2. Should we limit this Jira to branch-0.10 only? Or perhaps HS2 or Tez has moved to an incompatible change > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642405#comment-16642405 ] Gopal V commented on TEZ-3888: -- bq. this is causing NoSuchMethod errors in HiveServer2. Which version is this? Since the public interface for HS2 goes over HTTP, CVE-2009-1523 needs to go back in. > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642411#comment-16642411 ] Jonathan Eagles commented on TEZ-3888: -- Not sure the HS2 version. Only the dependency. Pretty much different across the board. java.lang.NoSuchMethodError: org.eclipse.jetty.io.ssl.SslConnection.setRenegotiationLimit(I)V https://abi-laboratory.pro/index.php?view=compat_report&lang=java&l=jetty&v1=9.3.23.v20180228&v2=9.4.0.v20161208&obj=7f297&kind=bin Tez 0.9.2-SNAPSHOT: 9.3.22.v20171030 HS2: 9.3.8.v20160314 Hadoop 3.x: 9.3.19.v20170502 > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
[ https://issues.apache.org/jira/browse/TEZ-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642412#comment-16642412 ] Gopal V commented on TEZ-3888: -- Sure, standardizing the versions for ABI makes sense - I think HIVE-19421 moved HS2 ahead of where you are. > Update Jetty to org.eclipse.jetty 9.x > - > > Key: TEZ-3888 > URL: https://issues.apache.org/jira/browse/TEZ-3888 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3888.1.patch > > > mortbay Jetty 6 is no longer supported and has multiple CVEs. > Tez can't be used in scenarios where compliance against vulnerability > scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3990) The number of shuffle penalties for a host/inputAttemptIdentifier should be capped
[ https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642481#comment-16642481 ] Jonathan Eagles commented on TEZ-3990: -- [~kshukla]. Couple more very minor things I noticed that I should have caught before. - please add "final" to maxPenaltyTime - please add "shuffle" to the tez runtime parameter if you think that is correct? Also, go ahead and file the subsequent jira along with the new penalties feature for unordered. > The number of shuffle penalties for a host/inputAttemptIdentifier should be > capped > -- > > Key: TEZ-3990 > URL: https://issues.apache.org/jira/browse/TEZ-3990 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1, 0.10.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3990.001.patch, TEZ-3990.002.patch, > TEZ-3990.003.patch, TEZ-3990.004.patch > > > In a scenario where the same mapId fetches fail, the penalty code allows > adding the same Host/InputAttemptIdentifier over and over with revised > penalty time that grows exponentially. It should at some point drop the > retrying and report failure to the AM asap to allow the job to rectify the > upstream output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3497) Upgrade Jetty to 9.3.X
[ https://issues.apache.org/jira/browse/TEZ-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-3497. -- Resolution: Duplicate > Upgrade Jetty to 9.3.X > -- > > Key: TEZ-3497 > URL: https://issues.apache.org/jira/browse/TEZ-3497 > Project: Apache Tez > Issue Type: Improvement >Reporter: darion yaphet >Assignee: darion yaphet >Priority: Major > Attachments: TEZ-3497.1.patch > > > Jetty 6.X have not keep in maintain and should upgrade to the current version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3961) Tez UI web.xml tries to reach out to java.sun.com for validation after moving to jetty-9
[ https://issues.apache.org/jira/browse/TEZ-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642516#comment-16642516 ] Jonathan Eagles commented on TEZ-3961: -- [~kshukla], this looks similar to YARN-8401. Can you provide a patch? > Tez UI web.xml tries to reach out to java.sun.com for validation after moving > to jetty-9 > > > Key: TEZ-3961 > URL: https://issues.apache.org/jira/browse/TEZ-3961 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > > Tez UI can throw a 503 error when hosted on a server that cannot reach public > IPs like java.sun.com which are listed as servers for DTDs in web.xml. This > behavior change comes from moving to jetty 9 (Tez and Hadoop 3.0) which > removed provided schemas that were being shipped with earlier versions. It is > suboptimal even in cases where public IPs are accessible to fetch the DTD for > a very very simple web.xml file. We can choose to either remove the DTD > validation or add dependency explicitly to org.eclipse.jetty.toolchain » > jetty-osgi-servlet-api to allow for this jetty change to not affect the > behavior of tez-ui. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
