[jira] [Resolved] (TEZ-4120) eliminate the S3 creation in initWriter
[ https://issues.apache.org/jira/browse/TEZ-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang resolved TEZ-4120. Resolution: Invalid > eliminate the S3 creation in initWriter > --- > > Key: TEZ-4120 > URL: https://issues.apache.org/jira/browse/TEZ-4120 > Project: Apache Tez > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > Attachments: TEZ-4120.1.patch > > > There is unnecessary fs object creation in TEZ which is not used in Hive. > Such fs object creation should be eliminated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4120) eliminate the S3 creation in initWriter
[ https://issues.apache.org/jira/browse/TEZ-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang updated TEZ-4120: --- Attachment: TEZ-4120.1.patch > eliminate the S3 creation in initWriter > --- > > Key: TEZ-4120 > URL: https://issues.apache.org/jira/browse/TEZ-4120 > Project: Apache Tez > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > Attachments: TEZ-4120.1.patch > > > There is unnecessary fs object creation in TEZ which is not used in Hive. > Such fs object creation should be eliminated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4120) eliminate the S3 creation in initWriter
Richard Zhang created TEZ-4120: -- Summary: eliminate the S3 creation in initWriter Key: TEZ-4120 URL: https://issues.apache.org/jira/browse/TEZ-4120 Project: Apache Tez Issue Type: Bug Reporter: Richard Zhang There is unnecessary fs object creation in TEZ which is not used in Hive. Such fs object creation should be eliminated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TEZ-4120) eliminate the S3 creation in initWriter
[ https://issues.apache.org/jira/browse/TEZ-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned TEZ-4120: -- Assignee: Richard Zhang > eliminate the S3 creation in initWriter > --- > > Key: TEZ-4120 > URL: https://issues.apache.org/jira/browse/TEZ-4120 > Project: Apache Tez > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > > There is unnecessary fs object creation in TEZ which is not used in Hive. > Such fs object creation should be eliminated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4114) Upgrade to Jetty 9.4
[
https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027942#comment-17027942
]
Jonathan Turner Eagles commented on TEZ-4114:
-
Fixing this jira is more complicated than just updating the jetty version. Tez
branch-0.10 is targeting Hadoop 3.1.3 as it's minimum version. A little bit of
digging helped me catch up to everyone else's understanding. Documenting it
here for clarity. HADOOP-16152 is committed to trunk in hadoop and back-ported
to branch-3.1 as part of HADOOP-16703 which you had a part. This jira will be
released in 3.1.4 and isn't in a community released version.
I see no urgency based on HADOOP-16152 or HADOOP-16703 to upgrade the community
version of tez until that jira is in a release version of Hadoop. A good
work-around is to build tez and passing the jetty version on the command line
or to keep an internal version of tez which patches can be applied to, which I
assume is a best practice among distributors of hadoop software.
For example,
{code}
mvn clean install -DskipTests -Djetty.version=9.4.20.v20190813
or even
mvn clean install -DskipTests -Djetty.version=9.4.26.v20200117
{code}
One more thing. Could you help me understand why TEZ-4114.02.patch used
9.4.26.v20200117 jetty version instead of 9.4.20.v20190813 which is aligned to
Hadoop? Our general rule in tez in to track the version in the Hadoop release
unless there is sufficient reason.
Once Hadoop is release we can revisit this. We'll need to be more thorough in
that we'll need to analyze our dependencies
For example even with the patch above I see multiple jetty versions being
shipped in tez-dist/target/tez-0.10.1-SNAPSHOT/lib. While the minimal
distribution in tez-dist/target/tez-0.10.1-SNAPSHOT-minimal/lib seems fine.
> Upgrade to Jetty 9.4
>
>
> Key: TEZ-4114
> URL: https://issues.apache.org/jira/browse/TEZ-4114
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch
>
>
> Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152
> Hive: "in progress", only abandoned jiras
> https://issues.apache.org/jira/browse/HIVE-21961
> https://issues.apache.org/jira/browse/HIVE-21211
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4114) Upgrade to Jetty 9.4
[
https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027903#comment-17027903
]
TezQA commented on TEZ-4114:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 11s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | tez.common.TestTezCommonUtils |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/266/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992374/TEZ-4114.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit xml compile |
| uname | Linux 4911187be4ea 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / cf111b1 |
| Default Java | 1.8.0_242 |
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/266/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/266/testReport/ |
| Max. process+thread count | 95 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/266/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Upgrade to Jetty 9.4
>
>
> Key: TEZ-4114
> URL: https://issues.apache.org/jira/browse/TEZ-4114
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch
>
>
> Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152
> Hive: "in progress", only abandoned jiras
> https://issues.apache.org/jira/browse/HIVE-21961
> https://issues.apache.org/jira/browse/HIVE-21211
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027850#comment-17027850 ] Siyao Meng edited comment on TEZ-4114 at 1/31/20 11:10 PM: --- [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our internal (CDH/CDP) build L0 has passed (together with changes to other components). I'm submitting a rev 02 patch (just to bump up to 9.4.26, which was the latest version we tested). Please kindly review :) was (Author: smeng): [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our internal (CDH/CDP) build L0 has passed (together with changes to other components). I'm submitting a rev 02 patch. Please kindly review :) > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated TEZ-4114: Attachment: (was: TEZ-4114.02.patch) > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated TEZ-4114: Attachment: TEZ-4114.02.patch > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027850#comment-17027850 ] Siyao Meng edited comment on TEZ-4114 at 1/31/20 10:43 PM: --- [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our internal (CDH/CDP) build L0 has passed (together with changes to other components). I'm submitting a rev 02 patch. Please kindly review :) was (Author: smeng): [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our internal build L0 has passed. I'm submitting a rev 02 patch. Please kindly review :) > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027850#comment-17027850 ] Siyao Meng edited comment on TEZ-4114 at 1/31/20 10:42 PM: --- [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our internal build L0 has passed. I'm submitting a rev 02 patch. Please kindly review :) was (Author: smeng): [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our canary L0 has passed in [CDPD-4752|https://jira.cloudera.com/browse/CDPD-4752?focusedCommentId=2640606=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2640606]. I'm submitting a rev 002 patch with a new jetty version 9.4.26 (the version we tested in CDPD-4752) and submit it to gerrit. (Expect precommits in gerrit to fail because of the upgrade since other components are not upgraded at the moment. lucene-solr, hadoop, tez and CDPD commits must go in roughly the same time for the CDH-Build to pass) Please kindly review. :) Submitted Tez jetty upgrade commit to Gerrit: https://gerrit.sjc.cloudera.com/c/cdh/tez/+/68205 > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4114) Upgrade to Jetty 9.4
[
https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027854#comment-17027854
]
TezQA commented on TEZ-4114:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} TEZ-4114 does not apply to master. Rebase required? Wrong Branch?
See https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | TEZ-4114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992372/TEZ-4114.02.patch |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/265/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Upgrade to Jetty 9.4
>
>
> Key: TEZ-4114
> URL: https://issues.apache.org/jira/browse/TEZ-4114
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch
>
>
> Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152
> Hive: "in progress", only abandoned jiras
> https://issues.apache.org/jira/browse/HIVE-21961
> https://issues.apache.org/jira/browse/HIVE-21211
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027850#comment-17027850 ] Siyao Meng commented on TEZ-4114: - [~abstractdog] It seems that this isn't really blocked by TEZ-4100. As our canary L0 has passed in [CDPD-4752|https://jira.cloudera.com/browse/CDPD-4752?focusedCommentId=2640606=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2640606]. I'm submitting a rev 002 patch with a new jetty version 9.4.26 (the version we tested in CDPD-4752) and submit it to gerrit. (Expect precommits in gerrit to fail because of the upgrade since other components are not upgraded at the moment. lucene-solr, hadoop, tez and CDPD commits must go in roughly the same time for the CDH-Build to pass) Please kindly review. :) Submitted Tez jetty upgrade commit to Gerrit: https://gerrit.sjc.cloudera.com/c/cdh/tez/+/68205 > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4114) Upgrade to Jetty 9.4
[ https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated TEZ-4114: Attachment: TEZ-4114.02.patch > Upgrade to Jetty 9.4 > > > Key: TEZ-4114 > URL: https://issues.apache.org/jira/browse/TEZ-4114 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch > > > Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152 > Hive: "in progress", only abandoned jiras > https://issues.apache.org/jira/browse/HIVE-21961 > https://issues.apache.org/jira/browse/HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4106) Add Exponential Smooth RuntimeEstimator to the speculator
[
https://issues.apache.org/jira/browse/TEZ-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027846#comment-17027846
]
TezQA commented on TEZ-4106:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
7s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} tez-dag: The patch generated 2 new + 27
unchanged - 16 fixed = 29 total (was 43) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
12s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/264/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4106 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992369/TEZ-4106.005.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 96ecba85fcac 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / cf111b1 |
| Default Java | 1.8.0_242 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/264/artifact/out/diff-checkstyle-tez-dag.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/264/testReport/ |
| Max. process+thread count | 234 (vs. ulimit of 5500) |
| modules | C: tez-api tez-dag U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/264/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was
[jira] [Commented] (TEZ-4070) SSLFactory not closed in DAGClientTimelineImpl caused native memory issues
[
https://issues.apache.org/jira/browse/TEZ-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027843#comment-17027843
]
Jonathan Turner Eagles commented on TEZ-4070:
-
I'll put this in my queue to review. Thanks for supplying a patch.
> SSLFactory not closed in DAGClientTimelineImpl caused native memory issues
> --
>
> Key: TEZ-4070
> URL: https://issues.apache.org/jira/browse/TEZ-4070
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Xun REN
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4070.01.patch, TEZ-4070.02.patch, TEZ-4070.03.patch
>
>
> Hi,
> Recently, we're facing native memory issues on Redhat servers. It crashed
> completely our servers.
> *Context:*
> - HDP-2.6.5
> - Redhat 7.4
> *Problem:*
> After upgrading from HDP-2.6.2 to HDP-2.6.5, after several days running, our
> HiveServer2 can eat up to more than 100GB memory. However, we have configured
> Xmx20G and MaxMetaspace to 10GB.
> After searching, we have found the similar issue here:
> https://issues.apache.org/jira/browse/YARN-5309
> This is fixed in the hadoop-common module. Our version includes already this
> issue, however, we still have the problem.
> After searching, I have found that in the class
> org.apache.tez.dag.api.client.TimelineReaderFactory of Tez, if HTTPS is used
> for YARN, it will create SSLFactory which is not destroyed after utilization.
> TimelineReaderFactory is referenced in the class DAGClientTimelineImpl.
> If ATS is used and DAG is completed, the method switchToTimelineClient in the
> class DAGClientImpl will be called. It will close the previous HTTPClient,
> but not the SSLFactory inside. And the SSLFactory will create a thread for
> each connection. Finally, we will get thousands of threads consuming a lot
> native memories.
> {code:java}
> private void switchToTimelineClient() throws IOException, TezException {
> realClient.close();
> realClient = new DAGClientTimelineImpl(appId, dagId, conf, frameworkClient,
> (int) (2 * PRINT_STATUS_INTERVAL_MILLIS));
> if (LOG.isDebugEnabled()) {
> LOG.debug("dag completed switching to DAGClientTimelineImpl");
> }
> }{code}
> I have checked on another environment which is still on HDP-2.6.2, we also
> have a lot of running threads holding by SSLFactory. That means the problem
> is zoomed in the version HDP-2.6.5
>
> *How to reproduce the problem:*
> 1. Use Tez as Hive execution engine
> 2. Launch a Beeline session for Hive
> 3. Do a select with a simple where clause on a table
> 4. Repeat steps 2-3 in order to open different connections (it is the case
> for a shared cluster with multiple clients).
> Finally, you can check in the thread dump file, that a lot of threads are
> named "Truststore reloader thread". And the native memory usage is very high
> by doing the command "top" or "ps".
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4081) Container release idle timeout exception for equal min and max values
[
https://issues.apache.org/jira/browse/TEZ-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027839#comment-17027839
]
Jonathan Turner Eagles edited comment on TEZ-4081 at 1/31/20 9:22 PM:
--
Seems like a good fix. The added test doesn't seem to fail when I remove the
changes from main.
For example, I expect this to fail, but it succeeds. So the test doesn't appear
to test the code changes.
{noformat}
patch -p0 < TEZ-4081.01.patch
git checkout
tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java
git checkout
tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
mvn clean test -Dtest=TestTaskScheduler -pl '!tez-ui'
{noformat}
was (Author: jeagles):
Seems like a good fix. The added test doesn't seem to fail when I remove the
changes from main.
For example, I expect this to fail, but it succeeds. So the test doesn't appear
to test the code changes.
{code}
patch -p0 < TEZ-4081.01.patch
git checkout
tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java
git checkout
tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
mvn clean test -Dtest=TestTaskScheduler -pl '!tez-ui'
{code}
> Container release idle timeout exception for equal min and max values
> -
>
> Key: TEZ-4081
> URL: https://issues.apache.org/jira/browse/TEZ-4081
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Turner Eagles
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4081.01.patch
>
>
> Documentation is that Max must be greater than or equal to the minimum values.
> {code}
> /**
>* Int value. The maximum amount of time to hold on to a container if no
> task can be
>* assigned to it immediately. Only active when reuse is enabled. The value
>* must be +ve and >=
>* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS.
>* Containers will have an expire time set to a random value between
>* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS &&
>* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS. This
>* creates a graceful reduction in the amount of idle resources held
>*/
> @ConfigurationScope(Scope.AM)
> @ConfigurationProperty(type="long")
> public static final String TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS
> =
> TEZ_AM_PREFIX + "container.idle.release-timeout-max.millis";
> public static final long
> TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS_DEFAULT = 1l;
> {code}
> Yet equal values of min and max cause this exception.
> {code}
> Service Error: Error in TaskScheduler for handling Task De-allocation,
> eventType=S_TA_ENDED, scheduler=[0:TezYarn],
> taskAttemptId=attempt_1564035733375_382652_1_00_51_0,
> eventType=TASK_SCHEDULER_SERVICE_FATAL_ERROR,
> exception=org.apache.commons.math3.exception.NumberIsTooLargeException: lower
> bound (60,000) must be strictly less than upper bound (60,000)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4081) Container release idle timeout exception for equal min and max values
[
https://issues.apache.org/jira/browse/TEZ-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027839#comment-17027839
]
Jonathan Turner Eagles commented on TEZ-4081:
-
Seems like a good fix. The added test doesn't seem to fail when I remove the
changes from main.
For example, I expect this to fail, but it succeeds. So the test doesn't appear
to test the code changes.
{code}
patch -p0 < TEZ-4081.01.patch
git checkout
tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java
git checkout
tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java
mvn clean test -Dtest=TestTaskScheduler -pl '!tez-ui'
{code}
> Container release idle timeout exception for equal min and max values
> -
>
> Key: TEZ-4081
> URL: https://issues.apache.org/jira/browse/TEZ-4081
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Turner Eagles
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4081.01.patch
>
>
> Documentation is that Max must be greater than or equal to the minimum values.
> {code}
> /**
>* Int value. The maximum amount of time to hold on to a container if no
> task can be
>* assigned to it immediately. Only active when reuse is enabled. The value
>* must be +ve and >=
>* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS.
>* Containers will have an expire time set to a random value between
>* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS &&
>* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS. This
>* creates a graceful reduction in the amount of idle resources held
>*/
> @ConfigurationScope(Scope.AM)
> @ConfigurationProperty(type="long")
> public static final String TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS
> =
> TEZ_AM_PREFIX + "container.idle.release-timeout-max.millis";
> public static final long
> TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS_DEFAULT = 1l;
> {code}
> Yet equal values of min and max cause this exception.
> {code}
> Service Error: Error in TaskScheduler for handling Task De-allocation,
> eventType=S_TA_ENDED, scheduler=[0:TezYarn],
> taskAttemptId=attempt_1564035733375_382652_1_00_51_0,
> eventType=TASK_SCHEDULER_SERVICE_FATAL_ERROR,
> exception=org.apache.commons.math3.exception.NumberIsTooLargeException: lower
> bound (60,000) must be strictly less than upper bound (60,000)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4119) TestSpeculation is flaky
[ https://issues.apache.org/jira/browse/TEZ-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027833#comment-17027833 ] Ahmed Hussein commented on TEZ-4119: Thanks [~abstractdog], I will take a look. > TestSpeculation is flaky > > > Key: TEZ-4119 > URL: https://issues.apache.org/jira/browse/TEZ-4119 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: jstack.log, jstack4.log, jstack6.log, > org.apache.tez.dag.app.TestSpeculation-output.txt > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TEZ-4119) TestSpeculation is flaky
[ https://issues.apache.org/jira/browse/TEZ-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned TEZ-4119: -- Assignee: Ahmed Hussein (was: László Bodor) > TestSpeculation is flaky > > > Key: TEZ-4119 > URL: https://issues.apache.org/jira/browse/TEZ-4119 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: Ahmed Hussein >Priority: Major > Attachments: jstack.log, jstack4.log, jstack6.log, > org.apache.tez.dag.app.TestSpeculation-output.txt > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4119) TestSpeculation is flaky
[ https://issues.apache.org/jira/browse/TEZ-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027824#comment-17027824 ] László Bodor commented on TEZ-4119: --- Hi [~ahussein]! I'm not yet familiar with speculation, feel free to take this over! > TestSpeculation is flaky > > > Key: TEZ-4119 > URL: https://issues.apache.org/jira/browse/TEZ-4119 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: jstack.log, jstack4.log, jstack6.log, > org.apache.tez.dag.app.TestSpeculation-output.txt > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4106) Add Exponential Smooth RuntimeEstimator to the speculator
[ https://issues.apache.org/jira/browse/TEZ-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated TEZ-4106: --- Attachment: TEZ-4106.005.patch > Add Exponential Smooth RuntimeEstimator to the speculator > - > > Key: TEZ-4106 > URL: https://issues.apache.org/jira/browse/TEZ-4106 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: TEZ-4106.001.patch, TEZ-4106.002.patch, > TEZ-4106.003.patch, TEZ-4106.004.patch, TEZ-4106.005.patch > > > Tez speculator implements start-end runtime estimator. Similar to > [MAPREDUCE-7208|https://issues.apache.org/jira/browse/MAPREDUCE-7208], we > need to implement an adaptive estimator based on smooth Exponential -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4106) Add Exponential Smooth RuntimeEstimator to the speculator
[
https://issues.apache.org/jira/browse/TEZ-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027822#comment-17027822
]
Ahmed Hussein commented on TEZ-4106:
Thanks [~jeagles] for the feedback.
{quote}Let's clean up the TezConfiguration names if possible. Does it make
sense to put them under a top level tez.am.speculation name? Right now there is
speculator, speculative, speculation, so it may not be possible to be match
perfectly with old configurations.
{quote}
Sure thing.
{quote}Also, TEZ-4119 has been filed to address the flaky tests in
TestSpeculation. Do we need to change the patch to account for this?
{quote}
I will address TEZ-4119 separately without changing the current patch.
> Add Exponential Smooth RuntimeEstimator to the speculator
> -
>
> Key: TEZ-4106
> URL: https://issues.apache.org/jira/browse/TEZ-4106
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: TEZ-4106.001.patch, TEZ-4106.002.patch,
> TEZ-4106.003.patch, TEZ-4106.004.patch
>
>
> Tez speculator implements start-end runtime estimator. Similar to
> [MAPREDUCE-7208|https://issues.apache.org/jira/browse/MAPREDUCE-7208], we
> need to implement an adaptive estimator based on smooth Exponential
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4119) TestSpeculation is flaky
[ https://issues.apache.org/jira/browse/TEZ-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027808#comment-17027808 ] Ahmed Hussein commented on TEZ-4119: Hi [~abstractdog], thanks for reporting this issue. I recently worked on a similar flaky test case for the MAPREDUCE-7259 (testSpeculateSuccessfulWithUpdateEvents fails Intermittently). I agree with you that this could be caused by timing issues that makes the blocking thread misses the speculator thread. Have you been able to make any progress on that? If not, let me know if you want me to take over. > TestSpeculation is flaky > > > Key: TEZ-4119 > URL: https://issues.apache.org/jira/browse/TEZ-4119 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: jstack.log, jstack4.log, jstack6.log, > org.apache.tez.dag.app.TestSpeculation-output.txt > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
