Re: [PR] Bump dependency-check-maven from 1.3.6 to 3.2.0 [tez]
ayushtkn commented on PR #252: URL: https://github.com/apache/tez/pull/252#issuecomment-2134403820 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (TEZ-4564) TezClient to expose Tez AM host:port
[ https://issues.apache.org/jira/browse/TEZ-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849884#comment-17849884 ] Ayush Saxena commented on TEZ-4564: --- Committed to master. Thanx [~abstractdog] for the contribution!!! > TezClient to expose Tez AM host:port > > > Key: TEZ-4564 > URL: https://issues.apache.org/jira/browse/TEZ-4564 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.4 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently, there is no easy way to retrieve the AM's host and (RPC) port from > a TezClient (or even a DagClient). While implementing HIVE-28095 I'm thinking > about it to be useful as we might be interested in it later when it comes to > query tracking/history. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (TEZ-4564) TezClient to expose Tez AM host:port
[ https://issues.apache.org/jira/browse/TEZ-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved TEZ-4564. --- Resolution: Fixed > TezClient to expose Tez AM host:port > > > Key: TEZ-4564 > URL: https://issues.apache.org/jira/browse/TEZ-4564 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.4 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently, there is no easy way to retrieve the AM's host and (RPC) port from > a TezClient (or even a DagClient). While implementing HIVE-28095 I'm thinking > about it to be useful as we might be interested in it later when it comes to > query tracking/history. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] TEZ-4564: TezClient to expose Tez AM host:port [tez]
ayushtkn merged PR #356: URL: https://github.com/apache/tez/pull/356 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4355: Unit test precommit improvements - full coverage [tez]
tez-yetus commented on PR #359: URL: https://github.com/apache/tez/pull/359#issuecomment-2134065949 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 5m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 44s | master passed | | +1 :green_heart: | compile | 1m 43s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | compile | 1m 35s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | javadoc | 1m 8s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 59s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 36s | the patch passed | | +1 :green_heart: | compile | 1m 43s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javac | 1m 43s | the patch passed | | +1 :green_heart: | compile | 1m 34s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | javac | 1m 34s | the patch passed | | +1 :green_heart: | shellcheck | 0m 1s | The patch generated 0 new + 0 unchanged - 144 fixed = 0 total (was 144) | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 57s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | ||| _ Other Tests _ | | +1 :green_heart: | unit | 4m 28s | tez-dag in the patch passed. | | -1 :x: | unit | 64m 47s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. | | | | 97m 25s | | | Reason | Tests | |---:|:--| | Failed junit tests | tez.analyzer.TestAnalyzer | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/359 | | JIRA Issue | TEZ-4355 | | Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit xml compile | | uname | Linux 56081489a99d 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | build-tools/tez-personality.sh | | git revision | master / 30e9cdbff | | Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | whitespace | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/3/artifact/out/whitespace-eol.txt | | unit | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/3/artifact/out/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/3/testReport/ | | Max. process+thread count | 1523 (vs. ulimit of 5500) | | modules | C: tez-dag . U: . | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/3/console | | versions | git=2.34.1 maven=3.6.3 shellcheck=0.7.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contac
Re: [PR] TEZ-4355: Unit test precommit improvements - full coverage [tez]
tez-yetus commented on PR #359: URL: https://github.com/apache/tez/pull/359#issuecomment-2133991375 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/3/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4355: Unit test precommit improvements - full coverage [tez]
tez-yetus commented on PR #359: URL: https://github.com/apache/tez/pull/359#issuecomment-2133980639 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 3s | The patch generated 0 new + 0 unchanged - 144 fixed = 0 total (was 144) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 51s | The patch does not generate ASF License warnings. | | | | 1m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/359 | | JIRA Issue | TEZ-4355 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 8844f37da383 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | build-tools/tez-personality.sh | | git revision | master / 30e9cdbff | | Max. process+thread count | 51 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/2/console | | versions | git=2.34.1 maven=3.6.3 shellcheck=0.7.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4355: Unit test precommit improvements - full coverage [tez]
tez-yetus commented on PR #359: URL: https://github.com/apache/tez/pull/359#issuecomment-2133979454 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/2/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4355: Unit test precommit improvements - full coverage [tez]
tez-yetus commented on PR #359: URL: https://github.com/apache/tez/pull/359#issuecomment-2133967579 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 12m 55s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 1s | The patch generated 0 new + 0 unchanged - 144 fixed = 0 total (was 144) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 18s | The patch does not generate ASF License warnings. | | | | 15m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/359 | | JIRA Issue | TEZ-4355 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 3dbaa11ff714 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/tez.sh | | git revision | master / 30e9cdbff | | Max. process+thread count | 51 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-359/1/console | | versions | git=2.34.1 maven=3.6.3 shellcheck=0.7.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (TEZ-4355) Unit test precommit improvements - full coverage
[ https://issues.apache.org/jira/browse/TEZ-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4355: -- Summary: Unit test precommit improvements - full coverage (was: Unit test precommit improvements - parallel, full coverage) > Unit test precommit improvements - full coverage > > > Key: TEZ-4355 > URL: https://issues.apache.org/jira/browse/TEZ-4355 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > 1. What about running all unit tests in precommit? With the current precommit > load in Tez project, it's worth trying (however it needs some flakiness fixes) > 2. Run tests in splits in a parallel fashion: 2 different, deterministic > splits could be a) tez-tests module vs. b) all the rest -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] TEZ-4564: TezClient to expose Tez AM host:port [tez]
tez-yetus commented on PR #356: URL: https://github.com/apache/tez/pull/356#issuecomment-2133607294 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 26m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 5m 45s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 11m 45s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | compile | 1m 10s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | checkstyle | 1m 22s | master passed | | +1 :green_heart: | javadoc | 1m 11s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 59s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +0 :ok: | spotbugs | 1m 18s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 0m 50s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | checkstyle | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 35s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | findbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 14s | tez-api in the patch passed. | | +1 :green_heart: | unit | 4m 57s | tez-dag in the patch passed. | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 68m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-356/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/356 | | JIRA Issue | TEZ-4564 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile | | uname | Linux b21fb11c36e6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/tez.sh | | git revision | master / 30e9cdbff | | Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-356/4/testReport/ | | Max. process+thread count | 425 (vs. ulimit of 5500) | | modules | C: tez-api tez-dag U: . | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-356/4/console | | versions | git=2.34.1 maven=3.6.3 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4564: TezClient to expose Tez AM host:port [tez]
abstractdog commented on code in PR #356:
URL: https://github.com/apache/tez/pull/356#discussion_r1615656673
##
tez-dag/src/main/java/org/apache/tez/client/LocalClient.java:
##
@@ -204,6 +206,9 @@ public ApplicationReport getApplicationReport(ApplicationId
appId) {
report.setProgress(dagAppMaster.getProgress());
report.setAMRMToken(null);
+this.amHost = dagAppMaster.getAppNMHost();
+this.amPort = dagAppMaster.getRpcPort();
Review Comment:
getRpcPort is the actual RPC port of the Tez AM itself, to which the
dagClient connects and what's actually targeted here
nmPort/nmHttpPort are the ports of the corresponding NodeManager and might
be used for exposing container logs (nmHttpPort)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] TEZ-4564: TezClient to expose Tez AM host:port [tez]
abstractdog commented on code in PR #356:
URL: https://github.com/apache/tez/pull/356#discussion_r1615644419
##
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
##
@@ -1286,4 +1286,12 @@ public static ApplicationId appIdfromString(String
appIdStr) {
+ appIdStr, n);
}
}
+
+ public String getAmHost() {
+return frameworkClient.getAmHost();
+ }
+
+ public int getAmPort() {
+return frameworkClient.getAmPort();
+ }
Review Comment:
right, I'll make this change
a returned null value can mean that frameworkClient is not initialized at
all, or initialized but hasn't connected to the AM, from the client's point of
view, those are the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
