Re: [PR] TEZ-4552. Upgrade protobuf to 3.23.4. [tez]
slfan1989 commented on PR #345: URL: https://github.com/apache/tez/pull/345#issuecomment-2076405395 > @slfan1989 @abstractdog Can we update protobuf version to 3.24.4? Hive is also using the same version Thanks for your message, I will try to upgrade to 3.24.4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4552. Upgrade protobuf to 3.23.4. [tez]
BilwaST commented on PR #345: URL: https://github.com/apache/tez/pull/345#issuecomment-2076402664 @slfan1989 @abstractdog Can we protobuf version to 3.24.4? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4553. Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535 [tez]
tez-yetus commented on PR #347: URL: https://github.com/apache/tez/pull/347#issuecomment-2075695818 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 24m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 14m 36s | master passed | | +1 :green_heart: | compile | 0m 33s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | compile | 0m 32s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | checkstyle | 1m 18s | master passed | | +1 :green_heart: | javadoc | 0m 48s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 37s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +0 :ok: | spotbugs | 1m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 23s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | checkstyle | 0m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | findbugs | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 19s | tez-api in the patch passed. | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 51m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/347 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile | | uname | Linux e9a2ad51aff2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/tez.sh | | git revision | master / f080031f5 | | Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/testReport/ | | Max. process+thread count | 335 (vs. ulimit of 5500) | | modules | C: tez-api U: tez-api | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/console | | versions | git=2.34.1 maven=3.6.3 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] TEZ-4553. Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535 [tez]
jteagles commented on PR #347: URL: https://github.com/apache/tez/pull/347#issuecomment-2075604358 > Hi @jteagles > > * Is the maven change intentional? related to this? > * UI changes? > * How come changing the default be a solution to a problem, like avoiding Hang. > * Can you describe the problem in the description, X -> Y -> Hence hang > * You mentioned: TEZ-3535, it has ticket linked which talks about tez job getting hang, by any chance you aren't missing that? [TEZ-3491](https://issues.apache.org/jira/browse/TEZ-3491) This jira is changing the default scheduler away from the one with the known hang in TEZ-3535 and to the one added by TEZ-3491. I've been running the TEZ-3491 scheduler DagAwareTaskScheduler for 8 years. Fixed the extra commit changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (TEZ-4556) Apache Tez Release 0.10.4
László Bodor created TEZ-4556: - Summary: Apache Tez Release 0.10.4 Key: TEZ-4556 URL: https://issues.apache.org/jira/browse/TEZ-4556 Project: Apache Tez Issue Type: Improvement Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] TEZ-4555: Fail fast in LocalClient if the dirs (log, local) haven't been created [tez]
tez-yetus commented on PR #348: URL: https://github.com/apache/tez/pull/348#issuecomment-2074703530 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 22m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 14m 33s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | compile | 0m 40s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | checkstyle | 1m 29s | master passed | | +1 :green_heart: | javadoc | 0m 36s | master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 23s | master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +0 :ok: | spotbugs | 1m 45s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 | | +1 :green_heart: | javadoc | 0m 9s | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | +1 :green_heart: | findbugs | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 5m 19s | tez-dag in the patch passed. | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 52m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-348/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/348 | | JIRA Issue | TEZ-4555 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile | | uname | Linux ce2f748e3c8f 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/tez.sh | | git revision | master / f080031f5 | | Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06 | | Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-348/1/testReport/ | | Max. process+thread count | 252 (vs. ulimit of 5500) | | modules | C: tez-dag U: tez-dag | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-348/1/console | | versions | git=2.34.1 maven=3.6.3 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (TEZ-4555) Fail fast in LocalClient if the dirs (log, local) haven't been created
[
https://issues.apache.org/jira/browse/TEZ-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4555:
--
Summary: Fail fast in LocalClient if the dirs (log, local) haven't been
created (was: Fail fast in LocalClient if the dirs haven't been created)
> Fail fast in LocalClient if the dirs (log, local) haven't been created
> --
>
> Key: TEZ-4555
> URL: https://issues.apache.org/jira/browse/TEZ-4555
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> https://github.com/apache/tez/blob/f080031f5c72bc4bfd8090ccdc670bdc0f7fd090/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L332-L335
> {code}
> Path logDir = new Path(userDir, "localmode-log-dir");
> Path localDir = new Path(userDir, "localmode-local-dir");
> localFs.mkdirs(logDir);
> localFs.mkdirs(localDir);
> {code}
> in case of a non-writable local fs path (/base), this mkdirs silently returns
> with false, whereas I can see that it's not writable on my mac:
> {code}
> mkdir -p /base
> mkdir: /base: Read-only file system
> {code}
> leading to a confusing error message later:
> {code}
> 2024-04-24T02:03:52,101 ERROR [DAGAppMaster Thread] client.LocalClient: Error
> starting DAGAppMaster
> java.io.FileNotFoundException:
> /base/scratch/laszlobodor/_tez_session_dir/b76689bc-d25e-4d65-a339-44206ff57ce2/.tez/application_1713949431891_0001_wd/tez-conf.pb
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_292]
> at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_292]
> at java.io.FileInputStream.(FileInputStream.java:138)
> ~[?:1.8.0_292]
> at
> org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:84)
> ~[tez-common-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at
> org.apache.tez.client.LocalClient.createDAGAppMaster(LocalClient.java:394)
> ~[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at org.apache.tez.client.LocalClient$1.run(LocalClient.java:357)
> [tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> {code}
> actually, the fix should be done in HIVE-28212, but we need to fail fast here
> and give a hint to the user about the folder
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (TEZ-4555) Fail fast in LocalClient if the dirs haven't been created
[
https://issues.apache.org/jira/browse/TEZ-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4555:
--
Description:
https://github.com/apache/tez/blob/f080031f5c72bc4bfd8090ccdc670bdc0f7fd090/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L332-L335
{code}
Path logDir = new Path(userDir, "localmode-log-dir");
Path localDir = new Path(userDir, "localmode-local-dir");
localFs.mkdirs(logDir);
localFs.mkdirs(localDir);
{code}
in case of a non-writable local fs path (/base), this mkdirs silently returns
with false, whereas I can see that it's not writable on my mac:
{code}
mkdir -p /base
mkdir: /base: Read-only file system
{code}
leading to a confusing error message later:
{code}
2024-04-24T02:03:52,101 ERROR [DAGAppMaster Thread] client.LocalClient: Error
starting DAGAppMaster
java.io.FileNotFoundException:
/base/scratch/laszlobodor/_tez_session_dir/b76689bc-d25e-4d65-a339-44206ff57ce2/.tez/application_1713949431891_0001_wd/tez-conf.pb
(No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_292]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_292]
at java.io.FileInputStream.(FileInputStream.java:138)
~[?:1.8.0_292]
at
org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:84)
~[tez-common-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
at
org.apache.tez.client.LocalClient.createDAGAppMaster(LocalClient.java:394)
~[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
at org.apache.tez.client.LocalClient$1.run(LocalClient.java:357)
[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
{code}
actually, the fix should be done in HIVE-28212, but we need to fail fast here
and give a hint to the user about the folder
> Fail fast in LocalClient if the dirs haven't been created
> -
>
> Key: TEZ-4555
> URL: https://issues.apache.org/jira/browse/TEZ-4555
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> https://github.com/apache/tez/blob/f080031f5c72bc4bfd8090ccdc670bdc0f7fd090/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L332-L335
> {code}
> Path logDir = new Path(userDir, "localmode-log-dir");
> Path localDir = new Path(userDir, "localmode-local-dir");
> localFs.mkdirs(logDir);
> localFs.mkdirs(localDir);
> {code}
> in case of a non-writable local fs path (/base), this mkdirs silently returns
> with false, whereas I can see that it's not writable on my mac:
> {code}
> mkdir -p /base
> mkdir: /base: Read-only file system
> {code}
> leading to a confusing error message later:
> {code}
> 2024-04-24T02:03:52,101 ERROR [DAGAppMaster Thread] client.LocalClient: Error
> starting DAGAppMaster
> java.io.FileNotFoundException:
> /base/scratch/laszlobodor/_tez_session_dir/b76689bc-d25e-4d65-a339-44206ff57ce2/.tez/application_1713949431891_0001_wd/tez-conf.pb
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_292]
> at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_292]
> at java.io.FileInputStream.(FileInputStream.java:138)
> ~[?:1.8.0_292]
> at
> org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:84)
> ~[tez-common-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at
> org.apache.tez.client.LocalClient.createDAGAppMaster(LocalClient.java:394)
> ~[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at org.apache.tez.client.LocalClient$1.run(LocalClient.java:357)
> [tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> {code}
> actually, the fix should be done in HIVE-28212, but we need to fail fast here
> and give a hint to the user about the folder
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (TEZ-4555) Fail fast in LocalClient if the dirs haven't been created
[ https://issues.apache.org/jira/browse/TEZ-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned TEZ-4555: - Assignee: László Bodor > Fail fast in LocalClient if the dirs haven't been created > - > > Key: TEZ-4555 > URL: https://issues.apache.org/jira/browse/TEZ-4555 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (TEZ-4555) Fail fast in LocalClient if the dirs haven't been created
László Bodor created TEZ-4555: - Summary: Fail fast in LocalClient if the dirs haven't been created Key: TEZ-4555 URL: https://issues.apache.org/jira/browse/TEZ-4555 Project: Apache Tez Issue Type: Bug Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.10#820010)
