[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895259#comment-17895259 ] Istvan Toth commented on MAPREDUCE-7492: 3.4.1 has the same issue. > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.2 > > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated MAPREDUCE-7492: --- Fix Version/s: 3.4.2 (was: 3.4.1) > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.2 > > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7495) Broken sort for numeric columns in UI
Alexandre Helmer created MAPREDUCE-7495: --- Summary: Broken sort for numeric columns in UI Key: MAPREDUCE-7495 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7495 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps, yarn Affects Versions: 3.3.6 Reporter: Alexandre Helmer Some columns the datatables of the UI have the sType "numeric". For instance: the elapsed time in mappers or reducers in the Mapreduce job UI, or the number of completed mappers/reducers in the Job History. However, since the version 1.10 of the datatables library, the type "numeric" has been replaced by "num". As a consequence, the columns are now sorted lexicographically. Replacing "numeric" with "num" fixes the issue. Tested on version 3.3.6, but I believe it may apply since version 3.1.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[
https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17891247#comment-17891247
]
ASF GitHub Bot commented on MAPREDUCE-7494:
---
zeekling commented on code in PR #7119:
URL: https://github.com/apache/hadoop/pull/7119#discussion_r1807794593
##
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LineRecordReader.java:
##
@@ -302,6 +302,8 @@ public synchronized void close() throws IOException {
try {
if (in != null) {
in.close();
+ } else if (fileIn != null) {
+fileIn.close();
Review Comment:
why not do this in finally ?
> File stream leak when LineRecordReader is interrupted
> -
>
> Key: MAPREDUCE-7494
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.4.0, 3.3.6, 3.4.1
>Reporter: Davin Tjong
>Assignee: Davin Tjong
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> In LineRecordReader, we may open a filestream:
> ```
> fileIn = FutureIO.awaitFuture(builder.build());
> ```
> However, we may be interrupted or otherwise throw an error before `in` is
> initialized, for example here:
> ```
> fileIn.seek(start);
> ```
> If this happens, the caller has no way to close this file stream causing it
> to leak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[
https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890742#comment-17890742
]
ASF GitHub Bot commented on MAPREDUCE-7494:
---
hadoop-yetus commented on PR #7119:
URL: https://github.com/apache/hadoop/pull/7119#issuecomment-2420753722
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 54s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 26s | | branch-3.4 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 23s | | branch-3.4 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 23s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 32s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 24s | | branch-3.4 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 21s | | branch-3.4 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 57s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 20m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 18s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 86m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7119/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7119 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6083137a0dde 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1
10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / ee28a825c8e0a688520533289315bc6d00ab2540 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7119/3/testReport/ |
| Max. process+thread count | 1600 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-clien
[jira] [Commented] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[
https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890546#comment-17890546
]
ASF GitHub Bot commented on MAPREDUCE-7494:
---
hadoop-yetus commented on PR #7119:
URL: https://github.com/apache/hadoop/pull/7119#issuecomment-2420346310
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 57s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 27s | | branch-3.4 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 24s | | branch-3.4 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 27s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 32s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 22s | | branch-3.4 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 20s | | branch-3.4 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 57s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 20m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 21s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 93m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7119/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7119 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3e30e8a9786f 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1
10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 516c73c1de6409c535219318ebebdc51522b9440 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7119/2/testReport/ |
| Max. process+thread count | 1617 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-clien
[jira] [Commented] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[
https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890229#comment-17890229
]
ASF GitHub Bot commented on MAPREDUCE-7494:
---
hadoop-yetus commented on PR #7119:
URL: https://github.com/apache/hadoop/pull/7119#issuecomment-2417753994
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 37s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 44s | | branch-3.4 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 40s | | branch-3.4 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 42s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 47s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 35s | | branch-3.4 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 30s | | branch-3.4 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 27s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 34m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 23m 49s |
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7119/1/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
| hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 164m 16s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapreduce.split.TestJobSplitWriterWithEC |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7119/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7119 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b4cfc560ba5e 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 8e0646e88e33223a0142a79a4bdc528d301e8f14 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd
[jira] [Commented] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890175#comment-17890175 ] ASF GitHub Bot commented on MAPREDUCE-7494: --- davintjong-db opened a new pull request, #7119: URL: https://github.com/apache/hadoop/pull/7119 ### Description of PR In LineRecordReader, we may open a filestream: ``` fileIn = FutureIO.awaitFuture(builder.build()); ``` However, we may be interrupted or otherwise throw an error before `in` is initialized, for example here: ``` fileIn.seek(start); ``` If this happens, the caller has no way to close this file stream causing it to leak. ### How was this patch tested? Existing tests. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6, 3.4.1 >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890026#comment-17890026 ] ASF GitHub Bot commented on MAPREDUCE-7494: --- steveloughran commented on PR #7117: URL: https://github.com/apache/hadoop/pull/7117#issuecomment-2416434667 ok, merged to trunk. Can you push up a cherry to branch 3.4 for yetus to validate it, then I'll merge to there too > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6, 3.4.1 >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7494: -- Priority: Minor (was: Major) > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.4.0, 3.3.6, 3.4.1 >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7494. --- Resolution: Fixed > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6, 3.4.1 >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7494: -- Component/s: client > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6, 3.4.1 >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7494: -- Affects Version/s: 3.3.6 3.4.0 3.4.1 > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.4.0, 3.3.6, 3.4.1 >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7494: -- Fix Version/s: 3.5.0 > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7494: - Assignee: Davin Tjong > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Davin Tjong >Assignee: Davin Tjong >Priority: Major > Labels: pull-request-available > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7494) File stream leak when LineRecordReader is interrupted
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7494: -- Summary: File stream leak when LineRecordReader is interrupted (was: Fix potential file stream leaks caused by interruption) > File stream leak when LineRecordReader is interrupted > - > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Davin Tjong >Priority: Major > Labels: pull-request-available > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7494) Fix potential file stream leaks caused by interruption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890025#comment-17890025 ] ASF GitHub Bot commented on MAPREDUCE-7494: --- steveloughran merged PR #7117: URL: https://github.com/apache/hadoop/pull/7117 > Fix potential file stream leaks caused by interruption > -- > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Davin Tjong >Priority: Major > Labels: pull-request-available > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7494) Fix potential file stream leaks caused by interruption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889885#comment-17889885
]
ASF GitHub Bot commented on MAPREDUCE-7494:
---
hadoop-yetus commented on PR #7117:
URL: https://github.com/apache/hadoop/pull/7117#issuecomment-2415427712
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 18s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 20s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 88m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7117/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7117 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 103e9f314be0 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b30f7c7c207e379c329e94d500e682dca92359be |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7117/1/testReport/ |
| Max. process+thread count | 1640 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapredu
[jira] [Commented] (MAPREDUCE-7494) Fix potential file stream leaks caused by interruption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889866#comment-17889866 ] ASF GitHub Bot commented on MAPREDUCE-7494: --- davintjong-db opened a new pull request, #7117: URL: https://github.com/apache/hadoop/pull/7117 ### Description of PR In LineRecordReader, we may open a filestream: ``` fileIn = FutureIO.awaitFuture(builder.build()); ``` However, we may be interrupted or otherwise throw an error before `in` is initialized, for example here: ``` fileIn.seek(start); ``` If this happens, the caller has no way to close this file stream causing it to leak. ### How was this patch tested? Existing tests. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Fix potential file stream leaks caused by interruption > -- > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Davin Tjong >Priority: Major > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7494) Fix potential file stream leaks caused by interruption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated MAPREDUCE-7494: -- Labels: pull-request-available (was: ) > Fix potential file stream leaks caused by interruption > -- > > Key: MAPREDUCE-7494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Davin Tjong >Priority: Major > Labels: pull-request-available > > In LineRecordReader, we may open a filestream: > ``` > fileIn = FutureIO.awaitFuture(builder.build()); > ``` > However, we may be interrupted or otherwise throw an error before `in` is > initialized, for example here: > ``` > fileIn.seek(start); > ``` > If this happens, the caller has no way to close this file stream causing it > to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7494) Fix potential file stream leaks caused by interruption
Davin Tjong created MAPREDUCE-7494: -- Summary: Fix potential file stream leaks caused by interruption Key: MAPREDUCE-7494 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7494 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Davin Tjong In LineRecordReader, we may open a filestream: ``` fileIn = FutureIO.awaitFuture(builder.build()); ``` However, we may be interrupted or otherwise throw an error before `in` is initialized, for example here: ``` fileIn.seek(start); ``` If this happens, the caller has no way to close this file stream causing it to leak. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7331) Make temporary directory used by FileOutputCommitter configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888669#comment-17888669
]
Andrew Otto commented on MAPREDUCE-7331:
[The Wikimedia Foundation was just
hit|https://phabricator.wikimedia.org/T376882] by this _temporary directory
bug. We didn't quite realize, and have been losing small amounts of data for
over a year because of it.
Having the same temp directory used by multiple jobs seems like a strange
design decision, no? Is there something I'm missing? Is there any reason for
this at all?
I suppose, if a job fails and its _temporary directory is not cleaned up, the
next job will run and end up deleting _temporary. So I guess that's good? The
downside is this bug though, which seems much worse then leftover per-job
_temporary directories.
A configurable temp directory would certainly help, but, why not just make the
default temp directory be unique per job, e.g. {{{}_temporary_\{job_id}{}}}?
> Make temporary directory used by FileOutputCommitter configurable
> -
>
> Key: MAPREDUCE-7331
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7331
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
>Affects Versions: 3.0.0
> Environment: CDH 6.2.1 Hadoop 3.0.0
>Reporter: Bimalendu Choudhary
>Priority: Major
>
> Spark SQL applications uses FileOutputCommitter to commit and merge its files
> under a table directory. The hardcoded PENDING_DIR_NAME = _temporary
> directory results in multiple application using the same temporary directory.
> This casues unwanted results of one application interfering with other
> applications temporary files. Also one application ending up deleting
> temporary files of other. There is no way right now for applications to have
> there unique path to store the temporary files to avoid any interference from
> other totally independent applications. I think the temporary directory
> being used by FileOutputCommitter should be made configurable to let the
> caller call with with its own unique value as per the requirement and avoid
> it getting deleted or overwritten by other applications
> Something like:
> {quote}public static final String PENDING_DIR_NAME_DEFAULT = "_temporary";
> public static final String PENDING_DIR_NAME_DEFAULT =
> "mapreduce.fileoutputcommitter.tempdir";
> {quote}
>
> This can be used very efficiently by Spark applications to handle even stage
> failures where temporary directories from previous attempts cause problem and
> can help in so many situations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7493) Can't find the setInputPath method
[
https://issues.apache.org/jira/browse/MAPREDUCE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
王子健 updated MAPREDUCE-7493:
---
Summary: Can't find the setInputPath method (was: setInputPath)
> Can't find the setInputPath method
> --
>
> Key: MAPREDUCE-7493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mapreduce-client
>Affects Versions: 3.4.0
> Environment: * jdk8
> * window 10
>
> org.apache.hadoop
> hadoop-mapreduce-client-common
> 3.4.0
>
>
> org.apache.hadoop
> hadoop-mapreduce-client-app
> 3.4.0
>
>
> org.apache.hadoop
> hadoop-mapreduce-client-jobclient
> 3.4.0
>
>Reporter: 王子健
>Priority: Critical
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I performed a relatively simple operation, pulling hadoop 3.4.0's
> mapreduce-client and performing a wordCount test, but following the annotated
> information in org.apache.hadoop.mapreduce.Job:
> {code:java}
> //java
> job.setInputPath(new Path(“in”)); job.
> job. setOutputPath(new Path("out ”)); {code}
> I can't find any method about setInputPath, setOutputPath, I checked some
> previous version 2.X and found this situation, is it a problem? Also I would
> love to make a fix for him
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7493) setInputPath
王子健 created MAPREDUCE-7493:
--
Summary: setInputPath
Key: MAPREDUCE-7493
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7493
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mapreduce-client
Affects Versions: 3.4.0
Environment: * jdk8
* window 10
org.apache.hadoop
hadoop-mapreduce-client-common
3.4.0
org.apache.hadoop
hadoop-mapreduce-client-app
3.4.0
org.apache.hadoop
hadoop-mapreduce-client-jobclient
3.4.0
Reporter: 王子健
I performed a relatively simple operation, pulling hadoop 3.4.0's
mapreduce-client and performing a wordCount test, but following the annotated
information in org.apache.hadoop.mapreduce.Job:
{code:java}
//java
job.setInputPath(new Path(“in”)); job.
job. setOutputPath(new Path("out ”)); {code}
I can't find any method about setInputPath, setOutputPath, I checked some
previous version 2.X and found this situation, is it a problem? Also I would
love to make a fix for him
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883773#comment-17883773 ] ASF GitHub Bot commented on MAPREDUCE-7492: --- stoty commented on PR #7058: URL: https://github.com/apache/hadoop/pull/7058#issuecomment-2367525277 Fixed in another commit. > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883772#comment-17883772 ] ASF GitHub Bot commented on MAPREDUCE-7492: --- stoty closed pull request #7058: MAPREDUCE-7492 MiniHadoopClusterManager does not accept writeDetails … URL: https://github.com/apache/hadoop/pull/7058 > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883694#comment-17883694 ] Istvan Toth commented on MAPREDUCE-7492: Thank you [~ayushtkn]. > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated MAPREDUCE-7492: --- Fix Version/s: 3.5.0 3.4.1 Resolution: Duplicate Status: Resolved (was: Patch Available) > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883277#comment-17883277 ] ASF GitHub Bot commented on MAPREDUCE-7492: --- stoty commented on PR #7058: URL: https://github.com/apache/hadoop/pull/7058#issuecomment-2363790138 Yes, that patch has the same fix. Peculiar timing. > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[
https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883264#comment-17883264
]
ASF GitHub Bot commented on MAPREDUCE-7492:
---
hadoop-yetus commented on PR #7058:
URL: https://github.com/apache/hadoop/pull/7058#issuecomment-2363726043
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 4m 15s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7058/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 2m 55s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 17s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 15s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 114m 55s | |
hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 172m 10s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7058/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7058 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2d0b210d6db6 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 358fa863d0eb2865a1d0a5f4bb2d2f1d3f8064ba |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7058/1/testReport/ |
| Max. process+thread count | 1220 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch
[jira] [Updated] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated MAPREDUCE-7492: --- Status: Patch Available (was: Open) > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883217#comment-17883217 ] ASF GitHub Bot commented on MAPREDUCE-7492: --- stoty opened a new pull request, #7058: URL: https://github.com/apache/hadoop/pull/7058 …file path ### Description of PR Fix mapred minicluster -writeDetails option ### How was this patch tested? Built Hadoop and ran the command manually. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [NA] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [NA] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated MAPREDUCE-7492: -- Labels: pull-request-available (was: ) > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883215#comment-17883215 ] Istvan Toth commented on MAPREDUCE-7492: This works in 3.3.x, but is broken in 3.4.0 and trunk. > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated MAPREDUCE-7492: --- Description: It seems that the option is missing a hasArgs() method call, which is present for the setup of the other similar arguments. (was: It seems that the option is missing a hasArgs() method call, which is present for the setup of the other similar arfuments.) > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
[ https://issues.apache.org/jira/browse/MAPREDUCE-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated MAPREDUCE-7492: --- Issue Type: Bug (was: Improvement) > MiniHadoopClusterManager does not accept writeDetails file path > --- > > Key: MAPREDUCE-7492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mapreduce-client >Affects Versions: 3.4.0 >Reporter: Istvan Toth >Priority: Minor > > It seems that the option is missing a hasArgs() method call, which is present > for the setup of the other similar arfuments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7492) MiniHadoopClusterManager does not accept writeDetails file path
Istvan Toth created MAPREDUCE-7492: -- Summary: MiniHadoopClusterManager does not accept writeDetails file path Key: MAPREDUCE-7492 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7492 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mapreduce-client Affects Versions: 3.4.0 Reporter: Istvan Toth It seems that the option is missing a hasArgs() method call, which is present for the setup of the other similar arfuments. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7491) TaskAttemptImpl:Invalid event: TA_CONTAINER_COMPLETED at ASSIGNED
John Doe created MAPREDUCE-7491: --- Summary: TaskAttemptImpl:Invalid event: TA_CONTAINER_COMPLETED at ASSIGNED Key: MAPREDUCE-7491 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7491 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: John Doe `TA_CONTAINER_COMPLETED` can arrive at an assigned task attempt before `FAIL_CONTAINER_CLEANUP`, which causes an invalid state transition. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7484) Invalid transitions for JobImpl on COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-7484:
Fix Version/s: (was: 2.1.1-beta)
> Invalid transitions for JobImpl on COMMITTING
> -
>
> Key: MAPREDUCE-7484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: John Doe
>Priority: Major
>
> {code:java}
> 2024-08-11 21:45:54,744 ERROR
> [org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl] - Can't handle this
> event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_MAP_TASK_RESCHEDULED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:965)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testCheckJobCompleteSuccess(TestJobImpl.java:212)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7490) Exposing Default Handling of Misconfigurations to Users in ManifestCommitterConfig
LoggingResearch created MAPREDUCE-7490:
--
Summary: Exposing Default Handling of Misconfigurations to Users
in ManifestCommitterConfig
Key: MAPREDUCE-7490
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7490
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mapreduce-client
Affects Versions: 3.3.6
Environment: Version: {{`3.3.6`}}
Location:
{{{}`hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/ManifestCommitterConfig.java`{}}}，
in {{`createSubmitter`}} method, from Line 385 - 388.
Reporter: LoggingResearch
Attachments: ManifestCommitterConfigTest.java,
original-vs-log-enhanced.md
When users accidentally set the parameter
{{mapreduce.manifest.committer.io.threads}} to a negative value (e.g., -80 due
to a typographical error), the system automatically resets the value to the
default of 32. While this handling prevents errors, it may lead to performance
degradation. However, without any system output indicating that the default
value has been applied, users might mistakenly believe that the system is
performing optimally with their intended setting of 80.
*Expected Behavior:*
If the parameter {{mapreduce.manifest.committer.io.threads}} is set incorrectly
due to a typographical error, the system should output a warning to notify
users of the misconfiguration and the use of the default value.
*How-to-Fix:*
We propose improving system observability by adding logging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7489) Exposing Default Handling of Misconfigurations to Users in MasterPrincipal
LoggingResearch created MAPREDUCE-7489:
--
Summary: Exposing Default Handling of Misconfigurations to Users
in MasterPrincipal
Key: MAPREDUCE-7489
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7489
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mapreduce-client
Affects Versions: 3.3.6
Environment: Version: {{`3.3.6`}}
Location:
{{{}`hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java`{}}}，
in {{`getMasterPrincipal`}} method, from Line 51 - 59.
Reporter: LoggingResearch
Attachments: MasterTest.java, original-vs-log-enhanced.md
When users attempt to configure the {{MasterPrincipal}} for local mode using
the system user {{{}root{}}}, they might set {{mapreduce.framework.name}} to
{{local}} and {{mapreduce.jobtracker.kerberos.principal}} to {{{}root{}}},
following the configuration documentation. Although the job executes, it may
not behave as expected. Users may spend considerable time troubleshooting
before discovering that setting {{mapreduce.framework.name}} to {{classic}}
yields the desired outcome. This confusion arises from the silent default
handling in the code, which is counterintuitive, as "classic" is not typically
associated with local mode. Even worse, there is no system output to indicate
this default handling, leaving users puzzled.
*Expected Behavior:*
The system should provide output indicating to users any unseen default
handling regarding {{{}MasterPrincipal{}}}.
*How-to-Fix:*
We propose improving system transparency by adding logging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7488) Missing System Output for Misconfiguration in Local Mode Settings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879091#comment-17879091
]
LoggingResearch commented on MAPREDUCE-7488:
To make it clear on how to fix the issue, we upload the comparison between the
original version and the log-enhanced version in the attached file
`original-vs-log-enhanced.md`. The log-enhanced version is generated
automatically utilizing our proposed tool.
> Missing System Output for Misconfiguration in Local Mode Settings
> -
>
> Key: MAPREDUCE-7488
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7488
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mapreduce-client
>Affects Versions: 3.3.6
> Environment: Version: {{`3.3.6`}}
> Location:
> {{{}`hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java`{}}}，
> in {{`create`}} method, from Line 37 - 39.
>Reporter: LoggingResearch
>Priority: Major
> Attachments: LocalClientProtocolProviderTest.java,
> original-vs-log-enhanced.md
>
>
> Enabling {{local mode}} in {{mapreduce}} requires setting the
> {{mapreduce.framework.name}} configuration parameter to {{{}local{}}}.
> However, users who are unfamiliar with configuration parameters may struggle
> to do this correctly. The system currently lacks runtime output indicating
> whether this configuration has taken effect, leading to confusion and a
> higher likelihood of misconfiguration.
>
> *Expected Behavior:*
> If the system fails to activate the {{LocalJobRunner}} because the
> {{mapreduce.framework.name}} parameter is not set to {{{}local{}}}, a warning
> should be output to notify the user.
>
> *How-to-Fix:*
> We propose enhancing system observability and diagnosability by adding
> logging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7488) Missing System Output for Misconfiguration in Local Mode Settings
LoggingResearch created MAPREDUCE-7488:
--
Summary: Missing System Output for Misconfiguration in Local Mode
Settings
Key: MAPREDUCE-7488
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7488
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mapreduce-client
Affects Versions: 3.3.6
Environment: Version: {{`3.3.6`}}
Location:
{{{}`hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java`{}}}，
in {{`create`}} method, from Line 37 - 39.
Reporter: LoggingResearch
Attachments: LocalClientProtocolProviderTest.java,
original-vs-log-enhanced.md
Enabling {{local mode}} in {{mapreduce}} requires setting the
{{mapreduce.framework.name}} configuration parameter to {{{}local{}}}. However,
users who are unfamiliar with configuration parameters may struggle to do this
correctly. The system currently lacks runtime output indicating whether this
configuration has taken effect, leading to confusion and a higher likelihood of
misconfiguration.
*Expected Behavior:*
If the system fails to activate the {{LocalJobRunner}} because the
{{mapreduce.framework.name}} parameter is not set to {{{}local{}}}, a warning
should be output to notify the user.
*How-to-Fix:*
We propose enhancing system observability and diagnosability by adding logging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7487) Missing System Output for Misconfiguration in KeyFieldBasedPartitioner Settings
[
https://issues.apache.org/jira/browse/MAPREDUCE-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879090#comment-17879090
]
LoggingResearch commented on MAPREDUCE-7487:
To make it clear on how to fix the issue, we upload the comparison between the
original version and the log-enhanced version in the attached file
`original-vs-log-enhanced.md`. The log-enhanced version is generated
automatically utilizing our proposed tool.
> Missing System Output for Misconfiguration in KeyFieldBasedPartitioner
> Settings
> ---
>
> Key: MAPREDUCE-7487
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7487
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mapreduce-client
>Affects Versions: 3.3.6
> Environment: Version: {{`3.3.6`}}
> Location:
> {{{}hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java{}}}，
> in {{`setConf`}} method, from Line 76 - 77.
>Reporter: LoggingResearch
>Priority: Major
> Attachments: KeyFieldBasedPartitionerTest.java,
> original-vs-log-enhanced.md
>
>
> To effectively use {{{}KeyFieldBasedPartitioner{}}}, users can optionally set
> the {{mapreduce.partition.keypartitioner.options}} configuration parameter to
> define partitioning behavior. However, this parameter is not documented in
> {{{}mapreduce-default.xml{}}}, leaving users without guidance on how to
> properly configure it. This omission hinders users from fully leveraging the
> system’s configurability.
>
> *Expected Behavior:*
> The system should provide warnings or informative logs to guide users in case
> of misconfiguration. Relying solely on users to carefully read through the
> configuration documentation is insufficient, particularly when the necessary
> details are not included in the documentation.
>
> *How-to-Fix:*
> To improve user experience, we suggest enhancing the system with additional
> logging to expose these configurations and their correct usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7487) Missing System Output for Misconfiguration in KeyFieldBasedPartitioner Settings
LoggingResearch created MAPREDUCE-7487:
--
Summary: Missing System Output for Misconfiguration in
KeyFieldBasedPartitioner Settings
Key: MAPREDUCE-7487
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7487
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mapreduce-client
Affects Versions: 3.3.6
Environment: Version: {{`3.3.6`}}
Location:
{{{}hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java{}}}，
in {{`setConf`}} method, from Line 76 - 77.
Reporter: LoggingResearch
Attachments: KeyFieldBasedPartitionerTest.java,
original-vs-log-enhanced.md
To effectively use {{{}KeyFieldBasedPartitioner{}}}, users can optionally set
the {{mapreduce.partition.keypartitioner.options}} configuration parameter to
define partitioning behavior. However, this parameter is not documented in
{{{}mapreduce-default.xml{}}}, leaving users without guidance on how to
properly configure it. This omission hinders users from fully leveraging the
system’s configurability.
*Expected Behavior:*
The system should provide warnings or informative logs to guide users in case
of misconfiguration. Relying solely on users to carefully read through the
configuration documentation is insufficient, particularly when the necessary
details are not included in the documentation.
*How-to-Fix:*
To improve user experience, we suggest enhancing the system with additional
logging to expose these configurations and their correct usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7486) Handling Cluster Storage Capacity Exceeded Exception with Enhanced Logging
[
https://issues.apache.org/jira/browse/MAPREDUCE-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879081#comment-17879081
]
LoggingResearch commented on MAPREDUCE-7486:
To make it clear on how to fix the issue, we upload the comparison between the
original version and the log-enhanced version in the attached file
`original-vs-log-enhanced.md`. The log-enhanced version is generated
automatically utilizing our proposed tool.
> Handling Cluster Storage Capacity Exceeded Exception with Enhanced Logging
> --
>
> Key: MAPREDUCE-7486
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7486
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mapreduce-client
>Affects Versions: 3.3.6
> Environment: Version: {{`3.3.6`}}
> Location:
> {{{}`hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java`{}}}，
> in {{`reportError`}} method, starting from Line 241-250.
>Reporter: LoggingResearch
>Priority: Major
> Attachments: TestYarnChild.java, original-vs-log-enhanced.md
>
>
> The existing {{reportError}} method in {{YarnChild.java}} is responsible for
> handling exceptions during job execution. However, when the exception is due
> to the cluster storage capacity being exceeded, the method lacks sufficient
> logging, especially in cases where the job is not configured to fast fail.
> This can make it difficult for users to understand why a job did not fail
> immediately when the storage capacity was exceeded. The enhancement adds
> detailed logging to inform users about the configuration that prevents fast
> failure.
>
> *Expected Behavior:*
> When a {{ClusterStorageCapacityExceededException}} is encountered, the system
> should log whether the job is configured to fail fast. If fast fail is
> disabled, the log should advise users on how to enable it.
>
> *How-to-Fix:*
> We propose to *expose such a relationship by logging.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7485) Insufficient System Outputs to Address Misconfiguration in Shared Cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LoggingResearch updated MAPREDUCE-7485:
---
Attachment: SharedCacheConfigTest.java
> Insufficient System Outputs to Address Misconfiguration in Shared Cache
> ---
>
> Key: MAPREDUCE-7485
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7485
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mapreduce-client
>Affects Versions: 3.3.6
> Environment: Version: {{`3.3.6`}}
> Location{*}:{*}
> {{{}hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/SharedCacheConfig.java{}}},
> in the {{init}} method, starting from Line 46.
>Reporter: LoggingResearch
>Priority: Major
> Attachments: SharedCacheConfigTest.java
>
>
> To enable the {{{}shared cache{}}}, users must set specific configuration
> parameters: {{mapreduce.framework.name}} should be set to {{{}yarn{}}},
> {{yarn.sharedcache.enable}} should be {{{}true{}}}, and
> {{mapreduce.job.sharedcache.mode}} should be configured as {{enabled}} or one
> of {{{}jobjar, libjars, files, archives{}}}. This requirement is implicit in
> the code within {{{}org.apache.hadoop.mapreduce.SharedCacheConfig.java{}}}.
> However, these relationships between the code and configuration parameters
> are not documented in the configuration guides. The necessary configurations
> are not mentioned in either {{mapreduce-default.xml}} or
> {{{}yarn-default.xml{}}}. Consequently, users may be unable to enable the
> shared cache feature even if they thoroughly review the documentation. This
> lack of guidance negatively impacts user experience, system diagnosability,
> and observability.
>
> *Expected Behavior:*
> The system should provide clear guidance on misconfiguration related to the
> shared cache. Simply relying on users to meticulously read the configuration
> documentation is insufficient, especially when some configurations are
> undocumented.
>
> *How-to-Fix:*
> We recommend enhancing the logging mechanism to expose these configuration
> relationships. This will provide users with immediate feedback and guidance
> when misconfigurations occur.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7446) NegativeArraySizeException when running MR jobs with large data size
[ https://issues.apache.org/jira/browse/MAPREDUCE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877370#comment-17877370 ] Andrew Olson commented on MAPREDUCE-7446: - Note that this correction has been merged into the 3.2.x and 3.3.x branches but not released yet for them. > NegativeArraySizeException when running MR jobs with large data size > > > Key: MAPREDUCE-7446 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7446 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 3.4.0 >Reporter: Peter Szucs >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We are using bit shifting to double the byte array in IFile's > [nextRawValue|https://github.infra.cloudera.com/CDH/hadoop/blob/bef14a39c7616e3b9f437a6fb24fc7a55a676b57/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java#L437] > method to store the byte values in it. With large dataset it can easily > happen that we shift the leftmost bit when we are calculating the size of the > array, which can lead to a negative number as the array size, causing the > NegativeArraySizeException. > It would be safer to expand the backing array with a 1.5x factor, and have a > check not to extend Integer's max value during that. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7484) Invalid transitions for JobImpl on COMMITTING
John Doe created MAPREDUCE-7484:
---
Summary: Invalid transitions for JobImpl on COMMITTING
Key: MAPREDUCE-7484
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7484
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: John Doe
Fix For: 2.1.1-beta
{code:java}
2024-08-11 21:45:54,744 ERROR
[org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl] - Can't handle this event
at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_MAP_TASK_RESCHEDULED at COMMITTING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:965)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testCheckJobCompleteSuccess(TestJobImpl.java:212)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7480) Invalid transitions for JobImpl
John Doe created MAPREDUCE-7480:
---
Summary: Invalid transitions for JobImpl
Key: MAPREDUCE-7480
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7480
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.0.1-alpha
Reporter: John Doe
Fix For: 2.1.1-beta
{code:java}
2024-08-11 21:29:28,744 ERROR
[org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl] - Can't handle this event
at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_TASK_ATTEMPT_COMPLETED at COMMITTING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:965)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testCheckJobCompleteSuccess(TestJobImpl.java:212)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7479) Send kill event to recovered attempt in unassigned state
[
https://issues.apache.org/jira/browse/MAPREDUCE-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated MAPREDUCE-7479:
Fix Version/s: 2.0.3-alpha
(was: 2.0.0-alpha)
Target Version/s: (was: 2.0.1-alpha)
> Send kill event to recovered attempt in unassigned state
>
>
> Key: MAPREDUCE-7479
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7479
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0-alpha
>Reporter: John Doe
>Priority: Major
> Fix For: 2.0.3-alpha
>
>
> `TA_DIAGNOSTICS_UPDATE` can arrive at an unassigned task attempt before
> `TA_KILL` sent by RecoveryService, which causes an invalid state transition.
> {code:java}
> 2024-03-09 16:56:05,734 ERROR
> [org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl] - Can't handle
> this event at current state for attempt_0__m_00_0
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_DIAGNOSTICS_UPDATE at UNASSIGNED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:941)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:133)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:912)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:904)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:750)
> 2024-03-09 16:56:05,737 INFO
> [org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl] -
> attempt_0__m_00_0 TaskAttempt Transitioned from UNASSIGNED to KILLED
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7479) Send kill event to recovered attempt in unassigned state
[
https://issues.apache.org/jira/browse/MAPREDUCE-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated MAPREDUCE-7479:
Description:
`TA_DIAGNOSTICS_UPDATE` can arrive at an unassigned task attempt before
`TA_KILL` sent by RecoveryService, which causes an invalid state transition.
{code:java}
2024-03-09 16:56:05,734 ERROR
[org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl] - Can't handle
this event at current state for attempt_0__m_00_0
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_DIAGNOSTICS_UPDATE at UNASSIGNED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:941)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:133)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:912)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:904)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:750)
2024-03-09 16:56:05,737 INFO
[org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl] -
attempt_0__m_00_0 TaskAttempt Transitioned from UNASSIGNED to KILLED
{code}
was:`TA_DIAGNOSTICS_UPDATE` can arrive at an unassigned task attempt before
`TA_KILL` sent by RecoveryService, which causes an invalid state transition.
> Send kill event to recovered attempt in unassigned state
>
>
> Key: MAPREDUCE-7479
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7479
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0-alpha
>Reporter: John Doe
>Priority: Major
> Fix For: 2.0.0-alpha
>
>
> `TA_DIAGNOSTICS_UPDATE` can arrive at an unassigned task attempt before
> `TA_KILL` sent by RecoveryService, which causes an invalid state transition.
> {code:java}
> 2024-03-09 16:56:05,734 ERROR
> [org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl] - Can't handle
> this event at current state for attempt_0__m_00_0
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_DIAGNOSTICS_UPDATE at UNASSIGNED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:941)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:133)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:912)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:904)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:750)
> 2024-03-09 16:56:05,737 INFO
> [org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl] -
> attempt_0__m_00_0 TaskAttempt Transitioned from UNASSIGNED to KILLED
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7477) compare some Map-Reduce Framework Counters to point out or warn user there are issues with the results
[ https://issues.apache.org/jira/browse/MAPREDUCE-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867256#comment-17867256 ] Brahma Reddy Battula commented on MAPREDUCE-7477: - Hi [~wanghl] Thanks for reporting.. Can you Elaborate more on this an provide any changes which you are looking.? > compare some Map-Reduce Framework Counters to point out or warn user there > are issues with the results > -- > > Key: MAPREDUCE-7477 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7477 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mapreduce-client >Reporter: WangHL >Priority: Major > > Our project aims to ensure data uniformity, Map results use > System.currentTimeMillis()%numOfRed decide which Reduce task to send to. > In the case of unstable key output from repeated Map tasks, if a Map task is > rerun due to various reasons, it may cause result data omission or > duplication. > I want compare some Map-Reduce Framework Counters(MAP_OUTPUT_RECORDS, > COMBINER_INPUT_RECORDS, COMBINER_OUPUT_REDORDS, REDUCE_INPUT_RECORDS), to > point out or warn user there are issues with the results. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866430#comment-17866430
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
hadoop-yetus commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2231173900
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 1m 38s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 1m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 6s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/7/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new
+ 481 unchanged - 0 fixed = 482 total (was 481) |
| +1 :green_heart: | mvnsite | 1m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 30s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 49s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 162m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 7b18390bd17f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / de49c49ca8ccb99178568a01138f39902d812ac1 |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866425#comment-17866425
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
hadoop-yetus commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2231033210
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 31s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/8/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new
+ 481 unchanged - 0 fixed = 482 total (was 481) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 1m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 15s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 6m 49s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 107m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 929ba39f9639 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / de49c49ca8ccb99178568a01138f39902d812ac1 |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866384#comment-17866384
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
granewang commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2230772574
fix checkstyle of WhitespaceAfter,another one :'method call' child has
incorrect indentation level 10 ,is legacy code
> decode value of hive.query.string for the job Configuration which was encoded
> by hive
> -
>
> Key: MAPREDUCE-7472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7472
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, resourcemanager
>Affects Versions: 3.4.0, 3.3.3, 3.3.6
>Reporter: wangzhongwei
>Assignee: wangzhongwei
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-02-02-09-44-57-503.png
>
>
> the value of hive.query.string in job Configuration is URLEncoded by
> hive and written to hdfs,which shoud be decoded before rendered.
> the sql is :insert overwrite table test_hive_query_string
> partition(dt='2024-2-1') values("testquerystring",1); while it is not
> Decoded .we can see running job on yarn web site or finished job on
> jobhistory web site
> !image-2024-02-02-09-44-57-503.png|width=525,height=72!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866375#comment-17866375
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
hadoop-yetus commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2230729993
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 49s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 1m 40s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 1m 40s | | the patch passed |
| +1 :green_heart: | compile | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 1m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 4s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/6/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new
+ 481 unchanged - 0 fixed = 483 total (was 481) |
| +1 :green_heart: | mvnsite | 1m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 19s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 30s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 170m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux e456880a6c2c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 45f51dfefc58cc016fb5c643c33a2f03a875108e |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866369#comment-17866369
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
hadoop-yetus commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2230687034
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 31s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 51s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 1m 41s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 1m 41s | | the patch passed |
| +1 :green_heart: | compile | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 1m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/5/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new
+ 481 unchanged - 0 fixed = 483 total (was 481) |
| +1 :green_heart: | mvnsite | 0m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 11s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 17s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 170m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 31f73d95e50b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | t
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866329#comment-17866329
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
granewang commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2230421326
add UT for the pr
> decode value of hive.query.string for the job Configuration which was encoded
> by hive
> -
>
> Key: MAPREDUCE-7472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7472
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, resourcemanager
>Affects Versions: 3.4.0, 3.3.3, 3.3.6
>Reporter: wangzhongwei
>Assignee: wangzhongwei
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-02-02-09-44-57-503.png
>
>
> the value of hive.query.string in job Configuration is URLEncoded by
> hive and written to hdfs,which shoud be decoded before rendered.
> the sql is :insert overwrite table test_hive_query_string
> partition(dt='2024-2-1') values("testquerystring",1); while it is not
> Decoded .we can see running job on yarn web site or finished job on
> jobhistory web site
> !image-2024-02-02-09-44-57-503.png|width=525,height=72!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7478) [Decommission]Show Info Log for Repeated Useless refreshNode Operation
[ https://issues.apache.org/jira/browse/MAPREDUCE-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuchang resolved MAPREDUCE-7478. Resolution: Abandoned > [Decommission]Show Info Log for Repeated Useless refreshNode Operation > -- > > Key: MAPREDUCE-7478 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7478 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: wuchang >Priority: Major > > https://github.com/apache/hadoop/pull/6921 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7478) [Decommission]Show Info Log for Repeated Useless refreshNode Operation
wuchang created MAPREDUCE-7478: -- Summary: [Decommission]Show Info Log for Repeated Useless refreshNode Operation Key: MAPREDUCE-7478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7478 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: wuchang https://github.com/apache/hadoop/pull/6921 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7477) compare some Map-Reduce Framework Counters to point out or warn user there are issues with the results
WangHL created MAPREDUCE-7477: - Summary: compare some Map-Reduce Framework Counters to point out or warn user there are issues with the results Key: MAPREDUCE-7477 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7477 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mapreduce-client Reporter: WangHL Our project aims to ensure data uniformity, Map results use System.currentTimeMillis()%numOfRed decide which Reduce task to send to. In the case of unstable key output from repeated Map tasks, if a Map task is rerun due to various reasons, it may cause result data omission or duplication. I want compare some Map-Reduce Framework Counters(MAP_OUTPUT_RECORDS, COMBINER_INPUT_RECORDS, COMBINER_OUPUT_REDORDS, REDUCE_INPUT_RECORDS), to point out or warn user there are issues with the results. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856790#comment-17856790
]
ASF GitHub Bot commented on MAPREDUCE-7472:
---
hadoop-yetus commented on PR #6520:
URL: https://github.com/apache/hadoop/pull/6520#issuecomment-2182048365
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 44s | | trunk passed |
| +1 :green_heart: | compile | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 1m 38s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 7s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/4/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new
+ 471 unchanged - 0 fixed = 472 total (was 471) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 27s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 46s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 165m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6520/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d0033da0c6ad 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / abaec5cb668bc92061db35b9fceab12c84322365 |
| Def
[jira] [Updated] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangzhongwei updated MAPREDUCE-7472:
Affects Version/s: 3.3.6
3.4.0
> decode value of hive.query.string for the job Configuration which was encoded
> by hive
> -
>
> Key: MAPREDUCE-7472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7472
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, resourcemanager
>Affects Versions: 3.4.0, 3.3.3, 3.3.6
>Reporter: wangzhongwei
>Assignee: wangzhongwei
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-02-02-09-44-57-503.png
>
>
> the value of hive.query.string in job Configuration is URLEncoded by
> hive and written to hdfs,which shoud be decoded before rendered.
> the sql is :insert overwrite table test_hive_query_string
> partition(dt='2024-2-1') values("testquerystring",1); while it is not
> Decoded .we can see running job on yarn web site or finished job on
> jobhistory web site
> !image-2024-02-02-09-44-57-503.png|width=525,height=72!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7472) decode value of hive.query.string for the job Configuration which was encoded by hive
[
https://issues.apache.org/jira/browse/MAPREDUCE-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangzhongwei updated MAPREDUCE-7472:
Component/s: jobhistoryserver
resourcemanager
> decode value of hive.query.string for the job Configuration which was encoded
> by hive
> -
>
> Key: MAPREDUCE-7472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7472
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, resourcemanager
>Affects Versions: 3.3.3
>Reporter: wangzhongwei
>Assignee: wangzhongwei
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-02-02-09-44-57-503.png
>
>
> the value of hive.query.string in job Configuration is URLEncoded by
> hive and written to hdfs,which shoud be decoded before rendered.
> the sql is :insert overwrite table test_hive_query_string
> partition(dt='2024-2-1') values("testquerystring",1); while it is not
> Decoded .we can see running job on yarn web site or finished job on
> jobhistory web site
> !image-2024-02-02-09-44-57-503.png|width=525,height=72!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6925) CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848689#comment-17848689
]
Dhruv Kachhadia commented on MAPREDUCE-6925:
Is the fix for this coming soon?
a simple MR application with more than 120 counters fails wierdly when i set
mapreduce.job.counters.max different to default value:
Two different scenarios:
1. mapred-configs: max counters is set to 200
{code:java}
24/05/21 12:32:37 INFO mapreduce.Job: Task Id :
attempt_1716282029465_0013_m_00_0, Status : FAILED
Error:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcServerException):
IPC server unable to read call parameters: Too many counters: 121 max=120
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
at org.apache.hadoop.ipc.Client.call(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1455)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:251)
at com.sun.proxy.$Proxy8.statusUpdate(Unknown Source)
at org.apache.hadoop.mapred.Task.statusUpdate(Task.java:1317)
at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1350)
at org.apache.hadoop.mapred.Task.done(Task.java:1273)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:183)
{code}
2. Mapred-configs: Max counters set to 2
{code:java}
24/05/21 12:43:44 INFO mapreduce.Job: Task Id :
attempt_1716295266630_0002_m_00_2, Status : FAILED
Error: org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
counters: 3 max=2
at
org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
at
org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounterImpl(AbstractCounterGroup.java:123)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:113)
at
org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:130)
at
org.apache.hadoop.mapred.Counters$Group.findCounter(Counters.java:369)
at
org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:314)
at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:479)
at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:60)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:167)
at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:711)
at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:647)
at
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:71)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCounter(WrappedMapper.java:96)
at org.apache.hadoop.examples.MyCounter$Map.map(MyCounter.java:51)
at org.apache.hadoop.examples.MyCounter$Map.map(MyCounter.java:43)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800){code}
2nd option should be the ideal exception
So i feel the limits should be made same everywhere
> CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and
> YarnChild
> ---
>
> Key: MAPREDUCE-6925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the
[jira] [Resolved] (MAPREDUCE-7475) Fix non-idempotent unit tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved MAPREDUCE-7475. - Hadoop Flags: Reviewed Resolution: Fixed > Fix non-idempotent unit tests > - > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated MAPREDUCE-7475: Fix Version/s: 3.4.1 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) Fix non-idempotent unit tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847729#comment-17847729 ] Ayush Saxena commented on MAPREDUCE-7475: - Fix is merged both in trunk & branch-3.4 Thanx [~kaiyaok2] for the contribution!!! > Fix non-idempotent unit tests > - > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) Fix non-idempotent unit tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated MAPREDUCE-7475: Summary: Fix non-idempotent unit tests (was: 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)) > Fix non-idempotent unit tests > - > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847488#comment-17847488 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- kaiyaok2 commented on PR #6785: URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2118659087 @steveloughran Done in https://github.com/apache/hadoop/pull/6837 Please lmk if I shall do the same for https://github.com/apache/hadoop/pull/6793 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Fix Version/s: 3.5.0 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Affects Version/s: 3.4.0 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7475: - Assignee: Kaiyao Ke > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Priority: Minor (was: Major) > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Component/s: test > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847317#comment-17847317 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- steveloughran commented on PR #6785: URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2117659973 thanks! merged to trunk. Can you do a cherrypick to branch-3.4 and push that up as a PR too so that yetus can validate the backport. Then we can merge it in there too > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847316#comment-17847316 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- steveloughran merged PR #6785: URL: https://github.com/apache/hadoop/pull/6785 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7474. --- Fix Version/s: 3.3.9 3.5.0 3.4.1 Resolution: Fixed > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846683#comment-17846683 ] ASF GitHub Bot commented on MAPREDUCE-7474: --- steveloughran merged PR #6825: URL: https://github.com/apache/hadoop/pull/6825 > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846682#comment-17846682 ] ASF GitHub Bot commented on MAPREDUCE-7474: --- steveloughran merged PR #6824: URL: https://github.com/apache/hadoop/pull/6824 > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846680#comment-17846680 ] ASF GitHub Bot commented on MAPREDUCE-7474: --- steveloughran commented on PR #6824: URL: https://github.com/apache/hadoop/pull/6824#issuecomment-2112850821 update: ignoring yetus. the field is final, this is essentially a case class. > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846673#comment-17846673
]
ASF GitHub Bot commented on MAPREDUCE-7475:
---
hadoop-yetus commented on PR #6785:
URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2112803298
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 4m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 108m 49s | | trunk passed |
| +1 :green_heart: | compile | 7m 46s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 17m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 16m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 201m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 44s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 31s | | the patch passed |
| +1 :green_heart: | compile | 4m 53s | | the patch passed |
| +1 :green_heart: | javac | 4m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 9m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 7m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 201m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 25s | | The patch does not
generate ASF License warnings. |
| | | 556m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6785 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 f036a21db0b9 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / e4e557d43c7575dc5a49821dd609edb1927790c4 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/5/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/5/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> 2 tests are non-idempotent (passes in the first run but fails in repeated
> runs in the same JVM)
> ---
>
> Key: MAPREDUCE-7475
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Ubuntu 22.04, Java 17
>Reporter: Kaiyao Ke
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> 2 tests are not idempotent and fails upon repeated execution within the same
> JVM instance due to self-induced state pollution. Specifically, these tests
> try to make the directory TEST_ROOT_DIR and write to it. The tests do not
> clean up (remove) the directory after execution. Therefore, in the second
> execution, TEST_ROOT_DIR would already exist and the exception `Could not
> cre
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846393#comment-17846393
]
ASF GitHub Bot commented on MAPREDUCE-7475:
---
hadoop-yetus commented on PR #6785:
URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2110930846
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 93m 25s | | trunk passed |
| +1 :green_heart: | compile | 7m 17s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 01s | | trunk passed |
| +1 :green_heart: | mvnsite | 15m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 14m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 176m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 21s | | the patch passed |
| +1 :green_heart: | compile | 4m 10s | | the patch passed |
| +1 :green_heart: | javac | 4m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 33s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 7m 06s | | the patch passed |
| +1 :green_heart: | shadedclient | 178m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 31s | | The patch does not
generate ASF License warnings. |
| | | 487m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6785 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 285952dfb8f1 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 710e1ce492e199eeb586bfd26eb81c95bddbcadf |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/3/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/3/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> 2 tests are non-idempotent (passes in the first run but fails in repeated
> runs in the same JVM)
> ---
>
> Key: MAPREDUCE-7475
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Ubuntu 22.04, Java 17
>Reporter: Kaiyao Ke
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> 2 tests are not idempotent and fails upon repeated execution within the same
> JVM instance due to self-induced state pollution. Specifically, these tests
> try to make the directory TEST_ROOT_DIR and write to it. The tests do not
> clean up (remove) the directory after execution. Therefore, in the second
> execution, TEST_ROOT_DIR would already exist and the exception `Could not
> cre
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846266#comment-17846266 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- kaiyaok2 commented on PR #6785: URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2109873971 > it is just the variable name change, maybe changing to `testRootDir` should fix @ayushtkn Fixed. Thanks for the message > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846263#comment-17846263 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- ayushtkn commented on PR #6785: URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2109847341 We have the test results, everything is green, the checkstyle alarm is for the existing code, not introduced as part of this PR but I think since we are touching it now, maybe we can fix that as well @kaiyaok2 can you fix these checkstyle warnings https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6785/7/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt it is just the variable name change, maybe changing to ``testRootDir`` should fix > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846257#comment-17846257
]
ASF GitHub Bot commented on MAPREDUCE-7475:
---
hadoop-yetus commented on PR #6785:
URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2109819556
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 43m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 10m 50s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6785/7/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 34s | | trunk passed |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 4s |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6785/7/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new
+ 26 unchanged - 2 fixed = 28 total (was 28) |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 9m 43s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 8m 57s | | hadoop-mapreduce-client-app in
the patch passed. |
| +1 :green_heart: | unit | 120m 16s | |
hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 311m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6785/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6785 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux cbd8f53faff6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4e98ad73e02de48c4078314ec240f4106faf81dc |
| Default Java | Red Hat, Inc.-1.8.0_412-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6785/7/testReport/ |
| Max. process+thread count | 1150 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6785/7/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[
https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846146#comment-17846146
]
ASF GitHub Bot commented on MAPREDUCE-7474:
---
hadoop-yetus commented on PR #6825:
URL: https://github.com/apache/hadoop/pull/6825#issuecomment-2109190422
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 14m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 14m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 40m 34s | | branch-3.3 passed |
| +1 :green_heart: | compile | 19m 32s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 3m 3s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 4m 9s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 2m 35s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 7m 15s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 40m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 13s | | the patch passed |
| +1 :green_heart: | compile | 18m 42s | | the patch passed |
| +1 :green_heart: | javac | 18m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 50s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6825/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 22 unchanged - 0 fixed = 23 total (was
22) |
| +1 :green_heart: | mvnsite | 3m 59s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 32s | | No new issues. |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed |
| +1 :green_heart: | spotbugs | 7m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 8m 49s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 159m 20s | | hadoop-mapreduce-project in the
patch passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 407m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6825/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6825 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck
shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs
checkstyle markdownlint xmllint |
| uname | Linux f6a1aab3dfb3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / d16eb0a75adcc81bbc90e7c8122632eb8ead494a |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6825/1/testReport/ |
| Max. process+thread count | 1238 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project hadoop-tools/hadoop-azure U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6825/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically gener
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[
https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846122#comment-17846122
]
ASF GitHub Bot commented on MAPREDUCE-7474:
---
hadoop-yetus commented on PR #6824:
URL: https://github.com/apache/hadoop/pull/6824#issuecomment-2109077510
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +0 :ok: | mvndep | 4m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 45s | | branch-3.4 passed |
| +1 :green_heart: | compile | 9m 33s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 8m 57s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 16s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 2m 19s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 2m 3s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 54s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 0s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 21m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 8m 57s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 57s | | the patch passed |
| +1 :green_heart: | compile | 8m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 8s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6824/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 22 unchanged - 0 fixed = 23 total (was
22) |
| +1 :green_heart: | mvnsite | 2m 6s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 10s | | No new issues. |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 33s | | hadoop-mapreduce-client-core in
the patch passed. |
| -1 :x: | unit | 119m 38s |
[/patch-unit-hadoop-mapreduce-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6824/1/artifact/out/patch-unit-hadoop-mapreduce-project.txt)
| hadoop-mapreduce-project in the patch failed. |
| -1 :x: | unit | 0m 34s |
[/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6824/1/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| +0 :ok: | asflicense | 0m 35s | | ASF License check generated no
output? |
| | | 281m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapreduce.v2.TestMRJobs |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6824/1/artifact
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846064#comment-17846064 ] ASF GitHub Bot commented on MAPREDUCE-7474: --- steveloughran opened a new pull request, #6825: URL: https://github.com/apache/hadoop/pull/6825 Improve task commit resilience everywhere and add an option to reduce delete IO requests on job cleanup (relevant for ABFS and HDFS). Task Commit Resilience -- Task manifest saving is re-attempted on failure; the number of attempts made is configurable with the option: mapreduce.manifest.committer.manifest.save.attempts * The default is 5. * The minimum is 1; asking for less is ignored. * A retry policy adds 500ms of sleep per attempt. * Move from classic rename() to commitFile() to rename the file, after calling getFileStatus() to get its length and possibly etag. This becomes a rename() on gcs/hdfs anyway, but on abfs it does reach the ResilientCommitByRename callbacks in abfs, which report on the outcome to the caller...which is then logged at WARN. * New statistic task_stage_save_summary_file to distinguish from other saving operations (job success/report file). This is only saved to the manifest on task commit retries, and provides statistics on all previous unsuccessful attempts to save the manifests + test changes to match the codepath changes, including improvements in fault injection. Directory size for deletion --- New option mapreduce.manifest.committer.cleanup.parallel.delete.base.first This attempts an initial attempt at deleting the base dir, only falling back to parallel deletes if there's a timeout. This option is disabled by default; Consider enabling it for abfs to reduce IO load. Consult the documentation for more details. Success file printing - The command to print a JSON _SUCCESS file from this committer and any S3A committer is now something which can be invoked from the mapred command: mapred successfile Contributed by Steve Loughran ### How was this patch tested? yetus's work, if happy will validate on abfs. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846062#comment-17846062 ] ASF GitHub Bot commented on MAPREDUCE-7474: --- steveloughran opened a new pull request, #6824: URL: https://github.com/apache/hadoop/pull/6824 backport of #6716 Improve task commit resilience everywhere and add an option to reduce delete IO requests on job cleanup (relevant for ABFS and HDFS). Task Commit Resilience -- Task manifest saving is re-attempted on failure; the number of attempts made is configurable with the option: mapreduce.manifest.committer.manifest.save.attempts * The default is 5. * The minimum is 1; asking for less is ignored. * A retry policy adds 500ms of sleep per attempt. * Move from classic rename() to commitFile() to rename the file, after calling getFileStatus() to get its length and possibly etag. This becomes a rename() on gcs/hdfs anyway, but on abfs it does reach the ResilientCommitByRename callbacks in abfs, which report on the outcome to the caller...which is then logged at WARN. * New statistic task_stage_save_summary_file to distinguish from other saving operations (job success/report file). This is only saved to the manifest on task commit retries, and provides statistics on all previous unsuccessful attempts to save the manifests + test changes to match the codepath changes, including improvements in fault injection. Directory size for deletion --- New option mapreduce.manifest.committer.cleanup.parallel.delete.base.first This attempts an initial attempt at deleting the base dir, only falling back to parallel deletes if there's a timeout. This option is disabled by default; Consider enabling it for abfs to reduce IO load. Consult the documentation for more details. Success file printing - The command to print a JSON _SUCCESS file from this committer and any S3A committer is now something which can be invoked from the mapred command: mapred successfile Contributed by Steve Loughran ### How was this patch tested? yetus's work, if happy will validate on abfs. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846061#comment-17846061 ] ASF GitHub Bot commented on MAPREDUCE-7474: --- steveloughran merged PR #6716: URL: https://github.com/apache/hadoop/pull/6716 > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846056#comment-17846056 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- steveloughran commented on PR #6785: URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2108690394 @ayushtkn the issue with @after is doesn't clean up reliably on job interrupt. but that's more of an issue with cloud stuff, isn't it? as here a `mvn clean` will do the cleanup. So yes, let's go with your proposal > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845386#comment-17845386
]
ASF GitHub Bot commented on MAPREDUCE-7475:
---
hadoop-yetus commented on PR #6785:
URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2104748348
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 85m 29s | | trunk passed |
| +1 :green_heart: | compile | 6m 18s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 14m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 13m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 157m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 15s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 37s | | the patch passed |
| +1 :green_heart: | compile | 4m 03s | | the patch passed |
| +1 :green_heart: | javac | 4m 03s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 160m 01s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 11s | | The patch does not
generate ASF License warnings. |
| | | 439m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6785 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 2bb0df688968 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 4e98ad73e02de48c4078314ec240f4106faf81dc |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/2/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> 2 tests are non-idempotent (passes in the first run but fails in repeated
> runs in the same JVM)
> ---
>
> Key: MAPREDUCE-7475
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Ubuntu 22.04, Java 17
>Reporter: Kaiyao Ke
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> 2 tests are not idempotent and fails upon repeated execution within the same
> JVM instance due to self-induced state pollution. Specifically, these tests
> try to make the directory TEST_ROOT_DIR and write to it. The tests do not
> clean up (remove) the directory after execution. Therefore, in the second
> execution, TEST_ROOT_DIR would already exist and the exception `Could not
> cre
[jira] [Commented] (MAPREDUCE-7402) fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
[
https://issues.apache.org/jira/browse/MAPREDUCE-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845205#comment-17845205
]
ASF GitHub Bot commented on MAPREDUCE-7402:
---
hadoop-yetus commented on PR #6622:
URL: https://github.com/apache/hadoop/pull/6622#issuecomment-2103907810
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 92m 25s | | trunk passed |
| +1 :green_heart: | compile | 5m 10s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 00s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 147m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 49s | | the patch passed |
| +1 :green_heart: | compile | 2m 35s | | the patch passed |
| +1 :green_heart: | javac | 2m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 05s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 05s | | the patch passed |
| +1 :green_heart: | shadedclient | 158m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 12m 00s |
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/4/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
| hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 5m 39s | | The patch does not
generate ASF License warnings. |
| | | 442m 08s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapred.TestTaskProgressReporter |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6622 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 759ee4f0c44d 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / dff82fdd3ca73a2fd2b8b9c0a97f50135f26 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/4/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/4/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
> -
>
> Key: MAPREDUCE-7402
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7402
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian Zhang
>Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
>
> *Describe：*
> In the method long computeBytesInMerges(int factor, int inMem), when
> factor is set to 1, and the initial numSegments>1, inMem=0, it will cause an
> infinite loop of while. Although setting factor to 1 is meaningless, and it
> will not be done in actual production, we need to remind users to avoid an
> infinite loo
[jira] [Commented] (MAPREDUCE-7402) fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
[
https://issues.apache.org/jira/browse/MAPREDUCE-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845201#comment-17845201
]
ASF GitHub Bot commented on MAPREDUCE-7402:
---
hadoop-yetus commented on PR #6622:
URL: https://github.com/apache/hadoop/pull/6622#issuecomment-2103892286
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 108m 33s | | trunk passed |
| +1 :green_heart: | compile | 5m 55s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 175m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 32s | | the patch passed |
| +1 :green_heart: | compile | 2m 49s | | the patch passed |
| +1 :green_heart: | javac | 2m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 09s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 187m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 14m 29s |
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
| hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | asflicense | 9m 11s | | The patch does not
generate ASF License warnings. |
| | | 526m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapred.TestTaskProgressReporter |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6622 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 cf5d9f159693 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / dff82fdd3ca73a2fd2b8b9c0a97f50135f26 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/3/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6622/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
> -
>
> Key: MAPREDUCE-7402
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7402
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian Zhang
>Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
>
> *Describe：*
> In the method long computeBytesInMerges(int factor, int inMem), when
> factor is set to 1, and the initial numSegments>1, inMem=0, it will cause an
> infinite loop of while. Although setting factor to 1 is meaningless, and it
> will not be done in actual production, we need to remind users to avoid an
> infinite loo
[jira] [Commented] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[
https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844932#comment-17844932
]
ASF GitHub Bot commented on MAPREDUCE-7474:
---
hadoop-yetus commented on PR #6716:
URL: https://github.com/apache/hadoop/pull/6716#issuecomment-2102258570
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 05s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 05s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 05s | | detect-secrets was not available.
|
| +0 :ok: | shellcheck | 0m 05s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 05s | | Shelldocs was not available. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 11 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 87m 02s | | trunk passed |
| +1 :green_heart: | compile | 38m 10s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 01s | | trunk passed |
| +1 :green_heart: | mvnsite | 16m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 14m 05s | | trunk passed |
| +1 :green_heart: | shadedclient | 165m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 13s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 11m 25s | | the patch passed |
| +1 :green_heart: | compile | 36m 11s | | the patch passed |
| +1 :green_heart: | javac | 36m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 16m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 13m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 174m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 07s | | The patch does not
generate ASF License warnings. |
| | | 532m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6716 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck
shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs
checkstyle markdownlint xmllint |
| uname | MINGW64_NT-10.0-17763 54377fa890b8 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 68dff782a05bdaaf1fcc1e4999e6fd404f07796b |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6716/4/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project hadoop-tools/hadoop-azure U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6716/4/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [ABFS] Improve commit resilience and performance in Manifest Committer
> --
>
> Key: MAPREDUCE-7474
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.4.0, 3.3.6
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> * Manifest committer is not resilient to rename failures on task commit
> without HADOOP-18012 rename recovery enabled.
> * large burst of delete calls noted: are they needed?
> relates to HADOOP-19093 but takes a more minimal approach with goal of
> c
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844750#comment-17844750 ] ASF GitHub Bot commented on MAPREDUCE-7475: --- kaiyaok2 commented on PR #6785: URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2101208144 Resolved the trailing space issues > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7402) fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
[
https://issues.apache.org/jira/browse/MAPREDUCE-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844696#comment-17844696
]
ASF GitHub Bot commented on MAPREDUCE-7402:
---
KeeProMise commented on PR #6622:
URL: https://github.com/apache/hadoop/pull/6622#issuecomment-2100839456
> Was gonna hit the merge button, but got curious.
>
> Earlier if the value was 1, was it an infinite loop always? or in just in
a corner case, if in just one corner case & now if we throw an exception, if
anyone has value of 1 earlier his jobs would be working but post this change it
would fail?
>
> can we just break the condition for infinite loop if that is the case &
preserve the value? or at worst if that is an illegal value, we put a warn log
& use the defaul?
>
> @brumi1024 / @szilard-nemeth any pointers?
@ayushtkn hi, to prevent throwing exceptions from affecting the task, I will
print a warning log instead of throwing exceptions. Please take a look again,
thanks.
> fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
> -
>
> Key: MAPREDUCE-7402
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7402
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian Zhang
>Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
>
> *Describe：*
> In the method long computeBytesInMerges(int factor, int inMem), when
> factor is set to 1, and the initial numSegments>1, inMem=0, it will cause an
> infinite loop of while. Although setting factor to 1 is meaningless, and it
> will not be done in actual production, we need to remind users to avoid an
> infinite loop due to carelessness.
> !screenshot-1.png|width=600,height=400!
> *How to reproduce:*
> This bug can easily occur by setting
> configuration.set("mapreduce.task.io.sort.factor", "1").
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844693#comment-17844693
]
ASF GitHub Bot commented on MAPREDUCE-7475:
---
hadoop-yetus commented on PR #6785:
URL: https://github.com/apache/hadoop/pull/6785#issuecomment-2100800444
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 01s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 109m 16s | | trunk passed |
| +1 :green_heart: | compile | 8m 00s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 17m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 16m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 205m 05s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 3m 04s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 35s | | the patch passed |
| +1 :green_heart: | compile | 4m 50s | | the patch passed |
| +1 :green_heart: | javac | 4m 50s | | the patch passed |
| -1 :x: | blanks | 0m 00s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/1/artifact/out/blanks-eol.txt)
| The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 2m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 9m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 8m 00s | | the patch passed |
| +1 :green_heart: | shadedclient | 204m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 30s | | The patch does not
generate ASF License warnings. |
| | | 563m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6785 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 1ac01a056e3d 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 6e7c3ec42a278683c5d0b34242e1889ca8a67612 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/1/testReport/
|
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6785/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> 2 tests are non-idempotent (passes in the first run but fails in repeated
> runs in the same JVM)
> ---
>
> Key: MAPREDUCE-7475
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Ubuntu 22.04, Java 17
>Reporter: Kaiyao Ke
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> 2 tests are not idempotent and fails upon repeated execution within the same
> JVM instance due to self-induced state pollution. Specifically, these tests
> try to make the directory TEST_ROOT_DIR and write
