Re: [PR] HADOOP-19082: Update AWS SDK V2 to 2.24.1 [hadoop]
HarshitGupta11 commented on PR #6568: URL: https://github.com/apache/hadoop/pull/6568#issuecomment-1963501387 > @HarshitGupta11 need to know where you qualified the SDK... Still running all the tests as mentioned, I have run the test suite under normal conditions in both kind of buckets but there are a couple of changes needed with CSE-KMS and access point buckets, trying to figure those out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have replica. [hadoop]
hadoop-yetus commented on PR #6509:
URL: https://github.com/apache/hadoop/pull/6509#issuecomment-1963495064
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 27s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged -
0 fixed = 6 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 203m 14s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 305m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6509 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f527b079541a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7e39505156b4a7744c57737165c29be8ec1a0e1 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/testReport/ |
| Max. process+thread count | 5053 (vs. ulimit of 5500) |
| modules | C:
Re: [PR] HADOOP-19082: Update AWS SDK V2 to 2.24.1 [hadoop]
HarshitGupta11 commented on PR #6568: URL: https://github.com/apache/hadoop/pull/6568#issuecomment-1963493934 > @HarshitGupta11 could you upgrade to the latest version, or at least `2.24.6` instead? We found an issue with the S3 async client not implementing abort which is fixed in `2.24.6`. This doesn't really affect us currently, but if someone is using Client side encryption, then it might cause issues. Related to #6164 Hi yeah, I have updated the ticket and ran the hadoop test suite again a normal and directory bucket in us-west-2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14837) Handle S3A "glacier" data
[ https://issues.apache.org/jira/browse/HADOOP-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820598#comment-17820598 ] ASF GitHub Bot commented on HADOOP-14837: - bpahuja commented on PR #6407: URL: https://github.com/apache/hadoop/pull/6407#issuecomment-1963476220 Hello @steveloughran, Just a gentle reminder to review the PR. Thanks > Handle S3A "glacier" data > - > > Key: HADOOP-14837 > URL: https://issues.apache.org/jira/browse/HADOOP-14837 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Bhavay Pahuja >Priority: Minor > Labels: pull-request-available > > SPARK-21797 covers how if you have AWS S3 set to copy some files to glacier, > they appear in the listing but GETs fail, and so does everything else > We should think about how best to handle this. > # report better > # if listings can identify files which are glaciated then maybe we could have > an option to filter them out > # test & see what happens -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-14837 : Support Read Restored Glacier Objects [hadoop]
bpahuja commented on PR #6407: URL: https://github.com/apache/hadoop/pull/6407#issuecomment-1963476220 Hello @steveloughran, Just a gentle reminder to review the PR. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[ https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820595#comment-17820595 ] ASF GitHub Bot commented on HADOOP-18910: - anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1963472510 Will run the test suite again on the backport PR to 3.4 as well... > ABFS: Adding Support for MD5 Hash based integrity verification of the request > content during transport > --- > > Key: HADOOP-18910 > URL: https://issues.apache.org/jira/browse/HADOOP-18910 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Azure Storage Supports Content-MD5 Request Headers in Both Read and Append > APIs. > Read: [Path - Read - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read] > Append: [Path - Update - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update] > This change is to make client-side changes to support them. In Read request, > we will send the appropriate header in response to which server will return > the MD5 Hash of the data it sends back. On Client we will tally this with the > MD5 hash computed from the data received. > In Append request, we will compute the MD5 Hash of the data that we are > sending to the server and specify that in appropriate header. Server on > finding that header will tally this with the MD5 hash it will compute on the > data received. > This whole Checksum Validation Support is guarded behind a config, Config is > by default disabled because with the use of "https" integrity of data is > preserved anyways. This is introduced as an additional data integrity check > which will have a performance impact as well. > Users can decide if they want to enable this or not by setting the following > config to *"true"* or *"false"* respectively. *Config: > "fs.azure.enable.checksum.validation"* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1963472510 Will run the test suite again on the backport PR to 3.4 as well... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17389. [FGL] Create RPC minimizes the scope of the global BM lock [hadoop]
ZanderXu opened a new pull request, #6590: URL: https://github.com/apache/hadoop/pull/6590 Only hold the BM lock when the last file exists for the Create RPC, so that the Create RPC can minimize the scope of the global BM lock. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820592#comment-17820592
]
ASF GitHub Bot commented on HADOOP-18910:
-
anujmodi2021 commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1963467491
AGGREGATED TEST RESULT On branch-3.4
HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
HNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 586, Failures: 0, Errors: 0, Skipped: 266
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[WARNING] Tests run: 586, Failures: 0, Errors: 0, Skipped: 266
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
Time taken: 21 mins 25 secs.
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1963467491
AGGREGATED TEST RESULT On branch-3.4
HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
HNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 586, Failures: 0, Errors: 0, Skipped: 266
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[WARNING] Tests run: 586, Failures: 0, Errors: 0, Skipped: 266
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
Time taken: 21 mins 25 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17388. [FGL] Create RPC supports this fine-grained locking I [hadoop]
ZanderXu opened a new pull request, #6589: URL: https://github.com/apache/hadoop/pull/6589 HDFS-17388. [FGL] Create RPC supports this fine-grained locking I. Replacing the global FSN lock with fine-grained locks for the Create RPC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. [hadoop]
anujmodi2021 commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1963453029 > can you please create a backport PR on branch-3.4 and run the tests? Sure, Will to that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[ https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820581#comment-17820581 ] ASF GitHub Bot commented on HADOOP-18910: - anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1963452101 > can you please create a backport PR on branch-3.4 and run the tests? Sure, Mukund. Will create one. Regarding, the failures you indicated above. I do not see these tests failing for me on either trunk or branch-3.4 (as of 26th Feb) Can you please share the configs you are using to run these tests? Like Auth type, Account type etc.?? > ABFS: Adding Support for MD5 Hash based integrity verification of the request > content during transport > --- > > Key: HADOOP-18910 > URL: https://issues.apache.org/jira/browse/HADOOP-18910 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Azure Storage Supports Content-MD5 Request Headers in Both Read and Append > APIs. > Read: [Path - Read - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read] > Append: [Path - Update - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update] > This change is to make client-side changes to support them. In Read request, > we will send the appropriate header in response to which server will return > the MD5 Hash of the data it sends back. On Client we will tally this with the > MD5 hash computed from the data received. > In Append request, we will compute the MD5 Hash of the data that we are > sending to the server and specify that in appropriate header. Server on > finding that header will tally this with the MD5 hash it will compute on the > data received. > This whole Checksum Validation Support is guarded behind a config, Config is > by default disabled because with the use of "https" integrity of data is > preserved anyways. This is introduced as an additional data integrity check > which will have a performance impact as well. > Users can decide if they want to enable this or not by setting the following > config to *"true"* or *"false"* respectively. *Config: > "fs.azure.enable.checksum.validation"* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1963452101 > can you please create a backport PR on branch-3.4 and run the tests? Sure, Mukund. Will create one. Regarding, the failures you indicated above. I do not see these tests failing for me on either trunk or branch-3.4 (as of 26th Feb) Can you please share the configs you are using to run these tests? Like Auth type, Account type etc.?? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18869) ABFS: Fixing Behavior of a File System APIs on root path
[
https://issues.apache.org/jira/browse/HADOOP-18869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820576#comment-17820576
]
ASF GitHub Bot commented on HADOOP-18869:
-
anujmodi2021 closed pull request #6274: HADOOP-18869: [Backport to branch-3.3]
[ABFS] Fixing Behavior of a File System APIs on root path.
URL: https://github.com/apache/hadoop/pull/6274
> ABFS: Fixing Behavior of a File System APIs on root path
>
>
> Key: HADOOP-18869
> URL: https://issues.apache.org/jira/browse/HADOOP-18869
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.6
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Following HDFS Apis are failing when called on a root path.
> {*}{*}{*}{*}{*}{*}
> |FS Call|Status|Error thrown to caller|
> |create()|Failing|Operation failed: "The request URI is invalid.", 400, PUT,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-02076119-21ed-4ada-bcd0-14afaae54013/?resource=file=90,
> InvalidUri, "The request URI is invalid.
> RequestId:1d23f8c2-d01f-0059-61b6-c60c2400
> Time:2023-08-04T09:29:55.4813818Z"|
> |createNonRecursive()|Failing|Runtime Exception:
> java.lang.IllegalArgumentException: null path (This is occuring because
> getParentPath is null and getFileStatus is called on null)|
> |setXAttr()|Failing|Operation failed: "The request URI is invalid.", 400,
> HEAD,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false=90|
> |getXAttr()|Failing|Operation failed: "The request URI is invalid.", 400,
> HEAD,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false=91|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18869: [Backport to branch-3.3] [ABFS] Fixing Behavior of a File System APIs on root path. [hadoop]
anujmodi2021 closed pull request #6274: HADOOP-18869: [Backport to branch-3.3] [ABFS] Fixing Behavior of a File System APIs on root path. URL: https://github.com/apache/hadoop/pull/6274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18874) ABFS: Adding Server returned request id in Exception method thrown to caller.
[ https://issues.apache.org/jira/browse/HADOOP-18874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820574#comment-17820574 ] ASF GitHub Bot commented on HADOOP-18874: - anujmodi2021 closed pull request #6285: HADOOP-18874: [Backport to 3.3] [ABFS] Adding Server returned request id in Exception Message thrown to caller. URL: https://github.com/apache/hadoop/pull/6285 > ABFS: Adding Server returned request id in Exception method thrown to caller. > - > > Key: HADOOP-18874 > URL: https://issues.apache.org/jira/browse/HADOOP-18874 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Each request made to Azure server has its unique ActivityId (rid) which is > returned in response of the request whether is succeed or fails. > When a HDFS call fails due to an error from Azure service, An > ABFSRestOperationException is throws to the caller. This task is to add a > server returned activity id (rid) in the exception message which can be used > to investigate the failure on service side. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18872) ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations
[ https://issues.apache.org/jira/browse/HADOOP-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820575#comment-17820575 ] ASF GitHub Bot commented on HADOOP-18872: - anujmodi2021 closed pull request #6284: HADOOP-18872: [Backport to 3.3] [ABFS] [BugFix] Misreporting Retry Count for Sub-sequential and Parallel Operations URL: https://github.com/apache/hadoop/pull/6284 > ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations > - > > Key: HADOOP-18872 > URL: https://issues.apache.org/jira/browse/HADOOP-18872 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anuj Modi >Priority: Major > Labels: Bug, pull-request-available > Fix For: 3.4.0 > > > There was a bug identified where retry count in the client correlation id was > wrongly reported for sub-sequential and parallel operations triggered by a > single file system call. This was due to reusing same tracing context for all > such calls. > We create a new tracing context as soon as HDFS call comes. We keep on > passing that same TC for all the client calls. > For instance, when we get a createFile call, we first call metadata > operations. If those metadata operations somehow succeeded after a few > retries, the tracing context will have that many retry count in it. Now when > actual call for create is made, same retry count will be used to construct > the headers(clientCorrelationId). Alhough the create operation never failed, > we will still see retry count from the previous request. > Fix is to use a new tracing context object for all the network calls made. > All the sub-sequential and parallel operations will have same primary request > Id to correlate them, yet they will have their own tracing of retry count. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17767) ABFS: Improve test scripts
[ https://issues.apache.org/jira/browse/HADOOP-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820573#comment-17820573 ] ASF GitHub Bot commented on HADOOP-17767: - anujmodi2021 closed pull request #6287: HADOOP-17767: [Backport to 3.3] [ABFS] Update test scripts (#3124) URL: https://github.com/apache/hadoop/pull/6287 > ABFS: Improve test scripts > -- > > Key: HADOOP-17767 > URL: https://issues.apache.org/jira/browse/HADOOP-17767 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Current test run scripts need manual update across all combinations in > runTests.sh for account name and is working off a single azure-auth-keys.xml > file. While having to test across accounts that span various geo, the config > file grows big and also needs a manual change for configs such as > fs.contract.test.[abfs/abfss] which has to be uniquely set. To use the script > across various combinations, dev to be aware of the names of all the > combinations defined in runTests.sh as well. > > These concerns are addressed in the new version of the scripts. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18872: [Backport to 3.3] [ABFS] [BugFix] Misreporting Retry Count for Sub-sequential and Parallel Operations [hadoop]
anujmodi2021 closed pull request #6284: HADOOP-18872: [Backport to 3.3] [ABFS] [BugFix] Misreporting Retry Count for Sub-sequential and Parallel Operations URL: https://github.com/apache/hadoop/pull/6284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18874: [Backport to 3.3] [ABFS] Adding Server returned request id in Exception Message thrown to caller. [hadoop]
anujmodi2021 closed pull request #6285: HADOOP-18874: [Backport to 3.3] [ABFS] Adding Server returned request id in Exception Message thrown to caller. URL: https://github.com/apache/hadoop/pull/6285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-17767: [Backport to 3.3] [ABFS] Update test scripts (#3124) [hadoop]
anujmodi2021 closed pull request #6287: HADOOP-17767: [Backport to 3.3] [ABFS] Update test scripts (#3124) URL: https://github.com/apache/hadoop/pull/6287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata [hadoop]
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1963416005
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 40s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 38s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 14s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 80m 45s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 177m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.recovery.TestCheckRemoveZKNodeRMStateStore |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 62f075e61cd1 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 1d50dd5546633b2c9991ce3f847086397d95acc5 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/testReport/ |
| Max. process+thread count | 934 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
Re: [PR] HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem [hadoop]
hadoop-yetus commented on PR #6588: URL: https://github.com/apache/hadoop/pull/6588#issuecomment-1963405904 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 21s | | https://github.com/apache/hadoop/pull/6588 does not apply to HDFS-17384. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6588 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6588/1/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17394. [FGL] Remove unused WriteHoldCount of FSNamesystemLock [hadoop]
hadoop-yetus commented on PR #6571: URL: https://github.com/apache/hadoop/pull/6571#issuecomment-1963403242 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/6571 does not apply to HDFS-17384. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6571 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6571/2/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem [hadoop]
ZanderXu merged PR #6588: URL: https://github.com/apache/hadoop/pull/6588 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem [hadoop]
ZanderXu opened a new pull request, #6588: URL: https://github.com/apache/hadoop/pull/6588 Cherry-pick HDFS-17393 from trunk to HDFS-17384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19052) Hadoop use Shell command to get the count of the hard link which takes a lot of time
[
https://issues.apache.org/jira/browse/HADOOP-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820550#comment-17820550
]
ASF GitHub Bot commented on HADOOP-19052:
-
liangyu-1 opened a new pull request, #6587:
URL: https://github.com/apache/hadoop/pull/6587
### Description of PR
As described in
[HADOOP_19052](https://issues.apache.org/jira/browse/HADOOP-19052). When we try
to append a file, we will execute method `getHardLinkCount` twice. When we
execute method `getHardLinkCount` java will start a new process to execute a
shell command and wait for it to fork. When the QPS of `append` execution is
very high, method `getHardLinkCount` will take a long time to finish which
will cause a long-time wait to acquire lock.
I used another method to get the linkCount of a file whose file store
supports the file attributes identified by the given file attribute view. This
method does not start a new process and will finish in very short time even if
the QPS of `append` execution is high.
### How was this patch tested?
I add a new UT testGetLinkCountFromFileAttribute and a public method
supportsHardLink to get whether or not this file store supports the file
attributes identified by the given file attribute view.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Hadoop use Shell command to get the count of the hard link which takes a lot
> of time
>
>
> Key: HADOOP-19052
> URL: https://issues.apache.org/jira/browse/HADOOP-19052
> Project: Hadoop Common
> Issue Type: Improvement
> Environment: Hadopp 3.3.4
>Reporter: liang yu
>Priority: Major
> Labels: pull-request-available
> Attachments: debuglog.png
>
>
> Using Hadoop 3.3.4
>
> When the QPS of `append` executions is very high, at a rate of above 1/s.
>
> We found that the write speed in hadoop is very slow. We traced some
> datanodes' log and find that there is a warning :
> {code:java}
> 2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl
> (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to
> acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0
> lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
> java.lang.Thread,getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
> org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
> org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
> org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
> org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:230)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver
> (DataXceiver.java:1313)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock
> (DataXceiver.java:764)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> Then we traced the method
> _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.
> java:1239),_ and print how long each command take to finish the execution,
> and find that it takes us 700ms to get the linkCount of the file which is
> really slow.
> !debuglog.png!
>
> We traced the code and find that java1.8 use a Shell Command to get the
> linkCount, in which execution it will start a new Process and wait for the
> Process to fork, when the QPS is very high, it will sometimes take a long
> time to fork the process.
> Here is the shell command.
> {code:java}
> stat -c%h /path/to/file
> {code}
>
> Solution:
> For the FileStore that supports the file attributes "unix", we can use the
> method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount,
> this method doesn't need to start a
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
tomscut commented on PR #6509: URL: https://github.com/apache/hadoop/pull/6509#issuecomment-1963271347 Good catch! The changes look good to me. Wait for the Jenkins. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19052) Hadoop use Shell command to get the count of the hard link which takes a lot of time
[
https://issues.apache.org/jira/browse/HADOOP-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820529#comment-17820529
]
ASF GitHub Bot commented on HADOOP-19052:
-
liangyu-1 closed pull request #6527: HADOOP-19052.Hadoop use Shell command to
get the count of the hard link which takes a lot of time
URL: https://github.com/apache/hadoop/pull/6527
> Hadoop use Shell command to get the count of the hard link which takes a lot
> of time
>
>
> Key: HADOOP-19052
> URL: https://issues.apache.org/jira/browse/HADOOP-19052
> Project: Hadoop Common
> Issue Type: Improvement
> Environment: Hadopp 3.3.4
>Reporter: liang yu
>Priority: Major
> Labels: pull-request-available
> Attachments: debuglog.png
>
>
> Using Hadoop 3.3.4
>
> When the QPS of `append` executions is very high, at a rate of above 1/s.
>
> We found that the write speed in hadoop is very slow. We traced some
> datanodes' log and find that there is a warning :
> {code:java}
> 2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl
> (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to
> acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0
> lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
> java.lang.Thread,getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
> org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
> org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
> org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
> org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:230)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver
> (DataXceiver.java:1313)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock
> (DataXceiver.java:764)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> Then we traced the method
> _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.
> java:1239),_ and print how long each command take to finish the execution,
> and find that it takes us 700ms to get the linkCount of the file which is
> really slow.
> !debuglog.png!
>
> We traced the code and find that java1.8 use a Shell Command to get the
> linkCount, in which execution it will start a new Process and wait for the
> Process to fork, when the QPS is very high, it will sometimes take a long
> time to fork the process.
> Here is the shell command.
> {code:java}
> stat -c%h /path/to/file
> {code}
>
> Solution:
> For the FileStore that supports the file attributes "unix", we can use the
> method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount,
> this method doesn't need to start a new process, and will return the result
> in a very short time.
>
> When we use this method to get the file linkCount, we rarely get the WARN log
> above when the QPS of append execution is high.
> .
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time [hadoop]
liangyu-1 closed pull request #6527: HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time URL: https://github.com/apache/hadoop/pull/6527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
hfutatzhanghb commented on PR #6509: URL: https://github.com/apache/hadoop/pull/6509#issuecomment-1963222489 @zhangshuyan0 @haiyang1987 @tasanuma @tomscut Sir, have updated unit test, please check it again~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19074) Transitive dependencies with CVEs in Hadoop distro
[
https://issues.apache.org/jira/browse/HADOOP-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820511#comment-17820511
]
ASF GitHub Bot commented on HADOOP-19074:
-
hadoop-yetus commented on PR #6586:
URL: https://github.com/apache/hadoop/pull/6586#issuecomment-1963035870
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 35m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 41m 11s | | trunk passed |
| +1 :green_heart: | compile | 19m 54s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 17m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 41s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 41s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 2m 31s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 38m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 53s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 18m 50s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 18m 50s | | the patch passed |
| +1 :green_heart: | compile | 17m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 17m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 56s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 89 unchanged - 0 fixed = 90 total (was
89) |
| +1 :green_heart: | mvnsite | 3m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 12s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 49s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 36s | | hadoop-project has no data from
spotbugs |
| -1 :x: | shadedclient | 2m 44s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 20m 29s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 7m 32s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 0m 46s | | hadoop-rumen in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 276m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6586 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux eaae40a63a9a 5.15.0-94-generic
Re: [PR] Commit to remove vulnerabilities raised for: HADOOP-19074 [hadoop]
hadoop-yetus commented on PR #6586:
URL: https://github.com/apache/hadoop/pull/6586#issuecomment-1963035870
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 35m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 41m 11s | | trunk passed |
| +1 :green_heart: | compile | 19m 54s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 17m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 41s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 41s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 2m 31s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 38m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 53s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 18m 50s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 18m 50s | | the patch passed |
| +1 :green_heart: | compile | 17m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 17m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 56s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 89 unchanged - 0 fixed = 90 total (was
89) |
| +1 :green_heart: | mvnsite | 3m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 12s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 49s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 36s | | hadoop-project has no data from
spotbugs |
| -1 :x: | shadedclient | 2m 44s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 20m 29s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 7m 32s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 0m 46s | | hadoop-rumen in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 276m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6586/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6586 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux eaae40a63a9a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5b8e2af9ac050bb814e67c2ed6c642d34c2986dc |
| Default Java |
Re: [PR] YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata [hadoop]
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1963034340
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 4m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 59m 51s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 17s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 23m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 5 unchanged - 0 fixed = 13 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| -1 :x: | shadedclient | 26m 52s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 30m 17s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +0 :ok: | asflicense | 0m 25s | | ASF License check generated no
output? |
| | | 155m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 9d0efe4fa13d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / c4094ebd48ed921b6b60587827eea68aa28bfc29 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/testReport/ |
| Max. process+thread count | 706 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
[jira] [Commented] (HADOOP-19041) further use of StandardCharsets
[
https://issues.apache.org/jira/browse/HADOOP-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820491#comment-17820491
]
ASF GitHub Bot commented on HADOOP-19041:
-
hadoop-yetus commented on PR #6449:
URL: https://github.com/apache/hadoop/pull/6449#issuecomment-196323
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 33 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 30m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 6s | | trunk passed |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 3s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 17m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 15m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 14m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 31s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6449/6/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 1m 2s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6449/6/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-httpfs in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 33m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 45s | | the patch passed |
| +1 :green_heart: | compile | 16m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 38s | | the patch passed |
| +1 :green_heart: | compile | 16m 14s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 34s | | root: The patch generated
0 new + 1282 unchanged - 24 fixed = 1282 total (was 1306) |
| +1 :green_heart: | mvnsite | 17m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 15m 3s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 14m 54s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 29m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 41s | | hadoop-maven-plugins in the
patch passed. |
| +1 :green_heart: | unit | 3m 31s | | hadoop-auth in the patch
passed. |
| +1 :green_heart: | unit | 19m 40s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 277m 11s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6449/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 6m 30s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | unit | 6m 6s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 5m 10s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 106m 19s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1
Re: [PR] HADOOP-19041. Use StandardCharsets in more places [hadoop]
hadoop-yetus commented on PR #6449:
URL: https://github.com/apache/hadoop/pull/6449#issuecomment-196323
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 33 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 30m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 6s | | trunk passed |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 3s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 17m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 15m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 14m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 31s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6449/6/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 1m 2s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6449/6/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-httpfs in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 33m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 45s | | the patch passed |
| +1 :green_heart: | compile | 16m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 38s | | the patch passed |
| +1 :green_heart: | compile | 16m 14s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 34s | | root: The patch generated
0 new + 1282 unchanged - 24 fixed = 1282 total (was 1306) |
| +1 :green_heart: | mvnsite | 17m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 15m 3s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 14m 54s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 29m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 41s | | hadoop-maven-plugins in the
patch passed. |
| +1 :green_heart: | unit | 3m 31s | | hadoop-auth in the patch
passed. |
| +1 :green_heart: | unit | 19m 40s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 277m 11s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6449/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 6m 30s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | unit | 6m 6s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 5m 10s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 106m 19s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 24m 54s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | unit | 28m 35s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 7m 56s | |
[jira] [Commented] (HADOOP-19074) Transitive dependencies with CVEs in Hadoop distro
[ https://issues.apache.org/jira/browse/HADOOP-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820484#comment-17820484 ] Prathap Sagar S commented on HADOOP-19074: -- [~ste...@apache.org] I have fixed the above vulnerabilities and create a pull request for the same. Can you please review it. [HADOOP-19074 #6586|[https://github.com/apache/hadoop/pull/6586]] > Transitive dependencies with CVEs in Hadoop distro > -- > > Key: HADOOP-19074 > URL: https://issues.apache.org/jira/browse/HADOOP-19074 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Prathap Sagar S >Priority: Major > Labels: pull-request-available > Attachments: HADOOP_CVE_LIST.xlsx > > > Our ongoing security scans are turning up several long-standing CVEs, even in > the most recent version of Hadoop, which is making it difficult for us to use > Hadoop in our echo system. A comprehensive list of all the long-standing CVEs > and the JARs holding them is attached. I'm asking for community assistance to > address these high-risk vulnerabilities as soon as possible. > > |Vulnerability ID|Severity|Package name|Package version|Package type|Package > path|Package suggested fix| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|12.0.1|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/guava-12.0.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2022-25647|High|com.google.code.gson:gson|2.8.5|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/hbase-shaded-gson-3.0.0.jar|v2.8.9| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/avro-1.9.2.jar|v1.11.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v1.11.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/common/lib/avro-1.9.2.jar|v1.11.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| >
[jira] [Updated] (HADOOP-19074) Transitive dependencies with CVEs in Hadoop distro
[ https://issues.apache.org/jira/browse/HADOOP-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19074: Labels: pull-request-available (was: ) > Transitive dependencies with CVEs in Hadoop distro > -- > > Key: HADOOP-19074 > URL: https://issues.apache.org/jira/browse/HADOOP-19074 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Prathap Sagar S >Priority: Major > Labels: pull-request-available > Attachments: HADOOP_CVE_LIST.xlsx > > > Our ongoing security scans are turning up several long-standing CVEs, even in > the most recent version of Hadoop, which is making it difficult for us to use > Hadoop in our echo system. A comprehensive list of all the long-standing CVEs > and the JARs holding them is attached. I'm asking for community assistance to > address these high-risk vulnerabilities as soon as possible. > > |Vulnerability ID|Severity|Package name|Package version|Package type|Package > path|Package suggested fix| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|12.0.1|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/guava-12.0.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2022-25647|High|com.google.code.gson:gson|2.8.5|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/hbase-shaded-gson-3.0.0.jar|v2.8.9| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/avro-1.9.2.jar|v1.11.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v1.11.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/common/lib/avro-1.9.2.jar|v1.11.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| >
[jira] [Commented] (HADOOP-19074) Transitive dependencies with CVEs in Hadoop distro
[ https://issues.apache.org/jira/browse/HADOOP-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820481#comment-17820481 ] ASF GitHub Bot commented on HADOOP-19074: - prathapsagar opened a new pull request, #6586: URL: https://github.com/apache/hadoop/pull/6586 1) Update Gauva to Version: 32.0.1-jre 2) Update Protobuf to Version: 3.21.12 2) Update Avro to Version: 1.11.3 4) Updated private access in the below files for Avro compability: - hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/TestAvroSerialization.java - hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobQueueChangeEvent.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobHistoryUtils.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Transitive dependencies with CVEs in Hadoop distro > -- > > Key: HADOOP-19074 > URL: https://issues.apache.org/jira/browse/HADOOP-19074 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Prathap Sagar S >Priority: Major > Attachments: HADOOP_CVE_LIST.xlsx > > > Our ongoing security scans are turning up several long-standing CVEs, even in > the most recent version of Hadoop, which is making it difficult for us to use > Hadoop in our echo system. A comprehensive list of all the long-standing CVEs > and the JARs holding them is attached. I'm asking for community assistance to > address these high-risk vulnerabilities as soon as possible. > > |Vulnerability ID|Severity|Package name|Package version|Package type|Package > path|Package suggested fix| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|12.0.1|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/guava-12.0.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2022-25647|High|com.google.code.gson:gson|2.8.5|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/hbase-shaded-gson-3.0.0.jar|v2.8.9| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| >
[PR] Commit to remove vulnerabilities raised for: HADOOP-19074 [hadoop]
prathapsagar opened a new pull request, #6586: URL: https://github.com/apache/hadoop/pull/6586 1) Update Gauva to Version: 32.0.1-jre 2) Update Protobuf to Version: 3.21.12 2) Update Avro to Version: 1.11.3 4) Updated private access in the below files for Avro compability: - hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/TestAvroSerialization.java - hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobQueueChangeEvent.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobHistoryUtils.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java - hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19088) upgrade to jersey-json 1.22.0
[
https://issues.apache.org/jira/browse/HADOOP-19088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820469#comment-17820469
]
ASF GitHub Bot commented on HADOOP-19088:
-
hadoop-yetus commented on PR #6585:
URL: https://github.com/apache/hadoop/pull/6585#issuecomment-1962926858
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 27m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 43m 6s | | trunk passed |
| -1 :x: | compile | 20m 15s |
[/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 38s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
|
| -1 :x: | mvnsite | 0m 38s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| -1 :x: | javadoc | 0m 38s |
[/branch-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 38s |
[/branch-javadoc-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-javadoc-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
|
| -1 :x: | shadedclient | 2m 53s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 39s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 37m 38s | | the patch passed |
| +1 :green_heart: | compile | 18m 59s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 18m 59s | | the patch passed |
| +1 :green_heart: | compile | 16m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | javac | 16m 26s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08 with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08 generated 44 new + 0 unchanged - 0 fixed =
44 total (was 0) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 25m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 8m 32s |
[/results-javadoc-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/results-javadoc-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 generated 1283 new + 0 unchanged - 0
fixed = 1283 total (was 0) |
| -1 :x: | javadoc | 8m 7s |
Re: [PR] HADOOP-19088. Use jersey-json 1.22.0 [hadoop]
hadoop-yetus commented on PR #6585:
URL: https://github.com/apache/hadoop/pull/6585#issuecomment-1962926858
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 27m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 43m 6s | | trunk passed |
| -1 :x: | compile | 20m 15s |
[/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 38s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
|
| -1 :x: | mvnsite | 0m 38s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| -1 :x: | javadoc | 0m 38s |
[/branch-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 38s |
[/branch-javadoc-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/branch-javadoc-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.
|
| -1 :x: | shadedclient | 2m 53s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 39s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 37m 38s | | the patch passed |
| +1 :green_heart: | compile | 18m 59s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 18m 59s | | the patch passed |
| +1 :green_heart: | compile | 16m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | javac | 16m 26s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08 with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08 generated 44 new + 0 unchanged - 0 fixed =
44 total (was 0) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 25m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 8m 32s |
[/results-javadoc-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/results-javadoc-javadoc-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 generated 1283 new + 0 unchanged - 0
fixed = 1283 total (was 0) |
| -1 :x: | javadoc | 8m 7s |
[/results-javadoc-javadoc-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6585/1/artifact/out/results-javadoc-javadoc-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08 with JDK Private
[jira] [Commented] (HADOOP-18950) upgrade avro to 1.11.3 due to CVE
[
https://issues.apache.org/jira/browse/HADOOP-18950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820446#comment-17820446
]
ASF GitHub Bot commented on HADOOP-18950:
-
hadoop-yetus commented on PR #4854:
URL: https://github.com/apache/hadoop/pull/4854#issuecomment-1962873755
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 27m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 44s | | trunk passed |
| +1 :green_heart: | compile | 17m 36s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 9s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 28m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 12m 43s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 49s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 20s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 2m 13s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4854/9/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| -1 :x: | spotbugs | 30m 59s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4854/9/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 5 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 62m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 47s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 45m 14s | | the patch passed |
| +1 :green_heart: | compile | 16m 58s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 58s | | the patch passed |
| +1 :green_heart: | compile | 16m 7s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 27s | | root: The patch generated
0 new + 445 unchanged - 1 fixed = 445 total (was 446) |
| +1 :green_heart: | mvnsite | 15m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 31s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 51s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 22s | | hadoop-project has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 23s | | hadoop-client-modules/hadoop-client
has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 23s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +1 :green_heart: | shadedclient | 63m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 790m 22s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4854/9/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1
[jira] [Updated] (HADOOP-18950) upgrade avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-18950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18950: Labels: pull-request-available (was: ) > upgrade avro to 1.11.3 due to CVE > - > > Key: HADOOP-18950 > URL: https://issues.apache.org/jira/browse/HADOOP-18950 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Xuze Yang >Priority: Major > Labels: pull-request-available > > [https://nvd.nist.gov/vuln/detail/CVE-2023-39410] > When deserializing untrusted or corrupted data, it is possible for a reader > to consume memory beyond the allowed constraints and thus lead to out of > memory on the system. This issue affects Java applications using Apache Avro > Java SDK up to and including 1.11.2. Users should update to apache-avro > version 1.11.3 which addresses this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18950: shaded avro jar [hadoop]
hadoop-yetus commented on PR #4854:
URL: https://github.com/apache/hadoop/pull/4854#issuecomment-1962873755
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 27m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 44s | | trunk passed |
| +1 :green_heart: | compile | 17m 36s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 9s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 28m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 12m 43s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 49s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 20s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 2m 13s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4854/9/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| -1 :x: | spotbugs | 30m 59s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4854/9/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 5 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 62m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 47s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 45m 14s | | the patch passed |
| +1 :green_heart: | compile | 16m 58s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 58s | | the patch passed |
| +1 :green_heart: | compile | 16m 7s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 27s | | root: The patch generated
0 new + 445 unchanged - 1 fixed = 445 total (was 446) |
| +1 :green_heart: | mvnsite | 15m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 31s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 51s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 22s | | hadoop-project has no data from
spotbugs |
| +0 :ok: | spotbugs | 0m 23s | | hadoop-client-modules/hadoop-client
has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 23s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +1 :green_heart: | shadedclient | 63m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 790m 22s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4854/9/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 1219m 23s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
