[PR] HDFS-17390. [FGL] FSDirectory supports this fine-grained locking [hadoop]
ZanderXu opened a new pull request, #6573: URL: https://github.com/apache/hadoop/pull/6573 FSDirectory should holds read or write lock of FSLock, since it just involves directory tree-related operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17387. [FGL] Abstract selectable locking mode [hadoop]
ZanderXu opened a new pull request, #6572: URL: https://github.com/apache/hadoop/pull/6572 Splitting the current global FNS lock into FSLock and BMLock. Abstract these lock mode and make them selectable, so that end-user can choose to use global lock or fine-grained lock through configuration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6566: URL: https://github.com/apache/hadoop/pull/6566#issuecomment-1958788319 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/5/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17394. [FGL] Remove unused WriteHoldCount of FSNamesystemLock [hadoop]
ZanderXu opened a new pull request, #6571:
URL: https://github.com/apache/hadoop/pull/6571
```
public int getWriteHoldCount() {
return this.fsLock.getWriteHoldCount(FSNamesystemLockMode.GLOBAL);
}
@Deprecated // dirLock is obsolete, use namesystem.fsLock instead
public int getWriteHoldCount() {
return namesystem.getWriteHoldCount();
}
// sanity check.
if (!hadDirReadLock || !hadFsnReadLock || hadDirWriteLock ||
hadFsnWriteLock || dir.getReadHoldCount() != 1 ||
fsn.getReadHoldCount() != 1) {
// cannot relinquish
return false;
}
```
`getWriteHoldCount` in FSNamesystem.java and FSDirectory.java is unused.
`dir.getReadHoldCount()` is useless as it's same as `fsn.getReadHoldCount()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819483#comment-17819483
]
ASF GitHub Bot commented on HADOOP-18910:
-
anujmodi2021 commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498704079
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsInvalidChecksumException.java:
##
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+
+/**
+ * Exception to wrap invalid checksum verification on client side.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class AbfsInvalidChecksumException extends AbfsRestOperationException {
+
+ private static final String ERROR_MESSAGE = "Checksum Validation Failed, MD5
Mismatch Error";
+
+ public AbfsInvalidChecksumException(final AbfsRestOperationException
abfsRestOperationException) {
+super(
+abfsRestOperationException != null
+? abfsRestOperationException.getStatusCode()
+: AzureServiceErrorCode.UNKNOWN.getStatusCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.getErrorCode().getErrorCode()
+: AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.toString()
Review Comment:
toString() will be better it has more information like the URL which was hit.
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498704079
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsInvalidChecksumException.java:
##
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+
+/**
+ * Exception to wrap invalid checksum verification on client side.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class AbfsInvalidChecksumException extends AbfsRestOperationException {
+
+ private static final String ERROR_MESSAGE = "Checksum Validation Failed, MD5
Mismatch Error";
+
+ public AbfsInvalidChecksumException(final AbfsRestOperationException
abfsRestOperationException) {
+super(
+abfsRestOperationException != null
+? abfsRestOperationException.getStatusCode()
+: AzureServiceErrorCode.UNKNOWN.getStatusCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.getErrorCode().getErrorCode()
+: AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.toString()
Review Comment:
toString() will be better it has more information like the URL which was hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6566:
URL: https://github.com/apache/hadoop/pull/6566#issuecomment-1958776146
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 46s | | trunk passed |
| +1 :green_heart: | compile | 17m 29s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 5s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 45s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 2m 45s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/4/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs
warnings. |
| -1 :x: | shadedclient | 4m 59s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 17s | | the patch passed |
| +1 :green_heart: | compile | 16m 42s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 42s | | the patch passed |
| +1 :green_heart: | compile | 16m 2s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 2s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/4/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 12s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 17 new + 243 unchanged - 2 fixed = 260 total (was
245) |
| +1 :green_heart: | mvnsite | 3m 30s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 2m 54s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 30s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 40s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 36m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 38s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 2m 44s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 224m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 434m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
hfutatzhanghb commented on code in PR #6509:
URL: https://github.com/apache/hadoop/pull/6509#discussion_r1498581949
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:
##
@@ -436,9 +442,17 @@ protected void recover() throws IOException {
"datanode={})", block, internalBlk, id, e);
}
}
- checkLocations(syncBlocks.size());
- final long safeLength = getSafeLength(syncBlocks);
+ final long safeLength;
+ if (dnNotHaveReplicaCnt + zeroLenReplicaCnt <= locs.length -
ecPolicy.getNumDataUnits()) {
+checkLocations(syncBlocks.size());
+safeLength = getSafeLength(syncBlocks);
+ } else {
+safeLength = 0;
+LOG.warn("Block recovery: More than {} datanodes do not have the
replica of block {}." +
Review Comment:
It seems useless here, have removed it . Originally, it was more than (9 -
6) datanodes ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
zhangshuyan0 commented on code in PR #6509:
URL: https://github.com/apache/hadoop/pull/6509#discussion_r1498579433
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:
##
@@ -436,9 +442,17 @@ protected void recover() throws IOException {
"datanode={})", block, internalBlk, id, e);
}
}
- checkLocations(syncBlocks.size());
- final long safeLength = getSafeLength(syncBlocks);
+ final long safeLength;
+ if (dnNotHaveReplicaCnt + zeroLenReplicaCnt <= locs.length -
ecPolicy.getNumDataUnits()) {
+checkLocations(syncBlocks.size());
+safeLength = getSafeLength(syncBlocks);
+ } else {
+safeLength = 0;
+LOG.warn("Block recovery: More than {} datanodes do not have the
replica of block {}." +
Review Comment:
What does this "More than" mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
hfutatzhanghb commented on code in PR #6509:
URL: https://github.com/apache/hadoop/pull/6509#discussion_r1498576377
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:
##
@@ -436,9 +442,17 @@ protected void recover() throws IOException {
"datanode={})", block, internalBlk, id, e);
}
}
- checkLocations(syncBlocks.size());
- final long safeLength = getSafeLength(syncBlocks);
+ final long safeLength;
+ if (dnNotHaveReplicaCnt + zeroLenReplicaCnt <= locs.length -
ecPolicy.getNumDataUnits()) {
+checkLocations(syncBlocks.size());
+safeLength = getSafeLength(syncBlocks);
+ } else {
+safeLength = 0;
+LOG.warn("Block recovery: More than {} datanodes do not have the
replica of block {}." +
Review Comment:
Fixed it. Thanks sir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
zhangshuyan0 commented on code in PR #6509:
URL: https://github.com/apache/hadoop/pull/6509#discussion_r1498565609
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:
##
@@ -436,9 +442,17 @@ protected void recover() throws IOException {
"datanode={})", block, internalBlk, id, e);
}
}
- checkLocations(syncBlocks.size());
- final long safeLength = getSafeLength(syncBlocks);
+ final long safeLength;
+ if (dnNotHaveReplicaCnt + zeroLenReplicaCnt <= locs.length -
ecPolicy.getNumDataUnits()) {
+checkLocations(syncBlocks.size());
+safeLength = getSafeLength(syncBlocks);
+ } else {
+safeLength = 0;
+LOG.warn("Block recovery: More than {} datanodes do not have the
replica of block {}." +
Review Comment:
Suggest printing out the value of `zeroLenReplicaCnt` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819454#comment-17819454
]
Kai Zheng commented on HADOOP-19085:
Hi [~han.liu], since you're working on this, would you take it?
[~ste...@apache.org] could you help assign, thanks! I failed because I can't
see his name.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-19085) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HADOOP-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng moved HDFS-17316 to HADOOP-19085:
---
Key: HADOOP-19085 (was: HDFS-17316)
Project: Hadoop Common (was: Hadoop HDFS)
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HADOOP-19085
> URL: https://issues.apache.org/jira/browse/HADOOP-19085
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819448#comment-17819448 ] ASF GitHub Bot commented on HADOOP-19065: - huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1958556471 Pipeline failed, but the local env launched docker image was successfully built and compiled. cc @tasanuma 1. Install Docker and run this command: ``` ./start-build-env.sh ``` 2. trigger compile: ``` mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true ``` ``` [INFO] Apache Hadoop Cloud Storage SUCCESS [ 3.984 s] [INFO] Apache Hadoop Cloud Storage Project SUCCESS [ 0.016 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 16:56 min [INFO] Finished at: 2024-02-22T02:39:26Z [INFO] xxx@886b0e5bc37b:~/hadoop$ ``` 3. check docker env protobuf version: ``` protoc --version ``` ``` libprotoc 3.21.12 ``` > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1958556471 Pipeline failed, but the local env launched docker image was successfully built and compiled. cc @tasanuma 1. Install Docker and run this command: ``` ./start-build-env.sh ``` 2. trigger compile: ``` mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true ``` ``` [INFO] Apache Hadoop Cloud Storage SUCCESS [ 3.984 s] [INFO] Apache Hadoop Cloud Storage Project SUCCESS [ 0.016 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 16:56 min [INFO] Finished at: 2024-02-22T02:39:26Z [INFO] xxx@886b0e5bc37b:~/hadoop$ ``` 3. check docker env protobuf version: ``` protoc --version ``` ``` libprotoc 3.21.12 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17352. Add configuration to control whether DN delete this replica from disk when client requests a missing block [hadoop]
haiyang1987 commented on code in PR #6559:
URL: https://github.com/apache/hadoop/pull/6559#discussion_r1498534965
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##
@@ -188,6 +188,10 @@ public class DFSConfigKeys extends CommonConfigurationKeys
{
public static final long DFS_DN_CACHED_DFSUSED_CHECK_INTERVAL_DEFAULT_MS =
60;
+ public static final String
DFS_DATANODE_DELETE_CORRUPT_REPLICA_FROM_DISK_ENABLE =
Review Comment:
Thanks @tomscut for your review.
I wll support dynamically configured later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11656 RMStateStore event queue blocked [hadoop]
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1958479227
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 49s | | trunk passed |
| +1 :green_heart: | compile | 7m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 6m 59s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 1s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 6m 51s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 6m 51s | | the patch passed |
| +1 :green_heart: | compile | 6m 57s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 6m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 47s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 71 unchanged
- 1 fixed = 83 total (was 72) |
| +1 :green_heart: | mvnsite | 1m 57s | | the patch passed |
| -1 :x: | javadoc | 0m 58s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-yarn-common in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 57s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-yarn-common in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 35s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 106m 58s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| -1 :x: | asflicense | 0m 58s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warnings. |
| | | 288m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| |
Re: [PR] HDFS-17299. Adding rack failure tolerance when creating a new file [hadoop]
hadoop-yetus commented on PR #6566: URL: https://github.com/apache/hadoop/pull/6566#issuecomment-1958262110 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6566/4/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19083) hadoop binary tarball to exclude aws v2 sdk
[
https://issues.apache.org/jira/browse/HADOOP-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819401#comment-17819401
]
ASF GitHub Bot commented on HADOOP-19083:
-
hadoop-yetus commented on PR #6570:
URL: https://github.com/apache/hadoop/pull/6570#issuecomment-1958145264
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 78m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 33m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-tools-dist in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 119m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6570/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6570 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux d60acba26022 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 31f9a71b8c85ff79f9aa2ac5754686e75741b58a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6570/1/testReport/ |
| Max. process+thread count | 691 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-tools-dist U:
hadoop-tools/hadoop-tools-dist |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6570/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> hadoop binary tarball to exclude aws v2 sdk
> ---
>
> Key: HADOOP-19083
> URL: https://issues.apache.org/jira
Re: [PR] HADOOP-19083. hadoop binaries aws sdk unless -Pawssdk builds [hadoop]
hadoop-yetus commented on PR #6570:
URL: https://github.com/apache/hadoop/pull/6570#issuecomment-1958145264
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 78m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 33m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-tools-dist in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 119m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6570/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6570 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux d60acba26022 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 31f9a71b8c85ff79f9aa2ac5754686e75741b58a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6570/1/testReport/ |
| Max. process+thread count | 691 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-tools-dist U:
hadoop-tools/hadoop-tools-dist |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6570/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[ https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819386#comment-17819386 ] ASF GitHub Bot commented on HADOOP-18910: - mukund-thakur commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1957878931 `./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:28:import java.nio.charset.StandardCharsets;:8: Unused import - java.nio.charset.StandardCharsets. [UnusedImports]` please fix this checkstyle error. > ABFS: Adding Support for MD5 Hash based integrity verification of the request > content during transport > --- > > Key: HADOOP-18910 > URL: https://issues.apache.org/jira/browse/HADOOP-18910 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Azure Storage Supports Content-MD5 Request Headers in Both Read and Append > APIs. > Read: [Path - Read - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read] > Append: [Path - Update - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update] > This change is to make client-side changes to support them. In Read request, > we will send the appropriate header in response to which server will return > the MD5 Hash of the data it sends back. On Client we will tally this with the > MD5 hash computed from the data received. > In Append request, we will compute the MD5 Hash of the data that we are > sending to the server and specify that in appropriate header. Server on > finding that header will tally this with the MD5 hash it will compute on the > data received. > This whole Checksum Validation Support is guarded behind a config, Config is > by default disabled because with the use of "https" integrity of data is > preserved anyways. This is introduced as an additional data integrity check > which will have a performance impact as well. > Users can decide if they want to enable this or not by setting the following > config to *"true"* or *"false"* respectively. *Config: > "fs.azure.enable.checksum.validation"* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
mukund-thakur commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1957878931 `./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:28:import java.nio.charset.StandardCharsets;:8: Unused import - java.nio.charset.StandardCharsets. [UnusedImports]` please fix this checkstyle error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819385#comment-17819385
]
ASF GitHub Bot commented on HADOOP-18910:
-
mukund-thakur commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498282476
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStream.java:
##
@@ -279,3 +283,17 @@ private void verifyAfterSeek(AbfsInputStream
abfsInputStream, long seekPos) thro
assertEquals(0, abfsInputStream.getBCursor());
}
}
+
Review Comment:
cut this
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStream.java:
##
@@ -279,3 +283,17 @@ private void verifyAfterSeek(AbfsInputStream
abfsInputStream, long seekPos) thro
assertEquals(0, abfsInputStream.getBCursor());
}
}
+
+//
+//
+//<<< HEAD
+//private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead,
+//int fileSize) throws IOException {
+//final AzureBlobFileSystem fs = getFileSystem();
+//getAbfsStore(fs).getAbfsConfiguration()
+//.setOptimizeFooterRead(optimizeFooterRead);
+//getAbfsStore(fs).getAbfsConfiguration()
+//.setIsChecksumValidationEnabled(true);
+//if (fileSize <= getAbfsStore(fs).getAbfsConfiguration()
+//.getReadBufferSize()) {
+//===
Review Comment:
cut this
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
mukund-thakur commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498282476
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStream.java:
##
@@ -279,3 +283,17 @@ private void verifyAfterSeek(AbfsInputStream
abfsInputStream, long seekPos) thro
assertEquals(0, abfsInputStream.getBCursor());
}
}
+
Review Comment:
cut this
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStream.java:
##
@@ -279,3 +283,17 @@ private void verifyAfterSeek(AbfsInputStream
abfsInputStream, long seekPos) thro
assertEquals(0, abfsInputStream.getBCursor());
}
}
+
+//
+//
+//<<< HEAD
+//private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead,
+//int fileSize) throws IOException {
+//final AzureBlobFileSystem fs = getFileSystem();
+//getAbfsStore(fs).getAbfsConfiguration()
+//.setOptimizeFooterRead(optimizeFooterRead);
+//getAbfsStore(fs).getAbfsConfiguration()
+//.setIsChecksumValidationEnabled(true);
+//if (fileSize <= getAbfsStore(fs).getAbfsConfiguration()
+//.getReadBufferSize()) {
+//===
Review Comment:
cut this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[ https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819384#comment-17819384 ] ASF GitHub Bot commented on HADOOP-18910: - mukund-thakur commented on code in PR #6069: URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498282476 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStream.java: ## @@ -279,3 +283,17 @@ private void verifyAfterSeek(AbfsInputStream abfsInputStream, long seekPos) thro assertEquals(0, abfsInputStream.getBCursor()); } } + Review Comment: cut this > ABFS: Adding Support for MD5 Hash based integrity verification of the request > content during transport > --- > > Key: HADOOP-18910 > URL: https://issues.apache.org/jira/browse/HADOOP-18910 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Azure Storage Supports Content-MD5 Request Headers in Both Read and Append > APIs. > Read: [Path - Read - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read] > Append: [Path - Update - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update] > This change is to make client-side changes to support them. In Read request, > we will send the appropriate header in response to which server will return > the MD5 Hash of the data it sends back. On Client we will tally this with the > MD5 hash computed from the data received. > In Append request, we will compute the MD5 Hash of the data that we are > sending to the server and specify that in appropriate header. Server on > finding that header will tally this with the MD5 hash it will compute on the > data received. > This whole Checksum Validation Support is guarded behind a config, Config is > by default disabled because with the use of "https" integrity of data is > preserved anyways. This is introduced as an additional data integrity check > which will have a performance impact as well. > Users can decide if they want to enable this or not by setting the following > config to *"true"* or *"false"* respectively. *Config: > "fs.azure.enable.checksum.validation"* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
mukund-thakur commented on code in PR #6069: URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498282476 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStream.java: ## @@ -279,3 +283,17 @@ private void verifyAfterSeek(AbfsInputStream abfsInputStream, long seekPos) thro assertEquals(0, abfsInputStream.getBCursor()); } } + Review Comment: cut this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19083) hadoop binary tarball to exclude aws v2 sdk
[ https://issues.apache.org/jira/browse/HADOOP-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19083: Labels: pull-request-available (was: ) > hadoop binary tarball to exclude aws v2 sdk > --- > > Key: HADOOP-19083 > URL: https://issues.apache.org/jira/browse/HADOOP-19083 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default. > This SDK brings the total size of the distribution to about 1 GB. > Proposed > * add a profile to include the aws sdk in the dist module > * disable it by default > Instead we document which version is needed. > The hadoop-aws and hadoop-cloud storage maven artifacts will declare their > dependencies, so apps building with those get to do the download. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19083) hadoop binary tarball to exclude aws v2 sdk
[ https://issues.apache.org/jira/browse/HADOOP-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819380#comment-17819380 ] ASF GitHub Bot commented on HADOOP-19083: - steveloughran opened a new pull request, #6570: URL: https://github.com/apache/hadoop/pull/6570 New profile awssdk must be set for hadoop-tools-dist to include the AWS SDK (so cuts binary size in half) * This is not yet documented * no attempt to be clever in LICENSE-binary and remove the declaration. ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > hadoop binary tarball to exclude aws v2 sdk > --- > > Key: HADOOP-19083 > URL: https://issues.apache.org/jira/browse/HADOOP-19083 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default. > This SDK brings the total size of the distribution to about 1 GB. > Proposed > * add a profile to include the aws sdk in the dist module > * disable it by default > Instead we document which version is needed. > The hadoop-aws and hadoop-cloud storage maven artifacts will declare their > dependencies, so apps building with those get to do the download. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19084) hadoop-common exports logback as a transitive dependency
Steve Loughran created HADOOP-19084:
---
Summary: hadoop-common exports logback as a transitive dependency
Key: HADOOP-19084
URL: https://issues.apache.org/jira/browse/HADOOP-19084
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.4.0, 3.5.0
Reporter: Steve Loughran
this is probably caused by HADOOP-18613:
ZK is pulling in some extra transitive stuff which surfaces in applications
which import hadoop-common into their poms. It doesn't seem to show up in our
distro, but downstream you get warnings about duplicate logging stuff
{code}
| +- org.apache.zookeeper:zookeeper:jar:3.8.3:compile
| | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.3:compile
| | | \- (org.apache.yetus:audience-annotations:jar:0.12.0:compile - omitted
for duplicate)
| | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile
| | +- (io.netty:netty-handler:jar:4.1.94.Final:compile - omitted for
conflict with 4.1.100.Final)
| | +- (io.netty:netty-transport-native-epoll:jar:4.1.94.Final:compile -
omitted for conflict with 4.1.100.Final)
| | +- (org.slf4j:slf4j-api:jar:1.7.30:compile - omitted for duplicate)
| | +- ch.qos.logback:logback-core:jar:1.2.10:compile
| | +- ch.qos.logback:logback-classic:jar:1.2.10:compile
| | | +- (ch.qos.logback:logback-core:jar:1.2.10:compile - omitted for
duplicate)
| | | \- (org.slf4j:slf4j-api:jar:1.7.32:compile - omitted for conflict with
1.7.30)
| | \- (commons-io:commons-io:jar:2.11.0:compile - omitted for conflict with
2.14.0)
{code}
proposed: exclude the zk dependencies we either override outselves or don't
need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819379#comment-17819379
]
ASF GitHub Bot commented on HADOOP-18910:
-
mukund-thakur commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498253744
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsInvalidChecksumException.java:
##
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+
+/**
+ * Exception to wrap invalid checksum verification on client side.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class AbfsInvalidChecksumException extends AbfsRestOperationException {
+
+ private static final String ERROR_MESSAGE = "Checksum Validation Failed, MD5
Mismatch Error";
+
+ public AbfsInvalidChecksumException(final AbfsRestOperationException
abfsRestOperationException) {
+super(
+abfsRestOperationException != null
+? abfsRestOperationException.getStatusCode()
+: AzureServiceErrorCode.UNKNOWN.getStatusCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.getErrorCode().getErrorCode()
+: AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.toString()
Review Comment:
do we need full toString here or just ex.getMessage()
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
mukund-thakur commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1498253744
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsInvalidChecksumException.java:
##
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+
+/**
+ * Exception to wrap invalid checksum verification on client side.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class AbfsInvalidChecksumException extends AbfsRestOperationException {
+
+ private static final String ERROR_MESSAGE = "Checksum Validation Failed, MD5
Mismatch Error";
+
+ public AbfsInvalidChecksumException(final AbfsRestOperationException
abfsRestOperationException) {
+super(
+abfsRestOperationException != null
+? abfsRestOperationException.getStatusCode()
+: AzureServiceErrorCode.UNKNOWN.getStatusCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.getErrorCode().getErrorCode()
+: AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+abfsRestOperationException != null
+? abfsRestOperationException.toString()
Review Comment:
do we need full toString here or just ex.getMessage()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] YARN-11656 RMStateStore event queue blocked [hadoop]
K0K0V0K opened a new pull request, #6569: URL: https://github.com/apache/hadoop/pull/6569 ### Description of PR Created a MultiDispatcher class which implements the Dispatcher interface. The dispatcher replace the AsyncDispatcher in the RMStateStore. The Dispatcher creates a separate metric object called Event metrics for "rm-state-store" where we can see - how many unhandled events are currently present in the event queue for the specific event type - how many events were handled for the specific event type - average execution time for the specific event ### How was this patch tested? - Unit test - Deployed cluster and load test with PI example job with the following implementations - - FileSystemRMStateStore - - LeveldbRMStateStore - - ZKRMStateStore - Check RM stop - start can't corrupt RM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[
https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819359#comment-17819359
]
ASF GitHub Bot commented on HADOOP-19065:
-
hadoop-yetus commented on PR #6526:
URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957664677
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | mvnsite | 0m 24s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| -1 :x: | shadedclient | 2m 14s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 9s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -1 :x: | mvnsite | 0m 29s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| -1 :x: | shadedclient | 2m 50s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 24s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +0 :ok: | asflicense | 0m 25s | | ASF License check generated no
output? |
| | | 11m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6526 |
| Optional Tests | dupname asflicense codespell detsecrets hadolint
shellcheck shelldocs mvnsite unit |
| uname | Linux e1a4968923ae 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62390b2dec0d3fad3332d908168a09f8914969df |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/testReport/ |
| Max. process+thread count | 51 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Improve Protocol Buffers install instructions.
> --
>
> Key: HADOOP-19065
> URL: https://issues.apache.org/jira/browse/HADOOP-19065
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: huangzhaobo
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526:
URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957664677
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | mvnsite | 0m 24s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| -1 :x: | shadedclient | 2m 14s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 9s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -1 :x: | mvnsite | 0m 29s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| -1 :x: | shadedclient | 2m 50s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 24s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +0 :ok: | asflicense | 0m 25s | | ASF License check generated no
output? |
| | | 11m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6526 |
| Optional Tests | dupname asflicense codespell detsecrets hadolint
shellcheck shelldocs mvnsite unit |
| uname | Linux e1a4968923ae 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62390b2dec0d3fad3332d908168a09f8914969df |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/testReport/ |
| Max. process+thread count | 51 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819356#comment-17819356 ] ASF GitHub Bot commented on HADOOP-19065: - hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957645773 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/console in case of problems. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957645773 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-19083) hadoop binary tarball to exclude aws v2 sdk
[ https://issues.apache.org/jira/browse/HADOOP-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-19083: --- Assignee: Steve Loughran > hadoop binary tarball to exclude aws v2 sdk > --- > > Key: HADOOP-19083 > URL: https://issues.apache.org/jira/browse/HADOOP-19083 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default. > This SDK brings the total size of the distribution to about 1 GB. > Proposed > * add a profile to include the aws sdk in the dist module > * disable it by default > Instead we document which version is needed. > The hadoop-aws and hadoop-cloud storage maven artifacts will declare their > dependencies, so apps building with those get to do the download. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[
https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819337#comment-17819337
]
ASF GitHub Bot commented on HADOOP-19065:
-
hadoop-yetus commented on PR #6526:
URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957424893
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 52m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 4m 31s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | mvnsite | 0m 9s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| -1 :x: | shadedclient | 6m 10s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 29s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -1 :x: | mvnsite | 0m 24s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | shadedclient | 1m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 24s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 67m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6526 |
| Optional Tests | dupname asflicense codespell detsecrets hadolint
shellcheck shelldocs mvnsite unit |
| uname | Linux 4f263dea2f4f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b0f9f311af3deb8273bc60238af2b4d620fd7ce8 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/testReport/ |
| Max. process+thread count | 56 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Improve Protocol Buffers install instructions.
> --
>
> Key: HADOOP-19065
> URL: https://issues.apache.org/jira/browse/HADOOP-19065
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: huangzhaobo
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526:
URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957424893
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 52m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 4m 31s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | mvnsite | 0m 9s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| -1 :x: | shadedclient | 6m 10s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 29s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -1 :x: | mvnsite | 0m 24s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | shadedclient | 1m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 24s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 67m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6526 |
| Optional Tests | dupname asflicense codespell detsecrets hadolint
shellcheck shelldocs mvnsite unit |
| uname | Linux 4f263dea2f4f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b0f9f311af3deb8273bc60238af2b4d620fd7ce8 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/testReport/ |
| Max. process+thread count | 56 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19083) hadoop binary tarball to exclude aws v2 sdk
Steve Loughran created HADOOP-19083: --- Summary: hadoop binary tarball to exclude aws v2 sdk Key: HADOOP-19083 URL: https://issues.apache.org/jira/browse/HADOOP-19083 Project: Hadoop Common Issue Type: Sub-task Components: build, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default. This SDK brings the total size of the distribution to about 1 GB. Proposed * add a profile to include the aws sdk in the dist module * disable it by default Instead we document which version is needed. The hadoop-aws and hadoop-cloud storage maven artifacts will declare their dependencies, so apps building with those get to do the download. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19071) Update maven-surefire-plugin from 3.0.0 to 3.2.5
[ https://issues.apache.org/jira/browse/HADOOP-19071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819336#comment-17819336 ] ASF GitHub Bot commented on HADOOP-19071: - shahrs87 commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1957394370 Thank you @steveloughran for your response. > Rather than say "hey, let's revert", I'd propose a "surefire update triggers test failures" and see what can be done about addressing them. because we can't stay frozen with surefire versions. I didn't mean to sound rude. Apologies if you felt that way. I understand that when upgrading maven, surefire or protobuf's version, it is very reasonable to see test failures. But in this case, the jenkins build is not running all tests, some of the tests are failing and some of the builds are getting stuck for 20-24 hours and is affecting all the developers. I spent atleast 2-3 days debugging the test failures assuming that this is caused by my patch before posting a [message](https://the-asf.slack.com/archives/CAD7C52Q3/p1708118456027809) on the slack channel and then @ayushtkn helped me. I am sure other developers have followed the same path. What I am suggesting is: Lets revert this change, unblock all developers and then we raise a new PR with the proposed surefire version change, make sure all the tests pass, make test improvements if needed and then push the change to trunk branch. WDYT? Cc @ayushtkn > Update maven-surefire-plugin from 3.0.0 to 3.2.5 > - > > Key: HADOOP-19071 > URL: https://issues.apache.org/jira/browse/HADOOP-19071 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, common >Affects Versions: 3.4.0, 3.5.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. [hadoop]
shahrs87 commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1957394370 Thank you @steveloughran for your response. > Rather than say "hey, let's revert", I'd propose a "surefire update triggers test failures" and see what can be done about addressing them. because we can't stay frozen with surefire versions. I didn't mean to sound rude. Apologies if you felt that way. I understand that when upgrading maven, surefire or protobuf's version, it is very reasonable to see test failures. But in this case, the jenkins build is not running all tests, some of the tests are failing and some of the builds are getting stuck for 20-24 hours and is affecting all the developers. I spent atleast 2-3 days debugging the test failures assuming that this is caused by my patch before posting a [message](https://the-asf.slack.com/archives/CAD7C52Q3/p1708118456027809) on the slack channel and then @ayushtkn helped me. I am sure other developers have followed the same path. What I am suggesting is: Lets revert this change, unblock all developers and then we raise a new PR with the proposed surefire version change, make sure all the tests pass, make test improvements if needed and then push the change to trunk branch. WDYT? Cc @ayushtkn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819335#comment-17819335 ] ASF GitHub Bot commented on HADOOP-18708: - steveloughran commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1957376970 @ahmarsuhail can you rebase this and we can merge in to branch-3.4. thanks > AWS SDK V2 - Implement CSE > -- > > Key: HADOOP-18708 > URL: https://issues.apache.org/jira/browse/HADOOP-18708 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > S3 Encryption client for SDK V2 is now available, so add client side > encryption back in. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19050) Add S3 Access Grants Support in S3A
[ https://issues.apache.org/jira/browse/HADOOP-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819334#comment-17819334 ] ASF GitHub Bot commented on HADOOP-19050: - steveloughran commented on PR #6544: URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1957375191 really focused purely on 3.4.0 shipping right now, not looking at stuff it doesn't need. sorry > Add S3 Access Grants Support in S3A > --- > > Key: HADOOP-19050 > URL: https://issues.apache.org/jira/browse/HADOOP-19050 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Jason Han >Assignee: Jason Han >Priority: Minor > Labels: pull-request-available > > Add support for S3 Access Grants > (https://aws.amazon.com/s3/features/access-grants/) in S3A. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]
steveloughran commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1957376970 @ahmarsuhail can you rebase this and we can merge in to branch-3.4. thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19050. Add S3 Access Grants Support in S3A [hadoop]
steveloughran commented on PR #6544: URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1957375191 really focused purely on 3.4.0 shipping right now, not looking at stuff it doesn't need. sorry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819318#comment-17819318 ] ASF GitHub Bot commented on HADOOP-19065: - hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957214381 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/console in case of problems. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957214381 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957114687 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 13m 2s | | Docker failed to build run-specific yetus/hadoop:tp-26055}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6526 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/3/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19082) S3A: Update AWS SDK V2 to 2.24.1
[
https://issues.apache.org/jira/browse/HADOOP-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819304#comment-17819304
]
ASF GitHub Bot commented on HADOOP-19082:
-
hadoop-yetus commented on PR #6568:
URL: https://github.com/apache/hadoop/pull/6568#issuecomment-1957074319
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 80m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 36m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 122m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6568 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 187f99e9288d 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6bde82e57031bb376c7c16136eb6c82597e507d6 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/testReport/ |
| Max. process+thread count | 702 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> S3A: Update AWS SDK V2 to 2.24.1
>
>
> Key: HADOOP-19082
> URL: https://issues.apache.org/jira/browse/HADOOP-19082
> Project: Hadoop Common
Re: [PR] HADOOP-19082: Update AWS SDK V2 to 2.24.1 [hadoop]
hadoop-yetus commented on PR #6568:
URL: https://github.com/apache/hadoop/pull/6568#issuecomment-1957074319
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 80m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 36m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 122m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6568 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 187f99e9288d 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6bde82e57031bb376c7c16136eb6c82597e507d6 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/testReport/ |
| Max. process+thread count | 702 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional c
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819302#comment-17819302 ] ASF GitHub Bot commented on HADOOP-19065: - hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957048566 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/3/console in case of problems. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957048566 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/3/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819301#comment-17819301 ] ASF GitHub Bot commented on HADOOP-19065: - huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957044724 @tasanuma Have completed, Whether or not need to modify the PR title? > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957044724 @tasanuma Have completed, Whether or not need to modify the PR title? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[
https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819300#comment-17819300
]
ASF GitHub Bot commented on HADOOP-19065:
-
hadoop-yetus commented on PR #6526:
URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957037784
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 41m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shadedclient | 34m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 82m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6526 |
| Optional Tests | dupname asflicense codespell detsecrets |
| uname | Linux 4adb15f13af3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f470b40885bbefc647bb019f6fce8d659898af94 |
| Max. process+thread count | 712 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Improve Protocol Buffers install instructions.
> --
>
> Key: HADOOP-19065
> URL: https://issues.apache.org/jira/browse/HADOOP-19065
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: huangzhaobo
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
hadoop-yetus commented on PR #6526:
URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1957037784
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 41m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shadedclient | 34m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 82m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6526 |
| Optional Tests | dupname asflicense codespell detsecrets |
| uname | Linux 4adb15f13af3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f470b40885bbefc647bb019f6fce8d659898af94 |
| Max. process+thread count | 712 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956867368 @tasanuma Thanks for the reminder. I will complete it after. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819285#comment-17819285 ] ASF GitHub Bot commented on HADOOP-19065: - huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956867368 @tasanuma Thanks for the reminder. I will complete it after. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819282#comment-17819282 ] ASF GitHub Bot commented on HADOOP-19065: - tasanuma commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956849499 @huangzhaobo99 Thanks for updating it, but there are still many mentions of Protocol Buffers 3.7.1 in BUILDING.txt. If we want to update all them to version 3.21.12, we firstly need to update the Dockerfile and install-protobuf.sh to install protobuf 3.21.12, and we must make sure that Hadoop can still be built successfully. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
tasanuma commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956849499 @huangzhaobo99 Thanks for updating it, but there are still many mentions of Protocol Buffers 3.7.1 in BUILDING.txt. If we want to update all them to version 3.21.12, we firstly need to update the Dockerfile and install-protobuf.sh to install protobuf 3.21.12, and we must make sure that Hadoop can still be built successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819277#comment-17819277 ] ASF GitHub Bot commented on HADOOP-19065: - huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956774830 > Although I'm not an expert in curl, I believe this change is fine as it aligns with the instruction for CMake. However, in the first place, protobuf 3.7.1 could be outdated. We have started using protobuf 3.21 by [HADOOP-19069](https://issues.apache.org/jira/browse/HADOOP-19069). @tasanuma Thanks for your reply and review. After the updated version was tested locally, it can be downloaded normally. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
huangzhaobo99 commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956774830 > Although I'm not an expert in curl, I believe this change is fine as it aligns with the instruction for CMake. However, in the first place, protobuf 3.7.1 could be outdated. We have started using protobuf 3.21 by [HADOOP-19069](https://issues.apache.org/jira/browse/HADOOP-19069). @tasanuma Thanks for your reply and review. After the updated version was tested locally, it can be downloaded normally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19082) Update AWS SDK V2 to 2.24.1
[ https://issues.apache.org/jira/browse/HADOOP-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819272#comment-17819272 ] ASF GitHub Bot commented on HADOOP-19082: - steveloughran commented on PR #6568: URL: https://github.com/apache/hadoop/pull/6568#issuecomment-1956747625 * you will need to update LICENSE-binary * and run the hadoop-aws tests -where? * and run the "qualifying an SDK update" steps -where and what problems surfaced? > Update AWS SDK V2 to 2.24.1 > --- > > Key: HADOOP-19082 > URL: https://issues.apache.org/jira/browse/HADOOP-19082 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > > Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging > w.r.t. imds module. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19082) S3A: Update AWS SDK V2 to 2.24.1
[ https://issues.apache.org/jira/browse/HADOOP-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-19082: Summary: S3A: Update AWS SDK V2 to 2.24.1 (was: Update AWS SDK V2 to 2.24.1) > S3A: Update AWS SDK V2 to 2.24.1 > > > Key: HADOOP-19082 > URL: https://issues.apache.org/jira/browse/HADOOP-19082 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > > Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging > w.r.t. imds module. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19082: Update AWS SDK V2 to 2.24.1 [hadoop]
steveloughran commented on PR #6568: URL: https://github.com/apache/hadoop/pull/6568#issuecomment-1956747625 * you will need to update LICENSE-binary * and run the hadoop-aws tests -where? * and run the "qualifying an SDK update" steps -where and what problems surfaced? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19065) Improve Protocol Buffers install instructions.
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819270#comment-17819270 ] ASF GitHub Bot commented on HADOOP-19065: - tasanuma commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956740552 Although I'm not an expert in curl, I believe this change is fine as it aligns with the instruction for CMake. However, in the first place, protobuf 3.7.1 could be outdated. We have started using protobuf 3.21 by HADOOP-19069. > Improve Protocol Buffers install instructions. > -- > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19065. Improve Protocol Buffers install instructions. [hadoop]
tasanuma commented on PR #6526: URL: https://github.com/apache/hadoop/pull/6526#issuecomment-1956740552 Although I'm not an expert in curl, I believe this change is fine as it aligns with the instruction for CMake. However, in the first place, protobuf 3.7.1 could be outdated. We have started using protobuf 3.21 by HADOOP-19069. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19082) Update AWS SDK V2 to 2.24.1
[ https://issues.apache.org/jira/browse/HADOOP-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19082: Labels: pull-request-available (was: ) > Update AWS SDK V2 to 2.24.1 > --- > > Key: HADOOP-19082 > URL: https://issues.apache.org/jira/browse/HADOOP-19082 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > > Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging > w.r.t. imds module. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19082) Update AWS SDK V2 to 2.24.1
[ https://issues.apache.org/jira/browse/HADOOP-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819266#comment-17819266 ] ASF GitHub Bot commented on HADOOP-19082: - HarshitGupta11 opened a new pull request, #6568: URL: https://github.com/apache/hadoop/pull/6568 Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module. ### How was this patch tested? Integration Test using ```mvn clean verify -Dparallel-tests -DtestsThreadCount=16 -Dscale -fae``` ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Update AWS SDK V2 to 2.24.1 > --- > > Key: HADOOP-19082 > URL: https://issues.apache.org/jira/browse/HADOOP-19082 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > > Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging > w.r.t. imds module. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-19082: Update AWS SDK V2 to 2.24.1 [hadoop]
HarshitGupta11 opened a new pull request, #6568: URL: https://github.com/apache/hadoop/pull/6568 Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module. ### How was this patch tested? Integration Test using ```mvn clean verify -Dparallel-tests -DtestsThreadCount=16 -Dscale -fae``` ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569
[
https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819229#comment-17819229
]
Steve Loughran commented on HADOOP-18197:
-
saw this in the context of surefire upgrade #6537.
java.lang.NoSuchMethodError:
java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; is a recurrent nightmare;
it's a change in the ByteBuffer classic which added an overloaded position()
method in java9 *which was then backported to java8*
some java8 jvms are happy, some fail. and if you compile with a newer JVM with
the change (openjdk, corretto) you generate code which doesn't work everywhere.
fixable in code with a cast, but you need to edit the code
{code}
((Buffer)buffer).position()l
{code}
> Update protobuf 3.7.1 to a version without CVE-2021-22569
> -
>
> Key: HADOOP-18197
> URL: https://issues.apache.org/jira/browse/HADOOP-18197
> Project: Hadoop Common
> Issue Type: Improvement
> Components: hadoop-thirdparty
>Affects Versions: thirdparty-1.2.0
>Reporter: Ivan Viaznikov
>Assignee: PJ Fanning
>Priority: Major
> Labels: pull-request-available, security
> Fix For: thirdparty-1.2.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The artifact `org.apache.hadoop:hadoop-common` brings in a dependency
> `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version
> released in 2013 and it contains a vulnerability
> [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569].
> Therefore, requesting you to clarify if this library version is going to be
> updated in the following releases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. [hadoop]
ayushtkn commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1956562170 & some are failing due to protobuf upgrade as part of hadoop thirdparty upgrade, I am checking all the HDFS tests, but little slowly https://issues.apache.org/jira/browse/HADOOP-18197?focusedCommentId=17818651&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17818651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19071) Update maven-surefire-plugin from 3.0.0 to 3.2.5
[ https://issues.apache.org/jira/browse/HADOOP-19071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819220#comment-17819220 ] ASF GitHub Bot commented on HADOOP-19071: - ayushtkn commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1956558036 Some tests are getting skipped as well as I mentioned on the jira https://issues.apache.org/jira/browse/HADOOP-19071?focusedCommentId=17818734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17818734 TestDirectoryScanner is a known flaky that ain’t related there is already a ticket somewhere > Update maven-surefire-plugin from 3.0.0 to 3.2.5 > - > > Key: HADOOP-19071 > URL: https://issues.apache.org/jira/browse/HADOOP-19071 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, common >Affects Versions: 3.4.0, 3.5.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. [hadoop]
ayushtkn commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1956558036 Some tests are getting skipped as well as I mentioned on the jira https://issues.apache.org/jira/browse/HADOOP-19071?focusedCommentId=17818734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17818734 TestDirectoryScanner is a known flaky that ain’t related there is already a ticket somewhere -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17352. Add configuration to control whether DN delete this replica from disk when client requests a missing block [hadoop]
tomscut commented on code in PR #6559:
URL: https://github.com/apache/hadoop/pull/6559#discussion_r1497454155
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##
@@ -188,6 +188,10 @@ public class DFSConfigKeys extends CommonConfigurationKeys
{
public static final long DFS_DN_CACHED_DFSUSED_CHECK_INTERVAL_DEFAULT_MS =
60;
+ public static final String
DFS_DATANODE_DELETE_CORRUPT_REPLICA_FROM_DISK_ENABLE =
Review Comment:
It would be handy if this could be configured dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19071) Update maven-surefire-plugin from 3.0.0 to 3.2.5
[ https://issues.apache.org/jira/browse/HADOOP-19071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819215#comment-17819215 ] ASF GitHub Bot commented on HADOOP-19071: - steveloughran commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1956544980 This is not good. But looking at the failures I don't know whether to categorise as "test runner regression" or "brittle tests failing under new test runner". Here are some of the ones I've looked at `TestDirectoryScanner.testThrottling` This test is measuring how long things took. it is way too brittle against timing changes, both slower and faster. ``` java.lang.AssertionError: Throttle is too permissive at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.testThrottling(TestDirectoryScanner.java:901) ``` I think the step here is to move to assertj so asserts fail with meaningful messages, see if the failure can be understood. Ideally you'd want a test which doesn't measure elapsed time, but instead uses counters in the code (here: of throttle events) to assert what took place. Test` TestBlockListAsLongs.testFuzz` See this painfully often else where -it means that the protobuf lib was built with a more recent version of java8 than the early oracle ones. Its fixable in your own build (use the older one) or cast ByteBuffer to Buffer. otherwise we need to make sure tests are on a more recent build. ``` java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; at org.apache.hadoop.thirdparty.protobuf.IterableByteBufferInputStream.read(IterableByteBufferInputStream.java:143) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.read(CodedInputStream.java:2080) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.tryRefillBuffer(CodedInputStream.java:2831) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2777) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawByte(CodedInputStream.java:2859) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64SlowPath(CodedInputStream.java:2648) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64(CodedInputStream.java:2641) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readSInt64(CodedInputStream.java:2497) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:419) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:397) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:375) at org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs.checkReport(TestBlockListAsLongs.java:156) at org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs.testFuzz(TestBlockListAsLongs.java:139) ``` test `TestDFSAdmin.testDecommissionDataNodesReconfig` ``` java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at org.apache.hadoop.hdfs.tools.TestDFSAdmin.testDecommissionDataNodesReconfig(TestDFSAdmin.java:1356) ``` not a very meaningful message. suspect that a different ordering of the threads is causing the assert to fail. 1. move to AssertJ 2. analyse error, see what the fix is. Test `TestCacheDirectives`. ``` at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:403) at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:362) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.waitForCachedBlocks(TestCacheDirectives.java:760) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.teardown(TestCacheDirectives.java:173) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ``` this is a timeout during teardown; after this subsequent tests are possibly going to fail. No obvious cause, though again I'd suspect race conditions. Rather than say "hey, let's revert", I'd propose a "surefire update triggers test failures" and see what can be done about addressing them. because we can't stay frozen with surefire versions. > Update maven-surefire-plugin from 3.0.0 to 3.2.5 > - > > Key: HADOOP-19071 > URL: https://issues.apache.org/jira/browse/HADOOP-19071 >
Re: [PR] HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. [hadoop]
steveloughran commented on PR #6537: URL: https://github.com/apache/hadoop/pull/6537#issuecomment-1956544980 This is not good. But looking at the failures I don't know whether to categorise as "test runner regression" or "brittle tests failing under new test runner". Here are some of the ones I've looked at `TestDirectoryScanner.testThrottling` This test is measuring how long things took. it is way too brittle against timing changes, both slower and faster. ``` java.lang.AssertionError: Throttle is too permissive at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.testThrottling(TestDirectoryScanner.java:901) ``` I think the step here is to move to assertj so asserts fail with meaningful messages, see if the failure can be understood. Ideally you'd want a test which doesn't measure elapsed time, but instead uses counters in the code (here: of throttle events) to assert what took place. Test` TestBlockListAsLongs.testFuzz` See this painfully often else where -it means that the protobuf lib was built with a more recent version of java8 than the early oracle ones. Its fixable in your own build (use the older one) or cast ByteBuffer to Buffer. otherwise we need to make sure tests are on a more recent build. ``` java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; at org.apache.hadoop.thirdparty.protobuf.IterableByteBufferInputStream.read(IterableByteBufferInputStream.java:143) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.read(CodedInputStream.java:2080) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.tryRefillBuffer(CodedInputStream.java:2831) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2777) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawByte(CodedInputStream.java:2859) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64SlowPath(CodedInputStream.java:2648) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64(CodedInputStream.java:2641) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readSInt64(CodedInputStream.java:2497) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:419) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:397) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:375) at org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs.checkReport(TestBlockListAsLongs.java:156) at org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs.testFuzz(TestBlockListAsLongs.java:139) ``` test `TestDFSAdmin.testDecommissionDataNodesReconfig` ``` java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at org.apache.hadoop.hdfs.tools.TestDFSAdmin.testDecommissionDataNodesReconfig(TestDFSAdmin.java:1356) ``` not a very meaningful message. suspect that a different ordering of the threads is causing the assert to fail. 1. move to AssertJ 2. analyse error, see what the fix is. Test `TestCacheDirectives`. ``` at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:403) at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:362) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.waitForCachedBlocks(TestCacheDirectives.java:760) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.teardown(TestCacheDirectives.java:173) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ``` this is a timeout during teardown; after this subsequent tests are possibly going to fail. No obvious cause, though again I'd suspect race conditions. Rather than say "hey, let's revert", I'd propose a "surefire update triggers test failures" and see what can be done about addressing them. because we can't stay frozen with surefire versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-ma
[jira] [Commented] (HADOOP-18516) [ABFS]: Support fixed SAS token config in addition to SAS Token Provider class
[
https://issues.apache.org/jira/browse/HADOOP-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819148#comment-17819148
]
ASF GitHub Bot commented on HADOOP-18516:
-
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497042989
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,33 +941,66 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The user can choose between a configured fixed sas token, and a user
+ * implementation of the SASTokenProvider interface. Preference will be given
+ * to SASTokenProvider class provided as the value of
"fs.azure.sas.token.provider.type".
+ * If above config is not set, it is expected that user wants to use a
+ * fixed SAS Token provided as value of "fs.azure.sas.fixed.token".
+ *
+ * If both the configs are not provided,
+ * initialization fails and {@link TokenAccessProviderException} is
thrown.
+ * If both are present, SASTokenProvider class will be used to
generate SAS Token.
+ * If only fixed SAS Token is configured, this will return null
+ * and Fixed SAS token will be used to sign requests.
+ *
+ * Avoid using a tokenProvider implementation just to read the configured
fixed token,
+ * as this could create confusion. Also,implementing the SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend
on the
+ * AbfsConfiguration with which a filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object.
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
Review Comment:
Setting configurations is upto users and it will be difficult to stop them
from setting both.
We have clear documentation in abfs,md file as well as javadocs that
SASTokenProvider will have preference.
> [ABFS]: Support fixed SAS token config in addition to SAS Token Provider class
> --
>
> Key: HADOOP-18516
> URL: https://issues.apache.org/jira/browse/HADOOP-18516
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.3.4
>Reporter: Sree Bhattacharyya
>Assignee: Anuj Modi
>Priority: Minor
> Labels: pull-request-available
>
> Introduce a new configuration for setting the fixed account/service SAS token
> in ABFS driver. This will be in addition to the implementations of the
> SASTokenProvider interface that can be used for obtaining a SAS Token from
> the user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497042989
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,33 +941,66 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The user can choose between a configured fixed sas token, and a user
+ * implementation of the SASTokenProvider interface. Preference will be given
+ * to SASTokenProvider class provided as the value of
"fs.azure.sas.token.provider.type".
+ * If above config is not set, it is expected that user wants to use a
+ * fixed SAS Token provided as value of "fs.azure.sas.fixed.token".
+ *
+ * If both the configs are not provided,
+ * initialization fails and {@link TokenAccessProviderException} is
thrown.
+ * If both are present, SASTokenProvider class will be used to
generate SAS Token.
+ * If only fixed SAS Token is configured, this will return null
+ * and Fixed SAS token will be used to sign requests.
+ *
+ * Avoid using a tokenProvider implementation just to read the configured
fixed token,
+ * as this could create confusion. Also,implementing the SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend
on the
+ * AbfsConfiguration with which a filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object.
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
Review Comment:
Setting configurations is upto users and it will be difficult to stop them
from setting both.
We have clear documentation in abfs,md file as well as javadocs that
SASTokenProvider will have preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18516) [ABFS]: Support fixed SAS token config in addition to SAS Token Provider class
[ https://issues.apache.org/jira/browse/HADOOP-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819137#comment-17819137 ] ASF GitHub Bot commented on HADOOP-18516: - saxenapranav commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497115703 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java: ## @@ -332,6 +335,8 @@ public AbfsRestOperation setFilesystemProperties(final String properties, final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder(); abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM); +appendSASTokenToQuery(ROOT_PATH, "", abfsUriQueryBuilder); Review Comment: `appendSASTokenToQuery` was not there before this patch. Is this added for fixed sas token only. If yes, then we would have to add if-condition to keep only fixedSasToken (if in config) applying to the container APIs, and in non-fixedSasToken case, `getSASToken` should not be called. Reason being, developers would already have their implementations and those might not be adaptable to the container APIs. > [ABFS]: Support fixed SAS token config in addition to SAS Token Provider class > -- > > Key: HADOOP-18516 > URL: https://issues.apache.org/jira/browse/HADOOP-18516 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.4 >Reporter: Sree Bhattacharyya >Assignee: Anuj Modi >Priority: Minor > Labels: pull-request-available > > Introduce a new configuration for setting the fixed account/service SAS token > in ABFS driver. This will be in addition to the implementations of the > SASTokenProvider interface that can be used for obtaining a SAS Token from > the user. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
saxenapranav commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497115703 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java: ## @@ -332,6 +335,8 @@ public AbfsRestOperation setFilesystemProperties(final String properties, final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder(); abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM); +appendSASTokenToQuery(ROOT_PATH, "", abfsUriQueryBuilder); Review Comment: `appendSASTokenToQuery` was not there before this patch. Is this added for fixed sas token only. If yes, then we would have to add if-condition to keep only fixedSasToken (if in config) applying to the container APIs, and in non-fixedSasToken case, `getSASToken` should not be called. Reason being, developers would already have their implementations and those might not be adaptable to the container APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17393. [FGL] Remove unused `cond` in FSNamesystem [hadoop]
ZanderXu opened a new pull request, #6567: URL: https://github.com/apache/hadoop/pull/6567 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
