[jira] [Commented] (HADOOP-19044) AWS SDK V2 - Update S3A region logic
[ https://issues.apache.org/jira/browse/HADOOP-19044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809275#comment-17809275 ] ASF GitHub Bot commented on HADOOP-19044: - virajjasani commented on PR #6479: URL: https://github.com/apache/hadoop/pull/6479#issuecomment-1903417294 Several tests failing, cause is being discussed on HADOOP-18975 > AWS SDK V2 - Update S3A region logic > - > > Key: HADOOP-19044 > URL: https://issues.apache.org/jira/browse/HADOOP-19044 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set > fs.s3a.endpoint to > s3.amazonaws.com here: > [https://github.com/apache/spark/blob/9a2f39318e3af8b3817dc5e4baf52e548d82063c/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L540] > > > HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is > set, or if a region can be parsed from fs.s3a.endpoint (which will happen in > this case, region will be US_EAST_1), cross region access is not enabled. > This will cause 400 errors if the bucket is not in US_EAST_1. > > Proposed: Updated the logic so that if the endpoint is the global > s3.amazonaws.com , cross region access is enabled. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19044. AWS SDK V2 - Update S3A region logic [hadoop]
virajjasani commented on PR #6479: URL: https://github.com/apache/hadoop/pull/6479#issuecomment-1903417294 Several tests failing, cause is being discussed on HADOOP-18975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17342. Fix DataNode may invalidates normal block causing missing block [hadoop]
haiyang1987 commented on code in PR #6464:
URL: https://github.com/apache/hadoop/pull/6464#discussion_r1461445666
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java:
##
@@ -2011,4 +2011,83 @@ public void tesInvalidateMissingBlock() throws Exception
{
cluster.shutdown();
}
}
+
+ @Test
+ public void testCheckFilesWhenInvalidateMissingBlock() throws Exception {
+long blockSize = 1024;
+int heartbeatInterval = 1;
+HdfsConfiguration c = new HdfsConfiguration();
+c.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, heartbeatInterval);
+c.setLong(DFS_BLOCK_SIZE_KEY, blockSize);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(c).
+numDataNodes(1).build();
+DataNodeFaultInjector oldDnInjector = DataNodeFaultInjector.get();
+try {
+ cluster.waitActive();
+ BlockReaderTestUtil util = new BlockReaderTestUtil(cluster, new
+ HdfsConfiguration(conf));
+ Path path = new Path("/testFile");
+ util.writeFile(path, 1);
+ String bpid = cluster.getNameNode().getNamesystem().getBlockPoolId();
+ DataNode dn = cluster.getDataNodes().get(0);
+ FsDatasetImpl dnFSDataset = (FsDatasetImpl) dn.getFSDataset();
+ List replicaInfos = dnFSDataset.getFinalizedBlocks(bpid);
+ assertEquals(1, replicaInfos.size());
+ DFSTestUtil.readFile(cluster.getFileSystem(), path);
+ LocatedBlock blk = util.getFileBlocks(path, 512).get(0);
+ ExtendedBlock block = blk.getBlock();
+
+ // Append a new block with an incremented generation stamp.
+ long newGS = block.getGenerationStamp() + 1;
+ dnFSDataset.append(block, newGS, 1024);
+ block.setGenerationStamp(newGS);
+
+ DataNodeFaultInjector injector = new DataNodeFaultInjector() {
+@Override
+public void delayGetMetaDataInputStream() {
+ try {
+Thread.sleep(8000);
+ } catch (InterruptedException e) {
+// Ignore exception.
+ }
+}
+ };
+ // Delay to getMetaDataInputStream.
+ DataNodeFaultInjector.set(injector);
+
+ ExecutorService executorService = Executors.newFixedThreadPool(2);
+ try {
+Future blockReaderFuture = executorService.submit(() -> {
+ try {
+// Submit tasks for reading block.
+BlockReaderTestUtil.getBlockReader(cluster.getFileSystem(), blk,
0, 512);
+ } catch (IOException e) {
+// Ignore exception.
+ }
+});
+
+Future finalizeBlockFuture = executorService.submit(() -> {
+ try {
+// Submit tasks for finalizing block.
+Thread.sleep(1000);
+dnFSDataset.finalizeBlock(block, false);
+ } catch (Exception e) {
+// Ignore exception
+ }
+});
+
+// Wait for both tasks to complete.
+blockReaderFuture.get();
+finalizeBlockFuture.get();
+ } finally {
+executorService.shutdown();
+ }
+
+ // Validate the replica is exits.
+ assertNotNull(dnFSDataset.getReplicaInfo(blk.getBlock()));
Review Comment:
If it is to verify whether the UT can reproduce FNE, we can add the
following code for verification
```
GenericTestUtils.LogCapturer logCapturer =
GenericTestUtils.LogCapturer.captureLogs(DataNode.LOG);
ReplicaInfo tmpReplicaInfo = dnFSDataset.getReplicaInfo(blk.getBlock());
// Check DN log for FileNotFoundException.
String expectedMsg = String.format("opReadBlock %s received exception " +
"java.io.FileNotFoundException: %s (No such file or
directory)",
blk.getBlock(), tmpReplicaInfo.getMetadataURI().getPath());
assertTrue("Expected log message not found in DN log.",
logCapturer.getOutput().contains(expectedMsg));
```
How about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] YARN-11649. [Federation] Create Override for Maximum Resource Capability in getNewApplication [hadoop]
jchanggg opened a new pull request, #6481: URL: https://github.com/apache/hadoop/pull/6481 ### Description of PR When getNewApplication is called against YARN Router with Federation on, its possible we get different maxResourceCapabilities on different calls. This is because getNewApplication is called against a random cluster on each call, which may return different maxResourceCapability based on the cluster that the call is executed on. ### How was this patch tested? Added to interceptor test cases ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18975) AWS SDK v2: extend support for FIPS endpoints
[
https://issues.apache.org/jira/browse/HADOOP-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809271#comment-17809271
]
Viraj Jasani edited comment on HADOOP-18975 at 1/22/24 7:33 AM:
{code:java}
fs.s3a.bucket.landsat-pds.endpoint.fips
true
Use the fips endpoint
{code}
[~ste...@apache.org] [~ahmar] do we really need fips enabled for landsat in
hadoop-tools/hadoop-aws/src/test/resources/core-site.xml ?
This is breaking several tests from full suite that i am running against
us-west-2 for PR [https://github.com/apache/hadoop/pull/6479]
e.g.
{code:java}
[ERROR]
testSelectOddRecordsIgnoreHeaderV1(org.apache.hadoop.fs.s3a.select.ITestS3Select)
Time elapsed: 2.917 s <<< ERROR!
java.lang.IllegalArgumentException: An endpoint cannot set when
fs.s3a.endpoint.fips is true : https://s3-us-west-2.amazonaws.com
at
org.apache.hadoop.util.Preconditions.checkArgument(Preconditions.java:213)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureEndpointAndRegion(DefaultS3ClientFactory.java:292)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:179)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:126)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:1063)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:677)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:171)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3702)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3653)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:555)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
at
org.apache.hadoop.fs.s3a.select.AbstractS3SelectTest.setup(AbstractS3SelectTest.java:304)
at
org.apache.hadoop.fs.s3a.select.ITestS3Select.setup(ITestS3Select.java:112)
{code}
[ERROR] Tests run: 1264, Failures: 4, Errors: 87, Skipped: 164
was (Author: vjasani):
{code:java}
fs.s3a.bucket.landsat-pds.endpoint.fips
true
Use the fips endpoint
{code}
[~ste...@apache.org] [~ahmar] do we really need fips enabled for landsat in
hadoop-tools/hadoop-aws/src/test/resources/core-site.xml ?
This is breaking several tests from full suite that i am running against
us-west-2 for PR [https://github.com/apache/hadoop/pull/6479]
e.g.
{code:java}
[ERROR]
testSelectOddRecordsIgnoreHeaderV1(org.apache.hadoop.fs.s3a.select.ITestS3Select)
Time elapsed: 2.917 s <<< ERROR!
java.lang.IllegalArgumentException: An endpoint cannot set when
fs.s3a.endpoint.fips is true : https://s3-us-west-2.amazonaws.com
at
org.apache.hadoop.util.Preconditions.checkArgument(Preconditions.java:213)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureEndpointAndRegion(DefaultS3ClientFactory.java:292)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:179)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:126)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:1063)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:677)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:171)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3702)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3653)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:555)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
at
org.apache.hadoop.fs.s3a.select.AbstractS3SelectTest.setup(AbstractS3SelectTest.java:304)
at
org.apache.hadoop.fs.s3a.select.ITestS3Select.setup(ITestS3Select.java:112)
{code}
> AWS SDK v2: extend support for FIPS endpoints
> --
>
> Key: HADOOP-18975
> URL: https://issues.apache.org/jira/browse/HADOOP-18975
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> v1 SDK supported FIPS just by changing the endpoint.
> Now we have a new builder setting to use.
> * add new fs.s3a.endpoint.fips option
> * pass it down
> * test
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HADOOP-18975) AWS SDK v2: extend support for FIPS endpoints
[
https://issues.apache.org/jira/browse/HADOOP-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809271#comment-17809271
]
Viraj Jasani commented on HADOOP-18975:
---
{code:java}
fs.s3a.bucket.landsat-pds.endpoint.fips
true
Use the fips endpoint
{code}
[~ste...@apache.org] [~ahmar] do we really need fips enabled for landsat in
hadoop-tools/hadoop-aws/src/test/resources/core-site.xml ?
This is breaking several tests from full suite that i am running against
us-west-2 for PR [https://github.com/apache/hadoop/pull/6479]
e.g.
{code:java}
[ERROR]
testSelectOddRecordsIgnoreHeaderV1(org.apache.hadoop.fs.s3a.select.ITestS3Select)
Time elapsed: 2.917 s <<< ERROR!
java.lang.IllegalArgumentException: An endpoint cannot set when
fs.s3a.endpoint.fips is true : https://s3-us-west-2.amazonaws.com
at
org.apache.hadoop.util.Preconditions.checkArgument(Preconditions.java:213)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureEndpointAndRegion(DefaultS3ClientFactory.java:292)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:179)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:126)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:1063)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:677)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:171)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3702)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3653)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:555)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
at
org.apache.hadoop.fs.s3a.select.AbstractS3SelectTest.setup(AbstractS3SelectTest.java:304)
at
org.apache.hadoop.fs.s3a.select.ITestS3Select.setup(ITestS3Select.java:112)
{code}
> AWS SDK v2: extend support for FIPS endpoints
> --
>
> Key: HADOOP-18975
> URL: https://issues.apache.org/jira/browse/HADOOP-18975
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> v1 SDK supported FIPS just by changing the endpoint.
> Now we have a new builder setting to use.
> * add new fs.s3a.endpoint.fips option
> * pass it down
> * test
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17342. Fix DataNode may invalidates normal block causing missing block [hadoop]
haiyang1987 commented on PR #6464: URL: https://github.com/apache/hadoop/pull/6464#issuecomment-1903406926 > > This is a bug fix after #5564 , do you have time to help review this? > > @smarthanwang I have a question about [HDFS-16985](https://issues.apache.org/jira/browse/HDFS-16985), Normally FileNotFoundException means that the meta file or data file maybe lost, so the replication on this datanode maybe corrupt, right? In your business(AWS EC2 + EBS) situation, you don't expect datanode to delete this replica directly, so [HDFS-16985](https://issues.apache.org/jira/browse/HDFS-16985) just remove the replica from the memory of DN. > > But I want to see that DN should directly delete this corrupt replica If it can ensure that the replica is corrupt, such as: meta file or data file is lost. So we can add a configure to control whether DN delete this replication from disk directly, such as: fs.datanode.delete.corrupt.replica.from.disk with a default value true. > > If `fs.datanode.delete.corrupt.replica.from.disk` is true, DN can delete this corrupt replica from disk directly. If `fs.datanode.delete.corrupt.replica.from.disk` is false, DN can just delete this corrupt replica from memory. > > @smarthanwang @zhangshuyan0 looking forward to your good ideas. Thanks @ZanderXu for your comment. I agree with add new param to control whether this scenario requires deleting the replica from the disk. from the datanode side, if it is confirmed that the replica is not exists (meta file or data file is lost), it seems maybe reasonable that this replica should also be deleted (residual meta file or data file) from the disk. thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11649. [Federation] Create Override for Maximum Resource Capability in getNewApplication [hadoop]
jchanggg closed pull request #6480: YARN-11649. [Federation] Create Override for Maximum Resource Capability in getNewApplication URL: https://github.com/apache/hadoop/pull/6480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] YARN-11649. [Federation] Create Override for Maximum Resource Capability in getNewApplication [hadoop]
jchanggg opened a new pull request, #6480: URL: https://github.com/apache/hadoop/pull/6480 ### Description of PR JIRA: [YARN-11649](https://issues.apache.org/jira/browse/YARN-11649). [Router] Create Override for Maximum Resource Capability in getNewApplication When getNewApplication is called against YARN Router with Federation on, its possible we get different maxResourceCapabilities on different calls. This is because getNewApplication is called against a random cluster on each call, which may return different maxResourceCapability based on the cluster that the call is executed on. ### How was this patch tested? Via adding test cases ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt [hadoop]
hadoop-yetus commented on PR #6476:
URL: https://github.com/apache/hadoop/pull/6476#issuecomment-1903401143
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 1s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 182m 26s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 269m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6476 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7566da582717 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e8da365c3ce79fd700bc10e5d74095be1120266c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/2/testReport/ |
| Max. process+thread count | 4423 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
--
[jira] [Commented] (HADOOP-19045) S3A: pass request timeouts down to sdk clients
[ https://issues.apache.org/jira/browse/HADOOP-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809263#comment-17809263 ] ASF GitHub Bot commented on HADOOP-19045: - virajjasani commented on PR #6470: URL: https://github.com/apache/hadoop/pull/6470#issuecomment-1903390154 Looks like this can also cover HADOOP-19022 > S3A: pass request timeouts down to sdk clients > -- > > Key: HADOOP-19045 > URL: https://issues.apache.org/jira/browse/HADOOP-19045 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > s3a client timeout settings are getting down to http client, but not sdk > timeouts, so you can't have a longer timeout than the default. This surfaces > in the inability to tune the timeouts for CreateSession calls even now the > latest SDK does pick it up -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19045. S3A: Validate CreateSession Timeout Propagation [hadoop]
virajjasani commented on PR #6470: URL: https://github.com/apache/hadoop/pull/6470#issuecomment-1903390154 Looks like this can also cover HADOOP-19022 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. [hadoop]
hadoop-yetus commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903380159
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 36s | | trunk passed |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 45s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 2m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 50s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 188m 55s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 294m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5ba014e5ba65 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 96da2c85a6d951d1897c0d994f5379e2ec1db9e2 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/14/testReport/ |
| Max. process+thread count | 4223 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6368/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git S
[jira] [Commented] (HADOOP-19044) AWS SDK V2 - Update S3A region logic
[ https://issues.apache.org/jira/browse/HADOOP-19044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809259#comment-17809259 ] ASF GitHub Bot commented on HADOOP-19044: - virajjasani commented on PR #6479: URL: https://github.com/apache/hadoop/pull/6479#issuecomment-1903370772 Testing against `us-west-2` in progress. > AWS SDK V2 - Update S3A region logic > - > > Key: HADOOP-19044 > URL: https://issues.apache.org/jira/browse/HADOOP-19044 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set > fs.s3a.endpoint to > s3.amazonaws.com here: > [https://github.com/apache/spark/blob/9a2f39318e3af8b3817dc5e4baf52e548d82063c/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L540] > > > HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is > set, or if a region can be parsed from fs.s3a.endpoint (which will happen in > this case, region will be US_EAST_1), cross region access is not enabled. > This will cause 400 errors if the bucket is not in US_EAST_1. > > Proposed: Updated the logic so that if the endpoint is the global > s3.amazonaws.com , cross region access is enabled. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19044. AWS SDK V2 - Update S3A region logic [hadoop]
virajjasani commented on PR #6479: URL: https://github.com/apache/hadoop/pull/6479#issuecomment-1903370772 Testing against `us-west-2` in progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19044) AWS SDK V2 - Update S3A region logic
[ https://issues.apache.org/jira/browse/HADOOP-19044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19044: Labels: pull-request-available (was: ) > AWS SDK V2 - Update S3A region logic > - > > Key: HADOOP-19044 > URL: https://issues.apache.org/jira/browse/HADOOP-19044 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set > fs.s3a.endpoint to > s3.amazonaws.com here: > [https://github.com/apache/spark/blob/9a2f39318e3af8b3817dc5e4baf52e548d82063c/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L540] > > > HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is > set, or if a region can be parsed from fs.s3a.endpoint (which will happen in > this case, region will be US_EAST_1), cross region access is not enabled. > This will cause 400 errors if the bucket is not in US_EAST_1. > > Proposed: Updated the logic so that if the endpoint is the global > s3.amazonaws.com , cross region access is enabled. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19044) AWS SDK V2 - Update S3A region logic
[ https://issues.apache.org/jira/browse/HADOOP-19044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809258#comment-17809258 ] ASF GitHub Bot commented on HADOOP-19044: - virajjasani opened a new pull request, #6479: URL: https://github.com/apache/hadoop/pull/6479 Jira: HADOOP-19044 > AWS SDK V2 - Update S3A region logic > - > > Key: HADOOP-19044 > URL: https://issues.apache.org/jira/browse/HADOOP-19044 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > > If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set > fs.s3a.endpoint to > s3.amazonaws.com here: > [https://github.com/apache/spark/blob/9a2f39318e3af8b3817dc5e4baf52e548d82063c/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L540] > > > HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is > set, or if a region can be parsed from fs.s3a.endpoint (which will happen in > this case, region will be US_EAST_1), cross region access is not enabled. > This will cause 400 errors if the bucket is not in US_EAST_1. > > Proposed: Updated the logic so that if the endpoint is the global > s3.amazonaws.com , cross region access is enabled. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-19044. AWS SDK V2 - Update S3A region logic [hadoop]
virajjasani opened a new pull request, #6479: URL: https://github.com/apache/hadoop/pull/6479 Jira: HADOOP-19044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] just for testing. [hadoop]
hadoop-yetus commented on PR #6478:
URL: https://github.com/apache/hadoop/pull/6478#issuecomment-1903344407
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 12s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6478/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 3 new + 24
unchanged - 0 fixed = 27 total (was 24) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 84m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6478/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6478 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ad2a0e64443b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7fa5182c7908db0598eac06649be3a23cf8dad7 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6478/1/testReport/ |
| Max. process+thread count | 697 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6478/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Ap
[jira] [Commented] (HADOOP-19039) Hadoop 3.4.0 Highlight big features and improvements.
[
https://issues.apache.org/jira/browse/HADOOP-19039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809252#comment-17809252
]
ASF GitHub Bot commented on HADOOP-19039:
-
hadoop-yetus commented on PR #6462:
URL: https://github.com/apache/hadoop/pull/6462#issuecomment-1903335855
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 83m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 126m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6462 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets |
| uname | Linux 16105fc906f3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c47dcf73c237e0a4e3ad0a1a2ef58322831d12d |
| Max. process+thread count | 539 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Hadoop 3.4.0 Highlight big features and improvements.
> -
>
> Key: HADOOP-19039
> URL: https://issues.apache.org/jira/browse/HADOOP-19039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion
> of numerous commits in this version. Therefore, highlighting significant
> features and improvements becomes crucial. I've completed the initial
> version and now seek the review of more experienced partner to ensure the
> finalization of the version's highlights.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19039. Hadoop 3.4.0 Highlight big features and improvements. [hadoop]
hadoop-yetus commented on PR #6462:
URL: https://github.com/apache/hadoop/pull/6462#issuecomment-1903335855
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 83m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 126m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6462 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets |
| uname | Linux 16105fc906f3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c47dcf73c237e0a4e3ad0a1a2ef58322831d12d |
| Max. process+thread count | 539 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19039) Hadoop 3.4.0 Highlight big features and improvements.
[
https://issues.apache.org/jira/browse/HADOOP-19039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809247#comment-17809247
]
ASF GitHub Bot commented on HADOOP-19039:
-
hadoop-yetus commented on PR #6462:
URL: https://github.com/apache/hadoop/pull/6462#issuecomment-1903290380
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 50m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 78m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6462 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets |
| uname | Linux ccfc63cea1cc 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c47dcf73c237e0a4e3ad0a1a2ef58322831d12d |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Hadoop 3.4.0 Highlight big features and improvements.
> -
>
> Key: HADOOP-19039
> URL: https://issues.apache.org/jira/browse/HADOOP-19039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion
> of numerous commits in this version. Therefore, highlighting significant
> features and improvements becomes crucial. I've completed the initial
> version and now seek the review of more experienced partner to ensure the
> finalization of the version's highlights.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19039. Hadoop 3.4.0 Highlight big features and improvements. [hadoop]
hadoop-yetus commented on PR #6462:
URL: https://github.com/apache/hadoop/pull/6462#issuecomment-1903290380
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 50m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 78m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6462 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets |
| uname | Linux ccfc63cea1cc 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c47dcf73c237e0a4e3ad0a1a2ef58322831d12d |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6462/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] just for testing. [hadoop]
hfutatzhanghb opened a new pull request, #6478: URL: https://github.com/apache/hadoop/pull/6478 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18045) Disable TestDynamometerInfra
[ https://issues.apache.org/jira/browse/HADOOP-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809238#comment-17809238 ] Takanobu Asanuma commented on HADOOP-18045: --- [~slfan1989] Thanks for updating it! > Disable TestDynamometerInfra > > > Key: HADOOP-18045 > URL: https://issues.apache.org/jira/browse/HADOOP-18045 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This test is broken and there is no fix provided for a long time. Let's > disable the test to reduce the noise in the daily qbt job. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. [hadoop]
hfutatzhanghb commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903178049 > Committed to trunk. Thanks for your contribution @hfutatzhanghb . @zhangshuyan0 Sir, Thanks a lot for your reviewing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] TEST [hadoop]
hfutatzhanghb opened a new pull request, #6477: URL: https://github.com/apache/hadoop/pull/6477 ### Description of PR triiger yetus ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. [hadoop]
zhangshuyan0 commented on PR #6368: URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903143954 Committed to trunk. Thanks for your contribution @hfutatzhanghb . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. [hadoop]
zhangshuyan0 merged PR #6368: URL: https://github.com/apache/hadoop/pull/6368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. [hadoop]
hfutatzhanghb commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903059289
> > Sir, very nice catch. I think below code may resolve the problem you
found. Please take a look when you are free. I will submit another PR to fix it
and Add UT.
> > ```java
> > if (!getStreamer().getAppendChunk()) {
> > int psize = 0;
> > if (blockSize == getStreamer().getBytesCurBlock()) {
> > psize = writePacketSize;
> > } else if (blockSize - getStreamer().getBytesCurBlock() +
PacketHeader.PKT_MAX_HEADER_LEN
> > < writePacketSize ) {
> > psize = (int)(blockSize - getStreamer().getBytesCurBlock()) +
PacketHeader.PKT_MAX_HEADER_LEN;
> > } else {
> > psize = (int) Math
> > .min(blockSize - getStreamer().getBytesCurBlock(),
writePacketSize);
> > }
> > computePacketChunkSize(psize, bytesPerChecksum);
> > }
> > ```
>
> Thank you very much for investing your time in fixing these bugs. The
above fixes did not take `ChecksumSize` into account, and it would be better
for us to discuss this issue in the new PR. Please check if the failed tests
are related to the modification of this PR. Thanks again.
@zhangshuyan0 Sir, Agree with you, let's discuss this issue in the new PR.
The failed tests were all passed in my local.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] chore: Update releases.md for branding [hadoop-site]
Hexiaoqiao commented on PR #51: URL: https://github.com/apache/hadoop-site/pull/51#issuecomment-1903033902 @tisonkun Thanks for your update. Is it the same thing with https://github.com/apache/hadoop-site/pull/46 (which is from Brand Management suggestions). Not sure if we need update registered trademark description every where or just Download page? From [1] I didn't get some explicit guide line. Any other description documents? Thank again. [1] https://apache.org/foundation/marks/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt [hadoop]
haiyang1987 commented on PR #6476: URL: https://github.com/apache/hadoop/pull/6476#issuecomment-1903017756 Thanks @ZanderXu for your review. Fixed the issue of UT, please help review it again, thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. [hadoop]
zhangshuyan0 commented on PR #6368:
URL: https://github.com/apache/hadoop/pull/6368#issuecomment-1903015641
> Sir, very nice catch. I think below code may resolve the problem you
found. Please take a look when you are free. I will submit another PR to fix it
and Add UT.
>
> ```java
> if (!getStreamer().getAppendChunk()) {
> int psize = 0;
> if (blockSize == getStreamer().getBytesCurBlock()) {
> psize = writePacketSize;
> } else if (blockSize - getStreamer().getBytesCurBlock() +
PacketHeader.PKT_MAX_HEADER_LEN
> < writePacketSize ) {
> psize = (int)(blockSize - getStreamer().getBytesCurBlock()) +
PacketHeader.PKT_MAX_HEADER_LEN;
> } else {
> psize = (int) Math
> .min(blockSize - getStreamer().getBytesCurBlock(),
writePacketSize);
> }
> computePacketChunkSize(psize, bytesPerChecksum);
> }
> ```
Thank you very much for investing your time in fixing these bugs. The above
fixes did not take `ChecksumSize` into account, and it would be better for us
to discuss this issue in the new PR.
Please check if the failed tests are related to the modification of this PR.
Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18045) Disable TestDynamometerInfra
[ https://issues.apache.org/jira/browse/HADOOP-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809218#comment-17809218 ] Shilun Fan commented on HADOOP-18045: - [~tasanuma] I've added 3.4.0 back to fix-version. Thanks again for the reminder! > Disable TestDynamometerInfra > > > Key: HADOOP-18045 > URL: https://issues.apache.org/jira/browse/HADOOP-18045 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This test is broken and there is no fix provided for a long time. Let's > disable the test to reduce the noise in the daily qbt job. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18045) Disable TestDynamometerInfra
[ https://issues.apache.org/jira/browse/HADOOP-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809074#comment-17809074 ] Shilun Fan edited comment on HADOOP-18045 at 1/22/24 2:47 AM: -- -3.3.2 release has been fixed, fix version removed 3.4.0- rollback was (Author: slfan1989): -3.3.2 release has been fixed, fix version removed 3.4.0- > Disable TestDynamometerInfra > > > Key: HADOOP-18045 > URL: https://issues.apache.org/jira/browse/HADOOP-18045 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This test is broken and there is no fix provided for a long time. Let's > disable the test to reduce the noise in the daily qbt job. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17919) Fix command line example in Hadoop Cluster Setup documentation
[
https://issues.apache.org/jira/browse/HADOOP-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809217#comment-17809217
]
Shilun Fan commented on HADOOP-17919:
-
[~iwasakims] Thank you for your reminder！I will re-add the 3.4.0 version in the
fix version.
> Fix command line example in Hadoop Cluster Setup documentation
> --
>
> Key: HADOOP-17919
> URL: https://issues.apache.org/jira/browse/HADOOP-17919
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Rintaro Ikeda
>Assignee: Rintaro Ikeda
>Priority: Minor
> Labels: docuentation, pull-request-available
> Fix For: 3.3.2, 3.2.4
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> About Hadoop cluster setup documentation
> ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html])
> The option is specified in the following example, but HDFS
> command ignores it.
> {noformat}
> `[hdfs]$ $HADOOP_HOME/bin/hdfs namenode -format `
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17919) Fix command line example in Hadoop Cluster Setup documentation
[
https://issues.apache.org/jira/browse/HADOOP-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17919:
Fix Version/s: 3.4.0
> Fix command line example in Hadoop Cluster Setup documentation
> --
>
> Key: HADOOP-17919
> URL: https://issues.apache.org/jira/browse/HADOOP-17919
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Rintaro Ikeda
>Assignee: Rintaro Ikeda
>Priority: Minor
> Labels: docuentation, pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> About Hadoop cluster setup documentation
> ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html])
> The option is specified in the following example, but HDFS
> command ignores it.
> {noformat}
> `[hdfs]$ $HADOOP_HOME/bin/hdfs namenode -format `
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17919) Fix command line example in Hadoop Cluster Setup documentation
[
https://issues.apache.org/jira/browse/HADOOP-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808977#comment-17808977
]
Shilun Fan edited comment on HADOOP-17919 at 1/22/24 2:46 AM:
--
-3.3.2 release has been fixed, fix version removed 3.4.0-
rollback.
was (Author: slfan1989):
3.3.2 release has been fixed, fix version removed 3.4.0
> Fix command line example in Hadoop Cluster Setup documentation
> --
>
> Key: HADOOP-17919
> URL: https://issues.apache.org/jira/browse/HADOOP-17919
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Rintaro Ikeda
>Assignee: Rintaro Ikeda
>Priority: Minor
> Labels: docuentation, pull-request-available
> Fix For: 3.3.2, 3.2.4
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> About Hadoop cluster setup documentation
> ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html])
> The option is specified in the following example, but HDFS
> command ignores it.
> {noformat}
> `[hdfs]$ $HADOOP_HOME/bin/hdfs namenode -format `
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19019) Parallel Maven Build Support for Apache Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809216#comment-17809216
]
ASF GitHub Bot commented on HADOOP-19019:
-
Hexiaoqiao commented on PR #6373:
URL: https://github.com/apache/hadoop/pull/6373#issuecomment-1902975164
If no more other concerns, I will check this PR into trunk for a short
while. @steveloughran
> Parallel Maven Build Support for Apache Hadoop
> --
>
> Key: HADOOP-19019
> URL: https://issues.apache.org/jira/browse/HADOOP-19019
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: caijialiang
>Priority: Major
> Labels: pull-request-available
> Attachments: patch11-HDFS-17287.diff
>
>
> The reason for the slow compilation: The Hadoop project has many modules, and
> the inability to compile them in parallel results in a slow process. For
> instance, the first compilation of Hadoop might take several hours, and even
> with local Maven dependencies, a subsequent compilation can still take close
> to 40 minutes, which is very slow.
> How to solve it: Use {{mvn dependency:tree}} and {{maven-to-plantuml}} to
> investigate the dependency issues that prevent parallel compilation.
> * Investigate the dependencies between project modules.
> * Analyze the dependencies in multi-module Maven projects.
> * Download {{{}maven-to-plantuml{}}}:
>
> {{wget
> [https://github.com/phxql/maven-to-plantuml/releases/download/v1.0/maven-to-plantuml-1.0.jar]}}
> * Generate a dependency tree:
>
> {{mvn dependency:tree > dep.txt}}
> * Generate a UML diagram from the dependency tree:
>
> {{java -jar maven-to-plantuml.jar --input dep.txt --output dep.puml}}
> For more information, visit: [maven-to-plantuml GitHub
> repository|https://github.com/phxql/maven-to-plantuml/tree/master].
>
> *Hadoop Parallel Compilation Submission Logic*
> # Reasons for Parallel Compilation Failure
> *
> ** In sequential compilation, as modules are compiled one by one in order,
> there are no errors because the compilation follows the module sequence.
> ** However, in parallel compilation, all modules are compiled
> simultaneously. The compilation order during multi-module concurrent
> compilation depends on the inter-module dependencies. If Module A depends on
> Module B, then Module B will be compiled before Module A. This ensures that
> the compilation order follows the dependencies between modules.
> But when Hadoop compiles in parallel, for example, compiling
> {{{}hadoop-yarn-project{}}}, the dependencies between modules are correct.
> The issue arises during the dist package stage. {{dist}} packages all other
> compiled modules.
> *Behavior of {{hadoop-yarn-project}} in Serial Compilation:*
> *
> ** In serial compilation, it compiles modules in the pom one by one in
> sequence. After all modules are compiled, it compiles
> {{{}hadoop-yarn-project{}}}. During the {{prepare-package}} stage, the
> {{maven-assembly-plugin}} plugin is executed for packaging. All packages are
> repackaged according to the description in
> {{{}hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml{}}}.
> *Behavior of {{hadoop-yarn-project}} in Parallel Compilation:*
> *
> ** Parallel compilation compiles modules according to the dependency order
> among them. If modules do not declare dependencies on each other through
> {{{}dependency{}}}, they are compiled in parallel. According to the
> dependency definition in the pom of {{{}hadoop-yarn-project{}}}, the
> dependencies are compiled first, followed by {{{}hadoop-yarn-project{}}},
> executing its {{{}maven-assembly-plugin{}}}.
> ** However, the files needed for packaging in
> {{hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml}} are
> not all included in the {{dependency}} of {{{}hadoop-yarn-project{}}}.
> Therefore, when compiling {{hadoop-yarn-project}} and executing
> {{{}maven-assembly-plugin{}}}, not all required modules are built yet,
> leading to errors in parallel compilation.
> *Solution:*
> *
> ** The solution is relatively straightforward: organize all modules from
> {{{}hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml{}}},
> and then declare them as dependencies in the pom of
> {{{}hadoop-yarn-project{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19019: Parallel Maven Build Support for Apache Hadoop [hadoop]
Hexiaoqiao commented on PR #6373: URL: https://github.com/apache/hadoop/pull/6373#issuecomment-1902975164 If no more other concerns, I will check this PR into trunk for a short while. @steveloughran -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17342. Fix DataNode may invalidates normal block causing missing block [hadoop]
ZanderXu commented on PR #6464: URL: https://github.com/apache/hadoop/pull/6464#issuecomment-1902959898 > This is a bug fix after https://github.com/apache/hadoop/pull/5564 , do you have time to help review this? @smarthanwang I have a question about HDFS-16985, Normally FileNotFoundException means that the meta file or data file maybe lost, so the replication on this datanode maybe corrupt, right? In your business(AWS EC2 + EBS) situation, you don't expect datanode to delete this replica directly, so HDFS-16985 just remove the replica from the memory of DN. But I want to see that DN should directly delete this corrupt replica If it can ensure that the replica is corrupt, such as: meta file or data file is lost. So we can add a configure to control whether DN delete this replication from disk directly, such as: fs.datanode.delete.corrupt.replica.from.disk with a default value true. If `fs.datanode.delete.corrupt.replica.from.disk` is true, DN can delete this corrupt replica from disk directly. If `fs.datanode.delete.corrupt.replica.from.disk` is false, DN can just delete this corrupt replica from memory. @smarthanwang @zhangshuyan0 looking forward to your good ideas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17342. Fix DataNode may invalidates normal block causing missing block [hadoop]
smarthanwang commented on code in PR #6464:
URL: https://github.com/apache/hadoop/pull/6464#discussion_r1461284042
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java:
##
@@ -2011,4 +2011,83 @@ public void tesInvalidateMissingBlock() throws Exception
{
cluster.shutdown();
}
}
+
+ @Test
+ public void testCheckFilesWhenInvalidateMissingBlock() throws Exception {
+long blockSize = 1024;
+int heartbeatInterval = 1;
+HdfsConfiguration c = new HdfsConfiguration();
+c.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, heartbeatInterval);
+c.setLong(DFS_BLOCK_SIZE_KEY, blockSize);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(c).
+numDataNodes(1).build();
+DataNodeFaultInjector oldDnInjector = DataNodeFaultInjector.get();
+try {
+ cluster.waitActive();
+ BlockReaderTestUtil util = new BlockReaderTestUtil(cluster, new
+ HdfsConfiguration(conf));
+ Path path = new Path("/testFile");
+ util.writeFile(path, 1);
+ String bpid = cluster.getNameNode().getNamesystem().getBlockPoolId();
+ DataNode dn = cluster.getDataNodes().get(0);
+ FsDatasetImpl dnFSDataset = (FsDatasetImpl) dn.getFSDataset();
+ List replicaInfos = dnFSDataset.getFinalizedBlocks(bpid);
+ assertEquals(1, replicaInfos.size());
+ DFSTestUtil.readFile(cluster.getFileSystem(), path);
+ LocatedBlock blk = util.getFileBlocks(path, 512).get(0);
+ ExtendedBlock block = blk.getBlock();
+
+ // Append a new block with an incremented generation stamp.
+ long newGS = block.getGenerationStamp() + 1;
+ dnFSDataset.append(block, newGS, 1024);
+ block.setGenerationStamp(newGS);
+
+ DataNodeFaultInjector injector = new DataNodeFaultInjector() {
+@Override
+public void delayGetMetaDataInputStream() {
+ try {
+Thread.sleep(8000);
+ } catch (InterruptedException e) {
+// Ignore exception.
+ }
+}
+ };
+ // Delay to getMetaDataInputStream.
+ DataNodeFaultInjector.set(injector);
+
+ ExecutorService executorService = Executors.newFixedThreadPool(2);
+ try {
+Future blockReaderFuture = executorService.submit(() -> {
+ try {
+// Submit tasks for reading block.
+BlockReaderTestUtil.getBlockReader(cluster.getFileSystem(), blk,
0, 512);
+ } catch (IOException e) {
+// Ignore exception.
+ }
+});
+
+Future finalizeBlockFuture = executorService.submit(() -> {
+ try {
+// Submit tasks for finalizing block.
+Thread.sleep(1000);
+dnFSDataset.finalizeBlock(block, false);
+ } catch (Exception e) {
+// Ignore exception
+ }
+});
+
+// Wait for both tasks to complete.
+blockReaderFuture.get();
+finalizeBlockFuture.get();
+ } finally {
+executorService.shutdown();
+ }
+
+ // Validate the replica is exits.
+ assertNotNull(dnFSDataset.getReplicaInfo(blk.getBlock()));
Review Comment:
Yes, It tests case the block file not found for any causes. But I am not
sure the situation as your description would lead to FNE, so I think the case
should be constructed and tested
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:
##
@@ -2416,11 +2419,21 @@ public void invalidateMissingBlock(String bpid, Block
block) {
// So remove if from volume map notify namenode is ok.
try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.BLOCK_POOl,
bpid)) {
- ReplicaInfo replica = volumeMap.remove(bpid, block);
- invalidate(bpid, replica);
+ // Check if this block is on the volume map.
+ ReplicaInfo replica = volumeMap.get(bpid, block);
+ // Double-check block or meta file existence when checkFiles as true.
+ if (replica != null && (!checkFiles ||
+ (!replica.blockDataExists() || !replica.metadataExists( {
+volumeMap.remove(bpid, block);
+invalidate(bpid, replica);
Review Comment:
Yes, get it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17919) Fix command line example in Hadoop Cluster Setup documentation
[
https://issues.apache.org/jira/browse/HADOOP-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809208#comment-17809208
]
Masatake Iwasaki commented on HADOOP-17919:
---
[~slfan1989] I think we should keep 3.4.0 in the "Fix Version/s" as we usually
do for clarity. Since both branch-3.3 and branch-3.4 are maintained, we can not
assume that 3.4.0 contains all fixes of 3.3.x (and previous patch releases).
> Fix command line example in Hadoop Cluster Setup documentation
> --
>
> Key: HADOOP-17919
> URL: https://issues.apache.org/jira/browse/HADOOP-17919
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.3.1, 3.4.0
>Reporter: Rintaro Ikeda
>Assignee: Rintaro Ikeda
>Priority: Minor
> Labels: docuentation, pull-request-available
> Fix For: 3.3.2, 3.2.4
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> About Hadoop cluster setup documentation
> ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html])
> The option is specified in the following example, but HDFS
> command ignores it.
> {noformat}
> `[hdfs]$ $HADOOP_HOME/bin/hdfs namenode -format `
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19043) S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
[
https://issues.apache.org/jira/browse/HADOOP-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809175#comment-17809175
]
ASF GitHub Bot commented on HADOOP-19043:
-
hadoop-yetus commented on PR #6465:
URL: https://github.com/apache/hadoop/pull/6465#issuecomment-1902752372
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 11s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 13s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 88m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6465/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6465 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a03003e782eb 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68f5c8da5e3d6918dfc1e914127c64540eee3c79 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6465/3/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6465/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
> ---
Re: [PR] HADOOP-19043. S3A: Regression: ITestS3AOpenCost fails on prefetch test runs [hadoop]
hadoop-yetus commented on PR #6465:
URL: https://github.com/apache/hadoop/pull/6465#issuecomment-1902752372
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 11s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 13s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 88m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6465/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6465 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a03003e782eb 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68f5c8da5e3d6918dfc1e914127c64540eee3c79 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6465/3/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6465/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
[jira] [Commented] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
[
https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809168#comment-17809168
]
ASF GitHub Bot commented on HADOOP-18610:
-
steveloughran commented on PR #5953:
URL: https://github.com/apache/hadoop/pull/5953#issuecomment-1902738972
you can use env var resolution within a hadoop core-site file; which lets
you at the values with defaults when unset. on locked down config loading
(oozie etc) then only the default is valid.
```
${env.LOCAL_DIRS:-some.default}/
```
so: no need to add explicit resolution, just document or set as default.
example, s3a uses temp dirs in yarn containers automatically.
```xml
fs.s3a.buffer.dir
${env.LOCAL_DIRS:-${hadoop.tmp.dir}}/s3a
Comma separated list of directories that will be used to
buffer file
uploads to.
Yarn container path will be used as default value on yarn applications,
otherwise fall back to hadoop.tmp.dir
```
> ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
> -
>
> Key: HADOOP-18610
> URL: https://issues.apache.org/jira/browse/HADOOP-18610
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools
>Affects Versions: 3.3.4
>Reporter: Haifeng Chen
>Priority: Critical
> Labels: pull-request-available
> Attachments: HADOOP-18610-preview.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity
> with with [Azure Active Directory (Azure AD) workload
> identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview]
> (preview), which integrate with the Kubernetes native capabilities to
> federate with any external identity providers. This approach is simpler to
> use and deploy.
> Refer to
> [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview|https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.]
> and [https://azure.github.io/azure-workload-identity/docs/introduction.html]
> for more details.
> The basic use scenario is to access Azure cloud resources (such as cloud
> storage) from Kubernetes (such as AKS) workload using Azure managed identity
> federated with Kubernetes service account. The credential environment
> variables in pod projected by Azure AD workload identity are like following:
> AZURE_AUTHORITY_HOST: (Injected by the webhook,
> [https://login.microsoftonline.com/])
> AZURE_CLIENT_ID: (Injected by the webhook)
> AZURE_TENANT_ID: (Injected by the webhook)
> AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook,
> /var/run/secrets/azure/tokens/azure-identity-token)
> The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON
> Web Token) client assertion token which we can use to request to
> AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId +
> "/oauth2/v2.0/token") for a AD token which can be used to directly access
> the Azure cloud resources.
> This approach is very common and similar among cloud providers such as AWS
> and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to
> handle the same case.
> The existing MsiTokenProvider can only handle the managed identity associated
> with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider
> which handle Azure Workload Identity case. For this, we need to add one
> method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used
> by WorkloadIdentityTokenProvider.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18610. ABFS OAuth2 Token Provider support for Azure Workload Identity [hadoop]
steveloughran commented on PR #5953:
URL: https://github.com/apache/hadoop/pull/5953#issuecomment-1902738972
you can use env var resolution within a hadoop core-site file; which lets
you at the values with defaults when unset. on locked down config loading
(oozie etc) then only the default is valid.
```
${env.LOCAL_DIRS:-some.default}/
```
so: no need to add explicit resolution, just document or set as default.
example, s3a uses temp dirs in yarn containers automatically.
```xml
fs.s3a.buffer.dir
${env.LOCAL_DIRS:-${hadoop.tmp.dir}}/s3a
Comma separated list of directories that will be used to
buffer file
uploads to.
Yarn container path will be used as default value on yarn applications,
otherwise fall back to hadoop.tmp.dir
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[ https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18883. - Fix Version/s: 3.5.0 Resolution: Fixed > Expect-100 JDK bug resolution: prevent multiple server calls > > > Key: HADOOP-18883 > URL: https://issues.apache.org/jira/browse/HADOOP-18883 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978]. > > With the current implementation of HttpURLConnection if server rejects the > “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be > thrown from 'expect100Continue()' method. > After the exception thrown, If we call any other method on the same instance > (ex getHeaderField(), or getHeaderFields()). They will internally call > getOuputStream() which invokes writeRequests(), which make the actual server > call. > In the AbfsHttpOperation, after sendRequest() we call processResponse() > method from AbfsRestOperation. Even if the conn.getOutputStream() fails due > to expect-100 error, we consume the exception and let the code go ahead. So, > we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which > will be triggered after getOutputStream is failed. These invocation will lead > to server calls. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[ https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809162#comment-17809162 ] ASF GitHub Bot commented on HADOOP-18883: - steveloughran merged PR #6022: URL: https://github.com/apache/hadoop/pull/6022 > Expect-100 JDK bug resolution: prevent multiple server calls > > > Key: HADOOP-18883 > URL: https://issues.apache.org/jira/browse/HADOOP-18883 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > > This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978]. > > With the current implementation of HttpURLConnection if server rejects the > “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be > thrown from 'expect100Continue()' method. > After the exception thrown, If we call any other method on the same instance > (ex getHeaderField(), or getHeaderFields()). They will internally call > getOuputStream() which invokes writeRequests(), which make the actual server > call. > In the AbfsHttpOperation, after sendRequest() we call processResponse() > method from AbfsRestOperation. Even if the conn.getOutputStream() fails due > to expect-100 error, we consume the exception and let the code go ahead. So, > we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which > will be triggered after getOutputStream is failed. These invocation will lead > to server calls. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls [hadoop]
steveloughran merged PR #6022: URL: https://github.com/apache/hadoop/pull/6022 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19019) Parallel Maven Build Support for Apache Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809161#comment-17809161
]
ASF GitHub Bot commented on HADOOP-19019:
-
steveloughran commented on PR #6373:
URL: https://github.com/apache/hadoop/pull/6373#issuecomment-1902729436
Who is going to merge this? @Hexiaoqiao?
> Parallel Maven Build Support for Apache Hadoop
> --
>
> Key: HADOOP-19019
> URL: https://issues.apache.org/jira/browse/HADOOP-19019
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: caijialiang
>Priority: Major
> Labels: pull-request-available
> Attachments: patch11-HDFS-17287.diff
>
>
> The reason for the slow compilation: The Hadoop project has many modules, and
> the inability to compile them in parallel results in a slow process. For
> instance, the first compilation of Hadoop might take several hours, and even
> with local Maven dependencies, a subsequent compilation can still take close
> to 40 minutes, which is very slow.
> How to solve it: Use {{mvn dependency:tree}} and {{maven-to-plantuml}} to
> investigate the dependency issues that prevent parallel compilation.
> * Investigate the dependencies between project modules.
> * Analyze the dependencies in multi-module Maven projects.
> * Download {{{}maven-to-plantuml{}}}:
>
> {{wget
> [https://github.com/phxql/maven-to-plantuml/releases/download/v1.0/maven-to-plantuml-1.0.jar]}}
> * Generate a dependency tree:
>
> {{mvn dependency:tree > dep.txt}}
> * Generate a UML diagram from the dependency tree:
>
> {{java -jar maven-to-plantuml.jar --input dep.txt --output dep.puml}}
> For more information, visit: [maven-to-plantuml GitHub
> repository|https://github.com/phxql/maven-to-plantuml/tree/master].
>
> *Hadoop Parallel Compilation Submission Logic*
> # Reasons for Parallel Compilation Failure
> *
> ** In sequential compilation, as modules are compiled one by one in order,
> there are no errors because the compilation follows the module sequence.
> ** However, in parallel compilation, all modules are compiled
> simultaneously. The compilation order during multi-module concurrent
> compilation depends on the inter-module dependencies. If Module A depends on
> Module B, then Module B will be compiled before Module A. This ensures that
> the compilation order follows the dependencies between modules.
> But when Hadoop compiles in parallel, for example, compiling
> {{{}hadoop-yarn-project{}}}, the dependencies between modules are correct.
> The issue arises during the dist package stage. {{dist}} packages all other
> compiled modules.
> *Behavior of {{hadoop-yarn-project}} in Serial Compilation:*
> *
> ** In serial compilation, it compiles modules in the pom one by one in
> sequence. After all modules are compiled, it compiles
> {{{}hadoop-yarn-project{}}}. During the {{prepare-package}} stage, the
> {{maven-assembly-plugin}} plugin is executed for packaging. All packages are
> repackaged according to the description in
> {{{}hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml{}}}.
> *Behavior of {{hadoop-yarn-project}} in Parallel Compilation:*
> *
> ** Parallel compilation compiles modules according to the dependency order
> among them. If modules do not declare dependencies on each other through
> {{{}dependency{}}}, they are compiled in parallel. According to the
> dependency definition in the pom of {{{}hadoop-yarn-project{}}}, the
> dependencies are compiled first, followed by {{{}hadoop-yarn-project{}}},
> executing its {{{}maven-assembly-plugin{}}}.
> ** However, the files needed for packaging in
> {{hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml}} are
> not all included in the {{dependency}} of {{{}hadoop-yarn-project{}}}.
> Therefore, when compiling {{hadoop-yarn-project}} and executing
> {{{}maven-assembly-plugin{}}}, not all required modules are built yet,
> leading to errors in parallel compilation.
> *Solution:*
> *
> ** The solution is relatively straightforward: organize all modules from
> {{{}hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml{}}},
> and then declare them as dependencies in the pom of
> {{{}hadoop-yarn-project{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19019: Parallel Maven Build Support for Apache Hadoop [hadoop]
steveloughran commented on PR #6373: URL: https://github.com/apache/hadoop/pull/6373#issuecomment-1902729436 Who is going to merge this? @Hexiaoqiao? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19046) S3A: update sdk versions
[
https://issues.apache.org/jira/browse/HADOOP-19046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809160#comment-17809160
]
ASF GitHub Bot commented on HADOOP-19046:
-
steveloughran merged PR #6467:
URL: https://github.com/apache/hadoop/pull/6467
> S3A: update sdk versions
>
>
> Key: HADOOP-19046
> URL: https://issues.apache.org/jira/browse/HADOOP-19046
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> Move up to the most recent versions of the v2 sdk, with a v1 update just to
> keep some CVE checking happy.
> {code}
> 1.12.599
> 2.23.5
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 [hadoop]
steveloughran merged PR #6467: URL: https://github.com/apache/hadoop/pull/6467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19043. S3A: Regression: ITestS3AOpenCost fails on prefetch test runs [hadoop]
steveloughran commented on PR #6465: URL: https://github.com/apache/hadoop/pull/6465#issuecomment-1902727835 applied my own review comments; tested against s3 express in usw2 with and without -prefetch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19043) S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
[
https://issues.apache.org/jira/browse/HADOOP-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809159#comment-17809159
]
ASF GitHub Bot commented on HADOOP-19043:
-
steveloughran commented on PR #6465:
URL: https://github.com/apache/hadoop/pull/6465#issuecomment-1902727835
applied my own review comments; tested against s3 express in usw2 with and
without -prefetch
> S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
> -
>
> Key: HADOOP-19043
> URL: https://issues.apache.org/jira/browse/HADOOP-19043
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> Getting test failures in the new ITestS3AOpenCost tests when run with
> {{-Dprefetch}}
> Thought I'd tested this, but clearly not
> * class cast failures on asserts (fix: skip)
> * bytes read different in one test: (fix: identify and address)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809158#comment-17809158
]
ASF GitHub Bot commented on HADOOP-19047:
-
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1461054737
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.Retries;
+import org.apache.hadoop.fs.s3a.S3ADataBlocks;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.impl.PutObjectOptions;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.s3a.Statistic.COMMITTER_MAGIC_MARKER_PUT;
+import static
org.apache.hadoop.fs.s3a.commit.CommitConstants.X_HEADER_MAGIC_MARKER;
+import static
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation;
+
+public class S3MagicCommitTracker extends MagicCommitTracker {
+
+ public S3MagicCommitTracker(Path path,
+ String bucket,
+ String originalDestKey,
+ String destKey,
+ String pendingsetKey,
+ WriteOperationHelper writer,
+ PutTrackerStatistics trackerStatistics) {
+super(path, bucket, originalDestKey, destKey, pendingsetKey, writer,
trackerStatistics);
+ }
+
+ @Override
+ public boolean aboutToComplete(String uploadId,
+ List parts,
+ long bytesWritten,
+ final IOStatistics iostatistics)
+ throws IOException {
+Preconditions.checkArgument(StringUtils.isNotEmpty(uploadId),
+"empty/null upload ID: "+ uploadId);
+Preconditions.checkArgument(parts != null,
+"No uploaded parts list");
+Preconditions.checkArgument(!parts.isEmpty(),
+"No uploaded parts to save");
+
+// put a 0-byte file with the name of the original under-magic path
+// Add the final file length as a header
+// this is done before the task commit, so its duration can be
+// included in the statistics
+Map headers = new HashMap<>();
+headers.put(X_HEADER_MAGIC_MARKER, Long.toString(bytesWritten));
+PutObjectRequest originalDestPut = writer.createPutObjectRequest(
+originalDestKey,
+0,
+new PutObjectOptions(true, null, headers), false);
+upload(originalDestPut, new ByteArrayInputStream(EMPTY));
+
+// build the commit summary
+SinglePendingCommit commitData = new SinglePendingCommit();
+commitData.touch(System.currentTimeMillis());
+commitData.setDestinationKey(getDestKey());
+commitData.setBucket(bucket);
+commitData.setUri(path.toUri().toString());
+commitData.setUploadId(uploadId);
+commitData.setText("");
+commitData.setLength(bytesWritten);
+commitData.bindCommitData(parts);
+commitData.setIOStatistics(
+new IOStatisticsSnapshot(iostatistics));
+
+byte[] bytes = commitData.toBytes(SinglePendingCommit.serializer());
Review Comment:
you know, the other thing to consider here is moving from json
serialization; IOStatisticsSnapshot already implements Serializable; adding
Hadoop Writable to it would make for faster ser/deser and marshalling than
through jackson
> Support InMemory Tracking Of S3A Magic Commits
> --
>
> Key: HADOOP-19047
> URL: https://issues.ap
Re: [PR] HADOOP-19047: Support InMemory Tracking Of S3A Magic Commits [hadoop]
steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1461054737
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.commit.magic;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.Retries;
+import org.apache.hadoop.fs.s3a.S3ADataBlocks;
+import org.apache.hadoop.fs.s3a.WriteOperationHelper;
+import org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit;
+import org.apache.hadoop.fs.s3a.impl.PutObjectOptions;
+import org.apache.hadoop.fs.s3a.statistics.PutTrackerStatistics;
+import org.apache.hadoop.fs.statistics.IOStatistics;
+import org.apache.hadoop.fs.statistics.IOStatisticsSnapshot;
+import org.apache.hadoop.util.Preconditions;
+import software.amazon.awssdk.services.s3.model.CompletedPart;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.hadoop.fs.s3a.Statistic.COMMITTER_MAGIC_MARKER_PUT;
+import static
org.apache.hadoop.fs.s3a.commit.CommitConstants.X_HEADER_MAGIC_MARKER;
+import static
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation;
+
+public class S3MagicCommitTracker extends MagicCommitTracker {
+
+ public S3MagicCommitTracker(Path path,
+ String bucket,
+ String originalDestKey,
+ String destKey,
+ String pendingsetKey,
+ WriteOperationHelper writer,
+ PutTrackerStatistics trackerStatistics) {
+super(path, bucket, originalDestKey, destKey, pendingsetKey, writer,
trackerStatistics);
+ }
+
+ @Override
+ public boolean aboutToComplete(String uploadId,
+ List parts,
+ long bytesWritten,
+ final IOStatistics iostatistics)
+ throws IOException {
+Preconditions.checkArgument(StringUtils.isNotEmpty(uploadId),
+"empty/null upload ID: "+ uploadId);
+Preconditions.checkArgument(parts != null,
+"No uploaded parts list");
+Preconditions.checkArgument(!parts.isEmpty(),
+"No uploaded parts to save");
+
+// put a 0-byte file with the name of the original under-magic path
+// Add the final file length as a header
+// this is done before the task commit, so its duration can be
+// included in the statistics
+Map headers = new HashMap<>();
+headers.put(X_HEADER_MAGIC_MARKER, Long.toString(bytesWritten));
+PutObjectRequest originalDestPut = writer.createPutObjectRequest(
+originalDestKey,
+0,
+new PutObjectOptions(true, null, headers), false);
+upload(originalDestPut, new ByteArrayInputStream(EMPTY));
+
+// build the commit summary
+SinglePendingCommit commitData = new SinglePendingCommit();
+commitData.touch(System.currentTimeMillis());
+commitData.setDestinationKey(getDestKey());
+commitData.setBucket(bucket);
+commitData.setUri(path.toUri().toString());
+commitData.setUploadId(uploadId);
+commitData.setText("");
+commitData.setLength(bytesWritten);
+commitData.bindCommitData(parts);
+commitData.setIOStatistics(
+new IOStatisticsSnapshot(iostatistics));
+
+byte[] bytes = commitData.toBytes(SinglePendingCommit.serializer());
Review Comment:
you know, the other thing to consider here is moving from json
serialization; IOStatisticsSnapshot already implements Serializable; adding
Hadoop Writable to it would make for faster ser/deser and marshalling than
through jackson
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsu
Re: [PR] HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt [hadoop]
hadoop-yetus commented on PR #6476:
URL: https://github.com/apache/hadoop/pull/6476#issuecomment-1902711189
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 188m 49s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 276m 1s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6476 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3e7372f9c854 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fc6654ef0d6cd5b65f68fba577390faf3ed57ae2 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/1/testReport/ |
| Max. process+thread count | 4194 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6476/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the
[jira] [Comment Edited] (HADOOP-18691) Add a CallerContext getter on the Schedulable interface
[
https://issues.apache.org/jira/browse/HADOOP-18691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809003#comment-17809003
]
Shilun Fan edited comment on HADOOP-18691 at 1/21/24 3:47 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0
> Add a CallerContext getter on the Schedulable interface
> ---
>
> Key: HADOOP-18691
> URL: https://issues.apache.org/jira/browse/HADOOP-18691
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Christos Bisias
>Assignee: Christos Bisias
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> We would like to add a default *{color:#00875a}CallerContext{color}* getter
> on the *{color:#00875a}Schedulable{color}* interface
> {code:java}
> default public CallerContext getCallerContext() {
> return null;
> } {code}
> and then override it on the
> *{color:#00875a}i{color}{color:#00875a}{*}pc/{*}Server.Call{color}* class
> {code:java}
> @Override
> public CallerContext getCallerContext() {
> return this.callerContext;
> } {code}
> to expose the already existing *{color:#00875a}callerContext{color}* field.
>
> This change will help us access the *{color:#00875a}CallerContext{color}* on
> an Apache Ozone *{color:#00875a}IdentityProvider{color}* implementation.
> On Ozone side the *{color:#00875a}FairCallQueue{color}* doesn't work with the
> Ozone S3G, because all users are masked under a special S3G user and there is
> no impersonation. Therefore, the FCQ reads only 1 user and becomes
> ineffective. We can use the *{color:#00875a}CallerContext{color}* field to
> store the current user and access it on the Ozone
> {*}{color:#00875a}IdentityProvider{color}{*}.
>
> This is a presentation with the proposed approach.
> [https://docs.google.com/presentation/d/1iChpCz_qf-LXiPyvotpOGiZ31yEUyxAdU4RhWMKo0c0/edit#slide=id.p]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18689) Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project
[ https://issues.apache.org/jira/browse/HADOOP-18689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18689: Fix Version/s: 3.4.0 > Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project > > > Key: HADOOP-18689 > URL: https://issues.apache.org/jira/browse/HADOOP-18689 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0, 3.3.9 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > PR from github depandabot > https://github.com/apache/hadoop/pull/5502 > Mentions CVE: https://nvd.nist.gov/vuln/detail/CVE-2023-1436 > Creating ticket for tracking. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18691) Add a CallerContext getter on the Schedulable interface
[
https://issues.apache.org/jira/browse/HADOOP-18691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18691:
Fix Version/s: 3.4.0
> Add a CallerContext getter on the Schedulable interface
> ---
>
> Key: HADOOP-18691
> URL: https://issues.apache.org/jira/browse/HADOOP-18691
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Christos Bisias
>Assignee: Christos Bisias
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> We would like to add a default *{color:#00875a}CallerContext{color}* getter
> on the *{color:#00875a}Schedulable{color}* interface
> {code:java}
> default public CallerContext getCallerContext() {
> return null;
> } {code}
> and then override it on the
> *{color:#00875a}i{color}{color:#00875a}{*}pc/{*}Server.Call{color}* class
> {code:java}
> @Override
> public CallerContext getCallerContext() {
> return this.callerContext;
> } {code}
> to expose the already existing *{color:#00875a}callerContext{color}* field.
>
> This change will help us access the *{color:#00875a}CallerContext{color}* on
> an Apache Ozone *{color:#00875a}IdentityProvider{color}* implementation.
> On Ozone side the *{color:#00875a}FairCallQueue{color}* doesn't work with the
> Ozone S3G, because all users are masked under a special S3G user and there is
> no impersonation. Therefore, the FCQ reads only 1 user and becomes
> ineffective. We can use the *{color:#00875a}CallerContext{color}* field to
> store the current user and access it on the Ozone
> {*}{color:#00875a}IdentityProvider{color}{*}.
>
> This is a presentation with the proposed approach.
> [https://docs.google.com/presentation/d/1iChpCz_qf-LXiPyvotpOGiZ31yEUyxAdU4RhWMKo0c0/edit#slide=id.p]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18689) Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project
[ https://issues.apache.org/jira/browse/HADOOP-18689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809005#comment-17809005 ] Shilun Fan edited comment on HADOOP-18689 at 1/21/24 3:46 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project > > > Key: HADOOP-18689 > URL: https://issues.apache.org/jira/browse/HADOOP-18689 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0, 3.3.9 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > PR from github depandabot > https://github.com/apache/hadoop/pull/5502 > Mentions CVE: https://nvd.nist.gov/vuln/detail/CVE-2023-1436 > Creating ticket for tracking. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18688) S3A audit header to include count of items in delete ops
[ https://issues.apache.org/jira/browse/HADOOP-18688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809008#comment-17809008 ] Shilun Fan edited comment on HADOOP-18688 at 1/21/24 3:46 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > S3A audit header to include count of items in delete ops > > > Key: HADOOP-18688 > URL: https://issues.apache.org/jira/browse/HADOOP-18688 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > it would be good to find out how many files were deleted in a DeleteObjects > call -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18688) S3A audit header to include count of items in delete ops
[ https://issues.apache.org/jira/browse/HADOOP-18688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18688: Fix Version/s: 3.4.0 > S3A audit header to include count of items in delete ops > > > Key: HADOOP-18688 > URL: https://issues.apache.org/jira/browse/HADOOP-18688 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > it would be good to find out how many files were deleted in a DeleteObjects > call -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18684) S3A filesystem to support binding to other URI schemes
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18684: Fix Version/s: 3.3.6 > S3A filesystem to support binding to other URI schemes > -- > > Key: HADOOP-18684 > URL: https://issues.apache.org/jira/browse/HADOOP-18684 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.5 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Allow s3a filesystem to be bindable to other filesystem schemas, especially > s3:// > * FileContext API has hard coded use of "s3a" > * S3AFileSystem.getScheme() needs to pick up the scheme of the URI passed to > initialize() > * plus tests -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18680) Insufficient heap during full test runs in Docker container.
[
https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809010#comment-17809010
]
Shilun Fan edited comment on HADOOP-18680 at 1/21/24 3:44 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0
> Insufficient heap during full test runs in Docker container.
>
>
> Key: HADOOP-18680
> URL: https://issues.apache.org/jira/browse/HADOOP-18680
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> During verification of releases on the 3.3 line, I often run out of heap
> during full test runs inside the Docker container. Let's increase the default
> in {{MAVEN_OPTS}} to match trunk.
> Additionally, on trunk, the settings are different in Dockerfile vs.
> Dockerfile_aarch64. We can align those.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18680) Insufficient heap during full test runs in Docker container.
[
https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18680:
Fix Version/s: 3.3.6
> Insufficient heap during full test runs in Docker container.
>
>
> Key: HADOOP-18680
> URL: https://issues.apache.org/jira/browse/HADOOP-18680
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> During verification of releases on the 3.3 line, I often run out of heap
> during full test runs inside the Docker container. Let's increase the default
> in {{MAVEN_OPTS}} to match trunk.
> Additionally, on trunk, the settings are different in Dockerfile vs.
> Dockerfile_aarch64. We can align those.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18662) ListFiles with recursive fails with FNF
[
https://issues.apache.org/jira/browse/HADOOP-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809012#comment-17809012
]
Shilun Fan edited comment on HADOOP-18662 at 1/21/24 3:43 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0
> ListFiles with recursive fails with FNF
> ---
>
> Key: HADOOP-18662
> URL: https://issues.apache.org/jira/browse/HADOOP-18662
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> Problem triggers in HDFS, but the change is in Hadoop-Common, Since the
> listFiles is defined in Hadoop-Common.
> Scenario:
> ListFiles With recursive:
> * Fetches a dir say /dir, which has some /dir/s1...s10
> * Recursive is set to true: It goes and tries on say /dir/s5 and /dir/s5 got
> deleted by that time
> * The entire operation fails with FNF
> Hive Cleaner uses listFiles with recursive true and this impacts that
> {noformat}
> 2023-03-06 07:45:48,331 ERROR
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner:
> [Cleaner-executor-thread-12]: Caught exception when cleaning, unable to
> complete cleaning of
> id:39762523,dbname:test,tableName:test_table,partName:null,state:,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:989,errorMessage:null,workerId:
> null,initiatorId: null java.io.FileNotFoundException: File
> hdfs:/cluster/warehouse/tablespace/managed/hive/test.db/test_table/.hive-staging_hive_2023-03-06_07-45-23_120_4659605113266849995-73550
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
> at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2332)
> at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2309)
> at
> org.apache.hadoop.util.functional.RemoteIterators$WrappingRemoteIterator.sourceHasNext(RemoteIterators.java:432)
> at
> org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.fetch(RemoteIterators.java:581)
> at
> org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.hasNext(RemoteIterators.java:602)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1435)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:287)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:214)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:114)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18671) Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-18671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18671: Fix Version/s: 3.4.0 > Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem > > > Key: HADOOP-18671 > URL: https://issues.apache.org/jira/browse/HADOOP-18671 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.6 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We are in the midst of enabling HBase and Solr to run on Ozone. > An obstacle is that HBase relies heavily on HDFS APIs and semantics for its > Write Ahead Log (WAL) file (similarly, for Solr's transaction log). We > propose to push up these HDFS APIs, i.e. recoverLease(), setSafeMode(), > isFileClosed() to FileSystem abstraction so that HBase and other applications > do not need to take on Ozone dependency at compile time. This work will > (hopefully) enable HBase to run on other storage system implementations in > the future. > There are other HDFS features that HBase uses, including hedged read and > favored nodes. Those are FS-specific optimizations and are not critical to > enable HBase on Ozone. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18662) ListFiles with recursive fails with FNF
[
https://issues.apache.org/jira/browse/HADOOP-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18662:
Fix Version/s: 3.4.0
> ListFiles with recursive fails with FNF
> ---
>
> Key: HADOOP-18662
> URL: https://issues.apache.org/jira/browse/HADOOP-18662
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> Problem triggers in HDFS, but the change is in Hadoop-Common, Since the
> listFiles is defined in Hadoop-Common.
> Scenario:
> ListFiles With recursive:
> * Fetches a dir say /dir, which has some /dir/s1...s10
> * Recursive is set to true: It goes and tries on say /dir/s5 and /dir/s5 got
> deleted by that time
> * The entire operation fails with FNF
> Hive Cleaner uses listFiles with recursive true and this impacts that
> {noformat}
> 2023-03-06 07:45:48,331 ERROR
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner:
> [Cleaner-executor-thread-12]: Caught exception when cleaning, unable to
> complete cleaning of
> id:39762523,dbname:test,tableName:test_table,partName:null,state:,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:989,errorMessage:null,workerId:
> null,initiatorId: null java.io.FileNotFoundException: File
> hdfs:/cluster/warehouse/tablespace/managed/hive/test.db/test_table/.hive-staging_hive_2023-03-06_07-45-23_120_4659605113266849995-73550
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
> at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2332)
> at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2309)
> at
> org.apache.hadoop.util.functional.RemoteIterators$WrappingRemoteIterator.sourceHasNext(RemoteIterators.java:432)
> at
> org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.fetch(RemoteIterators.java:581)
> at
> org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.hasNext(RemoteIterators.java:602)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1435)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:287)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:214)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:114)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18660) Filesystem Spelling Mistake
[ https://issues.apache.org/jira/browse/HADOOP-18660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18660: Fix Version/s: 3.4.0 > Filesystem Spelling Mistake > --- > > Key: HADOOP-18660 > URL: https://issues.apache.org/jira/browse/HADOOP-18660 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sebastian Baunsgaard >Assignee: Sebastian Baunsgaard >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > The fs/Filesystem reports errors always containing the spelling mistake > 'fileystem' > It is not the only place in Hadoop this is the case, but this is the easiest > to fix. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18652) Path.suffix raises NullPointerException
[
https://issues.apache.org/jira/browse/HADOOP-18652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18652:
Fix Version/s: 3.4.0
> Path.suffix raises NullPointerException
> ---
>
> Key: HADOOP-18652
> URL: https://issues.apache.org/jira/browse/HADOOP-18652
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Patrick Grandjean
>Assignee: Patrick Grandjean
>Priority: Minor
> Fix For: 3.4.0, 3.3.6
>
>
> Calling the Path.suffix method on root raises a NullPointerException. Tested
> with hadoop-client-api 3.3.2
> Scenario:
> {code:java}
> import org.apache.hadoop.fs.*
> Path root = new Path("/")
> root.getParent == null // true
> root.suffix("bar") // NPE is raised
> {code}
> Stack:
> {code:none}
> 23/03/03 15:13:18 ERROR Uncaught throwable from user code:
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:104)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at org.apache.hadoop.fs.Path.suffix(Path.java:361)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18660) Filesystem Spelling Mistake
[ https://issues.apache.org/jira/browse/HADOOP-18660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809013#comment-17809013 ] Shilun Fan edited comment on HADOOP-18660 at 1/21/24 3:41 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Filesystem Spelling Mistake > --- > > Key: HADOOP-18660 > URL: https://issues.apache.org/jira/browse/HADOOP-18660 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sebastian Baunsgaard >Assignee: Sebastian Baunsgaard >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > The fs/Filesystem reports errors always containing the spelling mistake > 'fileystem' > It is not the only place in Hadoop this is the case, but this is the easiest > to fix. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18658) snakeyaml dependency: upgrade to v2.0
[ https://issues.apache.org/jira/browse/HADOOP-18658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18658: Fix Version/s: 3.4.0 > snakeyaml dependency: upgrade to v2.0 > - > > Key: HADOOP-18658 > URL: https://issues.apache.org/jira/browse/HADOOP-18658 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > * [https://github.com/advisories/GHSA-mjmj-j48q-9wg2] > * I don't think this needs to go in v3.3.5 - since this CVE affects part of > snakeyaml that hadoop doesn't use -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18652) Path.suffix raises NullPointerException
[
https://issues.apache.org/jira/browse/HADOOP-18652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809015#comment-17809015
]
Shilun Fan edited comment on HADOOP-18652 at 1/21/24 3:41 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0
> Path.suffix raises NullPointerException
> ---
>
> Key: HADOOP-18652
> URL: https://issues.apache.org/jira/browse/HADOOP-18652
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Patrick Grandjean
>Assignee: Patrick Grandjean
>Priority: Minor
> Fix For: 3.4.0, 3.3.6
>
>
> Calling the Path.suffix method on root raises a NullPointerException. Tested
> with hadoop-client-api 3.3.2
> Scenario:
> {code:java}
> import org.apache.hadoop.fs.*
> Path root = new Path("/")
> root.getParent == null // true
> root.suffix("bar") // NPE is raised
> {code}
> Stack:
> {code:none}
> 23/03/03 15:13:18 ERROR Uncaught throwable from user code:
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:104)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at org.apache.hadoop.fs.Path.suffix(Path.java:361)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18647) x-ms-client-request-id to have some way that identifies retry of an API.
[ https://issues.apache.org/jira/browse/HADOOP-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809016#comment-17809016 ] Shilun Fan edited comment on HADOOP-18647 at 1/21/24 3:40 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > x-ms-client-request-id to have some way that identifies retry of an API. > > > Key: HADOOP-18647 > URL: https://issues.apache.org/jira/browse/HADOOP-18647 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.5 >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > In case primaryRequestId in x-ms-client-request-id is empty-string, the > retry's primaryRequestId has to contain last part of clientRequestId UUID. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18647) x-ms-client-request-id to have some way that identifies retry of an API.
[ https://issues.apache.org/jira/browse/HADOOP-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18647: Fix Version/s: 3.4.0 > x-ms-client-request-id to have some way that identifies retry of an API. > > > Key: HADOOP-18647 > URL: https://issues.apache.org/jira/browse/HADOOP-18647 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.5 >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > In case primaryRequestId in x-ms-client-request-id is empty-string, the > retry's primaryRequestId has to contain last part of clientRequestId UUID. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18646) Upgrade Netty to 4.1.89.Final
[ https://issues.apache.org/jira/browse/HADOOP-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18646: Fix Version/s: 3.4.0 > Upgrade Netty to 4.1.89.Final > - > > Key: HADOOP-18646 > URL: https://issues.apache.org/jira/browse/HADOOP-18646 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.4 >Reporter: Aleksandr Nikolaev >Assignee: Aleksandr Nikolaev >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > h4. Netty version - 4.1.89 has fix CVEs: > [CVE-2022-41881|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41881] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18637) S3A to support upload of files greater than 2 GB using DiskBlocks
[ https://issues.apache.org/jira/browse/HADOOP-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809018#comment-17809018 ] Shilun Fan edited comment on HADOOP-18637 at 1/21/24 3:39 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > S3A to support upload of files greater than 2 GB using DiskBlocks > - > > Key: HADOOP-18637 > URL: https://issues.apache.org/jira/browse/HADOOP-18637 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Use S3A Diskblocks to support the upload of files greater than 2 GB using > DiskBlocks. Currently, the max upload size of a single block is ~2GB. > cc: [~mthakur] [~ste...@apache.org] [~mehakmeet] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18646) Upgrade Netty to 4.1.89.Final
[ https://issues.apache.org/jira/browse/HADOOP-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809017#comment-17809017 ] Shilun Fan edited comment on HADOOP-18646 at 1/21/24 3:39 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Upgrade Netty to 4.1.89.Final > - > > Key: HADOOP-18646 > URL: https://issues.apache.org/jira/browse/HADOOP-18646 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.4 >Reporter: Aleksandr Nikolaev >Assignee: Aleksandr Nikolaev >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > h4. Netty version - 4.1.89 has fix CVEs: > [CVE-2022-41881|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41881] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809019#comment-17809019
]
Shilun Fan edited comment on HADOOP-18636 at 1/21/24 3:38 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0.-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0.
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
>
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18636:
Fix Version/s: 3.4.0
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
>
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18637) S3A to support upload of files greater than 2 GB using DiskBlocks
[ https://issues.apache.org/jira/browse/HADOOP-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18637: Fix Version/s: 3.4.0 > S3A to support upload of files greater than 2 GB using DiskBlocks > - > > Key: HADOOP-18637 > URL: https://issues.apache.org/jira/browse/HADOOP-18637 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Use S3A Diskblocks to support the upload of files greater than 2 GB using > DiskBlocks. Currently, the max upload size of a single block is ~2GB. > cc: [~mthakur] [~ste...@apache.org] [~mehakmeet] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18628) Server connection should log host name before returning VersionMismatch error
[
https://issues.apache.org/jira/browse/HADOOP-18628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18628:
Fix Version/s: 3.4.0
> Server connection should log host name before returning VersionMismatch error
> -
>
> Key: HADOOP-18628
> URL: https://issues.apache.org/jira/browse/HADOOP-18628
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> In env with dynamically changing IP addresses, debugging issue with the logs
> with only IP address becomes a bit difficult at times.
> {code:java}
> 2023-02-08 23:26:50,112 WARN [Socket Reader #1 for port 8485] ipc.Server -
> Incorrect RPC Header length from {IPV4}:36556 expected length:
> java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] got length:
> java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] {code}
> It would be better to log full hostname for the given IP address rather than
> only IP address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18628) Server connection should log host name before returning VersionMismatch error
[
https://issues.apache.org/jira/browse/HADOOP-18628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809020#comment-17809020
]
Shilun Fan edited comment on HADOOP-18628 at 1/21/24 3:37 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0
> Server connection should log host name before returning VersionMismatch error
> -
>
> Key: HADOOP-18628
> URL: https://issues.apache.org/jira/browse/HADOOP-18628
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> In env with dynamically changing IP addresses, debugging issue with the logs
> with only IP address becomes a bit difficult at times.
> {code:java}
> 2023-02-08 23:26:50,112 WARN [Socket Reader #1 for port 8485] ipc.Server -
> Incorrect RPC Header length from {IPV4}:36556 expected length:
> java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] got length:
> java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] {code}
> It would be better to log full hostname for the given IP address rather than
> only IP address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18622) Upgrade ant to 1.10.13
[ https://issues.apache.org/jira/browse/HADOOP-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18622: Fix Version/s: 3.4.0 > Upgrade ant to 1.10.13 > -- > > Key: HADOOP-18622 > URL: https://issues.apache.org/jira/browse/HADOOP-18622 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Aleksandr Nikolaev >Assignee: Aleksandr Nikolaev >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > Attachments: hadoop_dep.log > > > lnerabilities reported in org.apache.ant:ant:1.10.11 > * > [CVE-2022-23437|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23437] > * > [CVE-2020-14338|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14338] > suggested: org.apache.ant:ant ~> 1.10.13 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18625) Fix method name of RPC.Builder#setnumReaders
[ https://issues.apache.org/jira/browse/HADOOP-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18625: Fix Version/s: 3.3.6 > Fix method name of RPC.Builder#setnumReaders > - > > Key: HADOOP-18625 > URL: https://issues.apache.org/jira/browse/HADOOP-18625 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 3.3.4 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons
[ https://issues.apache.org/jira/browse/HADOOP-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809026#comment-17809026 ] Shilun Fan edited comment on HADOOP-18612 at 1/21/24 3:35 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Avoid mixing canonical and non-canonical when performing comparisons > > > Key: HADOOP-18612 > URL: https://issues.apache.org/jira/browse/HADOOP-18612 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5, 3.3.9 > Environment: Tests were run using the Hadoop development environment > docker image. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > The test mixes canonical and non-canonical paths and then perform > comparisons. We can avoid unexpected failures by ensuring that comparisons > are always made against canonical forms. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18620) Avoid using grizzly-http-* APIs
[ https://issues.apache.org/jira/browse/HADOOP-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18620: Fix Version/s: 3.4.0 > Avoid using grizzly-http-* APIs > --- > > Key: HADOOP-18620 > URL: https://issues.apache.org/jira/browse/HADOOP-18620 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > As discussed on the parent Jira HADOOP-15984, we do not have any > grizzly-http-servlet version available that uses Jersey 2 dependencies. > version 2.4.4 contains Jersey 1 artifacts: > [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/2.4.4/grizzly-http-servlet-2.4.4.pom] > The next higher version available is 3.0.0-M1 and it contains Jersey 3 > artifacts: > [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/3.0.0-M1/grizzly-http-servlet-3.0.0-M1.pom] > > Moreover, we do not use grizzly-http-* modules extensively. We use them only > for few tests such that we don't have to implement all the methods of > HttpServletResponse for our custom test classes. > We should get rid of grizzly-http-servlet, grizzly-http and > grizzly-http-server artifacts of org.glassfish.grizzly and rather implement > HttpServletResponse directly to avoid having to depend on grizzly upgrades as > part of overall Jersey upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18620) Avoid using grizzly-http-* APIs
[ https://issues.apache.org/jira/browse/HADOOP-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809025#comment-17809025 ] Shilun Fan edited comment on HADOOP-18620 at 1/21/24 3:35 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Avoid using grizzly-http-* APIs > --- > > Key: HADOOP-18620 > URL: https://issues.apache.org/jira/browse/HADOOP-18620 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > As discussed on the parent Jira HADOOP-15984, we do not have any > grizzly-http-servlet version available that uses Jersey 2 dependencies. > version 2.4.4 contains Jersey 1 artifacts: > [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/2.4.4/grizzly-http-servlet-2.4.4.pom] > The next higher version available is 3.0.0-M1 and it contains Jersey 3 > artifacts: > [https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/3.0.0-M1/grizzly-http-servlet-3.0.0-M1.pom] > > Moreover, we do not use grizzly-http-* modules extensively. We use them only > for few tests such that we don't have to implement all the methods of > HttpServletResponse for our custom test classes. > We should get rid of grizzly-http-servlet, grizzly-http and > grizzly-http-server artifacts of org.glassfish.grizzly and rather implement > HttpServletResponse directly to avoid having to depend on grizzly upgrades as > part of overall Jersey upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons
[ https://issues.apache.org/jira/browse/HADOOP-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18612: Fix Version/s: 3.4.0 > Avoid mixing canonical and non-canonical when performing comparisons > > > Key: HADOOP-18612 > URL: https://issues.apache.org/jira/browse/HADOOP-18612 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5, 3.3.9 > Environment: Tests were run using the Hadoop development environment > docker image. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > The test mixes canonical and non-canonical paths and then perform > comparisons. We can avoid unexpected failures by ensuring that comparisons > are always made against canonical forms. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18606: Fix Version/s: 3.4.0 > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809027#comment-17809027 ] Shilun Fan edited comment on HADOOP-18606 at 1/21/24 3:34 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18604) Add compile platform in the hadoop version output
[ https://issues.apache.org/jira/browse/HADOOP-18604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809028#comment-17809028 ] Shilun Fan edited comment on HADOOP-18604 at 1/21/24 3:33 PM: -- -3.3.6 release has been fixed, fix version removed 3.4.0- was (Author: slfan1989): 3.3.6 release has been fixed, fix version removed 3.4.0 > Add compile platform in the hadoop version output > - > > Key: HADOOP-18604 > URL: https://issues.apache.org/jira/browse/HADOOP-18604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Hadoop releases support both x86 and Aarch64, good to have a line indicating > this in the hadoop version output. > Inspired by: HDDS-7783 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18604) Add compile platform in the hadoop version output
[ https://issues.apache.org/jira/browse/HADOOP-18604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18604: Fix Version/s: 3.4.0 > Add compile platform in the hadoop version output > - > > Key: HADOOP-18604 > URL: https://issues.apache.org/jira/browse/HADOOP-18604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Hadoop releases support both x86 and Aarch64, good to have a line indicating > this in the hadoop version output. > Inspired by: HDDS-7783 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18597) Simplify single node instructions for creating directories for Map Reduce
[
https://issues.apache.org/jira/browse/HADOOP-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18597:
Fix Version/s: 3.4.0
> Simplify single node instructions for creating directories for Map Reduce
> -
>
> Key: HADOOP-18597
> URL: https://issues.apache.org/jira/browse/HADOOP-18597
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.4
>Reporter: Nikita Eshkeev
>Assignee: Nikita Eshkeev
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> The {{mkdir}} command supports the {{-p}} option which instructs {{hdfs}} to
> create all the parent directories if needed. The single nose setup
> instructions now ask a user to create both /user and /user/
> directories explicitly, which can be simplified to creating just the
> /user/ with the help from the -p option of mkdir
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809030#comment-17809030
]
Shilun Fan edited comment on HADOOP-18592 at 1/21/24 3:32 PM:
--
-3.3.6 release has been fixed, fix version removed 3.4.0.-
was (Author: slfan1989):
3.3.6 release has been fixed, fix version removed 3.4.0.
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18592:
Fix Version/s: 3.4.0
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18590) Publish SBOM artifacts
[ https://issues.apache.org/jira/browse/HADOOP-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18590: Fix Version/s: 3.4.0 > Publish SBOM artifacts > -- > > Key: HADOOP-18590 > URL: https://issues.apache.org/jira/browse/HADOOP-18590 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.6 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
