[jira] [Commented] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[ https://issues.apache.org/jira/browse/HADOOP-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309177#comment-17309177 ] Akira Ajisaka commented on HADOOP-16870: Thank you [~ahussein] for the suggestion. The migration is a big change, so I've committed this to a few branches. I found the spotbugs check is working well, and we need to backport to the rest of the branches. bq. Since spotbugs require JDK1.8.0+ run, do you think we should remove --findbugs-strict-precheck for branch-2.10? After the backport, Spotbugs is run with only JDK 1.8 in the hadoop-multibranch job, so I think we don't have to remove the flag. Now JDK 1.7 is used in only compile check, and findbugs is not run with JDK 1.7. It is configured in https://github.com/apache/hadoop/blob/branch-2.10/dev-support/Jenkinsfile#L162 > Use spotbugs-maven-plugin instead of findbugs-maven-plugin > -- > > Key: HADOOP-16870 > URL: https://issues.apache.org/jira/browse/HADOOP-16870 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2819: HDFS-15923. Authentication failed when rename accross sub clusters.
hadoop-yetus commented on pull request #2819:
URL: https://github.com/apache/hadoop/pull/2819#issuecomment-807928592
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 29m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2819/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 23m 44s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2819/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 134m 52s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterFederationRename |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2819/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2819 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 83666180296e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 22f65bf61522127f940c3cc7a6aed538d61c07d1 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2819/1/testReport/ |
| Max. process+thread count | 2061 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[
https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309155#comment-17309155
]
Yongjun Zhang commented on HADOOP-17338:
Sorry [~ste...@apache.org] I missed your message above. Have you created a new
jira? thanks.
> Intermittent S3AInputStream failures: Premature end of Content-Length
> delimited message body etc
>
>
> Key: HADOOP-17338
> URL: https://issues.apache.org/jira/browse/HADOOP-17338
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 2.10.2
>
> Attachments: HADOOP-17338.001.patch
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> We are seeing the following two kinds of intermittent exceptions when using
> S3AInputSteam:
> 1.
> {code:java}
> Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException:
> Premature end of Content-Length delimited message body (expected: 156463674;
> received: 150001089
> at
> com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
> at
> com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
> com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
> com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779)
> at
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 15 more
> {code}
> 2.
> {code:java}
> Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
> at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596)
> at sun.security.ssl.InputRecord.read(InputRecord.java:532)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990)
> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948)
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
> at
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
> at
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198)
> at
> com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
> at
> com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
> at
[jira] [Commented] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[
https://issues.apache.org/jira/browse/HADOOP-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309148#comment-17309148
]
Ahmed Hussein commented on HADOOP-16870:
Since spotbugs require JDK1.8.0+ run, do you think we should remove
{{--findbugs-strict-precheck}} for branch-2.10?
> Use spotbugs-maven-plugin instead of findbugs-maven-plugin
> --
>
> Key: HADOOP-16870
> URL: https://issues.apache.org/jira/browse/HADOOP-16870
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengzhuobinzzb opened a new pull request #2819: HDFS-15923. Authentication failed when rename accross sub clusters.
zhengzhuobinzzb opened a new pull request #2819: URL: https://github.com/apache/hadoop/pull/2819 Rename accross subcluster with RBF and Kerberos environment. Will encounter the following two errors: Save Object to journal. Precheck try to get src file status So, we need use Proxy UGI doAs create DistcpProcedure and TrashProcedure and submit Job. In patch i use proxy ugi doAs above method. It worked. But there are another strange thing and this patch not solve: Router use ugi itself to submit the Distcp job. But not user ugi or proxy ugi. This may cause excessive distcp permissions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2784: HDFS-15850. Superuser actions should be reported to external enforcers
hadoop-yetus commented on pull request #2784:
URL: https://github.com/apache/hadoop/pull/2784#issuecomment-807877116
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 48s | | trunk passed |
| +1 :green_heart: | compile | 5m 53s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 5m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | | the patch passed |
| +1 :green_heart: | compile | 6m 12s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 6m 12s | | the patch passed |
| +1 :green_heart: | compile | 5m 6s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 5m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | hadoop-hdfs-project: The
patch generated 0 new + 498 unchanged - 6 fixed = 498 total (was 504) |
| +1 :green_heart: | mvnsite | 2m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 3m 47s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/8/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 17m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 233m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 32s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2784/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch failed. |
| +0 :ok: | asflicense | 0m 32s | | ASF License check generated no
output? |
| | | 363m 23s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Possible null pointer dereference of r in
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.truncate(String, long,
String, String, long) Dereferenced at FSNamesystem.java:r in
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.truncate(String, long,
String, String, long) Dereferenced at FSNamesystem.java:[line 2325] |
| Failed junit tests | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
[jira] [Commented] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[
https://issues.apache.org/jira/browse/HADOOP-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309077#comment-17309077
]
Ahmed Hussein commented on HADOOP-16870:
I believe that HADOOP-16870 is missing in branch-2.10.
It causes an error
{code:bash}
ERROR: Unprocessed flag(s): --spotbugs-strict-precheck
{code}
We need to replace {{findbugs}} with {{spotbugs}}
> Use spotbugs-maven-plugin instead of findbugs-maven-plugin
> --
>
> Key: HADOOP-16870
> URL: https://issues.apache.org/jira/browse/HADOOP-16870
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[
https://issues.apache.org/jira/browse/HADOOP-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309077#comment-17309077
]
Ahmed Hussein edited comment on HADOOP-16870 at 3/26/21, 1:54 AM:
--
I believe that HADOOP-16870 is missing in branch-2.10.
It causes an error
{code:bash}
ERROR: Unprocessed flag(s): --spotbugs-strict-precheck
{code}
We need to replace {{findbugs}} with {{spotbugs}} in branch-2.10/jenkinsfile
was (Author: ahussein):
I believe that HADOOP-16870 is missing in branch-2.10.
It causes an error
{code:bash}
ERROR: Unprocessed flag(s): --spotbugs-strict-precheck
{code}
We need to replace {{findbugs}} with {{spotbugs}}
> Use spotbugs-maven-plugin instead of findbugs-maven-plugin
> --
>
> Key: HADOOP-16870
> URL: https://issues.apache.org/jira/browse/HADOOP-16870
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2748: HDFS-15879. Exclude slow nodes when choose targets for blocks
tomscut commented on a change in pull request #2748:
URL: https://github.com/apache/hadoop/pull/2748#discussion_r601954791
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.TestBlockStoragePolicy;
+import org.apache.hadoop.hdfs.server.namenode.NameNode;
+import org.apache.hadoop.net.Node;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Set;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(Parameterized.class)
+public class TestReplicationPolicyExcludeSlowNodes
+extends BaseReplicationPolicyTest {
+
+ public TestReplicationPolicyExcludeSlowNodes(String blockPlacementPolicy) {
+this.blockPlacementPolicy = blockPlacementPolicy;
+ }
+
+ @Parameterized.Parameters
+ public static Iterable data() {
+return Arrays.asList(new Object[][] {
+{ BlockPlacementPolicyDefault.class.getName() },
+{ BlockPlacementPolicyWithUpgradeDomain.class.getName() } });
+ }
+
+ @Override
+ DatanodeDescriptor[] getDatanodeDescriptors(Configuration conf) {
+conf.setBoolean(DFSConfigKeys
+.DFS_DATANODE_PEER_STATS_ENABLED_KEY,
+true);
+conf.setStrings(DFSConfigKeys
+.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
+"1s");
+conf.setBoolean(DFSConfigKeys
+.DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
+true);
+final String[] racks = {
+"/rack1",
+"/rack1",
+"/rack2",
+"/rack2",
+"/rack3",
+"/rack3"};
Review comment:
Hi @tasanuma , this PR is mainly for filtering slow nodes, which can set
the maximum number of slow nodes to be filtered, and we can set reasonable
parameters for the cluster size.
But if we want to add the RackFaultTolerant policy and pass the unit test,
we can also put the 6 DNs on 6 different racks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2693: Hadoop 16524 - resubmission following some unit test fixes
hadoop-yetus commented on pull request #2693:
URL: https://github.com/apache/hadoop/pull/2693#issuecomment-807831721
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 26s | | trunk passed |
| +1 :green_heart: | compile | 28m 38s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 9s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 16m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 9s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 9m 22s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 40m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 15m 36s | | the patch passed |
| +1 :green_heart: | compile | 31m 14s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 31m 14s | | the patch passed |
| +1 :green_heart: | compile | 20m 47s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 20m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 59s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2693/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 169 new + 92 unchanged - 11 fixed = 261 total
(was 103) |
| +1 :green_heart: | mvnsite | 12m 25s | | the patch passed |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 36s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 7m 57s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 32m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 22s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 0m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2693/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch failed. |
| -1 :x: | unit | 4m 38s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2693/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch failed. |
| -1 :x: | unit | 0m 53s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2693/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | unit | 4m 40s |
[/patch-unit-hadoop-yarn-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2693/5/artifact/out/patch-unit-hadoop-yarn-project.txt)
| hadoop-yarn-project in the patch failed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 376m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2693/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2693 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite
[jira] [Work logged] (HADOOP-17536) Suport for customer provided encrption key
[
https://issues.apache.org/jira/browse/HADOOP-17536?focusedWorklogId=572280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572280
]
ASF GitHub Bot logged work on HADOOP-17536:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 20:16
Start Date: 25/Mar/21 20:16
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#issuecomment-807407790
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 3s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 25s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2707/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 3 new + 7 unchanged - 0
fixed = 10 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 57s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 75m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2707/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2707 |
| Optional Tests | dupname asflicense mvnsite unit codespell shellcheck
shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle xml
|
| uname | Linux faa57a6d0dfc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 173bc3c7fef8543b8ab2602c1dde14c047fcb77f |
| Default Java | Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #2707: HADOOP-17536. ABFS: Supporting customer provided encryption key
hadoop-yetus commented on pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#issuecomment-807407790
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 3s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 25s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2707/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 3 new + 7 unchanged - 0
fixed = 10 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 57s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 75m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2707/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2707 |
| Optional Tests | dupname asflicense mvnsite unit codespell shellcheck
shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle xml
|
| uname | Linux faa57a6d0dfc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 173bc3c7fef8543b8ab2602c1dde14c047fcb77f |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2707/6/testReport/ |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #2818: HDFS-15922. Use memcpy for copying non-null terminated string in jni_helper.c
hadoop-yetus commented on pull request #2818:
URL: https://github.com/apache/hadoop/pull/2818#issuecomment-807347586
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 8s | | trunk passed |
| +1 :green_heart: | compile | 2m 42s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 44s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 52m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | cc | 2m 28s | |
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 60 unchanged
- 27 fixed = 60 total (was 87) |
| +1 :green_heart: | golang | 2m 28s | | the patch passed |
| +1 :green_heart: | javac | 2m 28s | | the patch passed |
| +1 :green_heart: | compile | 2m 33s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | cc | 2m 33s |
[/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/2/artifact/out/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt)
|
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 9 new
+ 51 unchanged - 36 fixed = 60 total (was 87) |
| +1 :green_heart: | golang | 2m 33s | | the patch passed |
| +1 :green_heart: | javac | 2m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 108m 25s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 183m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2818 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux db21b0266588 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3012b5101299fb3c175d5cc76130dc1e16014964 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/2/testReport/ |
| Max. process+thread count | 541 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To
[GitHub] [hadoop] hadoop-yetus commented on pull request #2818: HDFS-15922. Use memcpy for copying non-null terminated string in jni_helper.c
hadoop-yetus commented on pull request #2818:
URL: https://github.com/apache/hadoop/pull/2818#issuecomment-807326538
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 18s | | trunk passed |
| +1 :green_heart: | compile | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 2m 50s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 57m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 2m 39s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | cc | 2m 39s |
[/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/1/artifact/out/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 3 new + 69 unchanged
- 18 fixed = 72 total (was 87) |
| +1 :green_heart: | golang | 2m 39s | | the patch passed |
| +1 :green_heart: | javac | 2m 39s | | the patch passed |
| +1 :green_heart: | compile | 2m 40s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | cc | 2m 40s |
[/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/1/artifact/out/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt)
|
hadoop-hdfs-project_hadoop-hdfs-native-client-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 5 new
+ 67 unchanged - 20 fixed = 72 total (was 87) |
| +1 :green_heart: | golang | 2m 40s | | the patch passed |
| +1 :green_heart: | javac | 2m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 108m 32s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 191m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2818 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux b5a23a90c1f2 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9858cfcb5bc22ab53f92ebd8f5e3235a4d0583d8 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2818/1/testReport/ |
| Max. process+thread count | 623 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
[jira] [Updated] (HADOOP-17133) Implement HttpServer2 metrics
[ https://issues.apache.org/jira/browse/HADOOP-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-17133: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) I've merged the PR. Thank you for the work, [~aajisaka]! > Implement HttpServer2 metrics > - > > Key: HADOOP-17133 > URL: https://issues.apache.org/jira/browse/HADOOP-17133 > Project: Hadoop Common > Issue Type: Improvement > Components: httpfs, kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > I'd like to collect metrics (number of connections, average response time, > etc...) from HttpFS and KMS but there are no metrics for HttpServer2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17133) Implement HttpServer2 metrics
[ https://issues.apache.org/jira/browse/HADOOP-17133?focusedWorklogId=572234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572234 ] ASF GitHub Bot logged work on HADOOP-17133: --- Author: ASF GitHub Bot Created on: 25/Mar/21 19:09 Start Date: 25/Mar/21 19:09 Worklog Time Spent: 10m Work Description: Jing9 merged pull request #2145: URL: https://github.com/apache/hadoop/pull/2145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 572234) Time Spent: 1h 40m (was: 1.5h) > Implement HttpServer2 metrics > - > > Key: HADOOP-17133 > URL: https://issues.apache.org/jira/browse/HADOOP-17133 > Project: Hadoop Common > Issue Type: Improvement > Components: httpfs, kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > I'd like to collect metrics (number of connections, average response time, > etc...) from HttpFS and KMS but there are no metrics for HttpServer2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17536) Suport for customer provided encrption key
[
https://issues.apache.org/jira/browse/HADOOP-17536?focusedWorklogId=572233=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572233
]
ASF GitHub Bot logged work on HADOOP-17536:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 19:09
Start Date: 25/Mar/21 19:09
Worklog Time Spent: 10m
Work Description: bilaharith commented on a change in pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#discussion_r601769992
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClientCustomerProvidedKey.java
##
@@ -0,0 +1,623 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.CharBuffer;
+import java.nio.charset.CharacterCodingException;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.EnumSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assume;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.XAttrSetFlag;
+import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters.Mode;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformer;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformerInterface;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsAclHelper;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+import org.apache.hadoop.fs.azurebfs.services.AbfsHttpHeader;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.permission.AclEntry;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Lists;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_IDENTITY_TRANSFORM_CLASS;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_CLIENT_PROVIDED_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_ALGORITHM;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY_SHA256;
+import static
org.apache.hadoop.fs.azurebfs.constants.TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT;
+import static org.apache.hadoop.fs.azurebfs.utils.AclTestHelpers.aclEntry;
+import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
+import static org.apache.hadoop.fs.permission.AclEntryType.USER;
+import static org.apache.hadoop.fs.permission.FsAction.ALL;
+
+public class ITestAbfsClientCustomerProvidedKey
+extends AbstractAbfsIntegrationTest {
+
+ private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1";
+ private static final int INT_512 = 512;
+ private static final int INT_50 = 50;
+
+ private final IdentityTransformerInterface identityTransformer;
Review comment:
Removed
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
[GitHub] [hadoop] Jing9 merged pull request #2145: HADOOP-17133. Implement HttpServer2 metrics
Jing9 merged pull request #2145: URL: https://github.com/apache/hadoop/pull/2145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17536) Suport for customer provided encrption key
[
https://issues.apache.org/jira/browse/HADOOP-17536?focusedWorklogId=572232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572232
]
ASF GitHub Bot logged work on HADOOP-17536:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 19:08
Start Date: 25/Mar/21 19:08
Worklog Time Spent: 10m
Work Description: bilaharith commented on a change in pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#discussion_r601769765
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClientCustomerProvidedKey.java
##
@@ -0,0 +1,623 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.CharBuffer;
+import java.nio.charset.CharacterCodingException;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.EnumSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assume;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.XAttrSetFlag;
+import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters.Mode;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformer;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformerInterface;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsAclHelper;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+import org.apache.hadoop.fs.azurebfs.services.AbfsHttpHeader;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.permission.AclEntry;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Lists;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_IDENTITY_TRANSFORM_CLASS;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_CLIENT_PROVIDED_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_ALGORITHM;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY_SHA256;
+import static
org.apache.hadoop.fs.azurebfs.constants.TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT;
+import static org.apache.hadoop.fs.azurebfs.utils.AclTestHelpers.aclEntry;
+import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
+import static org.apache.hadoop.fs.permission.AclEntryType.USER;
+import static org.apache.hadoop.fs.permission.FsAction.ALL;
+
+public class ITestAbfsClientCustomerProvidedKey
+extends AbstractAbfsIntegrationTest {
+
+ private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1";
+ private static final int INT_512 = 512;
+ private static final int INT_50 = 50;
+
+ private final IdentityTransformerInterface identityTransformer;
+
+ public ITestAbfsClientCustomerProvidedKey() throws Exception {
+final Class
identityTransformerClass = getRawConfiguration()
+
[jira] [Work logged] (HADOOP-17536) Suport for customer provided encrption key
[
https://issues.apache.org/jira/browse/HADOOP-17536?focusedWorklogId=572231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572231
]
ASF GitHub Bot logged work on HADOOP-17536:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 19:08
Start Date: 25/Mar/21 19:08
Worklog Time Spent: 10m
Work Description: bilaharith commented on pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#issuecomment-807302693
> I can see the need for this.
>
> * One thing that it's not yet covered is the need to get this into
delegation tokens. Look at what was done for the S3A DT's there: the encryption
settings and secret key is picked up on the client and passed into the cluster
so users can submit jobs into a shared cluster, _without that shared cluster
having the encryption key_. Is this needed here? If so
AbfsDelegationTokenIdentifier will need to be extended.
> * is it ever the case that different stores will have different keys?
Yes, each account can have it's own key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 572231)
Time Spent: 1h 10m (was: 1h)
> Suport for customer provided encrption key
> --
>
> Key: HADOOP-17536
> URL: https://issues.apache.org/jira/browse/HADOOP-17536
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.4.0
>Reporter: Bilahari T H
>Assignee: Bilahari T H
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The data for a particular customer needs to be encrypted on account level. At
> server side the APIs will start accepting the encryption key as part of
> request headers. The data will be encrypted/decrypted with the given key at
> the server.
> Since the ABFS FileSystem APIs are implementations for Hadoop FileSystem APIs
> there is no direct way with which customer can pass the key to ABFS driver.
> In this case driver should have the following capabilities so that it can
> accept and pass the encryption key as one of the request headers.
> # There should be a way to configure the encryption key for different
> accounts.
> # If there is a key specified for a particular account, the same needs to be
> sent along with the request headers.
> *Config changes*
> They key for an account can be specified in the core-site as follows.
> fs.azure.account.client-provided-encryption-key.{account
> name}.dfs.core.windows.net
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2707: HADOOP-17536. ABFS: Supporting customer provided encryption key
bilaharith commented on a change in pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#discussion_r601769992
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClientCustomerProvidedKey.java
##
@@ -0,0 +1,623 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.CharBuffer;
+import java.nio.charset.CharacterCodingException;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.EnumSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assume;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.XAttrSetFlag;
+import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters.Mode;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformer;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformerInterface;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsAclHelper;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+import org.apache.hadoop.fs.azurebfs.services.AbfsHttpHeader;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.permission.AclEntry;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Lists;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_IDENTITY_TRANSFORM_CLASS;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_CLIENT_PROVIDED_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_ALGORITHM;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY_SHA256;
+import static
org.apache.hadoop.fs.azurebfs.constants.TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT;
+import static org.apache.hadoop.fs.azurebfs.utils.AclTestHelpers.aclEntry;
+import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
+import static org.apache.hadoop.fs.permission.AclEntryType.USER;
+import static org.apache.hadoop.fs.permission.FsAction.ALL;
+
+public class ITestAbfsClientCustomerProvidedKey
+extends AbstractAbfsIntegrationTest {
+
+ private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1";
+ private static final int INT_512 = 512;
+ private static final int INT_50 = 50;
+
+ private final IdentityTransformerInterface identityTransformer;
Review comment:
Removed
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -159,6 +185,18 @@ SharedKeyCredentials getSharedKeyCredentials() {
return requestHeaders;
}
+ private void addServerSideEncryptionHeaders(
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[GitHub] [hadoop] bilaharith commented on a change in pull request #2707: HADOOP-17536. ABFS: Supporting customer provided encryption key
bilaharith commented on a change in pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#discussion_r601769765
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClientCustomerProvidedKey.java
##
@@ -0,0 +1,623 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.CharBuffer;
+import java.nio.charset.CharacterCodingException;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.EnumSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assume;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.XAttrSetFlag;
+import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import
org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters.Mode;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformer;
+import org.apache.hadoop.fs.azurebfs.oauth2.IdentityTransformerInterface;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsAclHelper;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+import org.apache.hadoop.fs.azurebfs.services.AbfsHttpHeader;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.permission.AclEntry;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Lists;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_IDENTITY_TRANSFORM_CLASS;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_CLIENT_PROVIDED_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_ALGORITHM;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY;
+import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_ENCRYPTION_KEY_SHA256;
+import static
org.apache.hadoop.fs.azurebfs.constants.TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT;
+import static org.apache.hadoop.fs.azurebfs.utils.AclTestHelpers.aclEntry;
+import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
+import static org.apache.hadoop.fs.permission.AclEntryType.USER;
+import static org.apache.hadoop.fs.permission.FsAction.ALL;
+
+public class ITestAbfsClientCustomerProvidedKey
+extends AbstractAbfsIntegrationTest {
+
+ private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1";
+ private static final int INT_512 = 512;
+ private static final int INT_50 = 50;
+
+ private final IdentityTransformerInterface identityTransformer;
+
+ public ITestAbfsClientCustomerProvidedKey() throws Exception {
+final Class
identityTransformerClass = getRawConfiguration()
+.getClass(FS_AZURE_IDENTITY_TRANSFORM_CLASS, IdentityTransformer.class,
+IdentityTransformerInterface.class);
+try {
+ this.identityTransformer = identityTransformerClass
+ .getConstructor(Configuration.class)
+ .newInstance(getRawConfiguration());
+} catch (IllegalAccessException | InstantiationException |
IllegalArgumentException | InvocationTargetException | NoSuchMethodException e)
{
+
[GitHub] [hadoop] bilaharith commented on pull request #2707: HADOOP-17536. ABFS: Supporting customer provided encryption key
bilaharith commented on pull request #2707: URL: https://github.com/apache/hadoop/pull/2707#issuecomment-807302693 > I can see the need for this. > > * One thing that it's not yet covered is the need to get this into delegation tokens. Look at what was done for the S3A DT's there: the encryption settings and secret key is picked up on the client and passed into the cluster so users can submit jobs into a shared cluster, _without that shared cluster having the encryption key_. Is this needed here? If so AbfsDelegationTokenIdentifier will need to be extended. > * is it ever the case that different stores will have different keys? Yes, each account can have it's own key. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17536) Suport for customer provided encrption key
[
https://issues.apache.org/jira/browse/HADOOP-17536?focusedWorklogId=572228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572228
]
ASF GitHub Bot logged work on HADOOP-17536:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 19:07
Start Date: 25/Mar/21 19:07
Worklog Time Spent: 10m
Work Description: bilaharith commented on a change in pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#discussion_r601768977
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -88,6 +96,21 @@ private AbfsClient(final URL baseUrl, final
SharedKeyCredentials sharedKeyCreden
this.accountName = abfsConfiguration.getAccountName().substring(0,
abfsConfiguration.getAccountName().indexOf(AbfsHttpConstants.DOT));
this.authType = abfsConfiguration.getAuthType(accountName);
+String encryptionKey =
this.abfsConfiguration.getClientProvidedEncryptionKey();
+if (encryptionKey != null) {
+ this.clientProvidedEncryptionKey = encryptionKey;
+ try {
+MessageDigest digester = MessageDigest.getInstance("SHA-256");
+digester.update(encryptionKey.getBytes());
+this.clientProvidedEncryptionKeySHA =
Base64.getEncoder().encodeToString(digester.digest());
+ } catch (NoSuchAlgorithmException e) {
+throw new IOException(e);
Review comment:
Looks like this is a common Exception, do we need to mention explicitly?
https://stackoverflow.com/questions/48040299/java-security-nosuchalgorithmexception-sha3-384-messagedigest-not-available-w
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 572228)
Time Spent: 1h (was: 50m)
> Suport for customer provided encrption key
> --
>
> Key: HADOOP-17536
> URL: https://issues.apache.org/jira/browse/HADOOP-17536
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.4.0
>Reporter: Bilahari T H
>Assignee: Bilahari T H
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The data for a particular customer needs to be encrypted on account level. At
> server side the APIs will start accepting the encryption key as part of
> request headers. The data will be encrypted/decrypted with the given key at
> the server.
> Since the ABFS FileSystem APIs are implementations for Hadoop FileSystem APIs
> there is no direct way with which customer can pass the key to ABFS driver.
> In this case driver should have the following capabilities so that it can
> accept and pass the encryption key as one of the request headers.
> # There should be a way to configure the encryption key for different
> accounts.
> # If there is a key specified for a particular account, the same needs to be
> sent along with the request headers.
> *Config changes*
> They key for an account can be specified in the core-site as follows.
> fs.azure.account.client-provided-encryption-key.{account
> name}.dfs.core.windows.net
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2707: HADOOP-17536. ABFS: Supporting customer provided encryption key
bilaharith commented on a change in pull request #2707:
URL: https://github.com/apache/hadoop/pull/2707#discussion_r601768977
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -88,6 +96,21 @@ private AbfsClient(final URL baseUrl, final
SharedKeyCredentials sharedKeyCreden
this.accountName = abfsConfiguration.getAccountName().substring(0,
abfsConfiguration.getAccountName().indexOf(AbfsHttpConstants.DOT));
this.authType = abfsConfiguration.getAuthType(accountName);
+String encryptionKey =
this.abfsConfiguration.getClientProvidedEncryptionKey();
+if (encryptionKey != null) {
+ this.clientProvidedEncryptionKey = encryptionKey;
+ try {
+MessageDigest digester = MessageDigest.getInstance("SHA-256");
+digester.update(encryptionKey.getBytes());
+this.clientProvidedEncryptionKeySHA =
Base64.getEncoder().encodeToString(digester.digest());
+ } catch (NoSuchAlgorithmException e) {
+throw new IOException(e);
Review comment:
Looks like this is a common Exception, do we need to mention explicitly?
https://stackoverflow.com/questions/48040299/java-security-nosuchalgorithmexception-sha3-384-messagedigest-not-available-w
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17467) netgroup-user is not added to Groups.cache
[
https://issues.apache.org/jira/browse/HADOOP-17467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308914#comment-17308914
]
Ahmed Hussein commented on HADOOP-17467:
I see from the discussion in HADOOP-17079 that there are disagreements.
Therefore, I suggest that we fix the broken providers and file a separate Jira
that to address reverting the unnecessary providers changes.
WDYT [~daryn], [~Jim_Brennan], [~xyao] ?
> netgroup-user is not added to Groups.cache
> --
>
> Key: HADOOP-17467
> URL: https://issues.apache.org/jira/browse/HADOOP-17467
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.4.0
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> After the optimization in HADOOP-17079, both
> {{JniBasedUnixGroupsNetgroupMapping}} and
> {{ShellBasedUnixGroupsNetgroupMapping}} do not implement {{getGroupSet}}.
> As a result, {{Groups.load()}} load the cache calling {{fetchGroupSet}} which
> yield to the superclass {{JniBasedUnixGroupsMapping}} /
> {{ShellBasedUnixGroupsMapping}}.
> In other words, the groups mapping will never fetch from {{NetgroupCache}}.
> This alters the behavior of the implementation. Is there a reason to bypass
> loading. CC: [~xyao]
> This jira is to add missing implementation {{getGroupSet}} to
> {{JniBasedUnixGroupsNetgroupMapping}} and
> {{ShellBasedUnixGroupsNetgroupMapping}} .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17222) Create socket address leveraging URI cache
[
https://issues.apache.org/jira/browse/HADOOP-17222?focusedWorklogId=572207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-572207
]
ASF GitHub Bot logged work on HADOOP-17222:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 18:57
Start Date: 25/Mar/21 18:57
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2817:
URL: https://github.com/apache/hadoop/pull/2817#issuecomment-807284978
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 45s | | branch-3.3 passed |
| +1 :green_heart: | compile | 17m 27s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 44s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 4m 16s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 4m 8s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 8m 13s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 18m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 16m 37s | | the patch passed |
| +1 :green_heart: | javac | 16m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 37s | | root: The patch generated
0 new + 137 unchanged - 2 fixed = 137 total (was 139) |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 4m 7s | | the patch passed |
| +1 :green_heart: | spotbugs | 8m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 9s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 186m 44s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 354m 28s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2817 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 12842ed6431b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / e3ecfc277fa4243b04073af5d010a1fefc64afe7 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/testReport/ |
| Max. process+thread count | 3252 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2817: HADOOP-17222. Create socket address leveraging URI cache
hadoop-yetus commented on pull request #2817:
URL: https://github.com/apache/hadoop/pull/2817#issuecomment-807284978
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 45s | | branch-3.3 passed |
| +1 :green_heart: | compile | 17m 27s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 44s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 4m 16s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 4m 8s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 8m 13s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 18m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 16m 37s | | the patch passed |
| +1 :green_heart: | javac | 16m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 37s | | root: The patch generated
0 new + 137 unchanged - 2 fixed = 137 total (was 139) |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 4m 7s | | the patch passed |
| +1 :green_heart: | spotbugs | 8m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 9s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 186m 44s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 354m 28s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2817 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 12842ed6431b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / e3ecfc277fa4243b04073af5d010a1fefc64afe7 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/testReport/ |
| Max. process+thread count | 3252 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2817/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this
[GitHub] [hadoop] functioner commented on a change in pull request #2737: HDFS-15869. Network issue while FSEditLogAsync is executing RpcEdit.logSyncNotify can cause the namenode to hang
functioner commented on a change in pull request #2737: URL: https://github.com/apache/hadoop/pull/2737#discussion_r601739047 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java ## @@ -63,6 +68,9 @@ DFS_NAMENODE_EDITS_ASYNC_LOGGING_PENDING_QUEUE_SIZE_DEFAULT); editPendingQ = new ArrayBlockingQueue<>(editPendingQSize); + +// the thread pool size should be configurable later, and justified with a rationale +logSyncNotifyExecutor = Executors.newFixedThreadPool(10); Review comment: Sure. What should be the default value? Many users use the default value, so probably we shouldn't set it as 0 by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra opened a new pull request #2818: HDFS-15922. Replace strncpy with memcpy
GauthamBanasandra opened a new pull request #2818: URL: https://github.com/apache/hadoop/pull/2818 * strncpy reports a warning if the destination string isn't null terminated. * The scenario here is that the string is deliberately not null terminated since we want to imperatively suffix a PATH_SEPARATOR at the end. * Thus, the warning reported by strncpy even though valid, isn't applicable. * Hence we replace strncpy with memcpy which doesn't worry if the string is null terminated or not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17602) Upgrade JUnit to 4.13.1
[ https://issues.apache.org/jira/browse/HADOOP-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308747#comment-17308747 ] Ahmed Hussein commented on HADOOP-17602: Thank you [~aajisaka]! > Upgrade JUnit to 4.13.1 > --- > > Key: HADOOP-17602 > URL: https://issues.apache.org/jira/browse/HADOOP-17602 > Project: Hadoop Common > Issue Type: Bug > Components: build, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3 > > Attachments: HADOOP-17602.001.patch, > HADOOP-17602.branch-2.10.001.patch > > > A reported vulnerability reported in JUnit4.7-4.13. > The JUnit4 test rule [TemporaryFolder on unix-like systems does not limit > access to created > files|https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17531) DistCp: Reduce memory usage on copying huge directories
[ https://issues.apache.org/jira/browse/HADOOP-17531?focusedWorklogId=571936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571936 ] ASF GitHub Bot logged work on HADOOP-17531: --- Author: ASF GitHub Bot Created on: 25/Mar/21 15:05 Start Date: 25/Mar/21 15:05 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #2808: URL: https://github.com/apache/hadoop/pull/2808#issuecomment-806913688 @aajisaka does that make sense? Let me know your thoughts, Plan to conclude this by tomorrow EOD. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 571936) Time Spent: 8h 50m (was: 8h 40m) > DistCp: Reduce memory usage on copying huge directories > --- > > Key: HADOOP-17531 > URL: https://issues.apache.org/jira/browse/HADOOP-17531 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: MoveToStackIterator.patch, gc-NewD-512M-3.8ML.log > > Time Spent: 8h 50m > Remaining Estimate: 0h > > Presently distCp, uses the producer-consumer kind of setup while building the > listing, the input queue and output queue are both unbounded, thus the > listStatus grows quite huge. > Rel Code Part : > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L635 > This goes on bredth-first traversal kind of stuff(uses queue instead of > earlier stack), so if you have files at lower depth, it will like open up the > entire tree and the start processing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
ayushtkn commented on pull request #2808: URL: https://github.com/apache/hadoop/pull/2808#issuecomment-806913688 @aajisaka does that make sense? Let me know your thoughts, Plan to conclude this by tomorrow EOD. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] linyiqun commented on a change in pull request #2737: HDFS-15869. Network issue while FSEditLogAsync is executing RpcEdit.logSyncNotify can cause the namenode to hang
linyiqun commented on a change in pull request #2737: URL: https://github.com/apache/hadoop/pull/2737#discussion_r601568107 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java ## @@ -63,6 +68,9 @@ DFS_NAMENODE_EDITS_ASYNC_LOGGING_PENDING_QUEUE_SIZE_DEFAULT); editPendingQ = new ArrayBlockingQueue<>(editPendingQSize); + +// the thread pool size should be configurable later, and justified with a rationale +logSyncNotifyExecutor = Executors.newFixedThreadPool(10); Review comment: I prefer to make this improvement be more configurable to use. Can we make thread pool size be configurable in this PR? if the pool size is configured as 0, that means this improvements is disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16828) Zookeeper Delegation Token Manager fetch sequence number by batch
[ https://issues.apache.org/jira/browse/HADOOP-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308728#comment-17308728 ] Stephen O'Donnell commented on HADOOP-16828: Thanks for the information [~fengnanli]. This was a clean cherry-pick to branch-3.3, so I have backported the improvement to there. > Zookeeper Delegation Token Manager fetch sequence number by batch > - > > Key: HADOOP-16828 > URL: https://issues.apache.org/jira/browse/HADOOP-16828 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HADOOP-16828.001.patch, HADOOP-16828.002.patch, Screen > Shot 2020-01-25 at 2.25.06 PM.png, Screen Shot 2020-01-25 at 2.25.16 PM.png, > Screen Shot 2020-01-25 at 2.25.24 PM.png > > > Currently in ZKDelegationTokenSecretManager.java the seq number is > incremented by 1 each time there is a request for creating new token. This > will need to send traffic to Zookeeper server. With multiple managers > running, there is data contention going on. Also, since the current logic of > incrementing is using tryAndSet which is optimistic concurrency control > without locking. This data contention is having performance degradation when > the secret manager are under volume of traffic. > The change here is to fetching this seq number by batch instead of 1, which > will reduce the traffic sent to ZK and make many operations inside ZK secret > manager's memory. > After putting this into production we saw huge improvement to the RPC > processing latency of get delegationtoken calls. Also, since ZK takes less > traffic in this way. Other write calls, like renew and cancel delegation > tokens are benefiting from this change. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16828) Zookeeper Delegation Token Manager fetch sequence number by batch
[ https://issues.apache.org/jira/browse/HADOOP-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HADOOP-16828: --- Fix Version/s: 3.3.1 > Zookeeper Delegation Token Manager fetch sequence number by batch > - > > Key: HADOOP-16828 > URL: https://issues.apache.org/jira/browse/HADOOP-16828 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HADOOP-16828.001.patch, HADOOP-16828.002.patch, Screen > Shot 2020-01-25 at 2.25.06 PM.png, Screen Shot 2020-01-25 at 2.25.16 PM.png, > Screen Shot 2020-01-25 at 2.25.24 PM.png > > > Currently in ZKDelegationTokenSecretManager.java the seq number is > incremented by 1 each time there is a request for creating new token. This > will need to send traffic to Zookeeper server. With multiple managers > running, there is data contention going on. Also, since the current logic of > incrementing is using tryAndSet which is optimistic concurrency control > without locking. This data contention is having performance degradation when > the secret manager are under volume of traffic. > The change here is to fetching this seq number by batch instead of 1, which > will reduce the traffic sent to ZK and make many operations inside ZK secret > manager's memory. > After putting this into production we saw huge improvement to the RPC > processing latency of get delegationtoken calls. Also, since ZK takes less > traffic in this way. Other write calls, like renew and cancel delegation > tokens are benefiting from this change. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17511) Add an Audit plugin point for S3A auditing/context
[ https://issues.apache.org/jira/browse/HADOOP-17511?focusedWorklogId=571907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571907 ] ASF GitHub Bot logged work on HADOOP-17511: --- Author: ASF GitHub Bot Created on: 25/Mar/21 14:16 Start Date: 25/Mar/21 14:16 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2807: URL: https://github.com/apache/hadoop/pull/2807#issuecomment-806832078 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/2807 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/2807 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2807/4/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 571907) Time Spent: 12h 10m (was: 12h) > Add an Audit plugin point for S3A auditing/context > -- > > Key: HADOOP-17511 > URL: https://issues.apache.org/jira/browse/HADOOP-17511 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 12h 10m > Remaining Estimate: 0h > > Add a way for auditing tools to correlate S3 object calls with Hadoop FS API > calls. > Initially just to log/forward to an auditing service. > Later: let us attach them as parameters in S3 requests, such as opentrace > headeers or (my initial idea: http referrer header -where it will get into > the log) > Challenges > * ensuring the audit span is created for every public entry point. That will > have to include those used in s3guard tools, some defacto public APIs > * and not re-entered for active spans. s3A code must not call back into the > FS API points > * Propagation across worker threads -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2807: HADOOP-17511. Add audit/telemetry logging to S3A connector
hadoop-yetus commented on pull request #2807: URL: https://github.com/apache/hadoop/pull/2807#issuecomment-806832078 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/2807 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/2807 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2807/4/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13551) Collect AwsSdkMetrics in S3A FileSystem IOStatistics
[ https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13551. - Fix Version/s: 3.3.1 Resolution: Fixed > Collect AwsSdkMetrics in S3A FileSystem IOStatistics > > > Key: HADOOP-13551 > URL: https://issues.apache.org/jira/browse/HADOOP-13551 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > The S3A Connector has the ability to pass statistics collected by the AWS SDK > into the IOStatistics store of the (stream, FS) > But > * wiring up doesn't (yet) work > * its best if there was thread-context level collection, though FS-level > statistics would be a good start. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17305) ITestCustomSigner fails with gcs s3 compatible endpoint.
[ https://issues.apache.org/jira/browse/HADOOP-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17305: Fix Version/s: 3.3.1 > ITestCustomSigner fails with gcs s3 compatible endpoint. > - > > Key: HADOOP-17305 > URL: https://issues.apache.org/jira/browse/HADOOP-17305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1h > Remaining Estimate: 0h > > CC [~sseth] [~ste...@apache.org] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17222) Create socket address leveraging URI cache
[
https://issues.apache.org/jira/browse/HADOOP-17222?focusedWorklogId=571860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571860
]
ASF GitHub Bot logged work on HADOOP-17222:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 12:44
Start Date: 25/Mar/21 12:44
Worklog Time Spent: 10m
Work Description: sodonnel commented on a change in pull request #2817:
URL: https://github.com/apache/hadoop/pull/2817#discussion_r601450951
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
##
@@ -39,13 +39,16 @@
import java.nio.channels.SocketChannel;
import java.nio.channels.UnresolvedAddressException;
import java.util.Map.Entry;
+import java.util.concurrent.TimeUnit;
import java.util.regex.Pattern;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import javax.net.SocketFactory;
import org.apache.hadoop.security.AccessControlException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
Review comment:
I don't believe checkstyle checks for it or warns about it, however it
is good practice to keep them ordered. I sorted them in Intellij and there are
quite a few out of order, so I think it may cause more problems that it fixes
if we sort them all.
However in checking with Intellij, it moved com.google.common.cache.Cache ->
org.apache.hadoop.thirdparty.com.google.common.cache.Cache. This was part of
HADOOP-17288 which is on branch-3.3.
I will push a new commit with the new thirdparty change, but I think we
should just leave the order as it is. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 571860)
Time Spent: 5h 10m (was: 5h)
> Create socket address leveraging URI cache
> ---
>
> Key: HADOOP-17222
> URL: https://issues.apache.org/jira/browse/HADOOP-17222
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, hdfs-client
> Environment: HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: After Optimization remark.png, After optimization.svg,
> Before Optimization remark.png, Before optimization.svg
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Note：Not only the hdfs client can get the current benefit, all callers of
> NetUtils.createSocketAddr will get the benefit. Just use hdfs client as an
> example.
>
> Hdfs client selects best DN for hdfs Block. method call stack:
> DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair ->
> NetUtils.createSocketAddr
> NetUtils.createSocketAddr creates the corresponding InetSocketAddress based
> on the host and port. There are some heavier operations in the
> NetUtils.createSocketAddr method, for example: URI.create(target), so
> NetUtils.createSocketAddr takes more time to execute.
> The following is my performance report. The report is based on HBase calling
> hdfs. HBase is a high-frequency access client for hdfs, because HBase read
> operations often access a small DataBlock (about 64k) instead of the entire
> HFile. In the case of high frequency access, the NetUtils.createSocketAddr
> method is time-consuming.
> h3. Test Environment：
>
> {code:java}
> HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
> {code}
> h4. Before Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 4.86% of the entire CPU, and the creation of URIs accounts for a larger
> proportion.
> !Before Optimization remark.png!
> h3. Optimization ideas:
> NetUtils.createSocketAddr creates InetSocketAddress based on host and port.
> Here we can add Cache to InetSocketAddress. The key of Cache is host and
> port, and the value is InetSocketAddress.
> h4. After Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cache,
> and the ConcurrentHashMap.get() method gets data from
[GitHub] [hadoop] sodonnel commented on a change in pull request #2817: HADOOP-17222. Create socket address leveraging URI cache
sodonnel commented on a change in pull request #2817: URL: https://github.com/apache/hadoop/pull/2817#discussion_r601450951 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java ## @@ -39,13 +39,16 @@ import java.nio.channels.SocketChannel; import java.nio.channels.UnresolvedAddressException; import java.util.Map.Entry; +import java.util.concurrent.TimeUnit; import java.util.regex.Pattern; import java.util.*; import java.util.concurrent.ConcurrentHashMap; import javax.net.SocketFactory; import org.apache.hadoop.security.AccessControlException; +import com.google.common.cache.Cache; +import com.google.common.cache.CacheBuilder; Review comment: I don't believe checkstyle checks for it or warns about it, however it is good practice to keep them ordered. I sorted them in Intellij and there are quite a few out of order, so I think it may cause more problems that it fixes if we sort them all. However in checking with Intellij, it moved com.google.common.cache.Cache -> org.apache.hadoop.thirdparty.com.google.common.cache.Cache. This was part of HADOOP-17288 which is on branch-3.3. I will push a new commit with the new thirdparty change, but I think we should just leave the order as it is. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17133) Implement HttpServer2 metrics
[
https://issues.apache.org/jira/browse/HADOOP-17133?focusedWorklogId=571827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571827
]
ASF GitHub Bot logged work on HADOOP-17133:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 11:47
Start Date: 25/Mar/21 11:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2145:
URL: https://github.com/apache/hadoop/pull/2145#issuecomment-806585497
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 19s | | trunk passed |
| +1 :green_heart: | compile | 21m 19s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 11s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 20m 21s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 21s | | the patch passed |
| +1 :green_heart: | compile | 18m 14s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 4s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | unit | 6m 16s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 209m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2145/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2145 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 85e1c91b2690 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 453c37ef21f21aed28fcc7805ef65ff8a7a53064 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2145: HADOOP-17133. Implement HttpServer2 metrics
hadoop-yetus commented on pull request #2145:
URL: https://github.com/apache/hadoop/pull/2145#issuecomment-806585497
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 19s | | trunk passed |
| +1 :green_heart: | compile | 21m 19s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 11s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 20m 21s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 21s | | the patch passed |
| +1 :green_heart: | compile | 18m 14s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 4s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | unit | 6m 16s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 209m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2145/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2145 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 85e1c91b2690 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 453c37ef21f21aed28fcc7805ef65ff8a7a53064 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2145/3/testReport/ |
| Max. process+thread count | 1260 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-httpfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2145/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[GitHub] [hadoop] hadoop-yetus commented on pull request #2737: HDFS-15869. Network issue while FSEditLogAsync is executing RpcEdit.logSyncNotify can cause the namenode to hang
hadoop-yetus commented on pull request #2737:
URL: https://github.com/apache/hadoop/pull/2737#issuecomment-806583634
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 52s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2737/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 2 unchanged -
0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 730m 2s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2737/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 815m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestDistributedFileSystemWithECFile |
| | hadoop.hdfs.TestRollingUpgradeDowngrade |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS |
| | hadoop.hdfs.TestFetchImage |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages |
| | hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement |
| | hadoop.hdfs.server.namenode.TestListOpenFiles |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| |
[jira] [Work logged] (HADOOP-17548) ABFS: Toggle Store Mkdirs request overwrite parameter
[
https://issues.apache.org/jira/browse/HADOOP-17548?focusedWorklogId=571821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571821
]
ASF GitHub Bot logged work on HADOOP-17548:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 11:36
Start Date: 25/Mar/21 11:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2781:
URL: https://github.com/apache/hadoop/pull/2781#issuecomment-806579706
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 41s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 30s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 0m 59s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 16m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 99m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2781/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2781 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7b26bfeb0012 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 492a040ef939d2450188a12d891b17d78a4f75d0 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2781/2/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2781/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 571821)
Time Spent: 1h 20m (was: 1h 10m)
> ABFS: Toggle Store Mkdirs request overwrite parameter
> -
>
> Key: HADOOP-17548
> URL: https://issues.apache.org/jira/browse/HADOOP-17548
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sumangala Patki
>Assignee: Sumangala Patki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
[GitHub] [hadoop] hadoop-yetus commented on pull request #2781: HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter (#2729)
hadoop-yetus commented on pull request #2781:
URL: https://github.com/apache/hadoop/pull/2781#issuecomment-806579706
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 41s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 30s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 0m 59s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 16m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 99m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2781/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2781 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7b26bfeb0012 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 492a040ef939d2450188a12d891b17d78a4f75d0 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2781/2/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2781/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17602) Upgrade JUnit to 4.13.1
[ https://issues.apache.org/jira/browse/HADOOP-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17602: --- Fix Version/s: 3.2.3 2.10.2 3.1.5 3.4.0 3.3.1 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Committed to all the active branches. Thank you [~ahussein] for your contribution! > Upgrade JUnit to 4.13.1 > --- > > Key: HADOOP-17602 > URL: https://issues.apache.org/jira/browse/HADOOP-17602 > Project: Hadoop Common > Issue Type: Bug > Components: build, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3 > > Attachments: HADOOP-17602.001.patch, > HADOOP-17602.branch-2.10.001.patch > > > A reported vulnerability reported in JUnit4.7-4.13. > The JUnit4 test rule [TemporaryFolder on unix-like systems does not limit > access to created > files|https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17497) refactoring the signature of S3ClientFactory breaks hboss compile
[ https://issues.apache.org/jira/browse/HADOOP-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17497. - Resolution: Won't Fix HADOOP-13551 changed the args again, though by moving to a parameter object + builder is intended to be less brittle in future. going to have to patch HBoss as I couldn't come up with a design that was backwards compatible. > refactoring the signature of S3ClientFactory breaks hboss compile > - > > Key: HADOOP-17497 > URL: https://issues.apache.org/jira/browse/HADOOP-17497 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > ...restore removed S3ClientFactory method and call it as default by new one -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17133) Implement HttpServer2 metrics
[ https://issues.apache.org/jira/browse/HADOOP-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17133: --- Fix Version/s: (was: 3.4.0) Target Version/s: 3.4.0 > Implement HttpServer2 metrics > - > > Key: HADOOP-17133 > URL: https://issues.apache.org/jira/browse/HADOOP-17133 > Project: Hadoop Common > Issue Type: Improvement > Components: httpfs, kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > I'd like to collect metrics (number of connections, average response time, > etc...) from HttpFS and KMS but there are no metrics for HttpServer2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17133) Implement HttpServer2 metrics
[
https://issues.apache.org/jira/browse/HADOOP-17133?focusedWorklogId=571704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571704
]
ASF GitHub Bot logged work on HADOOP-17133:
---
Author: ASF GitHub Bot
Created on: 25/Mar/21 08:19
Start Date: 25/Mar/21 08:19
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #2145:
URL: https://github.com/apache/hadoop/pull/2145#discussion_r601183346
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2Metrics.java
##
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.http;
+
+import org.eclipse.jetty.server.handler.StatisticsHandler;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+
+/**
+ * This class collects all the metrics of Jetty's StatisticsHandler
+ * and expose them as Hadoop Metrics.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+@Metrics(name="HttpServer2", about="HttpServer2 metrics", context="http")
+public class HttpServer2Metrics {
+
+ private final StatisticsHandler handler;
+ private final int port;
+
+ @Metric("number of async requests requests that have expired")
Review comment:
Thank you! Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 571704)
Time Spent: 1h 20m (was: 1h 10m)
> Implement HttpServer2 metrics
> -
>
> Key: HADOOP-17133
> URL: https://issues.apache.org/jira/browse/HADOOP-17133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: httpfs, kms
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I'd like to collect metrics (number of connections, average response time,
> etc...) from HttpFS and KMS but there are no metrics for HttpServer2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2145: HADOOP-17133. Implement HttpServer2 metrics
aajisaka commented on a change in pull request #2145:
URL: https://github.com/apache/hadoop/pull/2145#discussion_r601183346
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2Metrics.java
##
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.http;
+
+import org.eclipse.jetty.server.handler.StatisticsHandler;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+
+/**
+ * This class collects all the metrics of Jetty's StatisticsHandler
+ * and expose them as Hadoop Metrics.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+@Metrics(name="HttpServer2", about="HttpServer2 metrics", context="http")
+public class HttpServer2Metrics {
+
+ private final StatisticsHandler handler;
+ private final int port;
+
+ @Metric("number of async requests requests that have expired")
Review comment:
Thank you! Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2748: HDFS-15879. Exclude slow nodes when choose targets for blocks
tomscut commented on a change in pull request #2748:
URL: https://github.com/apache/hadoop/pull/2748#discussion_r601108849
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.TestBlockStoragePolicy;
+import org.apache.hadoop.hdfs.server.namenode.NameNode;
+import org.apache.hadoop.net.Node;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Set;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(Parameterized.class)
+public class TestReplicationPolicyExcludeSlowNodes
+extends BaseReplicationPolicyTest {
+
+ public TestReplicationPolicyExcludeSlowNodes(String blockPlacementPolicy) {
+this.blockPlacementPolicy = blockPlacementPolicy;
+ }
+
+ @Parameterized.Parameters
+ public static Iterable data() {
+return Arrays.asList(new Object[][] {
Review comment:
Thanks @tasanuma for your advice. I will update it as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308426#comment-17308426 ] Siyao Meng commented on HADOOP-15566: - Thanks [~liuml07]. I have posted the v2 scope doc to reflect the change of using OpenTelemetry [^OpenTelemetry Support Scope Doc v2.pdf] [~liuml07] [~kiran.maturi] Please feel free to pick up any subtasks that hasn't been assigned or updated in a while. e.g. HADOOP-16286 > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566.000.WIP.patch, OpenTelemetry Support Scope > Doc v2.pdf, OpenTracing Support Scope Doc.pdf, Screen Shot 2018-06-29 at > 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 50m > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-15566: Attachment: OpenTelemetry Support Scope Doc v2.pdf > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566.000.WIP.patch, OpenTelemetry Support Scope > Doc v2.pdf, OpenTracing Support Scope Doc.pdf, Screen Shot 2018-06-29 at > 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 50m > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
