[hadoop] branch trunk updated (8d63734 -> c91ff8c)
This is an automated email from the ASF dual-hosted git repository. gabota pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 8d63734 HADOOP-16319. S3A Etag tests fail with default encryption enabled on bucket. add c91ff8c HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries (#1851). Contributed by Gabor Bota. No new revisions were added by this update. Summary of changes: .../apache/hadoop/fs/s3a/s3guard/S3GuardFsck.java | 33 +++- .../s3a/s3guard/S3GuardFsckViolationHandler.java | 61 +++--- .../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 38 ++ .../src/site/markdown/tools/hadoop-aws/s3guard.md | 8 ++- .../hadoop/fs/s3a/s3guard/ITestS3GuardFsck.java| 37 +++-- .../fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java | 34 6 files changed, 187 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 096533c2 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 096533c2 is described below commit 096533c2dc0afd51367030725d797480a22ba7e2 Author: Wei-Chiu Chuang AuthorDate: Wed Mar 18 06:27:13 2020 -0700 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. Contributed by Wei-Chiu Chuang. Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d --- hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties index 2d1c98b..df3b691 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties @@ -71,3 +71,6 @@ log4j.appender.FILE.layout=org.apache.log4j.PatternLayout log4j.appender.FILE.layout.ConversionPattern=%d{ISO8601} [%t] %-5p \ (%F:%L) %X{function} %X{resource} %X{user} %X{request} - \ %m%n + +# Supress KMS error log +log4j.logger.com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator=OFF \ No newline at end of file - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (096533c2 -> 367833c)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 096533c2 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. add 367833c HADOOP-16054. Update Dockerfile to use Bionic. No new revisions were added by this update. Summary of changes: dev-support/docker/Dockerfile | 46 ++- 1 file changed, 10 insertions(+), 36 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HADOOP-16920 ABFS: Make list page size configurable.
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 6ce5f87 HADOOP-16920 ABFS: Make list page size configurable. 6ce5f87 is described below commit 6ce5f8734f1864a2d628b23479cf3f6621b2fcb4 Author: bilaharith <52483117+bilahar...@users.noreply.github.com> AuthorDate: Wed Mar 18 19:44:18 2020 +0530 HADOOP-16920 ABFS: Make list page size configurable. Contributed by Bilahari T H. The page limit is set in "fs.azure.list.max.results"; default value is 500. There's currently a limit of 5000 in the store -there are no range checks in the client code so that limit can be changed on the server without any need to update the abfs connector. --- .../hadoop/fs/azurebfs/AbfsConfiguration.java | 14 .../fs/azurebfs/AzureBlobFileSystemStore.java | 4 +- .../fs/azurebfs/constants/ConfigurationKeys.java | 1 + .../constants/FileSystemConfigurations.java| 1 + .../apache/hadoop/fs/azurebfs/ITestAbfsClient.java | 90 ++ 5 files changed, 108 insertions(+), 2 deletions(-) diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java index 779f524..61fe3d8 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java @@ -125,6 +125,11 @@ public class AbfsConfiguration{ DefaultValue = MAX_CONCURRENT_WRITE_THREADS) private int maxConcurrentWriteThreads; + @IntegerConfigurationValidatorAnnotation(ConfigurationKey = AZURE_LIST_MAX_RESULTS, + MinValue = 1, + DefaultValue = DEFAULT_AZURE_LIST_MAX_RESULTS) + private int listMaxResults; + @IntegerConfigurationValidatorAnnotation(ConfigurationKey = AZURE_CONCURRENT_CONNECTION_VALUE_IN, MinValue = 1, DefaultValue = MAX_CONCURRENT_READ_THREADS) @@ -432,6 +437,10 @@ public class AbfsConfiguration{ return this.maxConcurrentReadThreads; } + public int getListMaxResults() { +return this.listMaxResults; + } + public boolean getTolerateOobAppends() { return this.tolerateOobAppends; } @@ -702,6 +711,11 @@ public class AbfsConfiguration{ this.disableOutputStreamFlush = disableOutputStreamFlush; } + @VisibleForTesting + void setListMaxResults(int listMaxResults) { +this.listMaxResults = listMaxResults; + } + private String getTrimmedPasswordString(String key, String defaultValue) throws IOException { String value = getPasswordString(key); if (StringUtils.isBlank(value)) { diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java index bbf3608..bff0e45 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java @@ -126,7 +126,6 @@ public class AzureBlobFileSystemStore implements Closeable { private static final String DATE_TIME_PATTERN = "E, dd MMM HH:mm:ss z"; private static final String TOKEN_DATE_PATTERN = "-MM-dd'T'HH:mm:ss.SSS'Z'"; private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1"; - private static final int LIST_MAX_RESULTS = 500; private static final int GET_SET_AGGREGATE_COUNT = 2; private final AbfsConfiguration abfsConfiguration; @@ -682,7 +681,8 @@ public class AzureBlobFileSystemStore implements Closeable { ArrayList fileStatuses = new ArrayList<>(); do { try (AbfsPerfInfo perfInfo = startTracking("listStatus", "listPath")) { -AbfsRestOperation op = client.listPath(relativePath, false, LIST_MAX_RESULTS, continuation); +AbfsRestOperation op = client.listPath(relativePath, false, +abfsConfiguration.getListMaxResults(), continuation); perfInfo.registerResult(op.getResult()); continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION); ListResultSchema retrievedSchema = op.getResult().getListResultSchema(); diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java index 3b0111e..a63e953 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/Confi
[hadoop] branch trunk updated: HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. ayushsaxena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 654db35f HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 654db35f is described below commit 654db35fa2a2bfabd8f844e9ca10ad8bfea859cf Author: Ayush Saxena AuthorDate: Wed Mar 18 20:55:34 2020 +0530 HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. --- hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html | 4 hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js | 2 ++ 2 files changed, 6 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html index 5053696..582420e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html @@ -329,6 +329,8 @@ Http Address Last contact Last Block Report + Used + Non DFS Used Capacity Blocks Block pool used @@ -342,6 +344,8 @@ {dnWebAddress} {lastContact}s {lastBlockReport}m +{used|fmt_bytes} +{nonDfsUsedSpace|fmt_bytes} {capacity|fmt_bytes} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js index 0d9de3e..9715c37 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js @@ -356,6 +356,8 @@ { 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent": 0}, { 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent": 0}, { 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent": 0}, + { 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent": 0}, + { 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent": 0}, { 'type': 'num' , "defaultContent": 0}, { 'orderDataType': 'ng-value', 'type': 'num' , "defaultContent": 0}, { 'type': 'string' , "defaultContent": ""} - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new 61915fb HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 61915fb is described below commit 61915fbd83ee625cb46900fa49ffbfc261a2647d Author: Wei-Chiu Chuang AuthorDate: Wed Mar 18 06:27:13 2020 -0700 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. Contributed by Wei-Chiu Chuang. Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d (cherry picked from commit 096533c2dc0afd51367030725d797480a22ba7e2) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties (cherry picked from commit 1d55e1e229d835a6743523b9cf4c1e97b4259128) --- hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties index 7378846..d45fe63 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties @@ -47,3 +47,6 @@ log4j.appender.DNMETRICSRFA.layout=org.apache.log4j.PatternLayout log4j.appender.DNMETRICSRFA.layout.ConversionPattern=%d{ISO8601} %m%n log4j.appender.DNMETRICSRFA.MaxBackupIndex=1 log4j.appender.DNMETRICSRFA.MaxFileSize=64MB + +# Supress KMS error log +log4j.logger.com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator=OFF - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new 1d55e1e HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 1d55e1e is described below commit 1d55e1e229d835a6743523b9cf4c1e97b4259128 Author: Wei-Chiu Chuang AuthorDate: Wed Mar 18 06:27:13 2020 -0700 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. Contributed by Wei-Chiu Chuang. Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d (cherry picked from commit 096533c2dc0afd51367030725d797480a22ba7e2) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties --- hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties index bd5a4dd..d45fe63 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties @@ -48,3 +48,5 @@ log4j.appender.DNMETRICSRFA.layout.ConversionPattern=%d{ISO8601} %m%n log4j.appender.DNMETRICSRFA.MaxBackupIndex=1 log4j.appender.DNMETRICSRFA.MaxFileSize=64MB +# Supress KMS error log +log4j.logger.com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator=OFF - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop-thirdparty] annotated tag rel/release-1.0.0 created (now 78ae206)
This is an automated email from the ASF dual-hosted git repository. vinayakumarb pushed a change to annotated tag rel/release-1.0.0 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git. at 78ae206 (tag) tagging 3395253a6235c40e5eac57a365fc1268d3f379af (commit) by Vinayakumar B on Thu Mar 19 00:38:41 2020 +0530 - Log - Hadoop Thirdparty 1.0.0 release -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEgM5LZnWY0WbA5Zgs5OrZw9AhWnEFAl5ycbkACgkQ5OrZw9Ah WnFcHQ/7B1Cxj5rGFIisvwtBsnMuAHAugxSXIPsrUl4hft3Hu15wRfugA64rp6+y n2vlM+veYTc4EHl0Aw+/KYXQVzD37ZqgBsAJqZZYBm8hqXIVj6F9QiVKgsMtbuA4 1ghoMwGpRkdKy5+qwcL08e8c1YtG/0Y8a/QcLn5YZSJaVxTQShLN/95KDP6AfuIc mr34SV5NkjxB7S+t6di2R4nfmd30nT1Tt5xwxRNl8+1H5n8i8AgcMaUuIU26ZLWD RYuL3qmije+YfGr42Nxf86D+FzKiYlUGEMlXU/LRhEsJHl0bfWAPy9aYkbsRbz+0 /bWQRrbJ8dc36gb52kJQTdSAO2i7w/b/RxSpROh4m6KzInWCGf6gS7HSJjuq7/BU xUrxQAlZVvzYzox2NlbQ1Gg/SpQLQPJp8N9gd7tHnQIqqRZ92umuv9WhGOhymqBA i6NLbelNfKLDZO4XAdwhhQEF7oV8GqIfOLTMcebfI0M8yHIgui81acRb4yBjMdUz kwV7ZHB3nAdX6qtjOtrQUgDhkQ0bJitJ87hzTdhSCD69yWiCI4A+/UohuuIkT+Im Til14BvZLaz159XJl7K6khdxkp+9Wl1u5D9KVWZpgM0FMGlaodw3M03s3enHVbIl z9Ar0UJqmzrqZmTwA8XWz272nDJVG20CbGnFToLBZwa/AwWdNfw= =olv4 -END PGP SIGNATURE- --- No new revisions were added by this update. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop-thirdparty] branch trunk updated: Finishing 1.0.0 release
This is an automated email from the ASF dual-hosted git repository. vinayakumarb pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/trunk by this push: new 80523cc Finishing 1.0.0 release 80523cc is described below commit 80523cce96186fd22e1c0c8f8e71fbc1274ff96d Author: Vinayakumar B AuthorDate: Thu Mar 19 01:10:44 2020 +0530 Finishing 1.0.0 release --- .../thirdparty-1.0.0/CHANGELOG.thirdparty-1.0.0.md | 55 ++ .../RELEASENOTES.thirdparty-1.0.0.md | 24 ++ 2 files changed, 79 insertions(+) diff --git a/src/site/markdown/release/thirdparty-1.0.0/CHANGELOG.thirdparty-1.0.0.md b/src/site/markdown/release/thirdparty-1.0.0/CHANGELOG.thirdparty-1.0.0.md new file mode 100644 index 000..9db3eb5 --- /dev/null +++ b/src/site/markdown/release/thirdparty-1.0.0/CHANGELOG.thirdparty-1.0.0.md @@ -0,0 +1,55 @@ + + +# Apache Hadoop Third-party Libs Changelog + +## Release thirdparty-1.0.0 - 2020-03-11 + + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-16895](https://issues.apache.org/jira/browse/HADOOP-16895) | [thirdparty] Revisit LICENSEs and NOTICEs | Major | . | Vinayakumar B | Vinayakumar B | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-16820](https://issues.apache.org/jira/browse/HADOOP-16820) | [thirdparty] ChangeLog and ReleaseNote are not packaged by createrelease script | Major | hadoop-thirdparty | Vinayakumar B | Vinayakumar B | +| [HADOOP-16824](https://issues.apache.org/jira/browse/HADOOP-16824) | [thirdparty] port HADOOP-16754 (Fix docker failed to build yetus/hadoop) to thirdparty Dockerfile | Major | . | Vinayakumar B | Vinayakumar B | +| [HADOOP-16919](https://issues.apache.org/jira/browse/HADOOP-16919) | [thirdparty] Handle release package related issues | Major | hadoop-thirdparty | Vinayakumar B | Vinayakumar B | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-16595](https://issues.apache.org/jira/browse/HADOOP-16595) | [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf | Major | hadoop-thirdparty | Vinayakumar B | Vinayakumar B | +| [HADOOP-16821](https://issues.apache.org/jira/browse/HADOOP-16821) | [pb-upgrade] Use 'o.a.h.thirdparty.protobuf' shaded prefix instead of 'protobuf\_3\_7' | Major | hadoop-thirdparty | Vinayakumar B | Vinayakumar B | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-16867](https://issues.apache.org/jira/browse/HADOOP-16867) | [thirdparty] Add shaded JaegerTracer | Major | . | Siyao Meng | Siyao Meng | + + diff --git a/src/site/markdown/release/thirdparty-1.0.0/RELEASENOTES.thirdparty-1.0.0.md b/src/site/markdown/release/thirdparty-1.0.0/RELEASENOTES.thirdparty-1.0.0.md new file mode 100644 index 000..1d48ea0 --- /dev/null +++ b/src/site/markdown/release/thirdparty-1.0.0/RELEASENOTES.thirdparty-1.0.0.md @@ -0,0 +1,24 @@ + + +# Apache Hadoop Third-party Libs thirdparty-1.0.0 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org