[jira] [Commented] (HADOOP-17767) ABFS: Improve test scripts
[ https://issues.apache.org/jira/browse/HADOOP-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613309#comment-17613309 ] ASF GitHub Bot commented on HADOOP-17767: - surendralilhore commented on PR #3124: URL: https://github.com/apache/hadoop/pull/3124#issuecomment-1269359761 @snvijaya , Thanks for update. Looks good to me. Just fix the check style warning. > ABFS: Improve test scripts > -- > > Key: HADOOP-17767 > URL: https://issues.apache.org/jira/browse/HADOOP-17767 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Current test run scripts need manual update across all combinations in > runTests.sh for account name and is working off a single azure-auth-keys.xml > file. While having to test across accounts that span various geo, the config > file grows big and also needs a manual change for configs such as > fs.contract.test.[abfs/abfss] which has to be uniquely set. To use the script > across various combinations, dev to be aware of the names of all the > combinations defined in runTests.sh as well. > > These concerns are addressed in the new version of the scripts. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on pull request #3124: HADOOP-17767. ABFS: Updates test scripts
surendralilhore commented on PR #3124: URL: https://github.com/apache/hadoop/pull/3124#issuecomment-1269359761 @snvijaya , Thanks for update. Looks good to me. Just fix the check style warning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17912) ABFS: Support for Encryption Context
[ https://issues.apache.org/jira/browse/HADOOP-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613289#comment-17613289 ] Pranav Saxena commented on HADOOP-17912: [~ste...@apache.org], requesting you to kindly review the PR: https://github.com/apache/hadoop/pull/3440. Thanks. > ABFS: Support for Encryption Context > > > Key: HADOOP-17912 > URL: https://issues.apache.org/jira/browse/HADOOP-17912 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Support for customer-provided encryption keys at the file level, superceding > the global (account-level) key use in HADOOP-17536. > ABFS driver will support an "EncryptionContext" plugin for retrieving > encryption information, the implementation for which should be provided by > the client. The keys/context retrieved will be sent via request headers to > the server, which will store the encryption context. Subsequent REST calls to > server that access data/user metadata of the file will require fetching the > encryption context through a GetFileProperties call and retrieving the key > from the custom provider, before sending the request. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4946: YARN-11317. [Federation] Refactoring Yarn Router's About Web Page.
slfan1989 commented on code in PR #4946:
URL: https://github.com/apache/hadoop/pull/4946#discussion_r988468315
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/dao/RouterClusterMetrics.java:
##
@@ -0,0 +1,310 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.webapp.dao;
+
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.records.Resource;
+import
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RouterClusterMetrics {
+
+ protected static final long BYTES_IN_MB = 1024 * 1024;
+ private static final Logger LOG =
LoggerFactory.getLogger(RouterClusterMetrics.class);
+
+ /** Application Information. **/
+ private String appsSubmitted = "N/A";
+ private String appsCompleted = "N/A";
+ private String appsPending = "N/A";
+ private String appsRunning = "N/A";
+ private String appsFailed = "N/A";
+ private String appsKilled = "N/A";
+
+ /** Memory Information. **/
+ private String totalMemory = "N/A";
+ private String reservedMemory = "N/A";
+ private String availableMemory = "N/A";
+ private String allocatedMemory = "N/A";
+ private String pendingMemory = "N/A";
+
+ /** VirtualCores Information. **/
+ private String reservedVirtualCores = "N/A";
+ private String availableVirtualCores = "N/A";
+ private String allocatedVirtualCores = "N/A";
+ private String pendingVirtualCores = "N/A";
+ private String totalVirtualCores = "N/A";
+
+ /** Resources Information. **/
+ private String usedResources = "N/A";
+ private String totalResources = "N/A";
+ private String reservedResources = "N/A";
+ private String allocatedContainers = "N/A";
+
+ /** Resource Percent Information. **/
+ private String utilizedMBPercent = "N/A";
+ private String utilizedVirtualCoresPercent = "N/A";
+
+ /** Node Information. **/
+ private String activeNodes = "N/A";
+ private String decommissioningNodes = "N/A";
+ private String decommissionedNodes = "N/A";
+ private String lostNodes = "N/A";
+ private String unhealthyNodes = "N/A";
+ private String rebootedNodes = "N/A";
+ private String shutdownNodes = "N/A";
+
+ public RouterClusterMetrics() {
+
+ }
+
+ public RouterClusterMetrics(ClusterMetricsInfo metrics) {
+if (metrics != null) {
+ /** Application Information Conversion. **/
+ conversionApplicationInformation(metrics);
+
+ /** Memory Information Conversion. **/
+ conversionMemoryInformation(metrics);
+
+ /** Resources Information Conversion. **/
+ conversionResourcesInformation(metrics);
+
+ /** Percent Information Conversion. **/
+ conversionResourcesPercent(metrics);
+
+ /** Node Information Conversion. **/
+ conversionNodeInformation(metrics);
+}
+ }
+
+ // Get Key Metric Information
+ public String getAppsSubmitted() {
+return appsSubmitted;
+ }
+
+ public String getAppsCompleted() {
+return appsCompleted;
+ }
+
+ public String getAppsPending() {
+return appsPending;
+ }
+
+ public String getAppsRunning() {
+return appsRunning;
+ }
+
+ public String getAppsFailed() {
+return appsFailed;
+ }
+
+ public String getAppsKilled() {
+return appsKilled;
+ }
+
+ public String getTotalMemory() {
+return totalMemory;
+ }
+
+ public String getReservedMemory() {
+return reservedMemory;
+ }
+
+ public String getAvailableMemory() {
+return availableMemory;
+ }
+
+ public String getAllocatedMemory() {
+return allocatedMemory;
+ }
+
+ public String getPendingMemory() {
+return pendingMemory;
+ }
+
+ public String getReservedVirtualCores() {
+return reservedVirtualCores;
+ }
+
+ public String getAvailableVirtualCores() {
+return availableVirtualCores;
+ }
+
+ public String
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4946: YARN-11317. [Federation] Refactoring Yarn Router's About Web Page.
slfan1989 commented on code in PR #4946:
URL: https://github.com/apache/hadoop/pull/4946#discussion_r988468178
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java:
##
@@ -1099,4 +1099,61 @@ private RMWebServices prepareWebServiceForValidation(
return webService;
}
+ @Test
+ public void testClusterSchedulerOverviewFifo() throws JSONException,
Exception {
+WebResource r = resource();
+ClientResponse response = r.path("ws").path("v1").path("cluster")
+.path("scheduler-overview").accept(MediaType.APPLICATION_JSON)
+.get(ClientResponse.class);
+
+assertEquals(MediaType.APPLICATION_JSON + "; " + JettyUtils.UTF_8,
+response.getType().toString());
+JSONObject json = response.getEntity(JSONObject.class);
+verifyClusterSchedulerOverView(json, "Fifo Scheduler");
+ }
+
+ public static void verifyClusterSchedulerOverView(
+ JSONObject json, String expectedSchedulerType) throws Exception {
+
+// why json contains 8 elements because we defined 8 fields
+assertEquals("incorrect number of elements in: " + json, 8, json.length());
+
+// 1.Verify that the schedulerType is as expected
+String schedulerType = json.getString("schedulerType");
+assertEquals(expectedSchedulerType, schedulerType);
+
+// 2.Verify that schedulingResourceType is as expected
+String schedulingResourceType = json.getString("schedulingResourceType");
+assertEquals("memory-mb (unit=Mi),vcores", schedulingResourceType);
+
+// 3.Verify that minimumAllocation is as expected
+JSONObject minimumAllocation = json.getJSONObject("minimumAllocation");
+String minMemory = minimumAllocation.getString("memory");
+String minVCores = minimumAllocation.getString("vCores");
+assertEquals("1024", minMemory);
+assertEquals("1", minVCores);
+
+// 4.Verify that maximumAllocation is as expected
+JSONObject maximumAllocation = json.getJSONObject("maximumAllocation");
+String maxMemory = maximumAllocation.getString("memory");
+String maxVCores = maximumAllocation.getString("vCores");
+assertEquals("8192", maxMemory);
+assertEquals("4", maxVCores);
+
+// 5.Verify that schedulerBusy is as expected
+int schedulerBusy = json.getInt("schedulerBusy");
+Assert.assertEquals(-1, schedulerBusy);
Review Comment:
Thanks a lot for your help reviewing the code, I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4927: YARN-11313. [Federation] Add SQLServer Script and Supported DB Version in Federation.md.
slfan1989 commented on PR #4927: URL: https://github.com/apache/hadoop/pull/4927#issuecomment-1269136639 @goiri Thank you very much for your help reviewing the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4974: HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang.
hadoop-yetus commented on PR #4974:
URL: https://github.com/apache/hadoop/pull/4974#issuecomment-1269124654
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 59s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 48s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 47s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 53s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 1m 21s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 38s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 25m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 57s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 125m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4974/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4974 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux eb9c479aa5cf 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 583322d4edb7e5bdbe590ee86d59baae5adab884 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4974/1/testReport/ |
| Max. process+thread count | 2115 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4974/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mkuchenbecker commented on a diff in pull request #4967: HDFS-16791 WIP - client protocol and Filesystem apis implemented and …
mkuchenbecker commented on code in PR #4967:
URL: https://github.com/apache/hadoop/pull/4967#discussion_r988403730
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##
@@ -3898,4 +3898,31 @@ public DatanodeInfo[] getSlowDatanodeStats() throws
IOException {
return dfs.slowDatanodeReport();
}
+ /* HDFS only */
+ public Path getEnclosingRoot(final Path path) throws IOException {
+statistics.incrementReadOps(1);
+storageStatistics.incrementOpCounter(OpType.GET_ENCLOSING_ROOT);
+Preconditions.checkNotNull(path);
+Path absF = fixRelativePart(path);
+return new FileSystemLinkResolver() {
+ @Override
+ public Path doCall(final Path p) throws IOException {
+return dfs.getEnclosingRoot(getPathName(p));
+ }
+
+ @Override
+ public Path next(final FileSystem fs, final Path p)
+ throws IOException {
+if (fs instanceof DistributedFileSystem) {
Review Comment:
Nit, I would reverse the condition and say:
if not instanceof then throw
// Logic
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java:
##
@@ -1919,6 +1927,14 @@ public Collection
getAllStoragePolicies()
}
return allPolicies;
}
+
+@Override
+public Path getEnclosingRoot(Path path) throws IOException {
Review Comment:
They both override; you could consider a static helper to DRY.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -1935,6 +1935,21 @@ public DatanodeInfo[] getSlowDatanodeReport() throws
IOException {
return rpcServer.getSlowDatanodeReport(true, 0);
}
+ @Override
+ public String getEnclosingRoot(String src) throws IOException {
+Path mountPath = new Path("/");
+if (subclusterResolver instanceof MountTableResolver) {
+ MountTableResolver mountTable = (MountTableResolver) subclusterResolver;
+ if (mountTable.getMountPoint(src) != null) {
+// unclear if this is the correct thing to do, probably depends on
default mount point / link fallback
+mountPath = new Path(mountTable.getMountPoint(src).getSourcePath());
Review Comment:
getMonutPoint can return null.
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto:
##
@@ -428,6 +428,14 @@ message GetPreferredBlockSizeResponseProto {
message GetSlowDatanodeReportRequestProto {
}
+message GetEnclosingRootRequestProto {
+ required string filename = 1;
Review Comment:
I'd use optional. Required has been deprecated in Proto3.
https://stackoverflow.com/questions/31801257/why-required-and-optional-is-removed-in-protocol-buffers-3
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java:
##
@@ -1888,4 +1889,11 @@ BatchedEntries listOpenFiles(long prevId,
@ReadOnly
DatanodeInfo[] getSlowDatanodeReport() throws IOException;
+ /**
+ * Get the enclosing root for a path.
+ */
+ @Idempotent
+ @ReadOnly(isCoordinated = true)
+ String getEnclosingRoot(String src) throws IOException;
Review Comment:
+1, Path is getting implicitly cast as string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin opened a new pull request, #4974: HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang.
xinglin opened a new pull request, #4974: URL: https://github.com/apache/hadoop/pull/4974 ### Description of PR Backport of HDFS-16628 from trunk to branch-3.3. Please merge this PR after https://github.com/apache/hadoop/pull/4962. HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. (cherry picked from commit f8c7e67fdcdf963ef9613c395058cc4e653cd6d7) ### How was this patch tested? mvn test -Dtest="TestRouterTrash" ``` [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hdfs.server.federation.router.TestRouterTrash [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.267 s - in org.apache.hadoop.hdfs.server.federation.router.TestRouterTrash [INFO] [INFO] Results: [INFO] [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 ``` ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18478) Upgrade jettison to 1.5.1 to mitigate CVE-2022-40149
[ https://issues.apache.org/jira/browse/HADOOP-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18478: Labels: pull-request-available (was: ) > Upgrade jettison to 1.5.1 to mitigate CVE-2022-40149 > > > Key: HADOOP-18478 > URL: https://issues.apache.org/jira/browse/HADOOP-18478 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade jettison to 1.5.1 to mitigate CVE-2022-40149 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18478) Upgrade jettison to 1.5.1 to mitigate CVE-2022-40149
[
https://issues.apache.org/jira/browse/HADOOP-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613227#comment-17613227
]
ASF GitHub Bot commented on HADOOP-18478:
-
hadoop-yetus commented on PR #4969:
URL: https://github.com/apache/hadoop/pull/4969#issuecomment-1268983041
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 34s | | trunk passed |
| +1 :green_heart: | compile | 25m 38s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 9s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 21m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 8m 53s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 42m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 25m 29s | | the patch passed |
| +1 :green_heart: | compile | 24m 57s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 24m 57s | | the patch passed |
| +1 :green_heart: | compile | 22m 1s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 20m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 22s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 39s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 40m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1079m 59s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4969/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 3m 0s | | The patch does not
generate ASF License warnings. |
| | | 1378m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4969/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4969 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux d8795663efc7 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0f41b636c4e21d826b2501b4423dbe17ad7c0307 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #4969: HADOOP-18478. Upgrade jettison to 1.5.1 to mitigate CVE-2022-40149
hadoop-yetus commented on PR #4969:
URL: https://github.com/apache/hadoop/pull/4969#issuecomment-1268983041
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 34s | | trunk passed |
| +1 :green_heart: | compile | 25m 38s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 9s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 21m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 8m 53s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 42m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 25m 29s | | the patch passed |
| +1 :green_heart: | compile | 24m 57s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 24m 57s | | the patch passed |
| +1 :green_heart: | compile | 22m 1s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 20m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 22s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 39s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 40m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1079m 59s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4969/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 3m 0s | | The patch does not
generate ASF License warnings. |
| | | 1378m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4969/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4969 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux d8795663efc7 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0f41b636c4e21d826b2501b4423dbe17ad7c0307 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4969/1/testReport/ |
| Max. process+thread count | 3434 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output |
[GitHub] [hadoop] goiri merged pull request #4927: YARN-11313. [Federation] Add SQLServer Script and Supported DB Version in Federation.md.
goiri merged PR #4927: URL: https://github.com/apache/hadoop/pull/4927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18471) An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input
[
https://issues.apache.org/jira/browse/HADOOP-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613199#comment-17613199
]
ASF GitHub Bot commented on HADOOP-18471:
-
FuzzingTeam commented on code in PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#discussion_r985712391
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DefaultStringifier.java:
##
@@ -158,6 +158,9 @@ public static K load(Configuration conf, String keyName,
public static void storeArray(Configuration conf, K[] items,
String keyName) throws IOException {
+if (items.length == 0) {
+ throw new IndexOutOfBoundsException();
+}
DefaultStringifier stringifier = new DefaultStringifier(conf,
GenericsUtil.getClass(items[0]));
Review Comment:
Thanks @steveloughran for the review.
Yes, the original code would trigger ArrayIndexOutOfBoundsException
automatically, and the new test passes without the new lines.
But we found the same exception has been handled in other places in the same
module and hence, propose to handle it.
> An unhandled ArrayIndexOutOfBoundsException in
> DefaultStringifier.storeArray() if provided with an empty input
> --
>
> Key: HADOOP-18471
> URL: https://issues.apache.org/jira/browse/HADOOP-18471
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, io
>Affects Versions: 3.3.4
>Reporter: FuzzingTeam
>Priority: Minor
> Labels: pull-request-available
>
> The code throws an unhandled ArrayIndexOutOfBoundsException when method
> _storeArray_ of DefaultStringifier.java is called with an empty array as
> input.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] FuzzingTeam commented on a diff in pull request #4957: HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in class DefaultStringifier
FuzzingTeam commented on code in PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#discussion_r985712391
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DefaultStringifier.java:
##
@@ -158,6 +158,9 @@ public static K load(Configuration conf, String keyName,
public static void storeArray(Configuration conf, K[] items,
String keyName) throws IOException {
+if (items.length == 0) {
+ throw new IndexOutOfBoundsException();
+}
DefaultStringifier stringifier = new DefaultStringifier(conf,
GenericsUtil.getClass(items[0]));
Review Comment:
Thanks @steveloughran for the review.
Yes, the original code would trigger ArrayIndexOutOfBoundsException
automatically, and the new test passes without the new lines.
But we found the same exception has been handled in other places in the same
module and hence, propose to handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18471) An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input
[
https://issues.apache.org/jira/browse/HADOOP-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613196#comment-17613196
]
ASF GitHub Bot commented on HADOOP-18471:
-
hadoop-yetus commented on PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#issuecomment-1265663464
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 51s | | trunk passed |
| +1 :green_heart: | compile | 23m 34s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 22m 50s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 50s | | the patch passed |
| +1 :green_heart: | compile | 20m 55s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 58s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 21s | | The patch does not
generate ASF License warnings. |
| | | 215m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4957 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d4139a80c560 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ff1834934e87473d03fbe34dad60d274a7019dcf |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/3/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> An unhandled
[GitHub] [hadoop] hadoop-yetus commented on pull request #4957: HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in class DefaultStringifier
hadoop-yetus commented on PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#issuecomment-1265663464
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 51s | | trunk passed |
| +1 :green_heart: | compile | 23m 34s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 22m 50s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 50s | | the patch passed |
| +1 :green_heart: | compile | 20m 55s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 58s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 21s | | The patch does not
generate ASF License warnings. |
| | | 215m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4957 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d4139a80c560 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ff1834934e87473d03fbe34dad60d274a7019dcf |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/3/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please
[GitHub] [hadoop] K0K0V0K commented on a diff in pull request #4655: YARN-11216. Avoid unnecessary reconstruction of ConfigurationProperties
K0K0V0K commented on code in PR #4655:
URL: https://github.com/apache/hadoop/pull/4655#discussion_r985932320
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -1200,7 +1207,23 @@ public ConfigurationProperties
getConfigurationProperties() {
public void reinitializeConfigurationProperties() {
// Props are always Strings, therefore this cast is safe
Map props = (Map) getProps();
-configurationProperties = new ConfigurationProperties(props);
+configurationProperties = new ConfigurationProperties(props, PREFIX);
+ }
+
+ @Override
+ public void set(String name, String value) {
+super.set(name, value);
+if (configurationProperties != null) {
Review Comment:
The getter will initialise the tree if it is not present in the object,
based on the configuration.
If we set values on this object but wont use the tree view of the data, then
the tree building will be wasted.
Why we should use the getter here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4927: YARN-11313. [Federation] Add SQLServer Script and Supported DB Version in Federation.md.
hadoop-yetus commented on PR #4927:
URL: https://github.com/apache/hadoop/pull/4927#issuecomment-1265586634
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 66m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 5m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 103m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4927/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4927 |
| Optional Tests | dupname asflicense codespell detsecrets mvnsite
markdownlint |
| uname | Linux 99306aa246b0 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e93e92d93e6b9e563c0c8fe28e0052127c95e2eb |
| Max. process+thread count | 661 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4927/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4961: YARN-11324. Fix some PBImpl classes to avoid NPE.
hadoop-yetus commented on PR #4961:
URL: https://github.com/apache/hadoop/pull/4961#issuecomment-1265584387
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4961/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 3m 36s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4961/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
| hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 110m 46s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.federation.store.impl.TestZookeeperFederationStateStore |
| |
hadoop.yarn.server.federation.policies.router.TestHashBasedRouterPolicy |
| |
hadoop.yarn.server.federation.policies.manager.TestPriorityBroadcastPolicyManager
|
| |
hadoop.yarn.server.federation.policies.TestFederationPolicyInitializationContextValidator
|
| | hadoop.yarn.server.federation.policies.router.TestRejectRouterPolicy |
| |
hadoop.yarn.server.federation.policies.router.TestUniformRandomRouterPolicy |
| |
hadoop.yarn.server.federation.policies.amrmproxy.TestHomeAMRMProxyPolicy |
| |
hadoop.yarn.server.federation.policies.manager.TestUniformBroadcastPolicyManager
|
| |
hadoop.yarn.server.federation.policies.amrmproxy.TestRejectAMRMProxyPolicy |
| |
hadoop.yarn.server.federation.policies.manager.TestRejectAllPolicyManager |
| |
hadoop.yarn.server.federation.policies.router.TestWeightedRandomRouterPolicy |
| |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4938: YARN-8041. [Router] Federation: Improve Router REST API Metrics.
hadoop-yetus commented on PR #4938:
URL: https://github.com/apache/hadoop/pull/4938#issuecomment-1265563157
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4938/15/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 5s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 99m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4938/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4938 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 62a0ef3f1d42 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bf0b4fc6d39bce2b1b5d6e29d2a14d48a82f5b9b |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4938/15/testReport/ |
| Max. process+thread count | 764 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
[GitHub] [hadoop] K0K0V0K commented on a diff in pull request #4655: YARN-11216. Avoid unnecessary reconstruction of ConfigurationProperties
K0K0V0K commented on code in PR #4655:
URL: https://github.com/apache/hadoop/pull/4655#discussion_r985825269
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ConfigurationProperties.java:
##
@@ -55,6 +59,17 @@ public ConfigurationProperties(Map props) {
storePropertiesInPrefixNodes(props);
}
+ /**
+ * A constructor defined in order to conform to the type used by
+ * {@code Configuration}. It must only be called by String keys and values.
+ * @param props properties to store
+ * @param whiteListPrefix only those properties will be in the nodes
+ *which starts with one of the provided prefixes.
+ */
+ public ConfigurationProperties(Map props, String...
whiteListPrefix) {
+this(Maps.filterKeys(props, key -> StringUtils.startsWithAny(key,
whiteListPrefix)));
Review Comment:
Thanks for the review @9uapaw and sorry for the late response
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ConfigurationProperties.java:
##
@@ -55,6 +59,17 @@ public ConfigurationProperties(Map props) {
storePropertiesInPrefixNodes(props);
}
+ /**
+ * A constructor defined in order to conform to the type used by
+ * {@code Configuration}. It must only be called by String keys and values.
+ * @param props properties to store
+ * @param whiteListPrefix only those properties will be in the nodes
+ *which starts with one of the provided prefixes.
+ */
+ public ConfigurationProperties(Map props, String...
whiteListPrefix) {
+this(Maps.filterKeys(props, key -> StringUtils.startsWithAny(key,
whiteListPrefix)));
Review Comment:
Thanks for the review @9uapaw and sorry for the late response
I dont think the lambda has performance issue here. I created a small perf
test to prove it, and based on it the vanilia solution is way slower than
guava. I suggest the Hashmap.put() method requires some mem allocation time
when the map reach the limit.
[benchmark_code.txt](https://github.com/apache/hadoop/files/9698166/benchmark_code.txt)
[benchmark_log.txt](https://github.com/apache/hadoop/files/9698167/benchmark_log.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18471) An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input
[
https://issues.apache.org/jira/browse/HADOOP-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613182#comment-17613182
]
ASF GitHub Bot commented on HADOOP-18471:
-
FuzzingTeam commented on code in PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#discussion_r985709429
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestDefaultStringifier.java:
##
@@ -107,6 +108,12 @@ public void testStoreLoadArray() throws IOException {
Integer[] array = new Integer[] {1,2,3,4,5};
+try {
Review Comment:
we have updated the code, please review.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DefaultStringifier.java:
##
@@ -158,6 +158,9 @@ public static K load(Configuration conf, String keyName,
public static void storeArray(Configuration conf, K[] items,
String keyName) throws IOException {
+if (items.length == 0) {
+ throw new IndexOutOfBoundsException();
+}
DefaultStringifier stringifier = new DefaultStringifier(conf,
GenericsUtil.getClass(items[0]));
Review Comment:
Thanks @steveloughran for the review.
Yes, the original code would trigger ArrayIndexOutOfBoundsException
automatically, and the new test passes without the new lines.
But we found the same exception has been handled in other places in the same
module and hence, request to handle it.
> An unhandled ArrayIndexOutOfBoundsException in
> DefaultStringifier.storeArray() if provided with an empty input
> --
>
> Key: HADOOP-18471
> URL: https://issues.apache.org/jira/browse/HADOOP-18471
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, io
>Affects Versions: 3.3.4
>Reporter: FuzzingTeam
>Priority: Minor
> Labels: pull-request-available
>
> The code throws an unhandled ArrayIndexOutOfBoundsException when method
> _storeArray_ of DefaultStringifier.java is called with an empty array as
> input.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] FuzzingTeam commented on a diff in pull request #4957: HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in class DefaultStringifier
FuzzingTeam commented on code in PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#discussion_r985709429
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestDefaultStringifier.java:
##
@@ -107,6 +108,12 @@ public void testStoreLoadArray() throws IOException {
Integer[] array = new Integer[] {1,2,3,4,5};
+try {
Review Comment:
we have updated the code, please review.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DefaultStringifier.java:
##
@@ -158,6 +158,9 @@ public static K load(Configuration conf, String keyName,
public static void storeArray(Configuration conf, K[] items,
String keyName) throws IOException {
+if (items.length == 0) {
+ throw new IndexOutOfBoundsException();
+}
DefaultStringifier stringifier = new DefaultStringifier(conf,
GenericsUtil.getClass(items[0]));
Review Comment:
Thanks @steveloughran for the review.
Yes, the original code would trigger ArrayIndexOutOfBoundsException
automatically, and the new test passes without the new lines.
But we found the same exception has been handled in other places in the same
module and hence, request to handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18471) An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input
[
https://issues.apache.org/jira/browse/HADOOP-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613174#comment-17613174
]
ASF GitHub Bot commented on HADOOP-18471:
-
steveloughran commented on code in PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#discussion_r985588594
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestDefaultStringifier.java:
##
@@ -107,6 +108,12 @@ public void testStoreLoadArray() throws IOException {
Integer[] array = new Integer[] {1,2,3,4,5};
+try {
Review Comment:
use our `LambdaTestUtils.intercept()` here; it's a reimplementation of
scalatest's and straightforward to use it.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DefaultStringifier.java:
##
@@ -158,6 +158,9 @@ public static K load(Configuration conf, String keyName,
public static void storeArray(Configuration conf, K[] items,
String keyName) throws IOException {
+if (items.length == 0) {
+ throw new IndexOutOfBoundsException();
+}
DefaultStringifier stringifier = new DefaultStringifier(conf,
GenericsUtil.getClass(items[0]));
Review Comment:
won't this trigger the same exception automatically? what does the test do
before the new lines are added
> An unhandled ArrayIndexOutOfBoundsException in
> DefaultStringifier.storeArray() if provided with an empty input
> --
>
> Key: HADOOP-18471
> URL: https://issues.apache.org/jira/browse/HADOOP-18471
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, io
>Affects Versions: 3.3.4
>Reporter: FuzzingTeam
>Priority: Minor
> Labels: pull-request-available
>
> The code throws an unhandled ArrayIndexOutOfBoundsException when method
> _storeArray_ of DefaultStringifier.java is called with an empty array as
> input.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4957: HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in class DefaultStringifier
steveloughran commented on code in PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#discussion_r985588594
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestDefaultStringifier.java:
##
@@ -107,6 +108,12 @@ public void testStoreLoadArray() throws IOException {
Integer[] array = new Integer[] {1,2,3,4,5};
+try {
Review Comment:
use our `LambdaTestUtils.intercept()` here; it's a reimplementation of
scalatest's and straightforward to use it.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DefaultStringifier.java:
##
@@ -158,6 +158,9 @@ public static K load(Configuration conf, String keyName,
public static void storeArray(Configuration conf, K[] items,
String keyName) throws IOException {
+if (items.length == 0) {
+ throw new IndexOutOfBoundsException();
+}
DefaultStringifier stringifier = new DefaultStringifier(conf,
GenericsUtil.getClass(items[0]));
Review Comment:
won't this trigger the same exception automatically? what does the test do
before the new lines are added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18429) MutableGaugeFloat#incr(float) get stuck in an infinite loop
[
https://issues.apache.org/jira/browse/HADOOP-18429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613173#comment-17613173
]
ASF GitHub Bot commented on HADOOP-18429:
-
hadoop-yetus commented on PR #4823:
URL: https://github.com/apache/hadoop/pull/4823#issuecomment-1265235069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 51s | | trunk passed |
| +1 :green_heart: | compile | 25m 42s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 21m 16s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 22m 43s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 43s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not
generate ASF License warnings. |
| | | 216m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4823/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4823 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bba9966bcacb 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 51ba2174c5e06c662cd960837d194d7e4e93e322 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4823/5/testReport/ |
| Max. process+thread count | 2218 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4823/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
>
[GitHub] [hadoop] hadoop-yetus commented on pull request #4823: HADOOP-18429. fix infinite loop in MutableGaugeFloat#incr(float)
hadoop-yetus commented on PR #4823:
URL: https://github.com/apache/hadoop/pull/4823#issuecomment-1265235069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 51s | | trunk passed |
| +1 :green_heart: | compile | 25m 42s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 21m 16s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 22m 43s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 43s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not
generate ASF License warnings. |
| | | 216m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4823/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4823 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bba9966bcacb 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 51ba2174c5e06c662cd960837d194d7e4e93e322 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4823/5/testReport/ |
| Max. process+thread count | 2218 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4823/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please
[GitHub] [hadoop] xiaoxiaopan118 commented on pull request #4207: HDFS-16545. Provide an option to balance rack level in Balancers
xiaoxiaopan118 commented on PR #4207: URL: https://github.com/apache/hadoop/pull/4207#issuecomment-1265148645 LGTM ！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17912) ABFS: Support for Encryption Context
[ https://issues.apache.org/jira/browse/HADOOP-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613160#comment-17613160 ] ASF GitHub Bot commented on HADOOP-17912: - pranavsaxena-microsoft commented on PR #3440: URL: https://github.com/apache/hadoop/pull/3440#issuecomment-1265014148 @steveloughran , requesting you to kindly review the PR. Thanks. > ABFS: Support for Encryption Context > > > Key: HADOOP-17912 > URL: https://issues.apache.org/jira/browse/HADOOP-17912 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Support for customer-provided encryption keys at the file level, superceding > the global (account-level) key use in HADOOP-17536. > ABFS driver will support an "EncryptionContext" plugin for retrieving > encryption information, the implementation for which should be provided by > the client. The keys/context retrieved will be sent via request headers to > the server, which will store the encryption context. Subsequent REST calls to > server that access data/user metadata of the file will require fetching the > encryption context through a GetFileProperties call and retrieving the key > from the custom provider, before sending the request. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18360) Update commons-csv from 1.0 to 1.9.0.
[
https://issues.apache.org/jira/browse/HADOOP-18360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613167#comment-17613167
]
ASF GitHub Bot commented on HADOOP-18360:
-
hadoop-yetus commented on PR #4928:
URL: https://github.com/apache/hadoop/pull/4928#issuecomment-1265088527
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 47s | | trunk passed |
| +1 :green_heart: | compile | 25m 37s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 2s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 34s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 70m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 26m 9s | | the patch passed |
| +1 :green_heart: | compile | 24m 58s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 24m 58s | | the patch passed |
| +1 :green_heart: | compile | 22m 12s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 20m 7s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 23s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 45s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 23s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 59m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1017m 3s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4928/3/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 1s | | The patch does not
generate ASF License warnings. |
| | | 1405m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4928/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4928 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
shellcheck shelldocs |
| uname | Linux 2afb681928bf 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1d839b6724a934007d48cbc0881464fff1abd8cf |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK
[jira] [Commented] (HADOOP-18471) An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input
[
https://issues.apache.org/jira/browse/HADOOP-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613169#comment-17613169
]
ASF GitHub Bot commented on HADOOP-18471:
-
hadoop-yetus commented on PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#issuecomment-1265105600
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 30s | | trunk passed |
| +1 :green_heart: | compile | 23m 26s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 58s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 22m 38s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 38s | | the patch passed |
| +1 :green_heart: | compile | 21m 28s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not
generate ASF License warnings. |
| | | 219m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4957 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e076bd633e98 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7dcfc810fb95b93d5873fcb14dbbf454acab9dd4 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/2/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> An unhandled
[GitHub] [hadoop] hadoop-yetus commented on pull request #4957: HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in class DefaultStringifier
hadoop-yetus commented on PR #4957:
URL: https://github.com/apache/hadoop/pull/4957#issuecomment-1265105600
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 30s | | trunk passed |
| +1 :green_heart: | compile | 23m 26s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 58s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 22m 38s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 38s | | the patch passed |
| +1 :green_heart: | compile | 21m 28s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 41s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not
generate ASF License warnings. |
| | | 219m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4957 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e076bd633e98 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7dcfc810fb95b93d5873fcb14dbbf454acab9dd4 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/2/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4957/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please
[GitHub] [hadoop] hadoop-yetus commented on pull request #4928: HADOOP-18360. Update commons-csv from 1.0 to 1.9.0.
hadoop-yetus commented on PR #4928:
URL: https://github.com/apache/hadoop/pull/4928#issuecomment-1265088527
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 47s | | trunk passed |
| +1 :green_heart: | compile | 25m 37s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 2s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 34s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 70m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 26m 9s | | the patch passed |
| +1 :green_heart: | compile | 24m 58s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 24m 58s | | the patch passed |
| +1 :green_heart: | compile | 22m 12s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 20m 7s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 23s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 45s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 23s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 59m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1017m 3s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4928/3/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 1s | | The patch does not
generate ASF License warnings. |
| | | 1405m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4928/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4928 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
shellcheck shelldocs |
| uname | Linux 2afb681928bf 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1d839b6724a934007d48cbc0881464fff1abd8cf |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
[GitHub] [hadoop] riyakhdl closed pull request #4959: YARN-6766 added HelperMethod for AppsBlock Class
riyakhdl closed pull request #4959: YARN-6766 added HelperMethod for AppsBlock Class URL: https://github.com/apache/hadoop/pull/4959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] pranavsaxena-microsoft commented on pull request #3440: HADOOP-17912. ABFS: Support for Encryption Context
pranavsaxena-microsoft commented on PR #3440: URL: https://github.com/apache/hadoop/pull/3440#issuecomment-1265014148 @steveloughran , requesting you to kindly review the PR. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4962: HDFS-16024: RBF: Rename data to the Trash should be based on src locations
hadoop-yetus commented on PR #4962:
URL: https://github.com/apache/hadoop/pull/4962#issuecomment-1264912310
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 38s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 51s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 42s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 54s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 1m 12s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 39s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 24m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 44s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 120m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4962/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4962 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 29ab37fcaf3b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 278374852bba7adfeb51f7fa3d2fcce709df337b |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4962/1/testReport/ |
| Max. process+thread count | 2592 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4962/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18360) Update commons-csv from 1.0 to 1.9.0.
[ https://issues.apache.org/jira/browse/HADOOP-18360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613148#comment-17613148 ] ASF GitHub Bot commented on HADOOP-18360: - slfan1989 commented on PR #4928: URL: https://github.com/apache/hadoop/pull/4928#issuecomment-1264773924 @ayushtkn Can you help review this pr? I think an upgrade is still necessary. For the deprecated method, I also made a replacement. > Update commons-csv from 1.0 to 1.9.0. > - > > Key: HADOOP-18360 > URL: https://issues.apache.org/jira/browse/HADOOP-18360 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > > commons-csv 1.0 is a very old jar, mvnrepository shows this jar as the 2014 > version, I have compiled and tested locally, I think this jar can be upgraded > to commons-csv 1.9 version. > The link to the release note is as follows: > [https://commons.apache.org/proper/commons-csv/changes-report.html] > We can see that the new version fixes some issues. > I read the code used, we use header related methods. We found that many > header-related methods have been upgraded. > *Release 1.1 – 2014-11-16* > CSVFormat#withHeader doesn't work well with #printComment, add > withHeaderComments(String...). > CSVFormat.EXCEL should ignore empty header names. > *Release 1.2 – 2015-08-24* > CSVFormat.with* methods clear the header comments. > *Release 1.3 – 2016-05-09* > Add shortcut method for using first record as header to CSVFormat. > Add withHeader(Class) to CSVFormat. > CSVPrinter doesn't skip creation of header record if skipHeaderRecord is set > to true. > Add IgnoreCase option for accessing header names. > *Release 1.5 – 2017-09-03* > Fix incorrect method name 'withFirstRowAsHeader' in user guide. > *Release 1.7 – 2019-06-01* > Cannot get headers in column order from CSVRecord. > *Release 1.8 – 2020-02-01* > CSVFormat#validate() does not account for allowDuplicateHeaderNames. > A single empty header is allowed when not allowing empty column headers. > *Release 1.9.0 – 2020-07-24* > Add possibility to use ResultSet header meta data as CSV header. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin opened a new pull request, #4962: HDFS-16024: RBF: Rename data to the Trash should be based on src locations
xinglin opened a new pull request, #4962: URL: https://github.com/apache/hadoop/pull/4962 (cherry picked from commit e18d8062126951b01d04d14374cc04053167735b) ### Description of PR backport of HDFS-16024 from trunk to branch-3.3. minor conflict resolved. The conflict is due to some code refactoring done in HDFS-15766. Not include HDFS-15766 as part of this PR (That should be done as another backport PR). ### How was this patch tested? mvn test -Dtest="TestRouterRpc" ``` [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hdfs.server.federation.router.TestRouterRpc [INFO] Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.225 s - in org.apache.hadoop.hdfs.server.federation.router.TestRouterRpc [INFO] [INFO] Results: [INFO] [INFO] Tests run: 47, Failures: 0, Errors: 0, Skipped: 0 ``` ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4961: YARN-11324. Fix some PBImpl classes to avoid NPE.
hadoop-yetus commented on PR #4961:
URL: https://github.com/apache/hadoop/pull/4961#issuecomment-1264827407
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4961/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 1m 35s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4961/1/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) |
| +1 :green_heart: | shadedclient | 27m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 3m 4s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4961/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
| hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 111m 41s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| | Null passed for non-null parameter of
convertToProtoFormat(SubClusterId) in
org.apache.hadoop.yarn.server.federation.store.records.impl.pb.ApplicationHomeSubClusterPBImpl.setHomeSubCluster(SubClusterId)
Method invoked at ApplicationHomeSubClusterPBImpl.java:of
convertToProtoFormat(SubClusterId) in
org.apache.hadoop.yarn.server.federation.store.records.impl.pb.ApplicationHomeSubClusterPBImpl.setHomeSubCluster(SubClusterId)
Method invoked at ApplicationHomeSubClusterPBImpl.java:[line 155] |
| | Null passed for non-null parameter of
[GitHub] [hadoop] andreoss commented on pull request #4910: YARN-11262: [hadoop-yarn-server-resourcemanager] Upgrade to Junit 5
andreoss commented on PR #4910: URL: https://github.com/apache/hadoop/pull/4910#issuecomment-1264823311 @ashutoshcipher can i continue to work on that or should i take something else? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4928: HADOOP-18360. Update commons-csv from 1.0 to 1.9.0.
slfan1989 commented on PR #4928: URL: https://github.com/apache/hadoop/pull/4928#issuecomment-1264773924 @ayushtkn Can you help review this pr? I think an upgrade is still necessary. For the deprecated method, I also made a replacement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 opened a new pull request, #4961: YARN-11324. Fix some PBImpl classes to avoid NPE.
slfan1989 opened a new pull request, #4961:
URL: https://github.com/apache/hadoop/pull/4961
JIRA: YARN-11324. Fix some PBImpl classes to avoid NPE.
When completing
[YARN-11323](https://issues.apache.org/jira/browse/YARN-11323), I found that
there is a bug in ApplicationHomeSubClusterPBImpl, which may cause a null
pointer exception when getting getApplicationId
```
@Test
public void testGetApplicationIdNullException() throws YarnException {
ApplicationId appId = ApplicationId.newInstance(Time.now(), 1);
ApplicationHomeSubCluster appHomeSC =
ApplicationHomeSubCluster.newInstance(
appId, subClusterId);
System.out.println(appHomeSC.getApplicationId());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4960: YARN-6766 HelperMethod added in AppsBlock class
hadoop-yetus commented on PR #4960:
URL: https://github.com/apache/hadoop/pull/4960#issuecomment-1264748308
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 1s | | trunk passed |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 25s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 211m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4960/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4698e47cacf8 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 13f76871cf60513f0980a9215d5b6c4c1f0d51c9 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4960/3/testReport/ |
| Max. process+thread count | 890 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4960/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated
[GitHub] [hadoop] riyakhdl commented on a diff in pull request #4960: YARN-6766 HelperMethod added in AppsBlock class
riyakhdl commented on code in PR #4960:
URL: https://github.com/apache/hadoop/pull/4960#discussion_r985277193
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java:
##
@@ -128,8 +128,15 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest
hsr) {
}
return true;
}
-
-
+ private long appInfoFunc;
+ public String printData(long appInfoFunc) {
+this.appInfoFunc=appInfoFunc;
Review Comment:
Was doing this to resolve the error : 'appInfoFunc' hides a field.
[HiddenField]
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java:
##
@@ -128,8 +128,15 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest
hsr) {
}
return true;
}
-
-
+ private long appInfoFunc;
+ public String printData(long appInfoFunc) {
+this.appInfoFunc=appInfoFunc;
Review Comment:
Was doing this to resolve the error : 'appInfoFunc' hides a field.
[HiddenField]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4960: YARN-6766 HelperMethod added in AppsBlock class
goiri commented on code in PR #4960:
URL: https://github.com/apache/hadoop/pull/4960#discussion_r985272139
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java:
##
@@ -128,8 +128,15 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest
hsr) {
}
return true;
}
-
-
+ private long appInfoFunc;
+ public String printData(long appInfoFunc) {
+this.appInfoFunc=appInfoFunc;
+if (appInfoFunc == -1) {
+ return "N/A";
+} else {
+ return String.valueOf(appInfoFunc);
Review Comment:
No need for else, just return.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java:
##
@@ -128,8 +128,15 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest
hsr) {
}
return true;
}
-
-
+ private long appInfoFunc;
+ public String printData(long appInfoFunc) {
Review Comment:
the name of printData() is a little too general.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java:
##
@@ -128,8 +128,15 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest
hsr) {
}
return true;
}
-
-
+ private long appInfoFunc;
+ public String printData(long appInfoFunc) {
+this.appInfoFunc=appInfoFunc;
Review Comment:
Why are we saving this? We are not using it later. Were you trying to cache
here?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerAppsBlock.java:
##
@@ -128,8 +128,15 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest
hsr) {
}
return true;
}
-
-
+ private long appInfoFunc;
+ public String printData(long appInfoFunc) {
+this.appInfoFunc=appInfoFunc;
+if (appInfoFunc == -1) {
+ return "N/A";
+} else {
+ return String.valueOf(appInfoFunc);
+}
+ }
Review Comment:
Breakline after and before the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4960: YARN-6766 HelperMethod added in AppsBlock class
hadoop-yetus commented on PR #4960:
URL: https://github.com/apache/hadoop/pull/4960#issuecomment-1264681215
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 8s | | trunk passed |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 45s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4960/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) |
| +1 :green_heart: | mvnsite | 0m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 103m 4s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 211m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4960/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fed8e4600920 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e7187ae612e62c284c3bc8bc85985197de1e7e7a |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4960/2/testReport/ |
| Max. process+thread count | 912 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4938: YARN-8041. [Router] Federation: Improve Router REST API Metrics.
hadoop-yetus commented on PR #4938:
URL: https://github.com/apache/hadoop/pull/4938#issuecomment-1264661770
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4938/14/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 57s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 103m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4938/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4938 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 18ebbe11d7b6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 649b960a3631102c95e49665a0de094b1f103a0d |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4938/14/testReport/ |
| Max. process+thread count | 778 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613101#comment-17613101
]
Steve Loughran commented on HADOOP-18481:
-
thanks, there's probably a few which can be silently migrated
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613069#comment-17613069
]
Ahmar Suhail commented on HADOOP-18481:
---
Opened [PR|https://github.com/apache/hadoop/pull/4973]. Will update class name
as part of V2 move
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18382) Upgrade AWS SDK to V2 - Prerequisites
[ https://issues.apache.org/jira/browse/HADOOP-18382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18382. - Resolution: Fixed this is in 3.4.0, but i'd like to get it in 3.3.5. [~ahmarsu] do you want to see about testing this cherrypicked into branch-3.3. and back. HADOOP-18481 would be a requirement first though > Upgrade AWS SDK to V2 - Prerequisites > -- > > Key: HADOOP-18382 > URL: https://issues.apache.org/jira/browse/HADOOP-18382 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > We want to update the AWS SDK to V2, before we do this we should warn on > things that will no longer supported. The following changes should be made: > > * > [getAmazonS3Client()|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1174] > - Warn that this method will be removed > * > [initCustomSigners()|https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/SignerManager.java#L65] > - Warn that the interface is changing, any custom signers will need to be > updated > * > [bindAWSClient|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L840] > - If DT is enabled, warn that credential providers interface is changing, > any custom cred providers used in binding classes will need to be updated > * > [buildAWSProviderList|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L618] > - if any SDK V1 cred providers are in this list, warn that these will be > removed > * > [S3ClientFactory|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java] > - Update javadocs to say this interface will be replaced by a V2 client > factory, mark interface deprecated? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
Steve Loughran created HADOOP-18481:
---
Summary: AWS v2 SDK warning to skip warning of
EnvironmentVariableCredentialsProvider
Key: HADOOP-18481
URL: https://issues.apache.org/jira/browse/HADOOP-18481
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Ahmar Suhail
looking at test output with the sdk warnings enabled, it is now always warning
of a v1 provider reference, even if the user hasn't set any
fs.s3a.credential.provider option
{code}
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
(LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK V1
credential providers will be removed once S3A is upgraded to SDK V2
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
com.amazonaws.auth.EnvironmentVariableCredentialsProvider
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
{code}
This is because the EnvironmentVariableCredentialsProvider provider is on the
default list of providers.
Everybody who is using the S3 a connector and who has not explicitly declared a
new set of providers excluding this one will be seeing the error message.
Proposed:
Don't warn on this provider. Instead with the v2 move the classname can be
patched to switch to a modified one.
The alternative would be to provide an s3a specific env var provider subclass
of this; and while that is potentially good in future it is a bit more effort
for the forthcoming 3.3.5 release.
And especially because and it will not be in previous versions people cannot
explicitly switch to it in their configs and be confident it will always be
there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18382) Upgrade AWS SDK to V2 - Prerequisites
[ https://issues.apache.org/jira/browse/HADOOP-18382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18382: Fix Version/s: 3.4.0 > Upgrade AWS SDK to V2 - Prerequisites > -- > > Key: HADOOP-18382 > URL: https://issues.apache.org/jira/browse/HADOOP-18382 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > We want to update the AWS SDK to V2, before we do this we should warn on > things that will no longer supported. The following changes should be made: > > * > [getAmazonS3Client()|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1174] > - Warn that this method will be removed > * > [initCustomSigners()|https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/SignerManager.java#L65] > - Warn that the interface is changing, any custom signers will need to be > updated > * > [bindAWSClient|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L840] > - If DT is enabled, warn that credential providers interface is changing, > any custom cred providers used in binding classes will need to be updated > * > [buildAWSProviderList|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L618] > - if any SDK V1 cred providers are in this list, warn that these will be > removed > * > [S3ClientFactory|https://github.com/apache/hadoop/blob/221eb2d68d5b52e4394fd36cb30d5ee9ffeea7f0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java] > - Update javadocs to say this interface will be replaced by a V2 client > factory, mark interface deprecated? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18480) upgrade AWS SDK for release 3.3.5
Steve Loughran created HADOOP-18480: --- Summary: upgrade AWS SDK for release 3.3.5 Key: HADOOP-18480 URL: https://issues.apache.org/jira/browse/HADOOP-18480 Project: Hadoop Common Issue Type: Sub-task Components: build, fs/s3 Affects Versions: 3.3.5 Reporter: Steve Loughran Assignee: Steve Loughran go up to the latest sdk through the usual qualification process. no doubt it'll be bigger... -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18159) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
[
https://issues.apache.org/jira/browse/HADOOP-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612986#comment-17612986
]
Steve Loughran commented on HADOOP-18159:
-
created https://github.com/aws/aws-sdk-java/issues/2860 for the AWS SDK team
> Certificate doesn't match any of the subject alternative names:
> [*.s3.amazonaws.com, s3.amazonaws.com]
> --
>
> Key: HADOOP-18159
> URL: https://issues.apache.org/jira/browse/HADOOP-18159
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.1, 3.3.2, 3.3.3
> Environment: hadoop 3.3.1
> httpclient 4.5.13
> JDK8
>Reporter: André F.
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h2. If you see this error message when trying to use s3a:// or gs:// URLs,
> look for copies of cos_api-bundle.jar on your classpath and remove them.
> Libraries which include shaded apache httpclient libraries
> (hadoop-client-runtime.jar, aws-java-sdk-bundle.jar,
> gcs-connector-shaded.jar, cos_api-bundle.jar) all load and use the unshaded
> resource mozilla/public-suffix-list.txt. If an out of date version of this is
> found on the classpath first, attempts to negotiate TLS connections may fail
> with the error "Certificate doesn't match any of the subject alternative
> names".
> In a hadoop installation, you can use the findclass tool to track down where
> the public-suffix-list.txt is coming from.
> {code}
> hadoop org.apache.hadoop.util.FindClass locate mozilla/public-suffix-list.txt
> {code}
> So far, the cos_api-bundle-5.6.19.jar appears to be the source of this
> problem.
>
> h2. bug report
> Trying to run any job after bumping our Spark version (which is now using
> Hadoop 3.3.1), lead us to the current exception while reading files on s3:
> {code:java}
> org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a:///.parquet: com.amazonaws.SdkClientException: Unable to
> execute HTTP request: Certificate for doesn't match
> any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]:
> Unable to execute HTTP request: Certificate for doesn't match any of
> the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170) at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3351)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4277)
> at {code}
>
> {code:java}
> Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for
> doesn't match any of the subject alternative names:
> [*.s3.amazonaws.com, s3.amazonaws.com]
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> at
> com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at
> com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy16.connect(Unknown Source)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> at
> com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
>
[jira] [Resolved] (HADOOP-15807) ITestS3AContractRootDir failure on non-S3Guarded bucket
[ https://issues.apache.org/jira/browse/HADOOP-15807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15807. - Resolution: Cannot Reproduce > ITestS3AContractRootDir failure on non-S3Guarded bucket > --- > > Key: HADOOP-15807 > URL: https://issues.apache.org/jira/browse/HADOOP-15807 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > Got a root test failure against S3 London, possibly consistency related. > The abstract test case should use eventually() here -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612944#comment-17612944 ] Steve Loughran commented on HADOOP-18448: - [~einavhol] in HADOOP-17705 there is documentation on a workaround known to work with s3a on older releases. if you try it on emr, let us know if it worked. > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Environment: Amazon EMR emr-6.5.0 cluster >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--conf', "spark.hadoop.fs.s3a.path.style.access=false" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16666) typo in TestNeworkBinding
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Carl Jones resolved HADOOP-1. Fix Version/s: 3.3.1 Target Version/s: 3.3.1 Resolution: Fixed Addressed by HADOOP-17271, resolving. > typo in TestNeworkBinding > - > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.1 > > > the test TestNeworkBinding should be TestNetworkBinding -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18479) Update jackson-databind to mitigate CVE-2022-42003
[ https://issues.apache.org/jira/browse/HADOOP-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18479: Component/s: build > Update jackson-databind to mitigate CVE-2022-42003 > -- > > Key: HADOOP-18479 > URL: https://issues.apache.org/jira/browse/HADOOP-18479 > Project: Hadoop Common > Issue Type: Improvement > Components: build, security >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > Update jackson-databind to mitigate CVE-2022-42003 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17942) abfs & s3a FS instantiate triggers warning about deprecated io.bytes.per.checksum
[
https://issues.apache.org/jira/browse/HADOOP-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17942.
-
Resolution: Duplicate
> abfs & s3a FS instantiate triggers warning about deprecated
> io.bytes.per.checksum
> -
>
> Key: HADOOP-17942
> URL: https://issues.apache.org/jira/browse/HADOOP-17942
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, conf
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If you don't turn off the deprecation log, you get told off about
> dfs.bytes-per-checksum
> {code}
> 2021-09-28 15:40:26,551 INFO Configuration.deprecation: io.bytes.per.checksum
> is deprecated. Instead, use dfs.bytes-per-checksum
> {code}
> proposed
> * find out where it's used/set
> * stop it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
