[jira] [Commented] (HADOOP-18840) Add enQueue time to RpcMetrics
[
https://issues.apache.org/jira/browse/HADOOP-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752283#comment-17752283
]
ASF GitHub Bot commented on HADOOP-18840:
-
xinglin commented on PR #5926:
URL: https://github.com/apache/hadoop/pull/5926#issuecomment-1670753258
> > @2005hithlj Please check the failed unit test and checkstyle.
>
> @Hexiaoqiao sir. The failed UT 'TestRPCCallBenchmark is not related to my
modification, and it will success in my modification branch in my local env.
>
> And Checkstyle Warn: RpcMetrics.java:122: @Metric("EQueue time")
MutableRate rpcEnQueueTime;:38: Variable 'rpcEnQueueTime' must be private and
have accessor methods. [VisibilityModifier] RpcMetrics.java:123:
MutableQuantiles[] rpcEnQueueTimeQuantiles;:22: Variable
'rpcEnQueueTimeQuantiles' must be private and have accessor methods.
[VisibilityModifier]
>
> If I follow the above prompts to modify, it will break the style of the
RpcMetrics class.
I don't know the unit test but yeah, for the checkstyle, we can ignore these
warnings: for metrics, we don't add visibility modifier. None of the existing
metrics has the visibility modifier.
> Add enQueue time to RpcMetrics
> --
>
> Key: HADOOP-18840
> URL: https://issues.apache.org/jira/browse/HADOOP-18840
> Project: Hadoop Common
> Issue Type: Improvement
> Components: rpc-server
>Reporter: Liangjun He
>Assignee: Liangjun He
>Priority: Minor
> Labels: pull-request-available
>
> Sometimes, the request time observed by the client is much longer than the
> queue + process time on the RPC server. Perhaps the RPC request 'waiting
> enQueue' took too long on the RPC server, so we should add enQueue time to
> RpcMetrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on pull request #5926: HADOOP-18840. Add enQueue time to RpcMetrics
xinglin commented on PR #5926:
URL: https://github.com/apache/hadoop/pull/5926#issuecomment-1670753258
> > @2005hithlj Please check the failed unit test and checkstyle.
>
> @Hexiaoqiao sir. The failed UT 'TestRPCCallBenchmark is not related to my
modification, and it will success in my modification branch in my local env.
>
> And Checkstyle Warn: RpcMetrics.java:122: @Metric("EQueue time")
MutableRate rpcEnQueueTime;:38: Variable 'rpcEnQueueTime' must be private and
have accessor methods. [VisibilityModifier] RpcMetrics.java:123:
MutableQuantiles[] rpcEnQueueTimeQuantiles;:22: Variable
'rpcEnQueueTimeQuantiles' must be private and have accessor methods.
[VisibilityModifier]
>
> If I follow the above prompts to modify, it will break the style of the
RpcMetrics class.
I don't know the unit test but yeah, for the checkstyle, we can ignore these
warnings: for metrics, we don't add visibility modifier. None of the existing
metrics has the visibility modifier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider
xinglin commented on PR #5878: URL: https://github.com/apache/hadoop/pull/5878#issuecomment-1670751349 > > Hi @goiri, > > could you take a look at this backport PR for branch-3.3 as well? thanks, > > You'd have to put a separate PR together I'd say. I am confused: this is a separate PR, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on a diff in pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider
xinglin commented on code in PR #5878:
URL: https://github.com/apache/hadoop/pull/5878#discussion_r1288010220
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java:
##
@@ -285,13 +323,67 @@ private synchronized NNProxyInfo
changeProxy(NNProxyInfo initial) {
}
currentIndex = (currentIndex + 1) % nameNodeProxies.size();
currentProxy = createProxyIfNeeded(nameNodeProxies.get(currentIndex));
-currentProxy.setCachedState(getHAServiceState(currentProxy));
+currentProxy.setCachedState(getHAServiceStateWithTimeout(currentProxy));
LOG.debug("Changed current proxy from {} to {}",
initial == null ? "none" : initial.proxyInfo,
currentProxy.proxyInfo);
return currentProxy;
}
+ /**
+ * Execute getHAServiceState() call with a timeout, to avoid a long wait when
+ * an NN becomes irresponsive to rpc requests
+ * (when a thread/heap dump is being taken, e.g.).
+ *
+ * For each getHAServiceState() call, a task is created and submitted to a
+ * threadpool for execution. We will wait for a response up to
+ * namenodeHAStateProbeTimeoutSec and cancel these requests if they time out.
+ *
+ * The implementation is split into two functions so that we can unit test
+ * the second function.
+ */
+ HAServiceState getHAServiceStateWithTimeout(final NNProxyInfo proxyInfo) {
+Callable getHAServiceStateTask = () ->
getHAServiceState(proxyInfo);
+
+try {
+ Future task =
+ nnProbingThreadPool.submit(getHAServiceStateTask);
Review Comment:
fixed. fits in one line with 100 characters. So, did not bother splitting
into two lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on a diff in pull request #5878: HDFS-17030 Limit wait time for getHAServiceState in ObserverReadProxyProvider
xinglin commented on code in PR #5878:
URL: https://github.com/apache/hadoop/pull/5878#discussion_r1288009483
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java:
##
@@ -285,13 +323,67 @@ private synchronized NNProxyInfo
changeProxy(NNProxyInfo initial) {
}
currentIndex = (currentIndex + 1) % nameNodeProxies.size();
currentProxy = createProxyIfNeeded(nameNodeProxies.get(currentIndex));
-currentProxy.setCachedState(getHAServiceState(currentProxy));
+currentProxy.setCachedState(getHAServiceStateWithTimeout(currentProxy));
LOG.debug("Changed current proxy from {} to {}",
initial == null ? "none" : initial.proxyInfo,
currentProxy.proxyInfo);
return currentProxy;
}
+ /**
+ * Execute getHAServiceState() call with a timeout, to avoid a long wait when
+ * an NN becomes irresponsive to rpc requests
+ * (when a thread/heap dump is being taken, e.g.).
+ *
+ * For each getHAServiceState() call, a task is created and submitted to a
+ * threadpool for execution. We will wait for a response up to
+ * namenodeHAStateProbeTimeoutSec and cancel these requests if they time out.
+ *
+ * The implementation is split into two functions so that we can unit test
+ * the second function.
+ */
+ HAServiceState getHAServiceStateWithTimeout(final NNProxyInfo proxyInfo) {
+Callable getHAServiceStateTask = () ->
getHAServiceState(proxyInfo);
+
+try {
+ Future task =
+ nnProbingThreadPool.submit(getHAServiceStateTask);
+ return getHAServiceStateWithTimeout(proxyInfo, task);
+} catch (RejectedExecutionException e) {
+ LOG.warn("Run out of threads to submit the request to query HA state. "
+ + "Ok to return null and we will fallback to use active NN to serve "
+ + "this request.");
+ return null;
+}
+ }
+
+ HAServiceState getHAServiceStateWithTimeout(final NNProxyInfo proxyInfo,
+ Future task) {
+HAServiceState state = null;
+try {
+ if (namenodeHAStateProbeTimeoutMs > 0) {
+state = task.get(namenodeHAStateProbeTimeoutMs, TimeUnit.MILLISECONDS);
+ } else {
+// Disable timeout by waiting indefinitely when
namenodeHAStateProbeTimeoutSec is set to 0
+// or a negative value.
+state = task.get();
+ }
+ LOG.debug("HA State for {} is {}", proxyInfo.proxyInfo, state);
+} catch (TimeoutException e) {
+ // Cancel the task on timeout
+ String msg = String.format("Cancel NN probe task due to timeout for %s",
proxyInfo.proxyInfo);
+ LOG.warn(msg, e);
+ if (task != null) {
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anujmodi2021 commented on pull request #5909: Hadoop 18826: [ABFS] Fix for Empty Relative Path Issue Leading to GetFileStatus("/") failure.
anujmodi2021 commented on PR #5909: URL: https://github.com/apache/hadoop/pull/5909#issuecomment-1670620744 > merged, doing a backport Thanks a lot Steve... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18840) Add enQueue time to RpcMetrics
[
https://issues.apache.org/jira/browse/HADOOP-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752231#comment-17752231
]
ASF GitHub Bot commented on HADOOP-18840:
-
2005hithlj commented on PR #5926:
URL: https://github.com/apache/hadoop/pull/5926#issuecomment-1670549206
> @2005hithlj Please check the failed unit test and checkstyle.
@Hexiaoqiao sir.
The failed UT 'TestRPCCallBenchmark is not related to my modification, and
it will success in my modification branch in my local env.
And Checkstyle Warn:
RpcMetrics.java:122: @Metric("EQueue time") MutableRate rpcEnQueueTime;:38:
Variable 'rpcEnQueueTime' must be private and have accessor methods.
[VisibilityModifier]
RpcMetrics.java:123: MutableQuantiles[] rpcEnQueueTimeQuantiles;:22:
Variable 'rpcEnQueueTimeQuantiles' must be private and have accessor methods.
[VisibilityModifier]
If I follow the above prompts to modify, it will break the style of the
RpcMetrics class.
> Add enQueue time to RpcMetrics
> --
>
> Key: HADOOP-18840
> URL: https://issues.apache.org/jira/browse/HADOOP-18840
> Project: Hadoop Common
> Issue Type: Improvement
> Components: rpc-server
>Reporter: Liangjun He
>Assignee: Liangjun He
>Priority: Minor
> Labels: pull-request-available
>
> Sometimes, the request time observed by the client is much longer than the
> queue + process time on the RPC server. Perhaps the RPC request 'waiting
> enQueue' took too long on the RPC server, so we should add enQueue time to
> RpcMetrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 2005hithlj commented on pull request #5926: HADOOP-18840. Add enQueue time to RpcMetrics
2005hithlj commented on PR #5926:
URL: https://github.com/apache/hadoop/pull/5926#issuecomment-1670549206
> @2005hithlj Please check the failed unit test and checkstyle.
@Hexiaoqiao sir.
The failed UT 'TestRPCCallBenchmark is not related to my modification, and
it will success in my modification branch in my local env.
And Checkstyle Warn:
RpcMetrics.java:122: @Metric("EQueue time") MutableRate rpcEnQueueTime;:38:
Variable 'rpcEnQueueTime' must be private and have accessor methods.
[VisibilityModifier]
RpcMetrics.java:123: MutableQuantiles[] rpcEnQueueTimeQuantiles;:22:
Variable 'rpcEnQueueTimeQuantiles' must be private and have accessor methods.
[VisibilityModifier]
If I follow the above prompts to modify, it will break the style of the
RpcMetrics class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400
[
https://issues.apache.org/jira/browse/HADOOP-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18826.
-
Fix Version/s: 3.4.0
3.3.9
Resolution: Fixed
> abfs getFileStatus(/) fails with "Value for one of the query parameters
> specified in the request URI is invalid.", 400
> --
>
> Key: HADOOP-18826
> URL: https://issues.apache.org/jira/browse/HADOOP-18826
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.1, 3.3.2, 3.3.5, 3.3.3, 3.3.4, 3.3.6
>Reporter: Sergey Shabalov
>Assignee: Anuj Modi
>Priority: Major
> Fix For: 3.4.0, 3.3.9
>
> Attachments: test_hadoop-azure-3_3_1-FileSystem_getFileStatus -
> Copy.zip
>
>
> I am using hadoop-azure-3.3.0.jar and have written code:
> {code:java}
> static final String ROOT_DIR =
> "abfs://ssh-test...@sshadlsgen2.dfs.core.windows.net",
> Configuration config = new Configuration();
> config.set("fs.defaultFS",ROOT_DIR);
> config.set("fs.adl.oauth2.access.token.provider.type","ClientCredential");
> config.set("fs.adl.oauth2.client.id","");
> config.set("fs.adl.oauth2.credential","");
> config.set("fs.adl.oauth2.refresh.url","");
> config.set("fs.azure.account.key.sshadlsgen2.dfs.core.windows.net",ACCESS_TOKEN);
>
> config.set("fs.azure.skipUserGroupMetadataDuringInitialization","true");
> FileSystem fs = FileSystem.get(config);
> System.out.println( "\nfs:'"+fs.toString()+"'");
> FileStatus status = fs.getFileStatus(new Path(ROOT_DIR)); // !!!
> Exception in 3.3.1-*
> System.out.println( "\nstatus:'"+status.toString()+"'");
> {code}
> It did work properly till 3.3.1.
> But in 3.3.1 it fails with exception:
> {code:java}
> Caused by: Operation failed: "Value for one of the query parameters specified
> in the request URI is invalid.", 400, HEAD,
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:218)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:181)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:179)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:942)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:924)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:846)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:507)
> {code}
> I performed some research and found:
> In hadoop-azure-3.3.0.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore{
> ...
> public FileStatus getFileStatus(final Path path) throws IOException {
> ...
> Line 604: op =
> client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH +
> AbfsHttpConstants.ROOT_PATH);
> ...
> }
> ...
> } {code}
> and this code produces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs//?upn=false&action=getAccessControl&timeout=90
> {code}
> There is finalizes slash in path part
> "...ssh-test-fs{*}{color:#de350b}//{color}{*}?upn=false..." This request does
> work properly.
> But since hadoop-azure-3.3.1.jar till latest hadoop-azure-3.3.6.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore {
> ...
> public FileStatus getFileStatus(final Path path) throws IOException {
> ...
> perfInfo.registerCallee("getAclStatus");
> Line 846: op = client.getAclStatus(getRelativePath(path));
> ...
> }
> ...
> }
> Line 1492:
> private String getRelativePath(final Path path) {
> ...
> return path.toUri().getPath();
> } {code}
> and this code prduces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
> {code}
> There is not finalizes slash in path part "...ssh-test-fs?upn=false..." It
> happens because the new code "path.toUri().getPath();" produces empty string.
> This request fails with message:
> {code:java}
> Caused by: Operation failed: "Value for one of the
[GitHub] [hadoop] steveloughran commented on pull request #5909: Hadoop 18826: [ABFS] Fix for Empty Relative Path Issue Leading to GetFileStatus("/") failure.
steveloughran commented on PR #5909: URL: https://github.com/apache/hadoop/pull/5909#issuecomment-167001 merged, doing a backport -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5909: Hadoop 18826: [ABFS] Fix for Empty Relative Path Issue Leading to GetFileStatus("/") failure.
steveloughran merged PR #5909: URL: https://github.com/apache/hadoop/pull/5909 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18757) S3A Committer only finalizes the commits in a single thread
[
https://issues.apache.org/jira/browse/HADOOP-18757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752147#comment-17752147
]
ASF GitHub Bot commented on HADOOP-18757:
-
steveloughran commented on code in PR #5706:
URL: https://github.com/apache/hadoop/pull/5706#discussion_r1287471261
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitContext.java:
##
@@ -236,7 +236,7 @@ private ExecutorService buildThreadPool(
.setDaemon(true)
.setNameFormat(THREAD_PREFIX + jobId + "-%d")
.build();
-return new HadoopThreadPoolExecutor(0, numThreads,
+return new HadoopThreadPoolExecutor(numThreads, numThreads,
Review Comment:
you are going to have to go with it for now; i think we should be thinking
about a 3.3.7 before long as we have some other abfs and s3a issues causing
pain...
> S3A Committer only finalizes the commits in a single thread
> ---
>
> Key: HADOOP-18757
> URL: https://issues.apache.org/jira/browse/HADOOP-18757
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.5
>Reporter: Moditha Hewasinghage
>Assignee: Moditha Hewasinghage
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> S3A Committer is being bottle-necked on the driver when finalizing the
> commits. It seems like only a single thread is being used to finalize the
> commit. In the experiment we are saving 36,000 files ending committing for
> almost 2 hours each file taking 0.1 - 0.5 seconds while all the executors
> stay idle while the driver commits. I have attached the driver log snippets
> to support this theory in comparison to spark 3.4.0.
> The most likely reason is the usage of
> [ThreadPoolExecutor]([https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html])
> in
> [https://github.com/apache/hadoop/blob/706d88266abcee09ed78fbaa0ad5f74d818ab0e9/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitContext.java#L239-L244]
> where an LinkedBlockingQueue is used with a corePoolSize of 0 which ends up
> with a single thread and no new threads created.
> From the documentation
> {code:java}
> Unbounded queues. Using an unbounded queue (for example a LinkedBlockingQueue
> without a predefined capacity) will cause new tasks to wait in the queue when
> all corePoolSize threads are busy. Thus, no more than corePoolSize threads
> will ever be created. (And the value of the maximumPoolSize therefore doesn't
> have any effect.) This may be appropriate when each task is completely
> independent of others, so tasks cannot affect each others execution; for
> example, in a web page server. While this style of queuing can be useful in
> smoothing out transient bursts of requests, it admits the possibility of
> unbounded work queue growth when commands continue to arrive on average
> faster than they can be processed.{code}
> Magic Committer spark 3.5.0-SNAPSHOT
> {code:java}
> 2023-05-26 15:35:04,852 DEBUG impl.CommitContext: creating thread pool of
> size 32
> 2023-05-26 15:35:04,922 INFO yarn.YarnAllocator: Driver requested a total
> number of 0 executor(s) for resource profile id: 0.
> 2023-05-26 15:35:07,910 INFO commit.AbstractS3ACommitter: Starting:
> committing the output of 36000 task(s)
> 2023-05-26 15:35:07,914 INFO commit.AbstractS3ACommitter: Starting: Loading
> and committing files in pendingset
> s3:date=20230520/__magic/job-b9167b1a-35a5-448
> e-8022-31e0f7457354/00/task_202305261454174742717183892533843_0031_m_00.pendingset
> 2023-05-26 15:35:07,941 DEBUG files.PersistentCommitData: Reading commit data
> from file
> s3:date=20230520/__magic/job-b9167b1a-35a5-448e-8022-31e0f7457354/00/
> task_202305261454174742717183892533843_0031_m_00.pendingset
> 2023-05-26 15:35:08,036 DEBUG impl.CommitContext: creating thread pool of
> size 32
> 2023-05-26 15:35:08,037 DEBUG impl.CommitOperations: Committing single commit
> SinglePendingCommit
> {version=2,
> uri='s3:date=20230520/__magic/job-b9167b1a-35a5-
> 448e-8022-31e0f7457354/00/tasks/attempt_202305261454174742717183892533843_0031_m_00_1920855/__base/rawRequestType=imp_dsp/part-0-b196e04e-c87b-43e1-97e0-7ad34ef0bedf.c000.zstd.parquet',
>
> destination='/date=20230520/rawRequestType=imp_dsp/part-0-b196e04e-c87b-43e1-97e0-7ad34ef0bedf.c000.zstd.parquet',
>
> uploadId='SCp78J9aYOmnPwrtfd5a.Q6B9Zu6olQw3eZcpyt.W.BCu6.M6fz54nlTe2ZYUDicSskFgPocsjlVXAbfiPn3Xu
> 26MzXNdWcg5j_kBGg9iWpeoWh4K21gt7bbKetML95MXAck15yP.VGFeOleoAspUg--',
> created=1685113555232, saved=1685113555232, size=110636173, date='Fri May 26
> 15:05:55 UTC 2023', jobId='b9167b1a-35a5-448e-8022-3
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5706: HADOOP-18757: Bump corePoolSize of HadoopThreadPoolExecutor in s3a committer
steveloughran commented on code in PR #5706: URL: https://github.com/apache/hadoop/pull/5706#discussion_r1287471261 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitContext.java: ## @@ -236,7 +236,7 @@ private ExecutorService buildThreadPool( .setDaemon(true) .setNameFormat(THREAD_PREFIX + jobId + "-%d") .build(); -return new HadoopThreadPoolExecutor(0, numThreads, +return new HadoopThreadPoolExecutor(numThreads, numThreads, Review Comment: you are going to have to go with it for now; i think we should be thinking about a 3.3.7 before long as we have some other abfs and s3a issues causing pain... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop-thirdparty] fredbalves86 commented on pull request #23: HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
fredbalves86 commented on PR #23: URL: https://github.com/apache/hadoop-thirdparty/pull/23#issuecomment-1670048204 @jojochuang Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18838) Some fs.s3a.* config values are different in sources and documentation
[
https://issues.apache.org/jira/browse/HADOOP-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18838:
Priority: Minor (was: Major)
> Some fs.s3a.* config values are different in sources and documentation
> --
>
> Key: HADOOP-18838
> URL: https://issues.apache.org/jira/browse/HADOOP-18838
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, fs/s3
>Affects Versions: 3.3.6
>Reporter: Maxim Martynov
>Priority: Minor
>
> For config option {{fs.s3a.retry.throttle.interval}} default value in source
> code is {{500ms}}:
> {code:java}
> public static final String RETRY_THROTTLE_INTERVAL_DEFAULT = "500ms";
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L921
> In {{core-default.xml}} it has value {{100ms}}, but in the description
> {{500ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 100ms
>
> Initial between retry attempts on throttled requests, +/- 50%. chosen at
> random.
> i.e. for an intial value of 3000ms, the initial delay would be in the
> range 1500ms to 4500ms.
> Backoffs are exponential; again randomness is used to avoid the
> thundering heard problem.
> 500ms is the default value used by the AWS S3 Retry policy.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1750
> This change introduced in HADOOP-16823.
> In Hadoop-AWS module documentation it has value {{1000ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 1000ms
>
> Interval between retry attempts on throttled requests.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md?plain=1#L1223
> File was created in HADOOP-13786, and value is left unchanged since when.
> In performance tuning page it has up-to-date value {{500ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 500ms
>
> Interval between retry attempts on throttled requests.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md?plain=1#L435
> This change introduced in HADOOP-15076.
> The same issue with:
> * {{fs.s3a.retry.throttle.limit}} - in source code it has value {{20}}, but
> in some documents still old value ${fs.s3a.attempts.maximum}
> * {{fs.s3a.connection.establish.timeout}} - in source code it has value
> {{50_000}}, in config file & documentation {{5_000}}
> * {{fs.s3a.attempts.maximum}} - in source code it has value {{10}}, in config
> file & documentation {{20}}
> * {{fs.s3a.threads.max}} - in source & documentation code it has value
> {{10}}, in config file {{64}}
> * {{fs.s3a.max.total.tasks}} - in source code & config it has value {{32}},
> in documentation {{5}}
> * {{fs.s3a.connection.maximum}} - in source code & config it has value
> {{96}}, in documentation {{15}} or {{30}}
> Please sync these values, outdated documentation is very painful to work with.
> As an idea, is it possible to use {{core-default.xml}} directly in
> documentation, or generate this documentation from docstrings in Java code?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18838) Some fs.s3a.* config values are different in sources and documentation
[
https://issues.apache.org/jira/browse/HADOOP-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752145#comment-17752145
]
Steve Loughran commented on HADOOP-18838:
-
you can actually get a dump of the live cluster settings from the hadoop hdfs
namenode; this ensures that you get the *real* values, rather than just the
default.
things are different in cloud-based deployment, which, given your minio
configuration issues, seems to be what you are doing. For them, can i point you
at cloudstore.jar, whose diagnostics handles things like s3a per bucket
settings, gradually ramps up io, including giving incremental diagnostics. you
may find it useful: https://github.com/steveloughran/cloudstore/tree/main
> Some fs.s3a.* config values are different in sources and documentation
> --
>
> Key: HADOOP-18838
> URL: https://issues.apache.org/jira/browse/HADOOP-18838
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.6
>Reporter: Maxim Martynov
>Priority: Major
>
> For config option {{fs.s3a.retry.throttle.interval}} default value in source
> code is {{500ms}}:
> {code:java}
> public static final String RETRY_THROTTLE_INTERVAL_DEFAULT = "500ms";
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L921
> In {{core-default.xml}} it has value {{100ms}}, but in the description
> {{500ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 100ms
>
> Initial between retry attempts on throttled requests, +/- 50%. chosen at
> random.
> i.e. for an intial value of 3000ms, the initial delay would be in the
> range 1500ms to 4500ms.
> Backoffs are exponential; again randomness is used to avoid the
> thundering heard problem.
> 500ms is the default value used by the AWS S3 Retry policy.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1750
> This change introduced in HADOOP-16823.
> In Hadoop-AWS module documentation it has value {{1000ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 1000ms
>
> Interval between retry attempts on throttled requests.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md?plain=1#L1223
> File was created in HADOOP-13786, and value is left unchanged since when.
> In performance tuning page it has up-to-date value {{500ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 500ms
>
> Interval between retry attempts on throttled requests.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md?plain=1#L435
> This change introduced in HADOOP-15076.
> The same issue with:
> * {{fs.s3a.retry.throttle.limit}} - in source code it has value {{20}}, but
> in some documents still old value ${fs.s3a.attempts.maximum}
> * {{fs.s3a.connection.establish.timeout}} - in source code it has value
> {{50_000}}, in config file & documentation {{5_000}}
> * {{fs.s3a.attempts.maximum}} - in source code it has value {{10}}, in config
> file & documentation {{20}}
> * {{fs.s3a.threads.max}} - in source & documentation code it has value
> {{10}}, in config file {{64}}
> * {{fs.s3a.max.total.tasks}} - in source code & config it has value {{32}},
> in documentation {{5}}
> * {{fs.s3a.connection.maximum}} - in source code & config it has value
> {{96}}, in documentation {{15}} or {{30}}
> Please sync these values, outdated documentation is very painful to work with.
> As an idea, is it possible to use {{core-default.xml}} directly in
> documentation, or generate this documentation from docstrings in Java code?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18838) Some fs.s3a.* config values are different in sources and documentation
[
https://issues.apache.org/jira/browse/HADOOP-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18838:
Component/s: documentation
> Some fs.s3a.* config values are different in sources and documentation
> --
>
> Key: HADOOP-18838
> URL: https://issues.apache.org/jira/browse/HADOOP-18838
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, fs/s3
>Affects Versions: 3.3.6
>Reporter: Maxim Martynov
>Priority: Major
>
> For config option {{fs.s3a.retry.throttle.interval}} default value in source
> code is {{500ms}}:
> {code:java}
> public static final String RETRY_THROTTLE_INTERVAL_DEFAULT = "500ms";
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L921
> In {{core-default.xml}} it has value {{100ms}}, but in the description
> {{500ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 100ms
>
> Initial between retry attempts on throttled requests, +/- 50%. chosen at
> random.
> i.e. for an intial value of 3000ms, the initial delay would be in the
> range 1500ms to 4500ms.
> Backoffs are exponential; again randomness is used to avoid the
> thundering heard problem.
> 500ms is the default value used by the AWS S3 Retry policy.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1750
> This change introduced in HADOOP-16823.
> In Hadoop-AWS module documentation it has value {{1000ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 1000ms
>
> Interval between retry attempts on throttled requests.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md?plain=1#L1223
> File was created in HADOOP-13786, and value is left unchanged since when.
> In performance tuning page it has up-to-date value {{500ms}}:
> {code:xml}
>
> fs.s3a.retry.throttle.interval
> 500ms
>
> Interval between retry attempts on throttled requests.
>
>
> {code}
> https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md?plain=1#L435
> This change introduced in HADOOP-15076.
> The same issue with:
> * {{fs.s3a.retry.throttle.limit}} - in source code it has value {{20}}, but
> in some documents still old value ${fs.s3a.attempts.maximum}
> * {{fs.s3a.connection.establish.timeout}} - in source code it has value
> {{50_000}}, in config file & documentation {{5_000}}
> * {{fs.s3a.attempts.maximum}} - in source code it has value {{10}}, in config
> file & documentation {{20}}
> * {{fs.s3a.threads.max}} - in source & documentation code it has value
> {{10}}, in config file {{64}}
> * {{fs.s3a.max.total.tasks}} - in source code & config it has value {{32}},
> in documentation {{5}}
> * {{fs.s3a.connection.maximum}} - in source code & config it has value
> {{96}}, in documentation {{15}} or {{30}}
> Please sync these values, outdated documentation is very painful to work with.
> As an idea, is it possible to use {{core-default.xml}} directly in
> documentation, or generate this documentation from docstrings in Java code?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18839) s3a client SSLException is raised after very long timeout "Unsupported or unrecognized SSL message"
[
https://issues.apache.org/jira/browse/HADOOP-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18839:
Summary: s3a client SSLException is raised after very long timeout
"Unsupported or unrecognized SSL message" (was: SSLException is raised after
very long timeout "Unsupported or unrecognized SSL message")
> s3a client SSLException is raised after very long timeout "Unsupported or
> unrecognized SSL message"
> ---
>
> Key: HADOOP-18839
> URL: https://issues.apache.org/jira/browse/HADOOP-18839
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.4
>Reporter: Maxim Martynov
>Priority: Minor
> Attachments: host.log, ssl.log
>
>
> I've tried to connect from PySpark to Minio running in docker.
> Installing PySpark and starting Minio:
> {code:bash}
> pip install pyspark==3.4.1
> docker run --rm -d --hostname minio --name minio -p 9000:9000 -p 9001:9001 -e
> MINIO_ACCESS_KEY=access -e MINIO_SECRET_KEY=Eevoh2wo0ui6ech0wu8oy
> 3feiR3eicha -e MINIO_ROOT_USER=admin -e
> MINIO_ROOT_PASSWORD=iepaegaigi3ofa9TaephieSo1iecaesh bitnami/minio:latest
> docker exec minio mc mb test-bucket
> {code}
> Then create Spark session:
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder\
> .config("spark.jars.packages",
> "org.apache.hadoop:hadoop-aws:3.3.4")\
> .config("spark.hadoop.fs.s3a.endpoint", "localhost:9000")\
> .config("spark.hadoop.fs.s3a.access.key", "access")\
> .config("spark.hadoop.fs.s3a.secret.key",
> "Eevoh2wo0ui6ech0wu8oy3feiR3eicha")\
> .config("spark.hadoop.fs.s3a.aws.credentials.provider",
> "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")\
> .getOrCreate()
> spark.sparkContext.setLogLevel("debug")
> {code}
> And try to access some object in a bucket:
> {code:python}
> import time
> begin = time.perf_counter()
> spark.read.format("csv").load("s3a://test-bucket/fake")
> end = time.perf_counter()
> py4j.protocol.Py4JJavaError: An error occurred while calling o40.load.
> : org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: Unsupported or unrecognized SSL message: Unable to execute HTTP
> request: Unsupported or unrecognized SSL message
> ...
> {code}
> [^ssl.log]
> {code:python}
> >>> print((end-begin)/60)
> 14.72387898775002
> {code}
> I was waiting almost *15 minutes* to get the exception from Spark. The reason
> was I tried to connect to endpoint with
> {{{}fs.s3a.connection.ssl.enabled=true{}}}, but Minio is configured to listen
> for HTTP protocol only.
> Is there any way to immediately raise exception if SSL connection cannot be
> established?
> If I try to pass wrong endpoint, like {{{}localhos:9000{}}}, I'll get
> exception like this in just 5 seconds:
> {code:java}
> : org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: test-bucket.localhos: Unable to execute HTTP request:
> test-bucket.localhos
> ...
> {code}
> [^host.log]
> {code:python}
> >>> print((end-begin)/60)
> 0.09500707178334172
> >>> end-begin
> 5.700424307000503
> {code}
> I know about options like {{fs.s3a.attempts.maximum}} and
> {{{}fs.s3a.retry.limit{}}}, setting them to 1 will cause raising exception
> just immediately. But this does not look right.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18839) SSLException is raised after very long timeout
[
https://issues.apache.org/jira/browse/HADOOP-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752144#comment-17752144
]
Steve Loughran commented on HADOOP-18839:
-
bq. The point of this issue is to provide better developer experience without
overriding default values. Is that possible?
yes, but that is going to need someone to write new code and tests. I am trying
to also suggest any short term workarounds.
> SSLException is raised after very long timeout
> --
>
> Key: HADOOP-18839
> URL: https://issues.apache.org/jira/browse/HADOOP-18839
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.4
>Reporter: Maxim Martynov
>Priority: Minor
> Attachments: host.log, ssl.log
>
>
> I've tried to connect from PySpark to Minio running in docker.
> Installing PySpark and starting Minio:
> {code:bash}
> pip install pyspark==3.4.1
> docker run --rm -d --hostname minio --name minio -p 9000:9000 -p 9001:9001 -e
> MINIO_ACCESS_KEY=access -e MINIO_SECRET_KEY=Eevoh2wo0ui6ech0wu8oy
> 3feiR3eicha -e MINIO_ROOT_USER=admin -e
> MINIO_ROOT_PASSWORD=iepaegaigi3ofa9TaephieSo1iecaesh bitnami/minio:latest
> docker exec minio mc mb test-bucket
> {code}
> Then create Spark session:
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder\
> .config("spark.jars.packages",
> "org.apache.hadoop:hadoop-aws:3.3.4")\
> .config("spark.hadoop.fs.s3a.endpoint", "localhost:9000")\
> .config("spark.hadoop.fs.s3a.access.key", "access")\
> .config("spark.hadoop.fs.s3a.secret.key",
> "Eevoh2wo0ui6ech0wu8oy3feiR3eicha")\
> .config("spark.hadoop.fs.s3a.aws.credentials.provider",
> "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")\
> .getOrCreate()
> spark.sparkContext.setLogLevel("debug")
> {code}
> And try to access some object in a bucket:
> {code:python}
> import time
> begin = time.perf_counter()
> spark.read.format("csv").load("s3a://test-bucket/fake")
> end = time.perf_counter()
> py4j.protocol.Py4JJavaError: An error occurred while calling o40.load.
> : org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: Unsupported or unrecognized SSL message: Unable to execute HTTP
> request: Unsupported or unrecognized SSL message
> ...
> {code}
> [^ssl.log]
> {code:python}
> >>> print((end-begin)/60)
> 14.72387898775002
> {code}
> I was waiting almost *15 minutes* to get the exception from Spark. The reason
> was I tried to connect to endpoint with
> {{{}fs.s3a.connection.ssl.enabled=true{}}}, but Minio is configured to listen
> for HTTP protocol only.
> Is there any way to immediately raise exception if SSL connection cannot be
> established?
> If I try to pass wrong endpoint, like {{{}localhos:9000{}}}, I'll get
> exception like this in just 5 seconds:
> {code:java}
> : org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: test-bucket.localhos: Unable to execute HTTP request:
> test-bucket.localhos
> ...
> {code}
> [^host.log]
> {code:python}
> >>> print((end-begin)/60)
> 0.09500707178334172
> >>> end-begin
> 5.700424307000503
> {code}
> I know about options like {{fs.s3a.attempts.maximum}} and
> {{{}fs.s3a.retry.limit{}}}, setting them to 1 will cause raising exception
> just immediately. But this does not look right.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18839) SSLException is raised after very long timeout "Unsupported or unrecognized SSL message"
[
https://issues.apache.org/jira/browse/HADOOP-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18839:
Summary: SSLException is raised after very long timeout "Unsupported or
unrecognized SSL message" (was: SSLException is raised after very long timeout)
> SSLException is raised after very long timeout "Unsupported or unrecognized
> SSL message"
>
>
> Key: HADOOP-18839
> URL: https://issues.apache.org/jira/browse/HADOOP-18839
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.4
>Reporter: Maxim Martynov
>Priority: Minor
> Attachments: host.log, ssl.log
>
>
> I've tried to connect from PySpark to Minio running in docker.
> Installing PySpark and starting Minio:
> {code:bash}
> pip install pyspark==3.4.1
> docker run --rm -d --hostname minio --name minio -p 9000:9000 -p 9001:9001 -e
> MINIO_ACCESS_KEY=access -e MINIO_SECRET_KEY=Eevoh2wo0ui6ech0wu8oy
> 3feiR3eicha -e MINIO_ROOT_USER=admin -e
> MINIO_ROOT_PASSWORD=iepaegaigi3ofa9TaephieSo1iecaesh bitnami/minio:latest
> docker exec minio mc mb test-bucket
> {code}
> Then create Spark session:
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder\
> .config("spark.jars.packages",
> "org.apache.hadoop:hadoop-aws:3.3.4")\
> .config("spark.hadoop.fs.s3a.endpoint", "localhost:9000")\
> .config("spark.hadoop.fs.s3a.access.key", "access")\
> .config("spark.hadoop.fs.s3a.secret.key",
> "Eevoh2wo0ui6ech0wu8oy3feiR3eicha")\
> .config("spark.hadoop.fs.s3a.aws.credentials.provider",
> "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")\
> .getOrCreate()
> spark.sparkContext.setLogLevel("debug")
> {code}
> And try to access some object in a bucket:
> {code:python}
> import time
> begin = time.perf_counter()
> spark.read.format("csv").load("s3a://test-bucket/fake")
> end = time.perf_counter()
> py4j.protocol.Py4JJavaError: An error occurred while calling o40.load.
> : org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: Unsupported or unrecognized SSL message: Unable to execute HTTP
> request: Unsupported or unrecognized SSL message
> ...
> {code}
> [^ssl.log]
> {code:python}
> >>> print((end-begin)/60)
> 14.72387898775002
> {code}
> I was waiting almost *15 minutes* to get the exception from Spark. The reason
> was I tried to connect to endpoint with
> {{{}fs.s3a.connection.ssl.enabled=true{}}}, but Minio is configured to listen
> for HTTP protocol only.
> Is there any way to immediately raise exception if SSL connection cannot be
> established?
> If I try to pass wrong endpoint, like {{{}localhos:9000{}}}, I'll get
> exception like this in just 5 seconds:
> {code:java}
> : org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: test-bucket.localhos: Unable to execute HTTP request:
> test-bucket.localhos
> ...
> {code}
> [^host.log]
> {code:python}
> >>> print((end-begin)/60)
> 0.09500707178334172
> >>> end-begin
> 5.700424307000503
> {code}
> I know about options like {{fs.s3a.attempts.maximum}} and
> {{{}fs.s3a.retry.limit{}}}, setting them to 1 will cause raising exception
> just immediately. But this does not look right.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18844) Add a way to get the IOStatistics of active filesystems in long-lived processes
Steve Loughran created HADOOP-18844:
---
Summary: Add a way to get the IOStatistics of active filesystems
in long-lived processes
Key: HADOOP-18844
URL: https://issues.apache.org/jira/browse/HADOOP-18844
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure, fs/s3
Affects Versions: 3.3.9
Reporter: Steve Loughran
we can configure abfs and s3a to print out the filesystem IOStats in close(),
but this doesn't let us see the state of long-lived processes, including
latencies, error rates, and other important information.
Proposed: we add a means by which all cached filesystems which provide
IOStatistics can log them to a file/console *while the process continues to run*
* a configurable scheduled thread which enumerates all open filesystem
instances, gets their iostats and if non empty dumps to a log...log
configuration can set that to the console or elsewhere. Good: can run without
intervention; bad: one more thread to manage.
* option to add a signal handler which will do this whenever a SIGUSR2 is
raised. JVM signal handling is a bit of a brittle feature, so it should be off
by default.
{{org.apache.hadoop.service.launcher.IrqHandler}} supports signal wire-up.
good: gives us an equivalent of kill -quit to get a view of iostats. bad: you
need to be on the machine
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
[ https://issues.apache.org/jira/browse/HADOOP-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-18843: - Target Version/s: 3.4.0, 3.3.7 > Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) > > > Key: HADOOP-18843 > URL: https://issues.apache.org/jira/browse/HADOOP-18843 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
[ https://issues.apache.org/jira/browse/HADOOP-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18843: Labels: pull-request-available (was: ) > Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) > > > Key: HADOOP-18843 > URL: https://issues.apache.org/jira/browse/HADOOP-18843 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop-thirdparty] jojochuang commented on pull request #23: HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
jojochuang commented on PR #23: URL: https://github.com/apache/hadoop-thirdparty/pull/23#issuecomment-1669990265 We'd also need to release a new hadoop-thirdparty version so the main hadoop repo can use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
Wei-Chiu Chuang created HADOOP-18843: Summary: Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) Key: HADOOP-18843 URL: https://issues.apache.org/jira/browse/HADOOP-18843 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.
hadoop-yetus commented on PR #5862:
URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1669986915
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 44s | | trunk passed |
| +1 :green_heart: | compile | 8m 4s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 55s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 36s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 9m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 20s | | the patch passed |
| +1 :green_heart: | compile | 7m 20s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | cc | 7m 20s | | the patch passed |
| +1 :green_heart: | javac | 7m 20s | | the patch passed |
| +1 :green_heart: | compile | 7m 8s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | cc | 7m 8s | | the patch passed |
| +1 :green_heart: | javac | 7m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 32s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 16s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 10m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 6s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 34s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 27s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 118m 23s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 14s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 41s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 379m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5862 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat xmllint |
| uname | Linux 7dc4ebab344d 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c372b1f01dc333cd6b506375270030859e33062f |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5547: HDFS-16977. Forbid assigned characters in pathname.
hadoop-yetus commented on PR #5547:
URL: https://github.com/apache/hadoop/pull/5547#issuecomment-1669950199
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 12s | | trunk passed |
| +1 :green_heart: | compile | 5m 37s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 5m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 5m 26s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 5m 26s | | the patch passed |
| +1 :green_heart: | compile | 5m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 5m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 18s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5547/5/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 3 new + 395 unchanged - 0 fixed =
398 total (was 395) |
| +1 :green_heart: | mvnsite | 2m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 29s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 221m 18s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5547/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 397m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5547/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5547 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux aa50f62265b1 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a01a6d6055d9974f79d70af278b580145acba95e |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.
[GitHub] [hadoop] hadoop-yetus commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete bloc
hadoop-yetus commented on PR #5855:
URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1669741496
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 7s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/12/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 202m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 297m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5855 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d0dcaa24cb96 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5af06d98849707bed42863172dc38247aba428c8 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/12/testReport/ |
| Max. process+thread count | 3592 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/12/console |
| versions
[jira] [Updated] (HADOOP-18842) Support Overwrite Directory On Commit For S3A Committers
[ https://issues.apache.org/jira/browse/HADOOP-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18842: Affects Version/s: 3.4.0 > Support Overwrite Directory On Commit For S3A Committers > > > Key: HADOOP-18842 > URL: https://issues.apache.org/jira/browse/HADOOP-18842 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > > The goal is to add a new kind of commit mechanism in which the destination > directory is cleared off before committing the file. > *Use Case* > In case of dynamicPartition insert overwrite queries, The destination > directory which needs to be overwritten are not known before the execution > and hence it becomes a challenge to clear off the destination directory. > > One approach to handle this is, The underlying engines/client will clear off > all the destination directories before calling the commitJob operation but > the issue with this approach is that, In case of failures while committing > the files, We might end up with the whole of previous data being deleted > making the recovery process difficult or time consuming. > > *Solution* > Based on mode of commit operation either *INSERT* or *OVERWRITE* , During > commitJob operations, The committer will map each destination directory with > the commits which needs to be added in the directory and if the mode is > *OVERWRITE* , The committer will delete the directory recursively and then > commit each of the files in the directory. So in case of failures (worst > case) The number of destination directory which will be deleted will be equal > to the number of threads if we do it in multi-threaded way as compared to the > whole data if it was done in the engine side. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18842) Support Overwrite Directory On Commit For S3A Committers
[ https://issues.apache.org/jira/browse/HADOOP-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18842: Component/s: fs/s3 > Support Overwrite Directory On Commit For S3A Committers > > > Key: HADOOP-18842 > URL: https://issues.apache.org/jira/browse/HADOOP-18842 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > > The goal is to add a new kind of commit mechanism in which the destination > directory is cleared off before committing the file. > *Use Case* > In case of dynamicPartition insert overwrite queries, The destination > directory which needs to be overwritten are not known before the execution > and hence it becomes a challenge to clear off the destination directory. > > One approach to handle this is, The underlying engines/client will clear off > all the destination directories before calling the commitJob operation but > the issue with this approach is that, In case of failures while committing > the files, We might end up with the whole of previous data being deleted > making the recovery process difficult or time consuming. > > *Solution* > Based on mode of commit operation either *INSERT* or *OVERWRITE* , During > commitJob operations, The committer will map each destination directory with > the commits which needs to be added in the directory and if the mode is > *OVERWRITE* , The committer will delete the directory recursively and then > commit each of the files in the directory. So in case of failures (worst > case) The number of destination directory which will be deleted will be equal > to the number of threads if we do it in multi-threaded way as compared to the > whole data if it was done in the engine side. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18842) Support Overwrite Directory On Commit For S3A Committers
[ https://issues.apache.org/jira/browse/HADOOP-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752038#comment-17752038 ] ASF GitHub Bot commented on HADOOP-18842: - steveloughran commented on PR #5931: URL: https://github.com/apache/hadoop/pull/5931#issuecomment-1669566586 this is very like the staging committer's partitioned overwrite, and is needed for the magic committer to support insert overwrite in spark, so will be good. Now HADOOP-16570 covers the scale problems with the staging committer, I'd hoped the manifest committer would be safe as the per-file data is so much smaller, but MAPREDUCE-7435 shows that no, you can't even include manifest (source, dest) rename lists without overloading the memory of a spark driver. The fix there involved streaming the pending data to the local fs and reading back in...I think this may be needed here too. Using the local fs avoids all s3 writeback/reading. The hardest bit of that PR, org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.EntryFileIO, will be on the classpath; maybe a more abstract superclass can be extracted, SinglePendingCommit data made Writable and then the same queue-based serialization used in this job commit: a pool of threads to read all .pendingset files, all then streamed to a temp file while that list of dirs to clean up is enumerated. > Support Overwrite Directory On Commit For S3A Committers > > > Key: HADOOP-18842 > URL: https://issues.apache.org/jira/browse/HADOOP-18842 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > > The goal is to add a new kind of commit mechanism in which the destination > directory is cleared off before committing the file. > *Use Case* > In case of dynamicPartition insert overwrite queries, The destination > directory which needs to be overwritten are not known before the execution > and hence it becomes a challenge to clear off the destination directory. > > One approach to handle this is, The underlying engines/client will clear off > all the destination directories before calling the commitJob operation but > the issue with this approach is that, In case of failures while committing > the files, We might end up with the whole of previous data being deleted > making the recovery process difficult or time consuming. > > *Solution* > Based on mode of commit operation either *INSERT* or *OVERWRITE* , During > commitJob operations, The committer will map each destination directory with > the commits which needs to be added in the directory and if the mode is > *OVERWRITE* , The committer will delete the directory recursively and then > commit each of the files in the directory. So in case of failures (worst > case) The number of destination directory which will be deleted will be equal > to the number of threads if we do it in multi-threaded way as compared to the > whole data if it was done in the engine side. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5931: [WIP][DRAFT] HADOOP-18842: Support Overwrite Directory On Commit For S3A Committers
steveloughran commented on PR #5931: URL: https://github.com/apache/hadoop/pull/5931#issuecomment-1669566586 this is very like the staging committer's partitioned overwrite, and is needed for the magic committer to support insert overwrite in spark, so will be good. Now HADOOP-16570 covers the scale problems with the staging committer, I'd hoped the manifest committer would be safe as the per-file data is so much smaller, but MAPREDUCE-7435 shows that no, you can't even include manifest (source, dest) rename lists without overloading the memory of a spark driver. The fix there involved streaming the pending data to the local fs and reading back in...I think this may be needed here too. Using the local fs avoids all s3 writeback/reading. The hardest bit of that PR, org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.EntryFileIO, will be on the classpath; maybe a more abstract superclass can be extracted, SinglePendingCommit data made Writable and then the same queue-based serialization used in this job commit: a pool of threads to read all .pendingset files, all then streamed to a temp file while that list of dirs to clean up is enumerated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669555702
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 13s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 31s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 1m 40s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 40s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 85m 58s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 190m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/28/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 89dae1d090e4 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a16b568ac37c0acd36c80f2628f1e7b85017cfa7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/28/testReport/ |
| Max. process+thread count | 929 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-yarn
[GitHub] [hadoop] hadoop-yetus commented on pull request #5934: YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator.
hadoop-yetus commented on PR #5934:
URL: https://github.com/apache/hadoop/pull/5934#issuecomment-1669554747
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 8s | | trunk passed |
| +1 :green_heart: | compile | 7m 50s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 28s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 1s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 52s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 6m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | | the patch passed |
| +1 :green_heart: | compile | 6m 58s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 6m 58s | | the patch passed |
| +1 :green_heart: | compile | 7m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 52s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5934/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 164 unchanged
- 0 fixed = 167 total (was 164) |
| +1 :green_heart: | mvnsite | 3m 35s | | the patch passed |
| -1 :x: | javadoc | 0m 42s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5934/1/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 generated 6 new + 0
unchanged - 0 fixed = 6 total (was 0) |
| -1 :x: | javadoc | 0m 37s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5934/1/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 6 new + 0
unchanged - 0 fixed = 6 total (was 0) |
| +1 :green_heart: | spotbugs | 7m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 56s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 6s | |
hadoop-
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
slfan1989 commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669545487 @zhengchenyu Thanks for the contribution! LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18837) Upgrade Okio to 3.4.0 due to CVE-2023-3635
[ https://issues.apache.org/jira/browse/HADOOP-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752027#comment-17752027 ] ASF GitHub Bot commented on HADOOP-18837: - steveloughran merged PR #5914: URL: https://github.com/apache/hadoop/pull/5914 > Upgrade Okio to 3.4.0 due to CVE-2023-3635 > -- > > Key: HADOOP-18837 > URL: https://issues.apache.org/jira/browse/HADOOP-18837 > Project: Hadoop Common > Issue Type: Task >Reporter: Rohit Kumar >Assignee: Rohit Kumar >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Upgrade Okio to 3.4.0 due to CVE-2023-3635 > GzipSource does not handle an exception that might be raised when parsing a > malformed gzip buffer. This may lead to denial of service of the Okio client > when handling a crafted GZIP archive, by using the GzipSource class. > CVSSv3 Score:- 7.5(High) > [https://nvd.nist.gov/vuln/detail/CVE-2023-3635] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5914: HADOOP-18837. Upgrade okio to 3.4.0 due to CVE-2023-3635
steveloughran merged PR #5914: URL: https://github.com/apache/hadoop/pull/5914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5913: HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication.
haiyang1987 commented on PR #5913: URL: https://github.com/apache/hadoop/pull/5913#issuecomment-1669411025 Thanks sir @Hexiaoqiao @tomscut help me review and merge !!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18233) Initialization race condition with TemporaryAWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18233:
Summary: Initialization race condition with TemporaryAWSCredentialsProvider
(was: Possible race condition with TemporaryAWSCredentialsProvider)
> Initialization race condition with TemporaryAWSCredentialsProvider
> --
>
> Key: HADOOP-18233
> URL: https://issues.apache.org/jira/browse/HADOOP-18233
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth, fs/s3
>Affects Versions: 3.3.1
> Environment: spark v3.2.0
> hadoop-aws v3.3.1
> java version 1.8.0_265 via zulu-8
>Reporter: Jason Sleight
>Assignee: Jimmy Wong
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> I'm in the process of upgrading spark+hadoop versions for my workflows and
> observing a weird behavior regression. I'm setting
> {code:java}
> spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
> spark.sql.catalogImplementation=hive
> spark.hadoop.aws.region=us-west-2
> ...many other things, I think these might be the relevant ones though...{code}
> in Spark config and I'm observing some non-fatal warnings/exceptions (see
> below for some examples). The warnings/exceptions randomly appear for some
> tasks, which causes them to fail, but then when Spark retries the task it
> will succeed. The initial tasks don't always fail either, just sometimes.
> I also found that if I switch to a SimpleAWSCredentials and use static keys,
> then I don't see any issues.
> My old setup was spark v3.0.2 with hadoop-aws v3.2.1 which also does not have
> these warnings/exceptions.
> From reading some other tickets I thought perhaps adding
> {code:java}
> spark.sql.hive.metastore.sharedPrefixes=com.amazonaws {code}
> would help, but it did not.
> Appreciate any suggestions for how to proceed or debug further :)
>
> Example stack traces:
> First one for an s3 read
> {code:java}
> WARN TaskSetManager: Lost task 27.0 in stage 4.0 (TID 29) ( executor
> 13): java.nio.file.AccessDeniedException:
> s3a://bucket/path/to/part.snappy.parquet:
> org.apache.hadoop.fs.s3a.CredentialInitializationException: Provider
> TemporaryAWSCredentialsProvider has no credentials
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:206)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3289)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
> at
> org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:268)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$1(ParquetFileFormat.scala:267)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(ParquetFileFormat.scala:270)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:164)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
> at
> org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:522)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.columnartorow_nextBatch_0$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
> at
> org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
> at
> org.apache.spark.sch
[GitHub] [hadoop-thirdparty] fredbalves86 commented on pull request #23: Guava version 32.0.1 bump to fix CVE-2023-2976
fredbalves86 commented on PR #23: URL: https://github.com/apache/hadoop-thirdparty/pull/23#issuecomment-1669393233 @jojochuang Can this be done? This is the only dependency of hadoop-common that still has guava 31.1-jre -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] YuanbenWang closed pull request #5547: HDFS-16977. Forbid assigned characters in pathname.
YuanbenWang closed pull request #5547: HDFS-16977. Forbid assigned characters in pathname. URL: https://github.com/apache/hadoop/pull/5547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 opened a new pull request, #5934: YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator.
slfan1989 opened a new pull request, #5934: URL: https://github.com/apache/hadoop/pull/5934 ### Description of PR JIRA: YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator. ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669146353
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 29s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 0m 43s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt)
| hadoop-yarn-server-web-proxy in trunk failed. |
| +1 :green_heart: | shadedclient | 23m 46s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 4s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 1m 43s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 43s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 59s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| -1 :x: | unit | 86m 19s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 202m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 410c096cdd46 4.15.0-213-generic #224-Ubuntu SMP Mo
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5924: HDFS-17140. Optimize the BPOfferService.reportBadBlocks() method
slfan1989 commented on code in PR #5924:
URL: https://github.com/apache/hadoop/pull/5924#discussion_r1286759399
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java:
##
@@ -291,9 +291,8 @@ public String toString() {
void reportBadBlocks(ExtendedBlock block,
String storageUuid, StorageType storageType) {
checkBlock(block);
+ReportBadBlockAction rbbAction = new ReportBadBlockAction(block,
storageUuid, storageType);
Review Comment:
Thank you very much for your explanation! Personally, I believe this
modification seems a bit forced and unnecessary.
Let's wait for @2005hithlj to give a detailed explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5933: HDFS-17145. Fix description of property dfs.namenode.file.close.num-committed-allowed.
hadoop-yetus commented on PR #5933:
URL: https://github.com/apache/hadoop/pull/5933#issuecomment-1669119407
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | shadedclient | 57m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | shadedclient | 23m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 198m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5933/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 287m 14s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5933/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5933 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux b57df667dd49 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fb8b16d54e29e3309d09130814e4f2c34dc6e1b2 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5933/1/testReport/ |
| Max. process+thread count | 3658 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5933/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
[GitHub] [hadoop] Hexiaoqiao commented on pull request #5913: HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication.
Hexiaoqiao commented on PR #5913: URL: https://github.com/apache/hadoop/pull/5913#issuecomment-1669086283 The failed unit test is not related to this changes. Committed to trunk. Thanks @haiyang1987 for your contribution and @tomscut reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao merged pull request #5913: HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication.
Hexiaoqiao merged PR #5913: URL: https://github.com/apache/hadoop/pull/5913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #5926: HADOOP-18840. Add enQueue time to RpcMetrics
Hexiaoqiao commented on PR #5926: URL: https://github.com/apache/hadoop/pull/5926#issuecomment-1669074483 @2005hithlj Please check the failed unit test and checkstyle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18840) Add enQueue time to RpcMetrics
[ https://issues.apache.org/jira/browse/HADOOP-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751924#comment-17751924 ] ASF GitHub Bot commented on HADOOP-18840: - Hexiaoqiao commented on PR #5926: URL: https://github.com/apache/hadoop/pull/5926#issuecomment-1669074483 @2005hithlj Please check the failed unit test and checkstyle. > Add enQueue time to RpcMetrics > -- > > Key: HADOOP-18840 > URL: https://issues.apache.org/jira/browse/HADOOP-18840 > Project: Hadoop Common > Issue Type: Improvement > Components: rpc-server >Reporter: Liangjun He >Assignee: Liangjun He >Priority: Minor > Labels: pull-request-available > > Sometimes, the request time observed by the client is much longer than the > queue + process time on the RPC server. Perhaps the RPC request 'waiting > enQueue' took too long on the RPC server, so we should add enQueue time to > RpcMetrics. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5924: HDFS-17140. Optimize the BPOfferService.reportBadBlocks() method
Hexiaoqiao commented on code in PR #5924:
URL: https://github.com/apache/hadoop/pull/5924#discussion_r1286718143
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java:
##
@@ -291,9 +291,8 @@ public String toString() {
void reportBadBlocks(ExtendedBlock block,
String storageUuid, StorageType storageType) {
checkBlock(block);
+ReportBadBlockAction rbbAction = new ReportBadBlockAction(block,
storageUuid, storageType);
Review Comment:
I am not worried about that, because `BPOfferService` is isolated for
different namespaces. But I am wonder what improvement here, save the cost to
create object and heap footprint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] huangzhaobo99 commented on pull request #5930: HDFS-17143. Optimize the logic for reconfigure ReadStrategy enable for Namenode.
huangzhaobo99 commented on PR #5930: URL: https://github.com/apache/hadoop/pull/5930#issuecomment-1669064696 Hi @slfan1989, Could you help review this when you have time? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] huangzhaobo99 commented on pull request #5930: HDFS-17143. Optimize the logic for reconfigure ReadStrategy enable for Namenode.
huangzhaobo99 commented on PR #5930: URL: https://github.com/apache/hadoop/pull/5930#issuecomment-1669062915 Those failed unit tests were unrelated to the change. And they work fine locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
