Li Rong created HADOOP-17847: -------------------------------- Summary: S3AInstrumentation Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics Key: HADOOP-17847 URL: https://issues.apache.org/jira/browse/HADOOP-17847 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.2.1 Environment: hadoop: 3.2.1
spark: 3.0.2 k8s server version: 1.18 aws.java.sdk.bundle.version:1.11.1033 Reporter: Li Rong Attachments: logs.txt When using hadoop s3a file upload for spark event Logs, the logs were queued up and not uploaded before the process is shut down: {code:java} // 21/08/13 12:22:39 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.) 21/08/13 12:22:39 WARN S3AInstrumentation: Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0, blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=106716, bytesUploaded=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=0 ms, queueDuration=0 ms, averageQueueTime=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}{code} details see logs attached -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org