[ https://issues.apache.org/jira/browse/HADOOP-17063 ]
Steve Loughran deleted comment on HADOOP-17063: ----------------------------------------- was (Author: ste...@apache.org): assuming this is in dir marker deletion, how about we disable retries in that case -now that recent clients are all happy with retained markers, and because permissions problems may be cause. > S3A deleteObjects hanging/retrying forever > ------------------------------------------ > > Key: HADOOP-17063 > URL: https://issues.apache.org/jira/browse/HADOOP-17063 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.2.1 > Environment: hadoop 3.2.1 > spark 2.4.4 > > Reporter: Dyno > Priority: Minor > Attachments: jstack_exec-34.log, jstack_exec-40.log, > jstack_exec-74.log > > > {code} > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523) > com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) > > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446) > > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365) > > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64) > org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685) > org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122) > > org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165) > > org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42) > > org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57) > > org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74) > > org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247) > > org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242) > > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394) > > org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248) > > org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170) > > org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169) > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) > org.apache.spark.scheduler.Task.run(Task.scala:123) > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) > org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > {code} > > we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator, > sometimes we see this hang with the stacktrace above. it looks like the > putObject never return, we have to kill the executor to make the job move > forward. > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org