lewyh opened a new issue, #7130:
URL: https://github.com/apache/hudi/issues/7130
**Describe the problem you faced**
When trying to write to existing COW table, PySpark application crashes with
`org.apache.hudi.exception.HoodieMetadataException: Failed to retrieve files in
partition s3://<REDACTED_S3_LOCATION>/2022-11-03 from metadata` error.
Error started occurring after many successful writes to the table.
(`<S3_LOCATION>` contains expected bucket/prefix in the actual output,
redacted here)
Read operations on the table succeed, but all attempts at writing are now
failing. We need to make this table writeable again.
**To Reproduce**
Not trivial to reproduce - only happening on a single table
**Expected behavior**
Writing to the table succeeds (as it did prior to this error occurring)
**Environment Description**
* Hudi version : 0.11.1
* Spark version : 3.1.1
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
PySpark application is running as an AWS Glue 3.0 ETL job. Hudi JAR is made
available to Spark application.
```
hudi_options = {
"hoodie.table.name": "product",
"hoodie.datasource.write.recordkey.field": "_uuid",
"hoodie.datasource.write.partitionpath.field": "_raw_partition",
"hoodie.datasource.write.precombine.field": "_raw_line_number",
"hoodie.datasource.write.keygenerator.class":
"org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.hive_style_partitioning": False,
"hoodie.datasource.write.drop.partition.columns": False,
"hoodie.keep.min.commits": 20,
"hoodie.keep.max.commits": 30,
"hoodie.write.concurrency.mode": "optimistic_concurrency_control",
"hoodie.write.lock.provider":
"org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
"hoodie.write.lock.dynamodb.table": <DYNAMODB_TABLE_NAME>,
"hoodie.write.lock.dynamodb.partition_key": "product",
"hoodie.write.lock.dynamodb.region": "eu-west-1",
"hoodie.write.lock.dynamodb.endpoint_url":
"dynamodb.eu-west-1.amazonaws.com",
"hoodie.cleaner.policy.failed.writes": "LAZY",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.support_timestamp": True,
"hoodie.datasource.hive_sync.database": <DATABASE_NAME>,
"hoodie.datasource.hive_sync.table": "product",
"hoodie.datasource.hive_sync.partition_fields": "_raw_partition",
"hoodie.datasource.hive_sync.partition_extractor_class":
"org.apache.hudi.hive.MultiPartKeysValueExtractor",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.mode": "hms",
}
df.write.format("hudi").options(**hudi_options).mode("append").save(f"s3://<REDACTED_S3_TABLE_LOCATION>/")
```
**Stacktrace**
Extended stracktrace, including a 500 Server error that occurs before the
final `HoodieMetadataException`
```
view.FileSystemViewManager
(FileSystemViewManager.java:createViewManager(232)): Creating View Manager with
storage type :REMOTE_FIRST
view.FileSystemViewManager
(FileSystemViewManager.java:createViewManager(252)): Creating remote first
table view
view.FileSystemViewManager
(FileSystemViewManager.java:createRemoteFileSystemView(200)): Creating remote
view for basePath s3://<REDACTED_S3_LOCATION>. Server=172.35.0.94:40355,
Timeout=300
view.FileSystemViewManager
(FileSystemViewManager.java:createInMemoryFileSystemView(161)): Creating
InMemory based view for basePath s3://<REDACTED_S3_LOCATION>
view.AbstractTableFileSystemView
(AbstractTableFileSystemView.java:resetFileGroupsReplaced(242)): Took 6 ms to
read 0 instants, 0 replaced file groups
util.ClusteringUtils
(ClusteringUtils.java:getAllFileGroupsInPendingClusteringPlans(137)): Found 0
files in pending clustering operations
view.RemoteHoodieTableFileSystemView
(RemoteHoodieTableFileSystemView.java:executeRequest(167)): Sending request :
(http://172.35.0.94:40355/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022-11-03&maxinstant=20221028163232442&basepath=s3%3A%2F%2F<REDACTED_S3_LOCATION>&lastinstantts=20221028164303229&timelinehash=8639d639619011fad6940d85244c6fd8e2a46366995563555af0577ac4a4899d)
view.PriorityBasedFileSystemView
(PriorityBasedFileSystemView.java:handleRuntimeException(128)): Got error
running preferred function. Trying secondary
org.apache.hudi.exception.HoodieRemoteException: status code: 500, reason
phrase: Server Error
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:241)
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesBeforeOrOn(RemoteHoodieTableFileSystemView.java:248)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:99)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFilesBeforeOrOn(PriorityBasedFileSystemView.java:144)
at
org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForPartition(HoodieIndexUtils.java:69)
at
org.apache.hudi.index.HoodieIndexUtils.lambda$getLatestBaseFilesForAllPartitions$ff6885d8$1(HoodieIndexUtils.java:89)
at
org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:137)
at
org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:480)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:486)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:58)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:49)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce.to(TraversableOnce.scala:309)
at scala.collection.TraversableOnce.to$(TraversableOnce.scala:307)
at scala.collection.AbstractIterator.to(Iterator.scala:1425)
at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:301)
at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:301)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1425)
at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:288)
at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:282)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1425)
at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1030)
at
org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.http.client.HttpResponseException: status code: 500,
reason phrase: Server Error
at
org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:70)
at
org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at
org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:179)
at
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:237)
... 35 more
view.AbstractTableFileSystemView
(AbstractTableFileSystemView.java:lambda$ensurePartitionLoadedCorrectly$9(300)):
Building file system view for partition (2022-11-03)
metadata.HoodieTableMetadataUtil
(HoodieTableMetadataUtil.java:getPartitionLatestMergedFileSlices(1000)):
Loading latest merged file slices for metadata table partition files
timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(129)):
Loaded instants upto : Option{val=[20221028164303229__deltacommit__COMPLETED]}
view.AbstractTableFileSystemView
(AbstractTableFileSystemView.java:resetFileGroupsReplaced(242)): Took 0 ms to
read 0 instants, 0 replaced file groups
util.ClusteringUtils
(ClusteringUtils.java:getAllFileGroupsInPendingClusteringPlans(137)): Found 0
files in pending clustering operations
view.AbstractTableFileSystemView
(AbstractTableFileSystemView.java:lambda$ensurePartitionLoadedCorrectly$9(300)):
Building file system view for partition (files)
view.AbstractTableFileSystemView
(AbstractTableFileSystemView.java:addFilesToView(150)): addFilesToView:
NumFiles=53, NumFileGroups=1, FileGroupsCreationTime=44, StoreTimeTaken=1
impl.MetricsConfig (MetricsConfig.java:loadFirst(136)): Cannot locate
configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(374)): Scheduled
Metric snapshot period at 10 second(s).
impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)): HBase metrics
system started
metrics.MetricRegistries (MetricRegistriesLoader.java:load(63)): Loaded
MetricRegistries class
org.apache.hudi.org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/files-0000_0-29-1021_20221028111515956001.hfile'
for reading
zlib.ZlibFactory (ZlibFactory.java:loadNativeZLib(59)): Successfully loaded
& initialized native-zlib library
metadata.HoodieBackedTableMetadata
(HoodieBackedTableMetadata.java:getBaseFileReader(435)): Opened metadata base
file from
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/files-0000_0-29-1021_20221028111515956001.hfile
at instant 20221028111515956001 in 942 ms
timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(129)):
Loaded instants upto : Option{val=[==>20221103092432772__commit__REQUESTED]}
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/20221028122607813.rollback' for reading
<34 LINES REMOVED FOR BREVITY, ALL READING .rollback files>
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/20221028164303229.rollback' for reading
table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(117)):
Loading HoodieTableMetaClient from s3://<REDACTED_S3_LOCATION>//.hoodie/metadata
table.HoodieTableConfig (HoodieTableConfig.java:<init>(242)): Loading table
properties from
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/.hoodie/hoodie.properties
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/.hoodie/hoodie.properties' for
reading
table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(136)):
Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE)
from s3://<REDACTED_S3_LOCATION>//.hoodie/metadata
timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(129)):
Loaded instants upto : Option{val=[20221028164303229__deltacommit__COMPLETED]}
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.1_0-36-1035'
for reading
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(225)): Scanning log file
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.1_0-36-1035',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(253)): Reading a data block
from file
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.1_0-36-1035
at instant 20221028112534117
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.2_0-29-700'
for reading
log.HoodieLogFormatReader (HoodieLogFormatReader.java:hasNext(115)): Moving
to the next reader for logfile
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.2_0-29-700',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(225)): Scanning log file
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.2_0-29-700',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(253)): Reading a data block
from file
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.2_0-29-700
at instant 20221028114451681
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.3_0-29-700'
for reading
log.HoodieLogFormatReader (HoodieLogFormatReader.java:hasNext(115)): Moving
to the next reader for logfile
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.3_0-29-700',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(225)): Scanning log file
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.3_0-29-700',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(253)): Reading a data block
from file
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.3_0-29-700
at instant 20221028115434620
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.4_0-51-743'
for reading
log.HoodieLogFormatReader (HoodieLogFormatReader.java:hasNext(115)): Moving
to the next reader for logfile
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.4_0-51-743',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(225)): Scanning log file
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.4_0-51-743',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(253)): Reading a data block
from file
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.4_0-51-743
at instant 20221028120141212
<180 LINES REMOVED FOR BREVITY, ALL Opening/Scanning/Reading/Moving log
FILES>
s3n.S3NativeFileSystem (S3NativeFileSystem.java:open(940)): Opening
's3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.50_0-89-2716'
for reading
log.HoodieLogFormatReader (HoodieLogFormatReader.java:hasNext(115)): Moving
to the next reader for logfile
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.50_0-89-2716',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(225)): Scanning log file
HoodieLogFile{pathStr='s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.50_0-89-2716',
fileLen=-1}
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(253)): Reading a data block
from file
s3://<REDACTED_S3_LOCATION>/.hoodie/metadata/files/.files-0000_20221028111515956001.log.50_0-89-2716
at instant 20221028162021361
log.AbstractHoodieLogRecordReader
(AbstractHoodieLogRecordReader.java:scanInternal(351)): Got exception when
reading log file
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
to execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5140)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5086)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1338)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:26)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:12)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:114)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:191)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:186)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.getObjectMetadata(AmazonS3LiteClient.java:96)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AbstractAmazonS3Lite.getObjectMetadata(AbstractAmazonS3Lite.java:43)
at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.getFileMetadataFromCacheOrS3(Jets3tNativeFileSystemStore.java:431)
at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:200)
at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:493)
at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.open(S3NativeFileSystem.java:936)
at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.open(S3NativeFileSystem.java:928)
at
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.open(EmrFileSystem.java:195)
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:187)
at
org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:474)
at
org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:114)
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.hasNext(HoodieLogFormatReader.java:110)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:223)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:192)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:110)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:103)
at
org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:63)
at
org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:51)
at
org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader$Builder.build(HoodieMetadataMergedLogRecordReader.java:230)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:506)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:468)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:414)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getOrCreateReaders$10(HoodieBackedTableMetadata.java:400)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:400)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getRecordsByKeys$0(HoodieBackedTableMetadata.java:209)
at java.util.HashMap.forEach(HashMap.java:1290)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:207)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:140)
at
org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:312)
at
org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:135)
at
org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:304)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:295)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestBaseFilesBeforeOrOn(AbstractTableFileSystemView.java:502)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:103)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFilesBeforeOrOn(PriorityBasedFileSystemView.java:144)
at
org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForPartition(HoodieIndexUtils.java:69)
at
org.apache.hudi.index.HoodieIndexUtils.lambda$getLatestBaseFilesForAllPartitions$ff6885d8$1(HoodieIndexUtils.java:89)
at
org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:137)
at
org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:480)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:486)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:58)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:49)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce.to(TraversableOnce.scala:309)
at scala.collection.TraversableOnce.to$(TraversableOnce.scala:307)
at scala.collection.AbstractIterator.to(Iterator.scala:1425)
at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:301)
at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:301)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1425)
at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:288)
at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:282)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1425)
at
org.apache.spark.rdd.RDD.$anonfun$collect$2(RD1667467780524,"D.scala:1030)
at
org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by:
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.ConnectionPoolTimeoutException:
Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.$Proxy55.get(Unknown
Source)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1331)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
... 85 more
executor.Executor (Logging.scala:logError(94)): Exception in task 0.0 in
stage 12.0 (TID 450)
org.apache.hudi.exception.HoodieMetadataException: Failed to retrieve files
in partition s3://<REDACTED_S3_LOCATION>/2022-11-03 from metadata
at
org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:137)
at
org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:304)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:295)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestBaseFilesBeforeOrOn(AbstractTableFileSystemView.java:502)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:103)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFilesBeforeOrOn(PriorityBasedFileSystemView.java:144)
at
org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForPartition(HoodieIndexUtils.java:69)
at
org.apache.hudi.index.HoodieIndexUtils.lambda$getLatestBaseFilesForAllPartitions$ff6885d8$1(HoodieIndexUtils.java:89)
at
org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:137)
at
org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:480)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:486)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:58)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:49)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce.to(TraversableOnce.scala:309)
at scala.collection.TraversableOnce.to$(TraversableOnce.scala:307)
at scala.collection.AbstractIterator.to(Iterator.scala:1425)
at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:301)
at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:301)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1425)
at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:288)
at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:282)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1425)
at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1030)
at
org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: Exception when reading
log file
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:352)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:192)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:110)
at
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:103)
at
org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:63)
at
org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:51)
at
org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader$Builder.build(HoodieMetadataMergedLogRecordReader.java:230)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:506)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:468)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:414)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getOrCreateReaders$10(HoodieBackedTableMetadata.java:400)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:400)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getRecordsByKeys$0(HoodieBackedTableMetadata.java:209)
at java.util.HashMap.forEach(HashMap.java:1290)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:207)
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:140)
at
org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:312)
at
org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:135)
... 39 more
Caused by:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5140)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5086)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1338)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:26)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.call.GetObjectMetadataCall.perform(GetObjectMetadataCall.java:12)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:114)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:191)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:186)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.getObjectMetadata(AmazonS3LiteClient.java:96)
at
com.amazon.ws.emr.hadoop.fs.s3.lite.AbstractAmazonS3Lite.getObjectMetadata(AbstractAmazonS3Lite.java:43)
at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.getFileMetadataFromCacheOrS3(Jets3tNativeFileSystemStore.java:431)
at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:200)
at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:493)
at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.open(S3NativeFileSystem.java:936)
at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.open(S3NativeFileSystem.java:928)
at
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.open(EmrFileSystem.java:195)
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:187)
at
org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:474)
at
org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:114)
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.hasNext(HoodieLogFormatReader.java:110)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:223)
... 57 more
Caused by:
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.ConnectionPoolTimeoutException:
Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.conn.$Proxy55.get(Unknown
Source)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1331)
at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
... 85 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
