HeadJk opened a new issue, #19670:
URL: https://github.com/apache/hudi/issues/19670
### Bug Description
**What happened:**
When attempting to switch from GLOBAL_BLOOM index to RECORD_INDEX for the
first time the write job fails when initializing the record index metadata.
The job consistantly fails during this operation:
`Bulk inserting at 20260818190857353010 into metadata table
transactions_all_metadata keyBy at `
The table size of the table being written to is ~2TB
There is some data skew in the table: some partitions have more data than
other partitions.
We ran this exact same hudi write job with RECORD_INDEX on a table which is
~500GB and it succeeded. So it appears hudi is failing to scale its record
index metadata generation on larger tables.
**What you expected:**
Hudi would generate the record_index metadata and perform the write job
without the job crashing.
**Steps to reproduce:**
1. Create a hudi table of approximately 2 TB using the configurations listed
below. But change RECORD_INDEX to GLOBAL_BLOOM.
2. Create an emr serverless spark job which perform a hudi write with the
following configurations listed below. The input data size is roughly a few GB
3. Wait for the job to crash.
### Environment
**Hudi version:**
`0.14.0-amzn-1`
**Query engine:** (Spark/Flink/Trino etc)
`Spark 3.5.0`
**Relevant configs:**
Hudi Configuration Options:
```hoodie.bootstrap.parallelism = 1920
hoodie.bulkinsert.shuffle.parallelism = 1920
hoodie.clean.automatic = false
hoodie.cleaner.policy.failed.writes = LAZY
hoodie.compact.inline = false
hoodie.datasource.hive_sync.partition_fields = warehouse,year,month
hoodie.datasource.write.hive_style_partitioning = true
hoodie.datasource.write.keygenerator.class =
org.apache.hudi.keygen.ComplexKeyGenerator
hoodie.datasource.write.operation = upsert
hoodie.datasource.write.partitionpath.field = warehouse,year,month
hoodie.datasource.write.payload.class =
org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
hoodie.datasource.write.precombine.field = CaptureDate
hoodie.datasource.write.reconcile.schema = true
hoodie.datasource.write.recordkey.field = uuid
hoodie.datasource.write.table.type = MERGE_ON_READ
hoodie.delete.shuffle.parallelism = 1920
hoodie.filesystem.operation.retry.enable = true
hoodie.filesystem.operation.retry.max_interval_ms = 5000
hoodie.filesystem.operation.retry.max_numbers = 10
hoodie.finalize.write.parallelism = 1920
hoodie.global.index.reconcile.parallelism = 1920
hoodie.global.simple.index.parallelism = 1920
hoodie.index.type = RECORD_INDEX
hoodie.insert.shuffle.parallelism = 1920
hoodie.keep.max.commits = 210
hoodie.keep.min.commits = 200
hoodie.markers.delete.parallelism = 1920
hoodie.meta.sync.metadata_file_listing = true
hoodie.metadata.enable = true
hoodie.metadata.record.index.enable = true
hoodie.parquet.max.file.size = 125829120
hoodie.parquet.small.file.limit = 100663296
hoodie.record.index.update.partition.path = true
hoodie.rollback.parallelism = 1920
hoodie.schema.on.read.enable = false
hoodie.simple.index.parallelism = 1920
hoodie.table.name = transactions_all
hoodie.table.services.enabled = true
hoodie.upsert.shuffle.parallelism = 1920
hoodie.write.concurrency.mode = OPTIMISTIC_CONCURRENCY_CONTROL
hoodie.write.lock.dynamodb.billing_mode = PAY_PER_REQUEST
hoodie.write.lock.dynamodb.endpoint_url = dynamodb.us-east-1.amazonaws.com
hoodie.write.lock.dynamodb.partition_key = ******
hoodie.write.lock.dynamodb.region = us-east-1
hoodie.write.lock.dynamodb.table = ******
hoodie.write.lock.provider =
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
```
EMR Serverless Configuration:
```
{
"applicationConfiguration": [
{
"classification": "emrfs-site",
"configurations": [],
"properties": {
"fs.s3.aimd.maxAttempts": "1000",
"fs.s3.aimd.enabled": "true",
"fs.s3.maxRetries": "50"
}
},
{
"classification": "hive-site",
"configurations": [],
"properties": {
"hive.metastore.client.factory.class":
"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
}
},
{
"classification": "spark-defaults",
"configurations": [],
"properties": {
"spark.executor.memory": "50G",
"spark.driver.memory": "50G",
"spark.emr-serverless.driver.disk": "200G",
"spark.driver.maxResultSize": "20G",
"spark.driver.cores": "8",
"spark.emr-serverless.memoryOverheadFactor": "0.2",
"spark.executor.cores": "8",
"spark.hadoop.fs.s3.serverSideEncryption.kms.keyId":
"******",
"spark.hadoop.fs.s3.enableServerSideEncryption": "true",
"spark.dynamicAllocation.maxExecutors": "200",
"spark.hadoop.fs.s3.maxConnections": "9999",
"spark.emr-serverless.executor.disk": "200G",
"spark.executorEnv.vfnEmrVersion": "emr-7.0.0",
"spark.executorEnv.vfnEnvironment": "prod",
"spark.serializer":
"org.apache.spark.serializer.KryoSerializer",
"spark.emr-serverless.driverEnv.vfnEnvironment": "prod",
"spark.jars":
"/usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/hudi/hudi-aws-bundle.jar",
"spark.emr-serverless.driverEnv.vfnEmrVersion": "emr-7.0.0"
}
}
],
"monitoringConfiguration": {
"managedPersistenceMonitoringConfiguration": {
"enabled": true,
"encryptionKeyArn": {}
},
"s3MonitoringConfiguration": {
"encryptionKeyArn": "******",
"logUri": "****&"
},
"cloudWatchLoggingConfiguration": {
"enabled": true,
"logGroupName": "*****",
"logStreamNamePrefix": {},
"encryptionKeyArn": {},
"logTypes": {
"SPARK_DRIVER": [
"STDOUT",
"STDERR"
],
"SPARK_EXECUTOR": [
"STDOUT",
"STDERR"
]
}
},
"prometheusMonitoringConfiguration": {}
},
"diskEncryptionConfiguration": {}
}
```
Spark Properties:
```
spark.app.id | 00g83gdn47a8r80b
spark.app.initial.jar.urls |
s3://***********,spark://[2600:1f10:4ace:6700:ad82:9311:5abc:965a]:38207/jars/hudi-aws-bundle-0.14.0-amzn-1.jar,spark://[2600:1f10:4ace:6700:ad82:9311:5abc:965a]:38207/jars/hudi-spark3.5-bundle_2.12-0.14.0-amzn-1.jar
spark.app.name | ******
spark.app.startTime | 1787083716722
spark.app.submitTime | 1787083716413
spark.authenticate | true
spark.blacklist.decommissioning.enabled | true
spark.blacklist.decommissioning.timeout | 1h
spark.decommissioning.timeout.threshold | 20
spark.default.parallelism | 1920
spark.driver.bindAddress | [fd00::2%eth0]
spark.driver.cores | 4
spark.driver.defaultJavaOptions | -XX:OnOutOfMemoryError='kill -9 %p'
spark.driver.extraClassPath |
/usr/lib/livy/rsc-jars/*:/usr/lib/livy/repl_2.12-jars/*:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/goodies/lib/emr-serverless-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/emrfs/conf:/docker/usr/share/aws/emr/emrfs/lib/*:/docker/usr/share/aws/emr/emrfs/auxlib/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr
/share/aws/emr/security/conf:/docker/usr/share/aws/emr/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/usr/share/aws/redshift/jdbc/RedshiftJDBC.jar:/usr/share/aws/redshift/spark-redshift/lib/*:/usr/share/aws/iceberg/lib/iceberg-emr-common.jar:/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar:/usr/lib/hudi/hudi-spark-bundle.jar:/usr/lib/hudi/hudi-aws-bundle.jar
spark.driver.extraJavaOptions | -Djava.net.preferIPv6Addresses=false
-XX:OnOutOfMemoryError='kill -9 %p' -XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false -XX:OnOutOfMemoryError='kill -9 %p'
spark.driver.extraLibraryPath |
/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native
spark.driver.host | [2600:1f10:4ace:6700:ad82:9311:5abc:965a]
spark.driver.maxResultSize | 5G
spark.driver.memory | 26G
spark.driver.port | 38207
spark.dynamicAllocation.enabled | true
spark.dynamicAllocation.initialExecutors | 3
spark.dynamicAllocation.maxExecutors | 62
spark.dynamicAllocation.minExecutors | 0
spark.dynamicAllocation.shuffleTracking.enabled | true
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout | 1s
spark.emr-serverless.allocation.executor.timeout | 300s
spark.emr-serverless.client.create.batch.size | 100
spark.emr-serverless.client.describe.batch.size | 100
spark.emr-serverless.client.release.batch.size | 100
spark.emr-serverless.driver.disk | 200G
spark.emr-serverless.driverEnv.SPARK_LOCAL_IP | [fd00::2%eth0]
spark.emr-serverless.executor.disk | 200G
spark.emr-serverless.fluentd.eventLog.custom.chunking.enabled | false
spark.emr-serverless.initialExecutorTimeout | 1200000
spark.emr-serverless.lakeformation.enabled |
spark.emr-serverless.maxPendingExecutors | 1000
spark.emr-serverless.memoryOverheadFactor | 0.1
spark.eventLog.dir | file:///var/log/spark/apps
spark.eventLog.enabled | true
spark.eventLog.rotation.enabled | true
spark.eventLog.rotation.interval | 300s
spark.eventLog.rotation.maxFilesToRetain | 2
spark.eventLog.rotation.minFileSize | 1m
spark.executor.cores | 4
spark.executor.defaultJavaOptions | -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+UseParallelGC -XX:InitiatingHeapOccupancyPercent=70
-XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraClassPath |
/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/goodies/lib/emr-serverless-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/emrfs/conf:/docker/usr/share/aws/emr/emrfs/lib/*:/docker/usr/share/aws/emr/emrfs/auxlib/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr/share/aws/emr/security/conf:/docker/usr/share/aws/emr
/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/usr/share/aws/redshift/jdbc/RedshiftJDBC.jar:/usr/share/aws/redshift/spark-redshift/lib/*:/usr/share/aws/iceberg/lib/iceberg-emr-common.jar:/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar:/usr/lib/hudi/hudi-spark-bundle.jar:/usr/lib/hudi/hudi-aws-bundle.jar
spark.executor.extraJavaOptions | -Djava.net.preferIPv6Addresses=false
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseParallelGC
-XX:InitiatingHeapOccupancyPercent=70 -XX:OnOutOfMemoryError='kill -9 %p'
-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false -verbose
:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseParallelGC
-XX:InitiatingHeapOccupancyPercent=70 -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraLibraryPath |
/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native
spark.executor.id | driver
spark.executor.instances | 3
spark.executor.memory | 26G
spark.executorEnv.vfnEmrVersion | emr-7.0.0
spark.executorEnv.vfnEnvironment | prod
spark.files.fetchFailure.unRegisterOutputOnHost | true
spark.hadoop.aws.region | us-east-1
spark.hadoop.dynamodb.region | us-east-1
spark.hadoop.fs.defaultFS | file:///
spark.hadoop.fs.s3.customAWSCredentialsProvider |
com.amazonaws.auth.DefaultAWSCredentialsProviderChain
spark.hadoop.fs.s3.enableServerSideEncryption | true
spark.hadoop.fs.s3.getObject.initialSocketTimeoutMilliseconds | 2000
spark.hadoop.fs.s3.maxConnections | 9999
spark.hadoop.fs.s3.serverSideEncryption.kms.keyId | *****
spark.hadoop.fs.s3a.aws.credentials.provider |
software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider
spark.hadoop.fs.s3a.committer.magic.enabled | true
spark.hadoop.fs.s3a.committer.name | magicv2
spark.hadoop.hive.metastore.client.factory.class |
com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version.emr_internal_use_only.EmrFileSystem
| 2
spark.hadoop.mapreduce.fileoutputcommitter.cleanup-failures.ignored.emr_internal_use_only.EmrFileSystem
| true
spark.hadoop.mapreduce.output.fs.optimized.committer.enabled | true
spark.hadoop.parquet.crypto.factory.class |
org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory
spark.hadoop.parquet.encryption.kms.client.class | *******
spark.history.fs.logDirectory | file:///var/log/spark/apps
spark.history.ui.port | 18080
spark.jars |
file:/usr/lib/hudi/hudi-spark3.5-bundle_2.12-0.14.0-amzn-1.jar,file:/usr/lib/hudi/hudi-aws-bundle-0.14.0-amzn-1.jar,s3://*******
spark.kryoserializer.buffer.max.mb | 256
spark.master | custom:emr-serverless
spark.repl.local.jars |
file:///usr/lib/hudi/hudi-spark3.5-bundle_2.12-0.14.0-amzn-1.jar,file:///usr/lib/hudi/hudi-aws-bundle-0.14.0-amzn-1.jar
spark.resourceManager.cleanupExpiredHost | true
spark.scheduler.mode | FIFO
spark.serializer | org.apache.spark.serializer.KryoSerializer
spark.sql.adaptive.enabled | true
spark.sql.catalogImplementation | hive
spark.sql.emr.internal.extensions |
com.amazonaws.emr.spark.EmrSparkSessionExtensions
spark.sql.hive.metastore.sharedPrefixes |
software.amazon.awssdk.services.dynamodb
spark.sql.legacy.avro.datetimeRebaseModeInRead | LEGACY
spark.sql.legacy.avro.datetimeRebaseModeInWrite | LEGACY
spark.sql.legacy.parquet.datetimeRebaseModeInRead | LEGACY
spark.sql.legacy.parquet.datetimeRebaseModeInWrite | LEGACY
spark.sql.legacy.parquet.int96RebaseModeInRead | LEGACY
spark.sql.legacy.parquet.int96RebaseModeInWrite | LEGACY
spark.sql.parquet.fs.optimized.committer.optimization-enabled | true
spark.sql.parquet.output.committer.class |
com.amazon.emr.committer.EmrOptimizedSparkSqlParquetOutputCommitter
spark.sql.shuffle.partitions | 1920
spark.ssl.internode.enabled | false
spark.ssl.ui.enabled | false
spark.stage.attempt.ignoreOnDecommissionFetchFailure | true
spark.submit.customResourceManager.submit.class |
org.apache.spark.deploy.emrserverless.submit.EmrServerlessClientApplication
spark.submit.deployMode | client
spark.submit.pyFiles |
spark.ui.custom.executor.log.url | /logs/{{CONTAINER_ID}}/{{FILE_NAME}}.gz
spark.ui.enabled | true
spark.ui.killEnabled | false
spark.ui.port | 4040
spark.yarn.heterogeneousExecutors.enabled | false
```
### Logs and Stack Trace
The job fails at this stage consistently:
Stage Id ▾ | Description | Submitted | Duration | Tasks: Succeeded/Total |
Input | Output | Shuffle Read | Shuffle Write | Failure Reason
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
16 | Bulk inserting at 20260818190857353010 into metadata table
transactions_all_metadatakeyBy at
SparkHoodieMetadataBulkInsertPartitioner.java:74+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.keyBy(JavaRDDLike.scala:45)
org.apache.hudi.metadata.SparkHoodieMetadataBulkInsertPartitioner.repartitionRecords(SparkHoodieMetadataBulkInsertPartitioner.java:74)
org.apache.hudi.metadata.SparkHoodieMetadataBulkInsertPartitioner.repartitionRecords(SparkHoodieMetadataBulkInsertPartitioner.java:40)
org.apache.hudi.table.action.commit.SparkBulkInsertHelper.bulkInsert(SparkBulkInsertHelper.java:126)
org.apache.hudi.table.action.commit.SparkBulkInsertHelper.bulkInsert(SparkBulkInsertHelper.java:81)
org.apache.hudi.table.action.deltacommit.SparkBulkInsertPreppedDeltaCommitActionExecutor.execute(SparkBulkInsertPreppedDeltaCommitActionExecutor.java:52)
org.apache.hudi.table.HoodieSparkMergeOnReadTable.bulkInsertPrepped(HoodieSparkMergeOnReadTable.java:139)
org.apache.hudi.table.HoodieSparkMer
geOnReadTable.bulkInsertPrepped(HoodieSparkMergeOnReadTable.java:88)
org.apache.hudi.client.SparkRDDWriteClient.bulkInsertPreppedRecords(SparkRDDWriteClient.java:237)
org.apache.hudi.client.SparkRDDWriteClient.bulkInsertPreppedRecords(SparkRDDWriteClient.java:63)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1129)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.bulkCommit(SparkHoodieBackedTableMetadataWriter.java:130)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:445)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.<init>(HoodieBackedTableMetadataWriter.java:182)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.<init>(SparkHoodieBackedTableMetadataWriter.java:95)
org.apache.hudi.metadata.SparkHoodi
eBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72)
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287)
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273)
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263)
| 2026/08/18 20:16:20 | 29 min | 25518/6573 (24760 failed) (65 killed: Stage
cancelled: Job aborted due to stage failure: Task 891 in stage 16.0 failed 4
times, most recent failure: Lost task 891.10 in stage 16.0 (TID 41417)
([2600:1f10:4ace:6700:201d:72db:f403:b6fd] executor 371): ExecutorLostFailure
(executor 371 exited caused by one of the running tasks) Reason: Unknown
executor exit code (137) (died from signal 9?) Driver stacktrace:) | 154.5 GiB
| | | 561.0 GiB | Job aborted due to stage failure: Task 891 in stage 16.0
failed 4 times, most recent failure: Lost task 891.10 in stage 16.0 (TID 41417)
([2600:1f10:4ace:6700:2
01d:72db:f403:b6fd] executor 371): ExecutorLostFailure (executor 371 exited
caused by one of the running tasks) Reason: Unknown executor exit code (137)
(died from signal 9?)+detailsJob aborted due to stage failure: Task 891 in
stage 16.0 failed 4 times, most recent failure: Lost task 891.10 in stage 16.0
(TID 41417) ([2600:1f10:4ace:6700:201d:72db:f403:b6fd] executor 371):
ExecutorLostFailure (executor 371 exited caused by one of the running tasks)
Reason: Unknown executor exit code (137) (died from signal 9?) Driver
stacktrace:
Before the failure above, the job appears to be creating 677 record index
file groups:
Stage Id ▾ | Description | Submitted | Duration | Tasks: Succeeded/Total |
Input | Output | Shuffle Read | Shuffle Write
-- | -- | -- | -- | -- | -- | -- | -- | --
15 | Creating 677 file groups for partition record_index with base fileId
record-index- at instant time 20260818190857353010foreach at
HoodieSparkEngineContext.java:155+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.foreach(JavaRDDLike.scala:45)
org.apache.hudi.client.common.HoodieSparkEngineContext.foreach(HoodieSparkEngineContext.java:155)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFileGroups(HoodieBackedTableMetadataWriter.java:748)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:441)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.<init>(HoodieBackedTableMetadataWriter.java:182)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.<init>(SparkHoodieBackedTableMetadataWriter.java:95)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.
create(SparkHoodieBackedTableMetadataWriter.java:72)
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287)
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273)
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263)
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1303)
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139)
org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224)
org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431)
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompu
te(commands.scala:75)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
| 2026/08/18 20:16:17 | 1 s | 677/677 | | | |
14 | Record Index: reading record keys from 6573 base filescount at
HoodieJavaRDD.java:115+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:45)
org.apache.hudi.data.HoodieJavaRDD.count(HoodieJavaRDD.java:115)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:435)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.<init>(HoodieBackedTableMetadataWriter.java:182)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.<init>(SparkHoodieBackedTableMetadataWriter.java:95)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72)
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287)
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.ja
va:273)
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263)
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1303)
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139)
org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224)
org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431)
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
| 2026/08/18 20:13:43 |
2.6 min | 6573/6573 | 2.0 TiB | | |
13 | Record Index: reading record keys from 6573 base filescount at
HoodieJavaRDD.java:115+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:45)
org.apache.hudi.data.HoodieJavaRDD.count(HoodieJavaRDD.java:115)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeRecordIndexPartition(HoodieBackedTableMetadataWriter.java:517)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:420)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278)
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.<init>(HoodieBackedTableMetadataWriter.java:182)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.<init>(SparkHoodieBackedTableMetadataWriter.java:95)
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72)
org.apache.hudi.client.SparkRDDWriteClient.initial
izeMetadataTable(SparkRDDWriteClient.java:287)
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273)
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263)
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1303)
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139)
org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224)
org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431)
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(comma
nds.scala:73) | 2026/08/18 20:09:11 | 4.5 min | 6573/6573 | | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
