Dear Spark Community,
I run a Structured Streaming Query to read json files from S3 into an Iceberg
table. This is my query:
```python
stream_reader = (
spark_session.readStream.format("json")
.schema(schema)
.option("maxFilesPerTrigger", 256_000)
.option("basePath", f"s3a://test-bucket/root_dir/")
.load(f"s3a://test-bucket/root_dir/2025/04/")
.coalesce(8)
.withColumn("object_key", input_file_name())
)
stream = (
stream_reader.writeStream.queryName(f"test_stream")
.format("iceberg")
.outputMode("append")
.option("checkpointLocation",
f"s3a://ttest-bucket/checkpoints/{uuid.uuid4()}/")
.trigger(processingTime="10 seconds")
.toTable(target_table_full_name)
)
```
My data on S3 has this structure:
```
root_dir/
└── 2025/
└── 04/
├── 15/
│ ├── 123e4567-e89b-12d3-a456-426614174003.json
│ └── 123e4567-e89b-12d3-a456-426614174004.json
├── 16/
│ ├── 123e4567-e89b-12d3-a456-426614174000.json
│ ├── 123e4567-e89b-12d3-a456-426614174001.json
│ └── 123e4567-e89b-12d3-a456-426614174002.json
└── 17/
├── 123e4567-e89b-12d3-a456-426614174005.json
└── 123e4567-e89b-12d3-a456-426614174006.json
```
These are millions of 1.5KB files.
I encounter issues with the initial listing: when I start the stream I see this
log:
```
Total size of serialized results of 51 tasks (1073.3 MiB) is bigger than
spark.driver.maxResultSize (1024.0 MiB)
```
51 is the number of sub directories I have in my test setup. It seems that
Spark recognises sub directories as partitions, and does the listing per
partition, but in the end still aggregates everything. This error happens for
total 5.5 mln files.
Setting maxFilesPerTrigger does not help to limit this initial listing either.
Please, give me a hint in how to handle this initial listing for potentially
billions of files.
My setup is a standalone Spark 3.5.1 cluster with Spark Connect.
Best regards,
Anastasiia
