d(config.get_parquet_dir()).count()} rows from
parquet')
Boris
From: Sachit Murarka
Sent: Wednesday, 16 June 2021 21:25
To: spark users
Subject: Small file problem
Hello Spark Users,
We are receiving too much small small files. About 3 million. Reading it using
spark.read itself taki
Hello Spark Users,
We are receiving too much small small files. About 3 million. Reading it
using spark.read itself taking long time and job is not proceeding further.
Is there any way to fasten this and proceed?
Regards
Sachit Murarka
https://stackoverflow.com/questions/56524539/how-to-handle-small-file-problem-in-spark-structured-streaming
Regards,
Shyam
