Thanks for your input.
After investigation, we essentially fixed the issue by re-adjusting the
size of the parquet files we generate. Bigger files, less of them.
It was interesting to explore the extreme limit in term of number of files
and how it impacts a single foreman at planning time. Good t
Ouch!
Looks like a logger was left behind in DEBUG mode. Can you manually turn that
off?
More memory would help in this case, because it seems that the foreman node is
the one running out of heap space as it goes through the metadata for all the
files. Is there a reason you are generating so
