AFAIK the process a spark program follows is:
1. A set of transformations are defined on a given input dataset.
2. At some point an action is called
1. In your case this is writing to your parquet file.
3. When that happens spark creates a logical plan and then a physical
plan
I am trying to understand if I should be concerned about this warning:
"WARN Utils:66 - Truncated the string representation of a plan since it
was too large. This behavior can be adjusted by setting
'spark.debug.maxToStringFields' in SparkEnv.conf"
It occurs while writing a data frame to
