AFAIK the process a spark program follows is: 1. A set of transformations are defined on a given input dataset. 2. At some point an action is called 1. In your case this is writing to your parquet file. 3. When that happens spark creates a logical plan and then a physical plan (This is largely where your transformations are optimized) to perform the transformations specified. 1. This is similar to what a sql engine does, it takes your raw SQL and turns it into something that it can execute to get the data you requested. 2. There are a set of artifacts generated, one of those artifacts would be the plan that you're seeing is being truncated.
The only time I'd be concerned about this would be if I was debugging the code and needed to see what was being truncated, it is after all a debug setting ('spark.debug.maxToStringFields') Good luck! Gary On 12 June 2017 at 15:10, Henry M <henrymanm...@gmail.com> wrote: > > > I am trying to understand if I should be concerned about this warning: > > "WARN Utils:66 - Truncated the string representation of a plan since it > was too large. This behavior can be adjusted by setting > 'spark.debug.maxToStringFields' > in SparkEnv.conf" > > It occurs while writing a data frame to parquet. > > Has any one on this list looked into this warning before and could help > explain what it means? > > Thank you for your help, > Henry >