Hi, I am debugging a program, and for some reason, a line calling the following is failing:
df.filter("sum(OpenAccounts) > 5").show It says it cannot find the column OpenAccounts, as if it was applying the sum() function and looking for a column called like that, where there is not. This works fine if I rename the column to something without parenthesis. I can't reproduce this issue in Spark Shell (1.6.0), any ideas on how can I analyze this? This is an aggregation result, with the default column names afterwards. PS: Workaround is to use toDF(cols) and rename all columns, but I am wondering if toDF has any impact on the RDD structure behind (e.g. repartitioning, cache, etc) Appreciated, Saif