I found some ways to get faster unit tests.In the meantime they had gone up
to about an hour.
Apparently defining columns in a for loop makes catalyst very slow, as it
blows up the logical plan with many projections:
final def castInts(dfIn: DataFrame, castToInts: String*): DataFrame = {
wrong
with our approach?
Regards,
Stefan Ackermann
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Disable-Spark-SQL-Optimizations-for-unit-tests-tp28380.html
Sent from the Apache Spark User List mailing list archiv
