Re: Speeding up Catalyst engine

2017-07-25 Thread Maciej Bryński
to use it on top of 2.2.0 ? > > > > Regards, > > -- > > Maciek Bryński > > > > > > - > Liang-Chi Hsieh | @viirya > Spark Technology Center > http://www.spark.tc/ > -- > View this messa

Re: Speeding up Catalyst engine

2017-07-24 Thread Liang-Chi Hsieh
t; -- > Maciek Bryński - Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Speeding-up-Catalyst-engine-tp22013p22014.html Sent from the

Speeding up Catalyst engine

2017-07-24 Thread Maciej Bryński
Hi Everyone, I'm trying to speed up my Spark streaming application and I have following problem. I'm using a lot of joins in my app and full catalyst analysis is triggered during every join. I found 2 options to speed up. 1) spark.sql.selfJoinAutoResolveAmbiguity option But looking at code: