Re: Spark SQL performance: version 1.6 vs version 1.5

Hi Tien-Dung, 1.6 plans single distinct aggregates like multiple distinct aggregates; this inherently causes some overhead but is more stable in case of high cardinalities. You can revert to the old behavior by setting the spark.sql.specializeSingleDistinctAggPlanning option to false. See also:

Re: Spark SQL performance: version 1.6 vs version 1.5

Hi Herman, We are very happy to receive your mail. Indeed, we can revert to the old behaviour of Spark SQL (the performance and the DAG are the same in both version). Many thanks and have a nice weekend, Tien-Dung PS: In order to revert, the setting value should be "true". On Fri, Feb 12, 2016

Spark SQL performance: version 1.6 vs version 1.5

Hi folks, I have compared the performance of Spark SQL version 1.6.0 and version 1.5.2. In a simple case, Spark 1.6.0 is quite faster than Spark 1.5.2. However in a more complex query - in our case it is an aggregation query with grouping sets, Spark SQL version 1.6.0 is very much slower than

Saving a Pipeline with DecisionTreeModel Spark ML

Hi all, I noticed that I cannot save a Pipeline containing a DecisionTree model similar to the way I can save one with a LogisticRegression model. It looks like DecisionTreeClassificationModel does not implement MLWritable. I describe a use case in this post

Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?

Yes you said it is only set in a props file, but why do you say that? because the resolution of your first question is that this is not differently handled. On Fri, Feb 12, 2016 at 11:11 PM, Jacek Laskowski wrote: > On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen

Re: Saving a Pipeline with DecisionTreeModel Spark ML

There is already JIRA tracking this https://issues.apache.org/jira/browse/SPARK-11888 On Fri, Feb 12, 2016 at 2:34 PM gstvolvr wrote: > Hi all, > > I noticed that I cannot save a Pipeline containing a DecisionTree model > similar to the way I can save one with a

Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?

I think that difference in the code is just an oversight. They actually do the same thing. Why do you say this property can only be set in a file? On Fri, Feb 12, 2016 at 9:39 PM, Jacek Laskowski wrote: > Hi devs, > > Following up on this, it appears that spark.worker.ui.port

Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?

Hi devs, Following up on this, it appears that spark.worker.ui.port can only be set in --properties-file. I wonder why conf/spark-defaults.conf is *not* used for the spark.worker.ui.port property? Any reason for the decision? Pozdrawiam, Jacek Jacek Laskowski |

Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?

On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen wrote: > I think that difference in the code is just an oversight. They > actually do the same thing. Correct. Just meant to know the reason if there was any. > Why do you say this property can only be set in a file? I said that