Re: [SQL][PYTHON] UDF improvements.

Thanks for your response Ryan. Here you are https://issues.apache.org/jira/browse/SPARK-19159 On 01/09/2017 07:30 PM, Ryan Blue wrote: > Maciej, this looks great. > > Could you open a JIRA issue for improving the @udf decorator and > possibly sub-tasks for the specific features from the gist?

Re: [SQL][CodeGen] Is there a way to set break point and debug the generated code?

It's unfortunately difficult to debug -- that's one downside of codegen. You can dump all the code via "explain codegen" though. That's typically enough for me to debug. On Tue, Jan 10, 2017 at 3:21 AM, dragonly wrote: > I am recently hacking into the SparkSQL and trying

Re: Spark performance tests

Thanks Adam, Kazuaki! On Tue, Jan 10, 2017 at 3:28 PM, Adam Roberts wrote: > Hi, I suggest HiBench and SparkSqlPerf, HiBench features many benchmarks > within it that exercise several components of Spark (great for stressing > core, sql, MLlib capabilities), SparkSqlPerf

[SQL][CodeGen] Is there a way to set break point and debug the generated code?

I am recently hacking into the SparkSQL and trying to add some new udts and functions, as well as some new Expression classes. I run into the problem of the return type of nullSafeEval method. In one of the new Expression classes, I want to return an array of my udt, and my code is like `return

Re: Spark performance tests

Hi, I suggest HiBench and SparkSqlPerf, HiBench features many benchmarks within it that exercise several components of Spark (great for stressing core, sql, MLlib capabilities), SparkSqlPerf features 99 TPC-DS queries (stressing the DataFrame API and therefore the Catalyst optimiser), both

Re: Spark performance tests

Hi, You may find several micro-benchmarks under https://github.com/apache/spark/tree/master/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark . Regards, Kazuaki Ishizaki From: Prasun Ratn To: Apache Spark Dev Date: