[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85530 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85530/testReport)**
 for PR 19872 at commit 
[`eeed9be`](https://github.com/apache/spark/commit/eeed9be030402ef73c56a2efc0f72e0d67945165).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85476/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85476 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85476/testReport)**
 for PR 19872 at commit 
[`2800344`](https://github.com/apache/spark/commit/28003442b6c7605363fef56ae40c294dd680d15f).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85476 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85476/testReport)**
 for PR 19872 at commit 
[`2800344`](https://github.com/apache/spark/commit/28003442b6c7605363fef56ae40c294dd680d15f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85446/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85446 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85446/testReport)**
 for PR 19872 at commit 
[`66a31f9`](https://github.com/apache/spark/commit/66a31f9d50dc93e8dc5c2c843101d76951ebf2c8).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85442/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85442 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85442/testReport)**
 for PR 19872 at commit 
[`99367a6`](https://github.com/apache/spark/commit/99367a6e0226a2e2dbd699b897c39d9ccc43e04b).
 * This patch **fails PySpark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[icexelloss]

Github user icexelloss commented on the issue:

https://github.com/apache/spark/pull/19872
  
@ueshin I pushed some more change to address your comments. There is one 
regression in existing test `SQLTests.test_udf_with_aggregate_function`. I will 
try to fix it tomorrow.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85446 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85446/testReport)**
 for PR 19872 at commit 
[`66a31f9`](https://github.com/apache/spark/commit/66a31f9d50dc93e8dc5c2c843101d76951ebf2c8).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85442 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85442/testReport)**
 for PR 19872 at commit 
[`99367a6`](https://github.com/apache/spark/commit/99367a6e0226a2e2dbd699b897c39d9ccc43e04b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85152/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85152 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85152/testReport)**
 for PR 19872 at commit 
[`ea5d6f3`](https://github.com/apache/spark/commit/ea5d6f319aa3b1bba20ad86a51e6efb65658e3d2).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85152 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85152/testReport)**
 for PR 19872 at commit 
[`ea5d6f3`](https://github.com/apache/spark/commit/ea5d6f319aa3b1bba20ad86a51e6efb65658e3d2).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19872
  
@ramacode2014 Hi, I'm not sure why you received notifications from this PR, 
but I guess you can unsubscribe by the "Unsubscribe" button in the right column 
of this page. Sorry for the inconvenience. Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[ramacode2014]

Github user ramacode2014 commented on the issue:

https://github.com/apache/spark/pull/19872
  
Please unsubscribe me from this spam

On Wed, Dec 20, 2017 at 10:46 AM, Takuya UESHIN 
wrote:

> *@ueshin* commented on this pull request.
> --
>
> In sql/core/src/main/scala/org/apache/spark/sql/execution/
> python/AggregateInPandasExec.scala
> :
>
> > +  // Add rows to queue to join later with the result.
> +  val projectedRowIter = grouped.map { case (groupingKey, rows) =>
> +queue.add(groupingKey.asInstanceOf[UnsafeRow])
> +rows
> +  }
> +
> +  val columnarBatchIter = new ArrowPythonRunner(
> +pyFuncs, bufferSize, reuseWorker,
> +PythonEvalType.SQL_PANDAS_GROUP_AGG_UDF, argOffsets, schema,
> +sessionLocalTimeZone, pandasRespectSessionTimeZone)
> +.compute(projectedRowIter, context.partitionId(), context)
> +
> +  val joined = new JoinedRow
> +  val resultProj = UnsafeProjection.create(output, output)
> +
> +  columnarBatchIter.map(_.rowIterator.next()).map{ outputRow =>
>
> Sorry, I meant columnarBatchIter.flatMap(_.rowIterator.asScala). I'd
> prefer this one.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



-- 




Best Regard,


Rama



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85138/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85138 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85138/testReport)**
 for PR 19872 at commit 
[`62c8f00`](https://github.com/apache/spark/commit/62c8f00b84ca600ea47ebb6db99dc86890099e4b).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85137/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85137 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85137/testReport)**
 for PR 19872 at commit 
[`1a197b7`](https://github.com/apache/spark/commit/1a197b760beef191615020cfec6fdaaa5c465fdb).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85138 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85138/testReport)**
 for PR 19872 at commit 
[`62c8f00`](https://github.com/apache/spark/commit/62c8f00b84ca600ea47ebb6db99dc86890099e4b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85137 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85137/testReport)**
 for PR 19872 at commit 
[`1a197b7`](https://github.com/apache/spark/commit/1a197b760beef191615020cfec6fdaaa5c465fdb).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85136 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85136/testReport)**
 for PR 19872 at commit 
[`ab91314`](https://github.com/apache/spark/commit/ab91314e8f89162f75493802a7f1fbd1e319d8ec).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85136/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #85136 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85136/testReport)**
 for PR 19872 at commit 
[`ab91314`](https://github.com/apache/spark/commit/ab91314e8f89162f75493802a7f1fbd1e319d8ec).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/19872
  
@icexelloss I'm sorry for the late response.
Actually I tried to implement prototypes of Pandas UDAF with partial 
aggregation and combining existing aggregate functions, but they are still much 
complicated (https://github.com/ueshin/apache-spark/pull/2, 
https://github.com/ueshin/apache-spark/pull/3, 
https://github.com/ueshin/apache-spark/pull/4). I was thinking about easier way 
to achieve that but not yet.
I've not looked into this pr yet but I guess we can start this pr and pick 
some functionalities from my prototypes if needed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84632 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84632/testReport)**
 for PR 19872 at commit 
[`37eff29`](https://github.com/apache/spark/commit/37eff294bc3825763fc438bfc4c291cbacfb0a0f).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84632/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84632 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84632/testReport)**
 for PR 19872 at commit 
[`37eff29`](https://github.com/apache/spark/commit/37eff294bc3825763fc438bfc4c291cbacfb0a0f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84631/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84631 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84631/testReport)**
 for PR 19872 at commit 
[`4332f28`](https://github.com/apache/spark/commit/4332f28bc32ea07c6ba5e55b4d66d70498d29abd).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84631 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84631/testReport)**
 for PR 19872 at commit 
[`4332f28`](https://github.com/apache/spark/commit/4332f28bc32ea07c6ba5e55b4d66d70498d29abd).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84630 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84630/testReport)**
 for PR 19872 at commit 
[`184b37f`](https://github.com/apache/spark/commit/184b37f49817488f8cc60f2c392c5ad746d23927).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84630/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84630 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84630/testReport)**
 for PR 19872 at commit 
[`184b37f`](https://github.com/apache/spark/commit/184b37f49817488f8cc60f2c392c5ad746d23927).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[icexelloss]

Github user icexelloss commented on the issue:

https://github.com/apache/spark/pull/19872
  
And to @holdenk 's question. Pandas group_agg udf fundamentally uses 
different physical plan than the existing java/scala udf and therefore it's 
hard to combine them together. I don't know a good way to do this, the closest 
is maybe to compute java/scala and python aggregation separately and join them 
together.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[icexelloss]

Github user icexelloss commented on the issue:

https://github.com/apache/spark/pull/19872
  
I end up removing `UDFColumn` class and using the existing `Aggregate` 
logical plan for pandas group_agg UDF. This reuses a lot of code being to 
existing `Aggregate` and minimize the code changes needed for pandas group_agg 
UDF.

The code works and three tests (test_basic, test_alias, test_multiple) 
passes now but the code is kind of messy. I am going on vacation next week but 
I will clean up the code and move this PR forward when I get back (Dec 16).

Thanks all.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84628/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84628 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84628/testReport)**
 for PR 19872 at commit 
[`3352050`](https://github.com/apache/spark/commit/335205037470228fa615def5d1246231b546c467).
 * This patch **fails Python style tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84628 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84628/testReport)**
 for PR 19872 at commit 
[`3352050`](https://github.com/apache/spark/commit/335205037470228fa615def5d1246231b546c467).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84446 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84446/testReport)**
 for PR 19872 at commit 
[`c1dc543`](https://github.com/apache/spark/commit/c1dc543f9ff6e77b310d1a38c3f3c2c4e8eeaf63).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class UDFColumn(Column):`
  * `case class AggregateInPandas(`
  * `case class AggregateInPandasExec(`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84446/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84446 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84446/testReport)**
 for PR 19872 at commit 
[`c1dc543`](https://github.com/apache/spark/commit/c1dc543f9ff6e77b310d1a38c3f3c2c4e8eeaf63).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[icexelloss]

Github user icexelloss commented on the issue:

https://github.com/apache/spark/pull/19872
  
> I thought @ueshin is working on this BTW.

Oh, I certainly don't want to duplicate @ueshin 's work. I am under the 
impression that @ueshin is working on two-stage PySpark UDAF with pandas_udf, 
but I cannot really find the Jira for it...

@ueshin can you point me to what you are working on so I don't overstep?



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84415/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84415 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84415/testReport)**
 for PR 19872 at commit 
[`a1058b8`](https://github.com/apache/spark/commit/a1058b8f91bc1093ef231bf41d6553d045788abc).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84415 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84415/testReport)**
 for PR 19872 at commit 
[`a1058b8`](https://github.com/apache/spark/commit/a1058b8f91bc1093ef231bf41d6553d045788abc).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19872
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84414/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84414 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84414/testReport)**
 for PR 19872 at commit 
[`4cfaf0e`](https://github.com/apache/spark/commit/4cfaf0e9723bcfbb74dfd1b9d1f5e30682bf072f).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[icexelloss]

Github user icexelloss commented on the issue:

https://github.com/apache/spark/pull/19872
  
cc @HyukjinKwon @holdenk @ueshin 

Passing some basic tests. I will work on this more next week to clean up 
and add more testing.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19872
  
**[Test build #84414 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84414/testReport)**
 for PR 19872 at commit 
[`4cfaf0e`](https://github.com/apache/spark/commit/4cfaf0e9723bcfbb74dfd1b9d1f5e30682bf072f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org