[jira] [Commented] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY

2017-10-07 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195929#comment-16195929 ] Tejas Patil commented on SPARK-0: - Does this repro with current spark trunk (over CLI) ? I could

[jira] [Commented] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY

2017-10-07 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195927#comment-16195927 ] Tejas Patil commented on SPARK-0: - Whats the full stack trace for `NullPointerException` ? >

[jira] [Commented] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195874#comment-16195874 ] Wes McKinney commented on SPARK-1: -- Hm, I'm not sure that using a MultiIndex is the right

[jira] [Commented] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195830#comment-16195830 ] Sean Owen commented on SPARK-22163: --- I'm going to assume you're saying that something like the stack

[jira] [Commented] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195817#comment-16195817 ] Sean Owen commented on SPARK-22163: --- Actually posting some code is also progress. What happens when you

[jira] [Updated] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22163: -- Description: The application objects can contain List and can be modified dynamically as well.

[jira] [Comment Edited] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195813#comment-16195813 ] Michael N edited comment on SPARK-22163 at 10/7/17 6:06 PM: Sean, my text

[jira] [Comment Edited] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195813#comment-16195813 ] Michael N edited comment on SPARK-22163 at 10/7/17 6:05 PM: Sean, my text

[jira] [Commented] (SPARK-21999) ConcurrentModificationException - Spark Streaming

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195814#comment-16195814 ] Michael N commented on SPARK-21999: --- Shixiong, Vadim, per your requests for the code,, I posted the

[jira] [Commented] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195813#comment-16195813 ] Michael N commented on SPARK-22163: --- Sean, my text description in the ticket clearly says "My

[jira] [Updated] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael N updated SPARK-22163: -- Description: The application objects can contain List and can be modified dynamically as well.

[jira] [Updated] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael N updated SPARK-22163: -- Description: The application objects can contain List and can be modified dynamically as well.

[jira] [Commented] (SPARK-21999) ConcurrentModificationException - Spark Streaming

2017-10-07 Thread Michael N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195810#comment-16195810 ] Michael N commented on SPARK-21999: --- Steve, your claim that I told the project "their design is wrong"

[jira] [Commented] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY

2017-10-07 Thread Dian Fay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195778#comment-16195778 ] Dian Fay commented on SPARK-0: -- I read the wrong line off Ambari, sorry! We're using 2.1.1. I'll

[jira] [Updated] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY

2017-10-07 Thread Dian Fay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fay updated SPARK-0: - Affects Version/s: (was: 1.6.3) 2.1.1 > Spark SQL: LATERAL VIEW OUTER null

[jira] [Comment Edited] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195750#comment-16195750 ] Li Jin edited comment on SPARK-1 at 10/7/17 3:32 PM: - Per [~leif] 's comment

[jira] [Commented] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195753#comment-16195753 ] Li Jin commented on SPARK-1: cc [~wesmckinn]. Wes, if I have an pyarrow.Table with Struct type

[jira] [Comment Edited] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195750#comment-16195750 ] Li Jin edited comment on SPARK-1 at 10/7/17 3:21 PM: - Also we might want

[jira] [Commented] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195750#comment-16195750 ] Li Jin commented on SPARK-1: Also we might want really consider Leif comment here about struct type:

[jira] [Comment Edited] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195736#comment-16195736 ] Li Jin edited comment on SPARK-1 at 10/7/17 2:49 PM: - -I think we should also

[jira] [Comment Edited] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195736#comment-16195736 ] Li Jin edited comment on SPARK-1 at 10/7/17 2:48 PM: - -I think we should also

[jira] [Comment Edited] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195736#comment-16195736 ] Li Jin edited comment on SPARK-1 at 10/7/17 2:46 PM: - I think we should also

[jira] [Commented] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195736#comment-16195736 ] Li Jin commented on SPARK-1: I think we should also add to the document is what are the behavior

[jira] [Updated] (SPARK-22216) Improving PySpark/Pandas interoperability

2017-10-07 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-22216: --- Issue Type: Epic (was: Umbrella) > Improving PySpark/Pandas interoperability >

[jira] [Commented] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY

2017-10-07 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195630#comment-16195630 ] Marco Gaido commented on SPARK-0: - Your version is quite old and Spark 1.6 is no longer

[jira] [Comment Edited] (SPARK-19428) Ability to select first row of groupby

2017-10-07 Thread Sohum Sachdev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195617#comment-16195617 ] Sohum Sachdev edited comment on SPARK-19428 at 10/7/17 9:21 AM: [~lminer]

[jira] [Commented] (SPARK-19428) Ability to select first row of groupby

2017-10-07 Thread Sohum Sachdev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195617#comment-16195617 ] Sohum Sachdev commented on SPARK-19428: --- [~lminer] This is a very interesting point you brought up.

[jira] [Assigned] (SPARK-22156) incorrect learning rate update equation when numIterations > 1

2017-10-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22156: - Assignee: kento nozawa > incorrect learning rate update equation when numIterations > 1 >

[jira] [Resolved] (SPARK-22156) incorrect learning rate update equation when numIterations > 1

2017-10-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22156. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19372

[jira] [Assigned] (SPARK-20396) groupBy().apply() with pandas udf in pyspark

2017-10-07 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reassigned SPARK-20396: --- Assignee: Li Jin > groupBy().apply() with pandas udf in pyspark >

[jira] [Updated] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-10-07 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-21190: Issue Type: Sub-task (was: New Feature) Parent: SPARK-22216 > SPIP: Vectorized UDFs in

[jira] [Updated] (SPARK-20396) groupBy().apply() with pandas udf in pyspark

2017-10-07 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-20396: Issue Type: Sub-task (was: New Feature) Parent: SPARK-22216 > groupBy().apply() with

[jira] [Updated] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-10-07 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-21404: Issue Type: Sub-task (was: Improvement) Parent: SPARK-22216 > Simple Vectorized Python

[jira] [Commented] (SPARK-22216) Improving PySpark/Pandas interoperability

2017-10-07 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195596#comment-16195596 ] Reynold Xin commented on SPARK-22216: - What you'd want to do is to move those to become subtasks. I