[jira] [Commented] (SPARK-19882) Pivot with null as the pivot value throws NPE

2017-03-09 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903643#comment-15903643
 ] 

Apache Spark commented on SPARK-19882:
--

User 'aray' has created a pull request for this issue:
https://github.com/apache/spark/pull/17226

> Pivot with null as the pivot value throws NPE
> -
>
> Key: SPARK-19882
> URL: https://issues.apache.org/jira/browse/SPARK-19882
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Hyukjin Kwon
>
> This seems a regression.
> - Spark 1.6
> {code}
> Seq(Tuple1(None), 
> Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()
> {code}
> prints
> {code}
> +++---+
> |   a|null|  1|
> +++---+
> |null|   0|  0|
> |   1|   0|  1|
> +++---+
> {code}
> - Current master
> {code}
> Seq(Tuple1(None), 
> Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()
> {code}
> prints
> {code}
> java.lang.NullPointerException was thrown.
> java.lang.NullPointerException
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:145)
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:143)
>   at scala.collection.immutable.List.map(List.scala:273)
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.(PivotFirst.scala:143)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$7$$anonfun$24.apply(Analyzer.scala:509)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19882) Pivot with null as the pivot value throws NPE

2017-03-09 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902977#comment-15902977
 ] 

Apache Spark commented on SPARK-19882:
--

User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/17224

> Pivot with null as the pivot value throws NPE
> -
>
> Key: SPARK-19882
> URL: https://issues.apache.org/jira/browse/SPARK-19882
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Hyukjin Kwon
>
> This seems a regression.
> - Spark 1.6
> {code}
> Seq(Tuple1(None), 
> Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()
> {code}
> prints
> {code}
> +++---+
> |   a|null|  1|
> +++---+
> |null|   0|  0|
> |   1|   0|  1|
> +++---+
> {code}
> - Current master
> {code}
> Seq(Tuple1(None), 
> Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()
> {code}
> prints
> {code}
> java.lang.NullPointerException was thrown.
> java.lang.NullPointerException
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:145)
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:143)
>   at scala.collection.immutable.List.map(List.scala:273)
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.(PivotFirst.scala:143)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$7$$anonfun$24.apply(Analyzer.scala:509)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19882) Pivot with null as the pivot value throws NPE

2017-03-09 Thread Hyukjin Kwon (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902965#comment-15902965
 ] 

Hyukjin Kwon commented on SPARK-19882:
--

Actually, there are two problems here.

It should not throw an exception and the results should be

{code}
+++---+
|   a|null|  1|
+++---+
|null|   1|  0|
|   1|   0|  1|
+++---+
{code}

given

{code}
scala> Seq(Tuple1(None), Tuple1(Some(1))).toDF("a").groupBy($"a").count().show()
++-+
|   a|count|
++-+
|null|1|
|   1|1|
++-+
{code}

> Pivot with null as the pivot value throws NPE
> -
>
> Key: SPARK-19882
> URL: https://issues.apache.org/jira/browse/SPARK-19882
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Hyukjin Kwon
>
> This seems a regression.
> - Spark 1.6
> {code}
> Seq(Tuple1(None), 
> Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()
> {code}
> prints
> {code}
> +++---+
> |   a|null|  1|
> +++---+
> |null|   0|  0|
> |   1|   0|  1|
> +++---+
> {code}
> - Current master
> {code}
> Seq(Tuple1(None), 
> Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()
> {code}
> prints
> {code}
> java.lang.NullPointerException was thrown.
> java.lang.NullPointerException
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:145)
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:143)
>   at scala.collection.immutable.List.map(List.scala:273)
>   at 
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.(PivotFirst.scala:143)
>   at 
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$7$$anonfun$24.apply(Analyzer.scala:509)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org