[jira] [Commented] (SPARK-15114) Column name generated by typed aggregate is super verbose

2016-05-11 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279623#comment-15279623
 ] 

Apache Spark commented on SPARK-15114:
--

User 'dilipbiswal' has created a pull request for this issue:
https://github.com/apache/spark/pull/13045

> Column name generated by typed aggregate is super verbose
> -
>
> Key: SPARK-15114
> URL: https://issues.apache.org/jira/browse/SPARK-15114
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Reporter: Yin Huai
>Priority: Critical
>
> {code}
> case class Person(name: String, email: String, age: Long)
> val ds = spark.read.json("/tmp/person.json").as[Person]
> import org.apache.spark.sql.expressions.scala.typed._
> ds.groupByKey(_ => 0).agg(sum(_.age))
> // org.apache.spark.sql.Dataset[(Int, Double)] = [value: int, 
> typedsumdouble(unresolveddeserializer(newInstance(class Person), age#0L, 
> email#1, name#2), upcast(value)): double]
> ds.groupByKey(_ => 0).agg(sum(_.age)).explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Final,isDistinct=false)],
>  output=[value#84,typedsumdouble(unresolveddeserializer(newInstance(class 
> $line15.$read$$iw$$iw$Person), age#0L, email#1, name#2), upcast(value))#91])
> : +- INPUT
> +- Exchange hashpartitioning(value#84, 200), None
>+- WholeStageCodegen
>   :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Partial,isDistinct=false)],
>  output=[value#84,value#97])
>   : +- INPUT
>   +- AppendColumns , newInstance(class 
> $line15.$read$$iw$$iw$Person), [input[0, int] AS value#84]
>  +- WholeStageCodegen
> :  +- Scan HadoopFiles[age#0L,email#1,name#2] Format: JSON, 
> PushedFilters: [], ReadSchema: struct
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15114) Column name generated by typed aggregate is super verbose

2016-05-10 Thread Dilip Biswal (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279229#comment-15279229
 ] 

Dilip Biswal commented on SPARK-15114:
--

Going to submit a PR for this tonight.

> Column name generated by typed aggregate is super verbose
> -
>
> Key: SPARK-15114
> URL: https://issues.apache.org/jira/browse/SPARK-15114
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Reporter: Yin Huai
>Priority: Critical
>
> {code}
> case class Person(name: String, email: String, age: Long)
> val ds = spark.read.json("/tmp/person.json").as[Person]
> import org.apache.spark.sql.expressions.scala.typed._
> ds.groupByKey(_ => 0).agg(sum(_.age))
> // org.apache.spark.sql.Dataset[(Int, Double)] = [value: int, 
> typedsumdouble(unresolveddeserializer(newInstance(class Person), age#0L, 
> email#1, name#2), upcast(value)): double]
> ds.groupByKey(_ => 0).agg(sum(_.age)).explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Final,isDistinct=false)],
>  output=[value#84,typedsumdouble(unresolveddeserializer(newInstance(class 
> $line15.$read$$iw$$iw$Person), age#0L, email#1, name#2), upcast(value))#91])
> : +- INPUT
> +- Exchange hashpartitioning(value#84, 200), None
>+- WholeStageCodegen
>   :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Partial,isDistinct=false)],
>  output=[value#84,value#97])
>   : +- INPUT
>   +- AppendColumns , newInstance(class 
> $line15.$read$$iw$$iw$Person), [input[0, int] AS value#84]
>  +- WholeStageCodegen
> :  +- Scan HadoopFiles[age#0L,email#1,name#2] Format: JSON, 
> PushedFilters: [], ReadSchema: struct
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15114) Column name generated by typed aggregate is super verbose

2016-05-05 Thread Dilip Biswal (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271972#comment-15271972
 ] 

Dilip Biswal commented on SPARK-15114:
--

[~yhuai] Sure Yin. I will give it a try.

> Column name generated by typed aggregate is super verbose
> -
>
> Key: SPARK-15114
> URL: https://issues.apache.org/jira/browse/SPARK-15114
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Reporter: Yin Huai
>Priority: Critical
>
> {code}
> case class Person(name: String, email: String, age: Long)
> val ds = spark.read.json("/tmp/person.json").as[Person]
> import org.apache.spark.sql.expressions.scala.typed._
> ds.groupByKey(_ => 0).agg(sum(_.age))
> // org.apache.spark.sql.Dataset[(Int, Double)] = [value: int, 
> typedsumdouble(unresolveddeserializer(newInstance(class Person), age#0L, 
> email#1, name#2), upcast(value)): double]
> ds.groupByKey(_ => 0).agg(sum(_.age)).explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Final,isDistinct=false)],
>  output=[value#84,typedsumdouble(unresolveddeserializer(newInstance(class 
> $line15.$read$$iw$$iw$Person), age#0L, email#1, name#2), upcast(value))#91])
> : +- INPUT
> +- Exchange hashpartitioning(value#84, 200), None
>+- WholeStageCodegen
>   :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Partial,isDistinct=false)],
>  output=[value#84,value#97])
>   : +- INPUT
>   +- AppendColumns , newInstance(class 
> $line15.$read$$iw$$iw$Person), [input[0, int] AS value#84]
>  +- WholeStageCodegen
> :  +- Scan HadoopFiles[age#0L,email#1,name#2] Format: JSON, 
> PushedFilters: [], ReadSchema: struct
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15114) Column name generated by typed aggregate is super verbose

2016-05-05 Thread Yin Huai (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271943#comment-15271943
 ] 

Yin Huai commented on SPARK-15114:
--

I think at here, we should use a UnresolvedAlias. Will you have time to try it 
out and see what will be a good way to generate the alias?

> Column name generated by typed aggregate is super verbose
> -
>
> Key: SPARK-15114
> URL: https://issues.apache.org/jira/browse/SPARK-15114
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Reporter: Yin Huai
>Priority: Critical
>
> {code}
> case class Person(name: String, email: String, age: Long)
> val ds = spark.read.json("/tmp/person.json").as[Person]
> import org.apache.spark.sql.expressions.scala.typed._
> ds.groupByKey(_ => 0).agg(sum(_.age))
> // org.apache.spark.sql.Dataset[(Int, Double)] = [value: int, 
> typedsumdouble(unresolveddeserializer(newInstance(class Person), age#0L, 
> email#1, name#2), upcast(value)): double]
> ds.groupByKey(_ => 0).agg(sum(_.age)).explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Final,isDistinct=false)],
>  output=[value#84,typedsumdouble(unresolveddeserializer(newInstance(class 
> $line15.$read$$iw$$iw$Person), age#0L, email#1, name#2), upcast(value))#91])
> : +- INPUT
> +- Exchange hashpartitioning(value#84, 200), None
>+- WholeStageCodegen
>   :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Partial,isDistinct=false)],
>  output=[value#84,value#97])
>   : +- INPUT
>   +- AppendColumns , newInstance(class 
> $line15.$read$$iw$$iw$Person), [input[0, int] AS value#84]
>  +- WholeStageCodegen
> :  +- Scan HadoopFiles[age#0L,email#1,name#2] Format: JSON, 
> PushedFilters: [], ReadSchema: struct
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15114) Column name generated by typed aggregate is super verbose

2016-05-04 Thread Dilip Biswal (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271351#comment-15271351
 ] 

Dilip Biswal commented on SPARK-15114:
--

[~yhuai] 
Currently we use the sql representation of the expression as the system 
generated alias name. like following.

case expr: Expression => Alias(expr, usePrettyExpression(expr).sql)()

Do we add a additional case for AggregateExpression and use toString() instead 
of sql() for a shorter name ? Like,

case aggExpr: AggregateExpression => Alias(aggExpr, 
usePrettyExpression(aggExpr).toString)()

> Column name generated by typed aggregate is super verbose
> -
>
> Key: SPARK-15114
> URL: https://issues.apache.org/jira/browse/SPARK-15114
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Reporter: Yin Huai
>Priority: Critical
>
> {code}
> case class Person(name: String, email: String, age: Long)
> val ds = spark.read.json("/tmp/person.json").as[Person]
> import org.apache.spark.sql.expressions.scala.typed._
> ds.groupByKey(_ => 0).agg(sum(_.age))
> // org.apache.spark.sql.Dataset[(Int, Double)] = [value: int, 
> typedsumdouble(unresolveddeserializer(newInstance(class Person), age#0L, 
> email#1, name#2), upcast(value)): double]
> ds.groupByKey(_ => 0).agg(sum(_.age)).explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Final,isDistinct=false)],
>  output=[value#84,typedsumdouble(unresolveddeserializer(newInstance(class 
> $line15.$read$$iw$$iw$Person), age#0L, email#1, name#2), upcast(value))#91])
> : +- INPUT
> +- Exchange hashpartitioning(value#84, 200), None
>+- WholeStageCodegen
>   :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Partial,isDistinct=false)],
>  output=[value#84,value#97])
>   : +- INPUT
>   +- AppendColumns , newInstance(class 
> $line15.$read$$iw$$iw$Person), [input[0, int] AS value#84]
>  +- WholeStageCodegen
> :  +- Scan HadoopFiles[age#0L,email#1,name#2] Format: JSON, 
> PushedFilters: [], ReadSchema: struct
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15114) Column name generated by typed aggregate is super verbose

2016-05-04 Thread Yin Huai (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270243#comment-15270243
 ] 

Yin Huai commented on SPARK-15114:
--

Seems we are missing this case in 
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Column.scala#L120-L148
 ?

> Column name generated by typed aggregate is super verbose
> -
>
> Key: SPARK-15114
> URL: https://issues.apache.org/jira/browse/SPARK-15114
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Reporter: Yin Huai
>Priority: Critical
>
> {code}
> case class Person(name: String, email: String, age: Long)
> val ds = spark.read.json("/tmp/person.json").as[Person]
> import org.apache.spark.sql.expressions.scala.typed._
> ds.groupByKey(_ => 0).agg(sum(_.age))
> // org.apache.spark.sql.Dataset[(Int, Double)] = [value: int, 
> typedsumdouble(unresolveddeserializer(newInstance(class Person), age#0L, 
> email#1, name#2), upcast(value)): double]
> ds.groupByKey(_ => 0).agg(sum(_.age)).explain
> == Physical Plan ==
> WholeStageCodegen
> :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Final,isDistinct=false)],
>  output=[value#84,typedsumdouble(unresolveddeserializer(newInstance(class 
> $line15.$read$$iw$$iw$Person), age#0L, email#1, name#2), upcast(value))#91])
> : +- INPUT
> +- Exchange hashpartitioning(value#84, 200), None
>+- WholeStageCodegen
>   :  +- TungstenAggregate(key=[value#84], 
> functions=[(TypedSumDouble($line15.$read$$iw$$iw$Person),mode=Partial,isDistinct=false)],
>  output=[value#84,value#97])
>   : +- INPUT
>   +- AppendColumns , newInstance(class 
> $line15.$read$$iw$$iw$Person), [input[0, int] AS value#84]
>  +- WholeStageCodegen
> :  +- Scan HadoopFiles[age#0L,email#1,name#2] Format: JSON, 
> PushedFilters: [], ReadSchema: struct
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org