[
https://issues.apache.org/jira/browse/SPARK-9614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661309#comment-14661309
]
Burak Yavuz commented on SPARK-9614:
------------------------------------
It used to work in Spark 1.4, without Tungsten. I fixed it for the 1.5 release,
but this could be manifesting itself somewhere else. Basically we need to be
careful with places that use executionPlan.toRdd.aggregate. Actually any place
that works on RDD[InternalRow] by a reduce, or aggregate, and doesn't copy the
value, but references the pointer.
> InternalRow representation during executionPlan.toRdd.aggregete possibly
> problematic
> ------------------------------------------------------------------------------------
>
> Key: SPARK-9614
> URL: https://issues.apache.org/jira/browse/SPARK-9614
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Burak Yavuz
> Priority: Blocker
>
> For example, in FrequentItems.scala, we have a specialized FrequentItems
> counter which is backed by a Mutable Map. The keys in the MutableMap are the
> elements in a column.
> After iterating through a partition, all the keys turn out to be the latest
> element seen in that partition!
> Assume a partition is composed of the elements ("0","1","2","3","4") for a
> column of a dataframe. The expected map is:
> {code}
> ("0" -> 1,"1" -> 1,"2" -> 1,"3" -> 1,"4" -> 1)
> {code}
> But once you print it out, it turns out to be:
> {code}
> ("4" -> 1,"4" -> 1,"4" -> 1,"4" -> 1,"4" -> 1)
> {code}
> There might be other instances where such behavior may be observed, and it
> must be analyzed before the release.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
