[
https://issues.apache.org/jira/browse/SPARK-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073145#comment-15073145
]
Tim Preece commented on SPARK-12319:
------------------------------------
Hi,
The failing test is already checked in. It is:
"org.apache.spark.sql.DatasetAggregatorSuite.typed aggregation: class input
with reordering"
The test only explicitly fails on Big Endian platforms. This is because an
integer takes an 8 byte slot in the Unsafe row. When the data corruption occurs
the BE integer ends up with the wrong value. I added print statements which
shows the data corruption on Little Endian as well, it just happens not to
effect the value of the LE integer, since the LE integer is in the other
4-bytes of the 8-byte slot.
> Address endian specific problems surfaced in 1.6
> ------------------------------------------------
>
> Key: SPARK-12319
> URL: https://issues.apache.org/jira/browse/SPARK-12319
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0
> Environment: Problems apparent on BE, LE could be impacted too
> Reporter: Adam Roberts
> Priority: Critical
>
> JIRA to cover endian specific problems - since testing 1.6 I've noticed
> problems with DataFrames on BE platforms, e.g.
> https://issues.apache.org/jira/browse/SPARK-9858
> [~joshrosen] [~yhuai]
> Current progress: using com.google.common.io.LittleEndianDataInputStream and
> com.google.common.io.LittleEndianDataOutputStream within UnsafeRowSerializer
> fixes three test failures in ExchangeCoordinatorSuite but I'm concerned
> around performance/wider functional implications
> "org.apache.spark.sql.DatasetAggregatorSuite.typed aggregation: class input
> with reordering" fails as we expect "one, 1" but instead get "one, 9" - we
> believe the issue lies within BitSetMethods.java, specifically around: return
> (wi << 6) + subIndex + java.lang.Long.numberOfTrailingZeros(word);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
