[
https://issues.apache.org/jira/browse/SPARK-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312233#comment-15312233
]
Adam Roberts commented on SPARK-12319:
--------------------------------------
Lowered priority, this also impacts Spark 2.0.0 but it's of questionable
importance, we did an analysis of this a while back and concluded that the
tests verify that the results of the DataFrame operations are correct and are
only failing here because of the slightly different compressed partition sizes
during the operations
> ExchangeCoordinatorSuite fails on big-endian platforms
> ------------------------------------------------------
>
> Key: SPARK-12319
> URL: https://issues.apache.org/jira/browse/SPARK-12319
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 1.6.0
> Environment: Problems apparent on BE, LE could be impacted too
> Reporter: Adam Roberts
> Priority: Minor
> Labels: big-endian
>
> JIRA to cover endian specific problems - since testing 1.6 I've noticed
> problems with DataFrames on BE platforms, e.g.
> https://issues.apache.org/jira/browse/SPARK-9858
> [~joshrosen] [~yhuai]
> Current progress: using com.google.common.io.LittleEndianDataInputStream and
> com.google.common.io.LittleEndianDataOutputStream within UnsafeRowSerializer
> fixes three test failures in ExchangeCoordinatorSuite but I'm concerned
> around performance/wider functional implications
> "org.apache.spark.sql.DatasetAggregatorSuite.typed aggregation: class input
> with reordering" fails as we expect "one, 1" but instead get "one, 9" - we
> believe the issue lies within BitSetMethods.java, specifically around: return
> (wi << 6) + subIndex + java.lang.Long.numberOfTrailingZeros(word);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
