[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569048#comment-15569048 ]
Pete Robbins commented on SPARK-17827: -------------------------------------- So this looks like the max field is being written as an Int into the UnsafeRow but is later read as a Long. Code stack to the write: java.lang.Thread.dumpStack(Thread.java:462) at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write(UnsafeRowWriter.java:149) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$generateResultProjection$1.apply(AggregationIterator.scala:232) at org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$generateResultProjection$1.apply(AggregationIterator.scala:221) at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:392) at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:79) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at org.apache.spark.sql.execution.aggregate.AggregationIterator.foreach(AggregationIterator.scala:35) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) at org.apache.spark.sql.execution.aggregate.AggregationIterator.to(AggregationIterator.scala:35) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) at org.apache.spark.sql.execution.aggregate.AggregationIterator.toBuffer(AggregationIterator.scala:35) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) at org.apache.spark.sql.execution.aggregate.AggregationIterator.toArray(AggregationIterator.scala:35) at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1927) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1927) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:785) > StatisticsColumnSuite failures on big endian platforms > ------------------------------------------------------ > > Key: SPARK-17827 > URL: https://issues.apache.org/jira/browse/SPARK-17827 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.0 > Environment: big endian > Reporter: Pete Robbins > Labels: big-endian > > https://issues.apache.org/jira/browse/SPARK-17073 > introduces new tests/function that fails on big endian platforms > Failing tests: > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > string column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > binary column > org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for > columns with different types > org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics > and load them from hive metastore > all fail in checkColStat eg: > java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:156) > at > org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40) > at > org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168) > at > org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33) > at > org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) > at > org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org