Cheng Lian created SPARK-4963: --------------------------------- Summary: SchemaRDD.sample may return wrong results Key: SPARK-4963 URL: https://issues.apache.org/jira/browse/SPARK-4963 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.2.0 Reporter: Cheng Lian
This {{sbt/sbt hive/console}} session can easily reproduce this issue: {code} sql("SELECT * FROM src WHERE key % 2 = 0"). sample(withReplacement = false, fraction = 0.05). registerTempTable("sampled") println(table("sampled").queryExecution) val query = sql("SELECT * FROM sampled WHERE key % 2 = 1") println(query.queryExecution) // Should print `true' println((1 to 10).map(_ => query.collect().isEmpty).reduce(_ && _)) {code} Notice that when fraction is less than 0.4, {{GapSamplingIterator}} is used to do the sampling. My guess is that there’s something to do with the underlying mutable row objects used in {{HiveTableScan}}, but haven't figured out the root cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org