[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gheorghe Gheorghe updated SPARK-21390: -------------------------------------- Description: Hello everybody, I've encountered a strange situation with the spark-shell. When I run the code below in my IDE the second test case prints as expected count "1". However, when I run the same code using the spark-shell in the second test case I get 0 back as a count. I've made sure that I'm running scala 2.11.8 and spark 2.0.1 in both my IDE and spark-shell. {code:java} import org.apache.spark.sql.Dataset case class SomeClass(field1:String, field2:String) val filterCondition: Seq[SomeClass] = Seq( SomeClass("00", "01") ) // Test 1 val filterMe1: Dataset[SomeClass] = Seq( SomeClass("00", "01") ).toDS println("Works fine!" +filterMe1.filter(filterCondition.contains(_)).count) // Test 2 case class OtherClass(field1:String, field2:String) val filterMe2 = Seq( OtherClass("00", "01"), OtherClass("00", "02")).toDS println("Fail, count should return 1: " + filterMe2.filter(x=> filterCondition.contains(SomeClass(x.field1, x.field2))).count) {code} Note if I transform the dataset first I get 1 back as expected. {code:java} println(filterMe2.map(x=> SomeClass(x.field1, x.field2)).filter(filterCondition.contains(_)).count) {code} Is this a bug? I can see that this filter function has been marked as experimental https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Dataset.html#filter(scala.Function1) was: Hello everybody, I've encountered a strange situation with the spark-shell. When I run the code below in my IDE the second test case prints as expected count "1". However, when I run the same code using the spark-shell in the second test case I get 0 back as a count. I've made sure that I'm running scala 2.11.8 and spark 2.0.1 in both my IDE and spark-shell. {code:java} import org.apache.spark.sql.Dataset case class SomeClass(field1:String, field2:String) val filterCondition: Seq[SomeClass] = Seq( SomeClass("00", "01") ) // Test 1 val filterMe1: Dataset[SomeClass] = Seq( SomeClass("00", "01") ).toDS println("Works fine!" +filterMe1.filter(filterCondition.contains(_)).count) // Test 2 case class OtherClass(field1:String, field2:String) val filterMe2 = Seq( OtherClass("00", "01"), OtherClass("00", "02")).toDS println("Fail, count should return 1: " + filterMe2.filter(x=> filterCondition.contains(SomeClass(x.field1, x.field2))).count) {code} Note if I do this it is printing 1 as expected. {code:java} println(filterMe2.map(x=> SomeClass(x.field1, x.field2)).filter(filterCondition.contains(_)).count) {code} Is this a bug? I can see that this filter function has been marked as experimental https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Dataset.html#filter(scala.Function1) > Dataset filter api inconsistency > -------------------------------- > > Key: SPARK-21390 > URL: https://issues.apache.org/jira/browse/SPARK-21390 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.1 > Reporter: Gheorghe Gheorghe > Priority: Minor > > Hello everybody, > I've encountered a strange situation with the spark-shell. > When I run the code below in my IDE the second test case prints as expected > count "1". However, when I run the same code using the spark-shell in the > second test case I get 0 back as a count. > I've made sure that I'm running scala 2.11.8 and spark 2.0.1 in both my IDE > and spark-shell. > {code:java} > import org.apache.spark.sql.Dataset > case class SomeClass(field1:String, field2:String) > val filterCondition: Seq[SomeClass] = Seq( SomeClass("00", "01") ) > // Test 1 > val filterMe1: Dataset[SomeClass] = Seq( SomeClass("00", "01") ).toDS > > println("Works fine!" +filterMe1.filter(filterCondition.contains(_)).count) > > // Test 2 > case class OtherClass(field1:String, field2:String) > > val filterMe2 = Seq( OtherClass("00", "01"), OtherClass("00", "02")).toDS > println("Fail, count should return 1: " + filterMe2.filter(x=> > filterCondition.contains(SomeClass(x.field1, x.field2))).count) > {code} > Note if I transform the dataset first I get 1 back as expected. > {code:java} > println(filterMe2.map(x=> SomeClass(x.field1, > x.field2)).filter(filterCondition.contains(_)).count) > {code} > Is this a bug? I can see that this filter function has been marked as > experimental > https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Dataset.html#filter(scala.Function1) -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org