Re: Fwd: Re: DataFrame equality does not working in 1.5.1

Hi, Michael It works find. scala> sqlContext.sql("SET spark.sql.inMemoryColumnarStorage.partitionPruning=false") res28: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> eventDF.filter($"entityType" === "user").select("entityId").distinct.count res29: Long = 2091 Thank you

Re: Fwd: Re: DataFrame equality does not working in 1.5.1

Hi Yanal! Yes, exactly. I read from csv file and convert to DF with column names. simply look like this. val eventDF = sc.textFile(eventFile).map(_.split(",")).filter(_.size >= 6) .map { e => // To do sometings }.toDF(eventTableColumns:_*).cache() The result of <=> function is

Re: Fwd: Re: DataFrame equality does not working in 1.5.1

In particular this is sounding like: https://issues.apache.org/jira/browse/SPARK-10859 On Fri, Nov 6, 2015 at 1:05 PM, Michael Armbrust wrote: > I would be great if you could try sql("SET > spark.sql.inMemoryColumnarStorage.partitionPruning=false") also, try Spark >

Re: Fwd: Re: DataFrame equality does not working in 1.5.1

I would be great if you could try sql("SET spark.sql.inMemoryColumnarStorage.partitionPruning=false") also, try Spark 1.5.2-RC2 On Fri, Nov 6, 2015 at 4:49 AM, Seongduk Cheon wrote: > Hi Yanal! > > Yes,