Re: Weird results with Spark SQL Outer joins

cally > pulling > >>>>>> out > >>>>>> selected columns from the query, but there is no roll up happening > or > >>>>>> anything that would possible make it suspicious that there is any > >>>>>> difference > &

Re: Weird results with Spark SQL Outer joins

matched based on three keys >>>>>> that >>>>>> are present in both tables (ad, account, and date), on top of this >>>>>> they are >>>>>> filtered by date being above 2016-01-03. Since all the joins are >>>

Re: Weird results with Spark SQL Outer joins

> > difference > >>>> > besides the type of joins. The tables are matched based on three > keys > >>>> > that > >>>> > are present in both tables (ad, account, and date), on top of this > >>>> > they are > >>>

Re: Weird results with Spark SQL Outer joins

mo_lt where date >>>> >>='2016-01-03'").count >>>> > >>>> > res14: Long = 34158 >>>> > >>>> > scala> sqlContext.sql("select * from dps_pin_promo_lt where date >>>> >>='2016-01-03&#

Re: Weird results with Spark SQL Outer joins

res15: Long = 42693 >>> > >>> > >>> > The above two queries filter out the data based on date used by the >>> joins of >>> > 2016-01-03 and you can see the row count between the two tables are >>> > different, which is why I am suspecting something is wrong w

Re: Weird results with Spark SQL Outer joins

joins in spark sql, because in this situation the right and outer joins >> may >> > produce the same results, but it should not be equal to the left join >> and >> > definitely not the inner join; unless I am missing something. >> > >> > >

Re: Weird results with Spark SQL Outer joins

l value is res16: Long = 42694 > > > > > > Thanks, > > > > > > KP > > > > > > > > > > On Mon, May 2, 2016 at 12:50 PM, Yong Zhang > wrote: > >> > >> We are still not sure what is the problem, if you cannot show u

Re: Weird results with Spark SQL Outer joins

ultSet row count as dps right outer join >> with swig on 3 columns, with same additional filters. >> >> Without knowing your data, I cannot see the reason that has to be a bug in >> the spark. >> >> Am I misunderstanding your bug? >> >> Yong >> >> ___

Re: Weird results with Spark SQL Outer joins

or left join. >> >> I think that there is no problem here, unless I am missing something. >> >> Regards, >> Gourav >> >> On Mon, May 2, 2016 at 7:48 PM, kpeng1 wrote: >> >> Also, the results of the inner query produced the same results: >> sq

Re: Weird results with Spark SQL Outer joins

ter join > with swig on 3 columns, with same additional filters. > > Without knowing your data, I cannot see the reason that has to be a bug in > the spark. > > Am I misunderstanding your bug? > > Yong > > ---------- > From: kpe...@gmail.com >

RE: Weird results with Spark SQL Outer joins

: Mon, 2 May 2016 12:11:18 -0700 Subject: Re: Weird results with Spark SQL Outer joins To: gourav.sengu...@gmail.com CC: user@spark.apache.org Gourav, I wish that was case, but I have done a select count on each of the two tables individually and they return back different number of rows

Re: Weird results with Spark SQL Outer joins

6-01-03'AND d.date >= >> '2016-01-03'").count() >> RESULT:23747 >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Weird-results-with-Spark-SQL-Outer-joins-tp26861p26863.htm

Re: Weird results with Spark SQL Outer joins

romo_lt s INNER JOIN > dps_pin_promo_lt d ON (s.date = d.date AND s.account = d.account AND s.ad > = > d.ad) WHERE s.date >= '2016-01-03'AND d.date >= '2016-01-03'").count() > RESULT:23747 > > > > -- > View this message in context: > http://apac

Re: Weird results with Spark SQL Outer joins

.date = d.date AND s.account = d.account AND s.ad = d.ad) WHERE s.date >= '2016-01-03'AND d.date >= '2016-01-03'").count() RESULT:23747 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Weird-results-with-Spark-SQL-Outer-jo

Re: Weird results with Spark SQL Outer joins

t;>> d.account AS >>> d_acc , s.ad as s_ad , d.ad as d_ad , s.spend AS s_spend , >>> d.spend_in_dollar AS d_spend FROM swig_pin_promo_lt s RIGHT OUTER JOIN >>> dps_pin_promo_lt d ON (s.date = d.date AND s.account = d.account AND >>> s.ad = >&g

Re: Weird results with Spark SQL Outer joins

_lt s RIGHT OUTER JOIN >> dps_pin_promo_lt d ON (s.date = d.date AND s.account = d.account AND >> s.ad = >> d.ad) WHERE s.date >= '2016-01-03'AND d.date >= >> '2016-01-03'").count() >> RESULT: 23747 >> >> Was wondering if some

Re: Weird results with Spark SQL Outer joins

nd AS s_spend , > d.spend_in_dollar AS d_spend FROM swig_pin_promo_lt s RIGHT OUTER JOIN > dps_pin_promo_lt d ON (s.date = d.date AND s.account = d.account AND s.ad > = > d.ad) WHERE s.date >= '2016-01-03'AND d.date >= '2016-01-03'").count() > RES

Weird results with Spark SQL Outer joins

ncountered this issues before. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Weird-results-with-Spark-SQL-Outer-joins-tp26861.html Sent from the Apache Spark User List mailing list archive at Nabble.com. ---