Re: Stream-stream join in structured streaming
Hi Youngwoo, We are aware that the current SQL spatial join does not support streaming tables. The Sedona support of Spark structured streaming is on our roadmap but it won't be available soon. Thanks, Jia On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) wrote: > Hi, > > I'm looking into spatial range query using Apache Sedona in spark > structured streaming. In my test code, I joined two streams using > `st_contains` like following: > > Dataset fenced = spark.sql( > > "SELECT * FROM poly, pts WHERE > > ST_Contains(ST_PolygonFromText(poly.value,','), > > ST_PointFromText(pts.value,','))"); > > > I created two streams from `sparkSession.readStream()` and created two > views 'poly' and 'pts' respectively. As you expect, I wanted to know > whether the polygon contains the point. > > However, It does not work and I get an exception like this: > > Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join > > without equality predicate is not supported;; > > Join Inner, > > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** > > > Looks like structured streaming has a restriction on that kind of join > query. > > Please let me know if there is a workaround for spatial joins and also, any > plans to support joins in structured streaming from Sedona. > > Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. > > Thanks, > Youngwoo >
Re: Stream-stream join in structured streaming
Got it, thank you Jia for noticing that! Youngwoo On Tue, Feb 2, 2021 at 11:56 AM Jia Yu wrote: > Hi Youngwoo, > > We are aware that the current SQL spatial join does not support streaming > tables. The Sedona support of Spark structured streaming is on our roadmap > but it won't be available soon. > > Thanks, > Jia > > On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) > wrote: > >> Hi, >> >> I'm looking into spatial range query using Apache Sedona in spark >> structured streaming. In my test code, I joined two streams using >> `st_contains` like following: >> >> Dataset fenced = spark.sql( >> > "SELECT * FROM poly, pts WHERE >> > ST_Contains(ST_PolygonFromText(poly.value,','), >> > ST_PointFromText(pts.value,','))"); >> >> >> I created two streams from `sparkSession.readStream()` and created two >> views 'poly' and 'pts' respectively. As you expect, I wanted to know >> whether the polygon contains the point. >> >> However, It does not work and I get an exception like this: >> >> Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join >> > without equality predicate is not supported;; >> > Join Inner, >> > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** >> >> >> Looks like structured streaming has a restriction on that kind of join >> query. >> >> Please let me know if there is a workaround for spatial joins and also, >> any >> plans to support joins in structured streaming from Sedona. >> >> Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. >> >> Thanks, >> Youngwoo >> >
Re: Stream-stream join in structured streaming
This could be highly interesting! From: Youngwoo Kim (김영우) Sent: Monday, February 1, 2021 7:03:36 PM To: Jia Yu Cc: dev@sedona.apache.org Subject: Re: Stream-stream join in structured streaming Got it, thank you Jia for noticing that! Youngwoo On Tue, Feb 2, 2021 at 11:56 AM Jia Yu wrote: > Hi Youngwoo, > > We are aware that the current SQL spatial join does not support streaming > tables. The Sedona support of Spark structured streaming is on our roadmap > but it won't be available soon. > > Thanks, > Jia > > On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) > wrote: > >> Hi, >> >> I'm looking into spatial range query using Apache Sedona in spark >> structured streaming. In my test code, I joined two streams using >> `st_contains` like following: >> >> Dataset fenced = spark.sql( >> > "SELECT * FROM poly, pts WHERE >> > ST_Contains(ST_PolygonFromText(poly.value,','), >> > ST_PointFromText(pts.value,','))"); >> >> >> I created two streams from `sparkSession.readStream()` and created two >> views 'poly' and 'pts' respectively. As you expect, I wanted to know >> whether the polygon contains the point. >> >> However, It does not work and I get an exception like this: >> >> Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join >> > without equality predicate is not supported;; >> > Join Inner, >> > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** >> >> >> Looks like structured streaming has a restriction on that kind of join >> query. >> >> Please let me know if there is a workaround for spatial joins and also, >> any >> plans to support joins in structured streaming from Sedona. >> >> Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. >> >> Thanks, >> Youngwoo >> >
