This could be highly interesting! ________________________________ From: Youngwoo Kim (김영우) <yw...@apache.org> Sent: Monday, February 1, 2021 7:03:36 PM To: Jia Yu <jiayu198...@gmail.com> Cc: dev@sedona.apache.org <dev@sedona.apache.org> Subject: Re: Stream-stream join in structured streaming
Got it, thank you Jia for noticing that! Youngwoo On Tue, Feb 2, 2021 at 11:56 AM Jia Yu <jiayu198...@gmail.com> wrote: > Hi Youngwoo, > > We are aware that the current SQL spatial join does not support streaming > tables. The Sedona support of Spark structured streaming is on our roadmap > but it won't be available soon. > > Thanks, > Jia > > On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) <yw...@apache.org> > wrote: > >> Hi, >> >> I'm looking into spatial range query using Apache Sedona in spark >> structured streaming. In my test code, I joined two streams using >> `st_contains` like following: >> >> Dataset<Row> fenced = spark.sql( >> > "SELECT * FROM poly, pts WHERE >> > ST_Contains(ST_PolygonFromText(poly.value,','), >> > ST_PointFromText(pts.value,','))"); >> >> >> I created two streams from `sparkSession.readStream()` and created two >> views 'poly' and 'pts' respectively. As you expect, I wanted to know >> whether the polygon contains the point. >> >> However, It does not work and I get an exception like this: >> >> Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join >> > without equality predicate is not supported;; >> > Join Inner, >> > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** >> >> >> Looks like structured streaming has a restriction on that kind of join >> query. >> >> Please let me know if there is a workaround for spatial joins and also, >> any >> plans to support joins in structured streaming from Sedona. >> >> Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. >> >> Thanks, >> Youngwoo >> >
