Re: Stream-stream join in structured streaming
Hi Youngwoo, We are aware that the current SQL spatial join does not support streaming tables. The Sedona support of Spark structured streaming is on our roadmap but it won't be available soon. Thanks, Jia On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) wrote: > Hi, > > I'm looking into spatial range query using Apache Sedona in spark > structured streaming. In my test code, I joined two streams using > `st_contains` like following: > > Dataset fenced = spark.sql( > > "SELECT * FROM poly, pts WHERE > > ST_Contains(ST_PolygonFromText(poly.value,','), > > ST_PointFromText(pts.value,','))"); > > > I created two streams from `sparkSession.readStream()` and created two > views 'poly' and 'pts' respectively. As you expect, I wanted to know > whether the polygon contains the point. > > However, It does not work and I get an exception like this: > > Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join > > without equality predicate is not supported;; > > Join Inner, > > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** > > > Looks like structured streaming has a restriction on that kind of join > query. > > Please let me know if there is a workaround for spatial joins and also, any > plans to support joins in structured streaming from Sedona. > > Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. > > Thanks, > Youngwoo >
Re: Stream-stream join in structured streaming
Got it, thank you Jia for noticing that! Youngwoo On Tue, Feb 2, 2021 at 11:56 AM Jia Yu wrote: > Hi Youngwoo, > > We are aware that the current SQL spatial join does not support streaming > tables. The Sedona support of Spark structured streaming is on our roadmap > but it won't be available soon. > > Thanks, > Jia > > On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) > wrote: > >> Hi, >> >> I'm looking into spatial range query using Apache Sedona in spark >> structured streaming. In my test code, I joined two streams using >> `st_contains` like following: >> >> Dataset fenced = spark.sql( >> > "SELECT * FROM poly, pts WHERE >> > ST_Contains(ST_PolygonFromText(poly.value,','), >> > ST_PointFromText(pts.value,','))"); >> >> >> I created two streams from `sparkSession.readStream()` and created two >> views 'poly' and 'pts' respectively. As you expect, I wanted to know >> whether the polygon contains the point. >> >> However, It does not work and I get an exception like this: >> >> Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join >> > without equality predicate is not supported;; >> > Join Inner, >> > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** >> >> >> Looks like structured streaming has a restriction on that kind of join >> query. >> >> Please let me know if there is a workaround for spatial joins and also, >> any >> plans to support joins in structured streaming from Sedona. >> >> Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. >> >> Thanks, >> Youngwoo >> >
[jira] [Closed] (SEDONA-5) Update Sedona documentation
[ https://issues.apache.org/jira/browse/SEDONA-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Yu closed SEDONA-5. --- Assignee: Jia Yu Resolution: Fixed > Update Sedona documentation > --- > > Key: SEDONA-5 > URL: https://issues.apache.org/jira/browse/SEDONA-5 > Project: Apache Sedona > Issue Type: Improvement >Reporter: Jia Yu >Assignee: Jia Yu >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > > The following items need to be corrected in the documentation website: > > # Download > # Tutorial > # API > # GeoSpark archive. The link to GeoSpark archive should be also added to the > previous three sections > In particular, Maven coordinates, and classpath should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Podling Sedona Report Reminder - February 2021
Dear Sedona mentors, I have submitted the podling report. Please read it and let me know if you have any questions. Thank you again for your time! Jia On Sun, Jan 31, 2021 at 1:59 PM Felix Cheung wrote: > Reminder > > > > From: jmcl...@apache.org > Sent: Wednesday, January 20, 2021 10:20:08 PM > To: d...@sedona.incubator.apache.org > Subject: Podling Sedona Report Reminder - February 2021 > > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare your quarterly board report. > > The board meeting is scheduled for Wed, 17 February 2021. > The report for your podling will form a part of the Incubator PMC > report. The Incubator PMC requires your report to be submitted 2 weeks > before the board meeting, to allow sufficient time for review and > submission (Wed, February 03). > > Please submit your report with sufficient time to allow the Incubator > PMC, and subsequently board members to review and digest. Again, the > very latest you should submit your report is 2 weeks prior to the board > meeting. > > Candidate names should not be made public before people are actually > elected, so please do not include the names of potential committers or > PPMC members in your report. > > Thanks, > > The Apache Incubator PMC > > Submitting your Report > > -- > > Your report should contain the following: > > * Your project name > * A brief description of your project, which assumes no knowledge of > the project or necessarily of its field > * A list of the three most important issues to address in the move > towards graduation. > * Any issues that the Incubator PMC or ASF Board might wish/need to be > aware of > * How has the community developed since the last report > * How has the project developed since the last report. > * How does the podling rate their own maturity. > > This should be appended to the Incubator Wiki page at: > > https://cwiki.apache.org/confluence/display/INCUBATOR/February2021 > > Note: This is manually populated. You may need to wait a little before > this page is created from a template. > > Note: The format of the report has changed to use markdown. > > Mentors > --- > > Mentors should review reports for their project(s) and sign them off on > the Incubator wiki page. Signing off reports shows that you are > following the project - projects that are not signed may raise alarms > for the Incubator PMC. > > Incubator PMC >
Re: Stream-stream join in structured streaming
This could be highly interesting! From: Youngwoo Kim (김영우) Sent: Monday, February 1, 2021 7:03:36 PM To: Jia Yu Cc: dev@sedona.apache.org Subject: Re: Stream-stream join in structured streaming Got it, thank you Jia for noticing that! Youngwoo On Tue, Feb 2, 2021 at 11:56 AM Jia Yu wrote: > Hi Youngwoo, > > We are aware that the current SQL spatial join does not support streaming > tables. The Sedona support of Spark structured streaming is on our roadmap > but it won't be available soon. > > Thanks, > Jia > > On Sun, Jan 31, 2021 at 9:31 PM Youngwoo Kim (김영우) > wrote: > >> Hi, >> >> I'm looking into spatial range query using Apache Sedona in spark >> structured streaming. In my test code, I joined two streams using >> `st_contains` like following: >> >> Dataset fenced = spark.sql( >> > "SELECT * FROM poly, pts WHERE >> > ST_Contains(ST_PolygonFromText(poly.value,','), >> > ST_PointFromText(pts.value,','))"); >> >> >> I created two streams from `sparkSession.readStream()` and created two >> views 'poly' and 'pts' respectively. As you expect, I wanted to know >> whether the polygon contains the point. >> >> However, It does not work and I get an exception like this: >> >> Caused by: org.apache.spark.sql.AnalysisException: Stream-stream join >> > without equality predicate is not supported;; >> > Join Inner, >> > **org.apache.spark.sql.geosparksql.expressions.ST_Contains$** >> >> >> Looks like structured streaming has a restriction on that kind of join >> query. >> >> Please let me know if there is a workaround for spatial joins and also, >> any >> plans to support joins in structured streaming from Sedona. >> >> Spark 3.0.1 and GeoSpark 1.3.2-SNAPSHOT for my test env. >> >> Thanks, >> Youngwoo >> >
