Re: Is Spark a good choice for geospatial/GIS applications? Is a community volunteer needed in this area?
Thanks for the info. It seems like the JTS library is exactly what I need (I'm not doing any raster processing at this point). So, once they successfully finish the Scala wrappers for JTS, I would theoretically be able to use Scala to write a Spark job that includes the JTS library, and then run it across a Spark cluster? That is absolutely fantastic! I'll have to look into contributing to the JTS wrapping effort. Thanks again! On 14-04-2014 at 2:25 PM, "Josh Marcus" wrote:Hey there, I'd encourage you to check out the development currently going on with the GeoTrellis project(http://github.com/geotrellis/geotrellis) or talk to the developers on irc (freenode, #geotrellis) as they're currently developing raster processing capabilities with spark as a backend, as well as scala wrappersfor JTS (for calculations about geometry features). --j On Wed, Apr 23, 2014 at 2:00 PM, neveroutgunned wrote: Greetings Spark users/devs! I'm interested in using Spark to process large volumes of data with a geospatial component, and I haven't been able to find much information on Spark's ability to handle this kind of operation. I don't need anything too complex; just distance between two points, point-in-polygon and the like. Does Spark (or possibly Shark) support this kind of query? Has anyone written a plugin/extension along these lines? If there isn't anything like this so far, then it seems like I have two options. I can either abandon Spark and fall back on Hadoop and Hive with the ESRI Tools extension, or I can stick with Spark and try to write/port a GIS toolkit. Which option do you think I should pursue? How hard is it for someone that's new to the Spark codebase to write an extension? Is there anyone else in the community that would be interested in having geospatial capability in Spark? Thanks for your help! ------------- View this message in context: Is Spark a good choice for geospatial/GIS applications? Is a community volunteer needed in this area? Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Is Spark a good choice for geospatial/GIS applications? Is a community volunteer needed in this area?
Hey there, I'd encourage you to check out the development currently going on with the GeoTrellis project (http://github.com/geotrellis/geotrellis) or talk to the developers on irc (freenode, #geotrellis) as they're currently developing raster processing capabilities with spark as a backend, as well as scala wrappers for JTS (for calculations about geometry features). --j On Wed, Apr 23, 2014 at 2:00 PM, neveroutgunned wrote: > Greetings Spark users/devs! I'm interested in using Spark to process large > volumes of data with a geospatial component, and I haven't been able to > find much information on Spark's ability to handle this kind of operation. > I don't need anything too complex; just distance between two points, > point-in-polygon and the like. > > Does Spark (or possibly Shark) support this kind of query? Has anyone > written a plugin/extension along these lines? > > If there isn't anything like this so far, then it seems like I have two > options. I can either abandon Spark and fall back on Hadoop and Hive with > the ESRI Tools extension, or I can stick with Spark and try to write/port a > GIS toolkit. Which option do you think I should pursue? How hard is it for > someone that's new to the Spark codebase to write an extension? Is there > anyone else in the community that would be interested in having geospatial > capability in Spark? > > Thanks for your help! > > -------------- > View this message in context: Is Spark a good choice for geospatial/GIS > applications? Is a community volunteer needed in this > area?<http://apache-spark-user-list.1001560.n3.nabble.com/Is-Spark-a-good-choice-for-geospatial-GIS-applications-Is-a-community-volunteer-needed-in-this-area-tp4685.html> > Sent from the Apache Spark User List mailing list > archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com. >
Re: Is Spark a good choice for geospatial/GIS applications? Is a community volunteer needed in this area?
I am not sure the kind of operations you wantyou can always put a list of lat-long inside a census-block and add census block as a vertex in graphx...and add edges within the census block... Looks like set of code that uses graphx primitives... Could you take a look at graphx apis and see if they suffice your GIS usecase ? I have not looked into ESRI extensions... On Wed, Apr 23, 2014 at 11:00 AM, neveroutgunned wrote: > Greetings Spark users/devs! I'm interested in using Spark to process large > volumes of data with a geospatial component, and I haven't been able to > find much information on Spark's ability to handle this kind of operation. > I don't need anything too complex; just distance between two points, > point-in-polygon and the like. > > Does Spark (or possibly Shark) support this kind of query? Has anyone > written a plugin/extension along these lines? > > If there isn't anything like this so far, then it seems like I have two > options. I can either abandon Spark and fall back on Hadoop and Hive with > the ESRI Tools extension, or I can stick with Spark and try to write/port a > GIS toolkit. Which option do you think I should pursue? How hard is it for > someone that's new to the Spark codebase to write an extension? Is there > anyone else in the community that would be interested in having geospatial > capability in Spark? > > Thanks for your help! > > ---------- > View this message in context: Is Spark a good choice for geospatial/GIS > applications? Is a community volunteer needed in this > area?<http://apache-spark-user-list.1001560.n3.nabble.com/Is-Spark-a-good-choice-for-geospatial-GIS-applications-Is-a-community-volunteer-needed-in-this-area-tp4685.html> > Sent from the Apache Spark User List mailing list > archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com. >
Is Spark a good choice for geospatial/GIS applications? Is a community volunteer needed in this area?
Greetings Spark users/devs! I'm interested in using Spark to process large volumes of data with a geospatial component, and I haven't been able to find much information on Spark's ability to handle this kind of operation. I don't need anything too complex; just distance between two points, point-in-polygon and the like. Does Spark (or possibly Shark) support this kind of query? Has anyone written a plugin/extension along these lines? If there isn't anything like this so far, then it seems like I have two options. I can either abandon Spark and fall back on Hadoop and Hive with the ESRI Tools extension, or I can stick with Spark and try to write/port a GIS toolkit. Which option do you think I should pursue? How hard is it for someone that's new to the Spark codebase to write an extension? Is there anyone else in the community that would be interested in having geospatial capability in Spark? Thanks for your help! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-Spark-a-good-choice-for-geospatial-GIS-applications-Is-a-community-volunteer-needed-in-this-area-tp4685.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
