Re: Lucene Spatial Future
On Wed, Apr 6, 2011 at 9:30 AM, Grant Ingersoll grant.ingers...@gmail.com wrote: By all means go for it. I don't see any reason not too. I guess in the end, I'm not sure what you are asking us to do. Do you want Lucene/Solr to remove all of our spatial support in favor of incorporating this new project or do you just want those who are interested in spatial to join the new project and it can be seen as an add on? Let's not confuse the issue... what is being discussed really has no impact on the basic spatial search that was added to Solr. As you said yourself, the Solr geo stuff uses very little of the spatial contrib stuff. This is about building and maintaining a spatial module, and the best place to do it (which I'll leave up to those doing the work... I'm pretty happy with basic point, radius, bounding-box). -Yonik http://www.lucenerevolution.org -- Lucene/Solr User Conference, May 25-26, San Francisco - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
The spatial API in google code takes a pretty different approach to spatial search in general. It is organized into three key packages: 1. core stuff, no lucene dependencies. Most of the math is here Aren't you just replicating what SIS is doing for this piece? If you don't have a JTS requirement, that means you are going to need equivalent math, right? Isn't that what SIS is about? This package defines the general interfaces and concepts used in the project. Things like SpatialOperations, Shape, PrefixGrid and DistanceCalculator -- these can then be backed by simple math, JTS, or eventually maybe SIS. The other key stuff in this package is the client side objects that to build spatial queries. Essentially everything that could be bundled with solrj. I could suggest a new ASF project, but there seems like too much overlap with SIS and very different philosophy on 3rd party libraries. In the end, osgeo.com seems like a more natural home and has better branding for spatial related work anyway. By all means go for it. I don't see any reason not too. I guess in the end, I'm not sure what you are asking us to do. Do you want Lucene/Solr to remove all of our spatial support in favor of incorporating this new project or do you just want those who are interested in spatial to join the new project and it can be seen as an add on? I'm trying to have an open discussion about what makes sense for spatial development. I don't *want* to start a new project... but I think we need a dev/test environment that can support the whole range of spatial needs -- without reinventing many wheels, this includes JTS. Lucene currently has LGPL compile dependencies, but they are on the way out, and (unless I'm missing something) i don't think folks are open to adding a JTS build/test dependency -- Maybe I should call a vote on the JTS issue, though i suspect most binding votes are -0 or -1. I *totally* understand if other people don't want JTS in the build system -- it is not a core concern to most people involved. If the lucene build/test environment does not support spatial development, this leads me to think about other places to host the project... wherever it makes the most sense. I would prefer staying within lucene because it is easiest for me. I don't want this to be competition or duplicate effort. I hope it lets us clean up the broken stuff from lucene and overtime deprecate the parts that are better supported elsewhere. I want the best spatial support available in solr out-of-the-box. If this project is eventually built and maintained outside of lucene, i would like the .jar distributed in solr. ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Apr 6, 2011, at 10:45 AM, Ryan McKinley wrote: I'm trying to have an open discussion about what makes sense for spatial development. I don't *want* to start a new project... but I think we need a dev/test environment that can support the whole range of spatial needs -- without reinventing many wheels, this includes JTS. Lucene currently has LGPL compile dependencies, but they are on the way out, and (unless I'm missing something) i don't think folks are open to adding a JTS build/test dependency -- Maybe I should call a vote on the JTS issue, though i suspect most binding votes are -0 or -1. I *totally* understand if other people don't want JTS in the build system -- it is not a core concern to most people involved. Until there is a specific patch that brings in and shows how JTS would be incorporated (via reflection and as a totally optional piece, presumably, per the ASF LGPL guidelines), there really isn't anything to vote on. I don't want this to be competition or duplicate effort. I hope it lets us clean up the broken stuff from lucene and overtime deprecate the parts that are better supported elsewhere. I totally agree. I hope I wasn't framing it that way. I'm just trying to understand what's being proposed. I can see advantages to both. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Apr 6, 2011, at 11:38 AM, Grant Ingersoll wrote: Until there is a specific patch that brings in and shows how JTS would be incorporated (via reflection and as a totally optional piece, presumably, per the ASF LGPL guidelines), there really isn't anything to vote on. I think what is being asked to vote on is deprecation/removal of Lucene's spatial contrib module with its replacement being an externally hosted ASL-licened module expressly designed to work with Lucene/Solr 4.0 and beyond (temporarily known as lucene-spatial-playground). What would stay is the _basic_ spatial support that got into Lucene/Solr 3.1. Furthermore, no future spatial work would be accepted on Lucene/Solr aside from support of the basic capability. This module isn't quite ready so perhaps the vote should wait till it is. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
-1. I *totally* understand if other people don't want JTS in the build system -- it is not a core concern to most people involved. Until there is a specific patch that brings in and shows how JTS would be incorporated (via reflection and as a totally optional piece, presumably, per the ASF LGPL guidelines), there really isn't anything to vote on. fair point -- the optional logistics are working in a maven build. I'm reluctant to convert to the ant build system if there is already strong opposition to the idea. If folks are OK with the idea, I will happily make concrete patch/branch that we could vote on. so maybe i'm just looking for a POLL not a vote -- find out if this is a non-starter or not (i am under the impression that it might be) FYI, the optional support is now handled by a static SpatialContextProvider' that you can ask for a SpatialContext. By default it makes a SimpleSpatialContext -- if you set some system properties, it uses reflection to load a different instance. Eventually, this should be replaced with the standard java service loader stuff (i think) ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Apr 6, 2011, at 12:06 PM, Smiley, David W. wrote: On Apr 6, 2011, at 11:38 AM, Grant Ingersoll wrote: Until there is a specific patch that brings in and shows how JTS would be incorporated (via reflection and as a totally optional piece, presumably, per the ASF LGPL guidelines), there really isn't anything to vote on. I think what is being asked to vote on is deprecation/removal of Lucene's spatial contrib module Just FYI, It is already deprecated in 3.x and slated for removal in 4.0. Someone just needs to axe the appropriate bits (and either move what's needed to Solr or to modules) with its replacement being an externally hosted ASL-licened module expressly designed to work with Lucene/Solr 4.0 and beyond (temporarily known as lucene-spatial-playground). What would stay is the _basic_ spatial support that got into Lucene/Solr 3.1. Furthermore, no future spatial work would be accepted on Lucene/Solr aside from support of the basic capability. That is the piece I was wondering about and why I said yesterday it isn't likely to work, as it will just fork. How do you tell people not to put in patches to L/S, especially when part of it is native and part of it isn't? - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Apr 6, 2011, at 2:08 PM, Grant Ingersoll wrote: with its replacement being an externally hosted ASL-licened module expressly designed to work with Lucene/Solr 4.0 and beyond (temporarily known as lucene-spatial-playground). What would stay is the _basic_ spatial support that got into Lucene/Solr 3.1. Furthermore, no future spatial work would be accepted on Lucene/Solr aside from support of the basic capability. That is the piece I was wondering about and why I said yesterday it isn't likely to work, as it will just fork. How do you tell people not to put in patches to L/S, especially when part of it is native and part of it isn't? I think risk of this is mitigated if the proposed external module is highly visible in L/S -- in other words, it's downloaded and packaged up as part of the distribution -- a jar, sitting along side the other contrib module jars (no JTS of course!). Users would be referred to this module for non-basic spatial via the wiki and community in general. Of course I would prominently mention this module in the 2nd edition of my book ;-) which is well underway. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Wed, Apr 6, 2011 at 2:08 PM, Grant Ingersoll grant.ingers...@gmail.com wrote: On Apr 6, 2011, at 12:06 PM, Smiley, David W. wrote: with its replacement being an externally hosted ASL-licened module expressly designed to work with Lucene/Solr 4.0 and beyond (temporarily known as lucene-spatial-playground). What would stay is the _basic_ spatial support that got into Lucene/Solr 3.1. Furthermore, no future spatial work would be accepted on Lucene/Solr aside from support of the basic capability. That is the piece I was wondering about and why I said yesterday it isn't likely to work, as it will just fork. How do you tell people not to put in patches to L/S, especially when part of it is native and part of it isn't? Right - there's no need to try and make promises about the future. It seems unrelated to the questions at hand here. -Yonik http://www.lucenerevolution.org -- Lucene/Solr User Conference, May 25-26, San Francisco - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
Right - there's no need to try and make promises about the future. It seems unrelated to the questions at hand here. To be clear... I don't see any of this as promises -- obviously nothing happens until there is somethign concrete to evaluate. The point of this thread (for me anyway) is to raise my concerns, see what people are thinking, and be transparent about my choices. This discussion has made me feel like the right choice (for me) is to pursue spatial development somewhere else -- likely osgeo -- and down the road figure out how that could/should fit with solr. ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module What about C: Spatial in Lucene/Solr w/ JTS component in Apache Extras as a drop in jar that implements the appropriate bindings? I personally have an interest in good spatial in L/S including more than just point search. This solution need not require any automated downloads/compiles, etc. and it can be noted that the support of that particular jar is handled on Apache Extras (or Githup or wherever) Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. I personally don't. I think we have enough committers who are active on spatial that we can sustain it once we have a good foundation in place (which we are close to) and we also have several contributors who have been active on spatial and are likely good candidates for being committers to work on it.
Re: Lucene Spatial Future
On Tue, Apr 5, 2011 at 9:34 AM, Grant Ingersoll gsing...@apache.org wrote: On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module What about C: Spatial in Lucene/Solr w/ JTS component in Apache Extras as a drop in jar that implements the appropriate bindings? I personally have an interest in good spatial in L/S including more than just point search. This solution need not require any automated downloads/compiles, etc. and it can be noted that the support of that particular jar is handled on Apache Extras (or Githup or wherever) If the suggestion is splitting the build so that the JTS shapes and math are not tested with the core, then i am -1 The spatial build and test system should be designed to best test spatial -- without jumping through serious hoops, splitting the compile/test setup only hurts the project. (unless you don't really care about the more complex stuff) I also want to make sure that the concerns of more complex features are a top priority of the whole framework -- developing/patching across two repos stinks. Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. You can always ask... but I think it is too late for a license change. This has been around to 10 years and the original copyright owner appears to be a defunct company. http://www.vividsolutions.com/jts/jtshome.htm (note http://tsusiatsoftware.net/jts/main.html Also check: http://tsusiatsoftware.net/jts/jts-history.html I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. I personally don't. I think we have enough committers who are active on spatial that we can sustain it once we have a good foundation in place (which we are close to) and we also have several contributors who have been active on spatial and are likely good candidates for being committers to work on it. Right now the lucene umbrella is pretty big -- the dev concerns of spatial are, and *should* be, of little concern to the core lucene development. I think spatial dev should live in a place where the spatial concerns are top priority. Just because it uses lucene does not mean it needs to be in the lucene umbrella. Perhaps moving the whole thing to apache-extras is OK, but that is not a visible place to attract participation etc. osgeo seems like a natural home since it will attract more spatial developers. Other then status quo... why should this live in the lucene repo? ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Lucene Spatial Future
Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I did already. Martin didn't respond but someone else did and the outlook is highly unlikely (see Ryan's points about its history). If I get news otherwise then I will respond ASAP but consider it a long shot. Grant, since you too have an interest in spatial, you too could be a developer on lucene-spatial-playground (I look forward to a better name). Just because there are folks interested in spatial involved with Lucene/Solr does not mean that the module needs to actually be in the Lucene/Solr's codebase itself. I think geospatial is both fairly specialized and also only desired by a fraction of Lucene/Solr users, particularly beyond the basic essentials satisfied with a pair of trie doubles (LatLonPoint) and a distance functionquery. If you consider that, then why would it be in Lucene/Solr? ~ David From: Grant Ingersoll [gsing...@apache.org] Sent: Tuesday, April 05, 2011 9:34 AM To: dev@lucene.apache.org Subject: Re: Lucene Spatial Future On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module What about C: Spatial in Lucene/Solr w/ JTS component in Apache Extras as a drop in jar that implements the appropriate bindings? I personally have an interest in good spatial in L/S including more than just point search. This solution need not require any automated downloads/compiles, etc. and it can be noted that the support of that particular jar is handled on Apache Extras (or Githup or wherever) Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. I personally don't. I think we have enough committers who are active on spatial that we can sustain it once we have a good foundation in place (which we are close to) and we also have several contributors who have been active on spatial and are likely good candidates for being committers to work on it. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
Forgive my ignorance, but: are there any technical reasons for spatial work to be in core? Or can all the spatial algos be safely (ie, won't lose much performance, if any) built on top of Lucene's (Solr's) APIs? Do we need to open up new APIs (in which case being part of one code base, released together, is a strong advantage)? For example, incremental field updates is a long needed and often requested feature in Lucene, but this really couldn't be easily/efficiently build on top -- it requires access to lots of inside details that are tracked during indexing. Mike http://blog.mikemccandless.com On Tue, Apr 5, 2011 at 11:10 AM, Smiley, David W. dsmi...@mitre.org wrote: Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I did already. Martin didn't respond but someone else did and the outlook is highly unlikely (see Ryan's points about its history). If I get news otherwise then I will respond ASAP but consider it a long shot. Grant, since you too have an interest in spatial, you too could be a developer on lucene-spatial-playground (I look forward to a better name). Just because there are folks interested in spatial involved with Lucene/Solr does not mean that the module needs to actually be in the Lucene/Solr's codebase itself. I think geospatial is both fairly specialized and also only desired by a fraction of Lucene/Solr users, particularly beyond the basic essentials satisfied with a pair of trie doubles (LatLonPoint) and a distance functionquery. If you consider that, then why would it be in Lucene/Solr? ~ David From: Grant Ingersoll [gsing...@apache.org] Sent: Tuesday, April 05, 2011 9:34 AM To: dev@lucene.apache.org Subject: Re: Lucene Spatial Future On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module What about C: Spatial in Lucene/Solr w/ JTS component in Apache Extras as a drop in jar that implements the appropriate bindings? I personally have an interest in good spatial in L/S including more than just point search. This solution need not require any automated downloads/compiles, etc. and it can be noted that the support of that particular jar is handled on Apache Extras (or Githup or wherever) Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. I personally don't. I think we have enough committers who are active on spatial that we can sustain it once we have a good foundation in place (which we are close to) and we also have several contributors who have been active on spatial and are likely good candidates for being committers to work on it. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Lucene Spatial Future
Mike: Lucene core supports what is required for efficient spatial algorithms to use it, without requiring modification to Lucene. So the answer to your question is no. ~ David From: Michael McCandless [luc...@mikemccandless.com] Sent: Tuesday, April 05, 2011 11:23 AM To: dev@lucene.apache.org Cc: Smiley, David W. Subject: Re: Lucene Spatial Future Forgive my ignorance, but: are there any technical reasons for spatial work to be in core? Or can all the spatial algos be safely (ie, won't lose much performance, if any) built on top of Lucene's (Solr's) APIs? Do we need to open up new APIs (in which case being part of one code base, released together, is a strong advantage)? For example, incremental field updates is a long needed and often requested feature in Lucene, but this really couldn't be easily/efficiently build on top -- it requires access to lots of inside details that are tracked during indexing. Mike http://blog.mikemccandless.com On Tue, Apr 5, 2011 at 11:10 AM, Smiley, David W. dsmi...@mitre.org wrote: Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I did already. Martin didn't respond but someone else did and the outlook is highly unlikely (see Ryan's points about its history). If I get news otherwise then I will respond ASAP but consider it a long shot. Grant, since you too have an interest in spatial, you too could be a developer on lucene-spatial-playground (I look forward to a better name). Just because there are folks interested in spatial involved with Lucene/Solr does not mean that the module needs to actually be in the Lucene/Solr's codebase itself. I think geospatial is both fairly specialized and also only desired by a fraction of Lucene/Solr users, particularly beyond the basic essentials satisfied with a pair of trie doubles (LatLonPoint) and a distance functionquery. If you consider that, then why would it be in Lucene/Solr? ~ David From: Grant Ingersoll [gsing...@apache.org] Sent: Tuesday, April 05, 2011 9:34 AM To: dev@lucene.apache.org Subject: Re: Lucene Spatial Future On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module What about C: Spatial in Lucene/Solr w/ JTS component in Apache Extras as a drop in jar that implements the appropriate bindings? I personally have an interest in good spatial in L/S including more than just point search. This solution need not require any automated downloads/compiles, etc. and it can be noted that the support of that particular jar is handled on Apache Extras (or Githup or wherever) Alternatively, has anyone simply asked JTS if they would dual license or something like that? I'm happy to do that, but let's coordinate so that we aren't all bombarding the guy. I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. I personally don't. I think we have enough committers who are active on spatial that we can sustain it once we have a good foundation in place (which we are close to) and we also have several contributors who have been active on spatial and are likely good candidates for being committers to work on it. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Tue, Apr 5, 2011 at 11:23 AM, Michael McCandless luc...@mikemccandless.com wrote: Forgive my ignorance, but: are there any technical reasons for spatial work to be in core? There is no reason it needs to be in core. As is, it is a maven build that depends on -SNAPSHOT and everything works great. Or can all the spatial algos be safely (ie, won't lose much performance, if any) built on top of Lucene's (Solr's) APIs? Do we need to open up new APIs (in which case being part of one code base, released together, is a strong advantage)? Not that i can think of -- it is right now using features that will get better when new things are added to lucene (for example the docvalues branch) but i think that is true of any project built on top of lucene. My push to have a Numeric field that both solr and lucene use was motivated by this project. Again, not a unique problem to spatial. Also, i recently changed some package protected constants in solr to protected -- so that an external library could make complex solr FieldTypes. For example, incremental field updates is a long needed and often requested feature in Lucene, but this really couldn't be easily/efficiently build on top -- it requires access to lots of inside details that are tracked during indexing. Nothing like that in spatial contrib. The closest I can think of (and this is a long shot) I can see writing a spatial codec that writes values to an rtree. But even this should be able to work within published (or SNAPSHOT) apis. More importantly, the people who would work on the core lucene problems vs the people who would work on the majority of spatial problems are pretty different. It would be nice if spatial contributions could not possibly break the lucene build. ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
Grant, since you too have an interest in spatial, you too could be a developer on lucene-spatial-playground (I look forward to a better name). Just because there are folks interested in spatial involved with Lucene/Solr does not mean that the module needs to actually be in the Lucene/Solr's codebase itself. I think geospatial is both fairly specialized and also only desired by a fraction of Lucene/Solr users, particularly beyond the basic essentials satisfied with a pair of trie doubles (LatLonPoint) and a distance functionquery. If you consider that, then why would it be in Lucene/Solr? my aspirations may be high, but I think spatial+lucene/solr can be a platform to rival PostGIS -- or at the very least good option when you need good text+spatial search. (PostGIS and oracle spatial are the only options now) Some key features that would get lots of attention from geo devlopers are: * native CS-W support from solr. This is super important for GIS catalogs. Actually now required by law in the EU! * native GeoRSS and opensearch geo from solr * lucene/solr as a geotools data source -- this opens up a huge range of off-the-shelf processing The people who care about these problems and the build system and dependencies to support them are pretty distinct from lucene core. ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: If we do elect for option A, I would also suggest we delete the spatial contrib (in 4.0) and have solr depend on the external .jar -- this way lucene users would have what they need directly with the external .jar, and solr users would get lots of fancy new stuff off-the-shelf. At the end of the day, Solr only uses a few methods in the Lucene contrib: the geohash stuff and a few other distance utils (some static methods that I moved from Solr down to Lucene). That's about it. Everything else is just FieldTypes and function queries (the tier stuff is broken b/c of the sinusoidal projection impl. and I didn't see a need for the geometry stuff). If we move function queries to modules, those utils could just be hidden in there (or they could just be put in Solr for that matter if function queries stay where they are) and there would be no need for any external dependency. Then, there is no spatial package at all and anyone who wishes to work on Spatial can go do it in the proposed external project if they need anything beyond point search. That being said, a separate package/project is what LocalLucene/LocalSolr was. You are of course free to do as you wish, but to me things that are first order supported by the community seem to stick better than those that don't and have the benefit of our extensive testing framework as well as resources, especially something as popular as spatial. I think, though, at the end of the day, you are just going to see, once again, a forking as people will likely be confused about where to contribute. Some will continue to contribute to Lucene/Solr b/c that is what they are working on (b/c they started w/ point search) and others will contribute to the external project. Nothing wrong with that, of course, people can do as they wish, it's just why I would prefer a single solution as part of our modules. If SIS or something else w/ a better license was as mature as JTS, we probably wouldn't even be having the discussion. On a different level, for me personally, and I really stress this just my personal choice, I don't have much interest in contributing to non-ASF (or similar foundation based open source projects) because they don't offer the same legal framework, branding, resources and other opportunities that the ASF does. -Grant - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
Let's just do it in Solr directly. I like the jar idea for JTS. We just need more committers to contribute and support the people doing the work. Bill On Tue, Apr 5, 2011 at 2:46 PM, Grant Ingersoll gsing...@apache.org wrote: On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: If we do elect for option A, I would also suggest we delete the spatial contrib (in 4.0) and have solr depend on the external .jar -- this way lucene users would have what they need directly with the external .jar, and solr users would get lots of fancy new stuff off-the-shelf. At the end of the day, Solr only uses a few methods in the Lucene contrib: the geohash stuff and a few other distance utils (some static methods that I moved from Solr down to Lucene). That's about it. Everything else is just FieldTypes and function queries (the tier stuff is broken b/c of the sinusoidal projection impl. and I didn't see a need for the geometry stuff). If we move function queries to modules, those utils could just be hidden in there (or they could just be put in Solr for that matter if function queries stay where they are) and there would be no need for any external dependency. Then, there is no spatial package at all and anyone who wishes to work on Spatial can go do it in the proposed external project if they need anything beyond point search. That being said, a separate package/project is what LocalLucene/LocalSolr was. You are of course free to do as you wish, but to me things that are first order supported by the community seem to stick better than those that don't and have the benefit of our extensive testing framework as well as resources, especially something as popular as spatial. I think, though, at the end of the day, you are just going to see, once again, a forking as people will likely be confused about where to contribute. Some will continue to contribute to Lucene/Solr b/c that is what they are working on (b/c they started w/ point search) and others will contribute to the external project. Nothing wrong with that, of course, people can do as they wish, it's just why I would prefer a single solution as part of our modules. If SIS or something else w/ a better license was as mature as JTS, we probably wouldn't even be having the discussion. On a different level, for me personally, and I really stress this just my personal choice, I don't have much interest in contributing to non-ASF (or similar foundation based open source projects) because they don't offer the same legal framework, branding, resources and other opportunities that the ASF does. -Grant - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
On Tue, Apr 5, 2011 at 2:46 PM, Grant Ingersoll gsing...@apache.org wrote: On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: If we do elect for option A, I would also suggest we delete the spatial contrib (in 4.0) and have solr depend on the external .jar -- this way lucene users would have what they need directly with the external .jar, and solr users would get lots of fancy new stuff off-the-shelf. At the end of the day, Solr only uses a few methods in the Lucene contrib: the geohash stuff and a few other distance utils (some static methods that I moved from Solr down to Lucene). That's about it. Everything else is just FieldTypes and function queries (the tier stuff is broken b/c of the sinusoidal projection impl. and I didn't see a need for the geometry stuff). If we move function queries to modules, those utils could just be hidden in there (or they could just be put in Solr for that matter if function queries stay where they are) and there would be no need for any external dependency. Then, there is no spatial package at all and anyone who wishes to work on Spatial can go do it in the proposed external project if they need anything beyond point search. The spatial API in google code takes a pretty different approach to spatial search in general. It is organized into three key packages: 1. core stuff, no lucene dependencies. Most of the math is here 2. lucene stuff 3. solr interface -- configure and manage the lucene implementations It does not rely on function queries (it will use ValueSorceQuery when necessary). The solr integration is handled via FieldType.getFieldQuery() This way we can send complex queries in the regular 'q' param or an 'fq' (or even a function query) -- syntax looks like: q=geo:Operation( shape ) Operation is one of: BBoxIntersects BBoxWithin Contains Intersects IsEqualTo IsDisjointTo Overlaps SimilarTo And the same is either a point, bbox, point+radius, or complex polygon q=geofield:Intersects( -10 20 -10 20 ) q=geofield:IsWithin( PointDistance( x y distance ) q=geofield:Intersects( POLYGON ((30 10, ...)) ) Different fields can be indexed with different strategies, but the interface stays the same Nothing wrong with that, of course, people can do as they wish, it's just why I would prefer a single solution as part of our modules. Hypothetically, if a stable Apache licensed spatial module were released by a reputable 3rd party foundation (osgeo.org) are you against including it in solr as a .jar? It could get tested directly in solr framework the same we do with carrot2 and tika. Assuming the legal stuff is up-to-snuff and the build is solid, I'm confused why spatial support for lucene needs to be in lucene modules? If SIS or something else w/ a better license was as mature as JTS, we probably wouldn't even be having the discussion. There is more to it then that -- the JTS license issue just slams the issue to the forefront so we can not ignore it. For me the bigger issue is that the development choices for a serious spatial module are not the same as for core lucene development. Also the people are potentially quite different (with overlap of course) On a different level, for me personally, and I really stress this just my personal choice, I don't have much interest in contributing to non-ASF (or similar foundation based open source projects) because they don't offer the same legal framework, branding, resources and other opportunities that the ASF does. This is why I suggest osgeo.org -- they are a foundation similar to ASF. They are the home to most of the key opensource geographic projects, including: PostGIS, GDAL, Geotools, Openlayers, and MapServer. Unlike the ASF, there is not a single license model across all projects, but they make sure work is licensed properly and have most of the same procedures as ASF (incubation, infrastructure, mentors, etc) I could suggest a new ASF project, but there seems like too much overlap with SIS and very different philosophy on 3rd party libraries. In the end, osgeo.com seems like a more natural home and has better branding for spatial related work anyway. ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
Hi, On Wed, Apr 6, 2011 at 6:46 AM, Grant Ingersoll gsing...@apache.org wrote: On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: If we do elect for option A, I would also suggest we delete the spatial contrib (in 4.0) and have solr depend on the external .jar -- this way lucene users would have what they need directly with the external .jar, and solr users would get lots of fancy new stuff off-the-shelf. At the end of the day, Solr only uses a few methods in the Lucene contrib: the geohash stuff and a few other distance utils (some static methods that I moved from Solr down to Lucene). That's about it. Everything else is just FieldTypes and function queries (the tier stuff is broken b/c of the sinusoidal projection impl. and I didn't see a need for the geometry stuff). If we move function queries to modules, those utils could just be hidden in there (or they could just be put in Solr for that matter if function queries stay where they are) and there would be no need for any external dependency. Then, there is no spatial package at all and anyone who wishes to work on Spatial can go do it in the proposed external project if they need anything beyond point search. That being said, a separate package/project is what LocalLucene/LocalSolr was. You are of course free to do as you wish, but to me things that are first order supported by the community seem to stick better than those that don't and have the benefit of our extensive testing framework as well as resources, especially something as popular as spatial. I think, though, at the end of the day, you are just going to see, once again, a forking as people will likely be confused about where to contribute. Some will continue to contribute to Lucene/Solr b/c that is what they are working on (b/c they started w/ point search) and others will contribute to the external project. Nothing wrong with that, of course, people can do as they wish, it's just why I would prefer a single solution as part of our modules. If SIS or something else w/ a better license was as mature as JTS, we probably wouldn't even be having the discussion. On a different level, for me personally, and I really stress this just my personal choice, I don't have much interest in contributing to non-ASF (or similar foundation based open source projects) because they don't offer the same legal framework, branding, resources and other opportunities that the ASF does. After some consideration, I agree with a lot of what you say here. Although it should be pointed out that bringing LocalLucene into Lucene did not help its functionality. Instead we are faced with deprecating/deleting it and starting from scratch basically. But that is also something we have to prevent from happing again, wherever the code is. Given that JTS, like it or not, is at the core of 'beyond point' spatial search at the moment, what are our options? You mentioned having the JTS components in Apache-extras, isn't that sort of splitting things up too? I am pretty keen to see this work stay within the ASF, but I'm struggling to see how we can do that at the moment. -Grant - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org -- Chris Male | Software Developer | JTeam BV.| www.jteam.nl
Re: Lucene Spatial Future
On Apr 3, 2011, at 3:50 PM, Ryan McKinley wrote: In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module It took some convincing by Ryan but I've come around to A -- a separate project that can be included just as say Carrot2 or Tika is today. Lucene/Solr can stick with the basics of geospatial (index points, filter point-distance or box, distance sort) that satisfy the majority of user needs. Anything else can be tossed, allowing Lucene/Solr to be more focused on its core capabilities. For more performant and rich geospatial features, users can get this proposed external project that is designed to easily work with Lucene/Solr. It would be good for this project to be featured prominently somehow; the wiki pages should be sufficient. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Lucene Spatial Future
Hello- I think it is worth discussing what we want to do with the lucene spatial contrib. If you have followed the spatial development, it started with a large contribution and has never had much love or attention. Grant did some great work to get point search working in solr, but much of the lucene spatial contrib does not work as expected, and it is not clear anyone intends to improve/fix it. (geometry and tier). I feel partly responsible for pushing the creation of this contrib, but never really using it -- the problems it solves do not apply to the work I have so I have not been able to give it much attention. Recently I have been working on what I hope a high level lucene spatial API could look like. This work has taken place at: https://lucene-spatial-playground.googlecode.com/svn/trunk/ The key idea is that there are many 'strategies' for spatial indexing -- you may work with simple xy coordinates, or you may need arbitrary geometry in a crazy projection. I hope a common API can be used for variety of approaches. The core api works directly with lucene and has nice bindings to solr. The other key concern is a good testing framework that works across all strategies. When I started this 'playground', I hoped to push for a spatial module within the Apache repo. I'm no longer sure that is the best path forward, and want to get other peoples opinion. My two concerns about a spatial module within lucene are community and 3rd party spatial tools 1. Community -- for the most part, developers involved in lucene are concerned with text search; spatial search is a nice-to-have feature, but not something that gets serious attention. I believe (perhaps naively) that with some clever indexing, lucene/solr could be a serious alternative to PostGIS. Our development environment should attract contribution from spatial folks. 2. 3rd party tools -- In general people working on complex geographic problems use JTS and other LGPL tools. There is some great work happening at Apache SIS now, but it is a long way from being a viable ASL alternative. Within ASF, it is legally ok to have build dependencies on LGPL. The Lucene contrib bdb even includes a dependency on a GPL(ish)! However, these dependencies are not recommended and only happen if the community (and PMC) think the tradeoff is worthwhile. Given the primary concerns of the lucene community, I totally understand why a build dependency on LGPL may not be acceptable. As designed the proposed spatial API does not require JTS. All geometry has a 'simple' implementation what works for points and boxes. I even refactored the JTS dependencies to different packages that could be hosted elsewhere. This works, but it is far from ideal because it makes testing the JTS implementations much more difficult -- given that the community of developers working on the code is likely to use the complex implementations, this separation is unacceptable (for me anyway). If I am going to spend the time to make good tests, i want to make sure the classes I use are a 1st class citizens. Likewise, if the real work goes into testing the external packages, it stinks if it does not apply to the simple implementations / base framework. I don't think the spatial developer community should be split across two code bases and build systems. In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. If we do elect for option A, I would also suggest we delete the spatial contrib (in 4.0) and have solr depend on the external .jar -- this way lucene users would have what they need directly with the external .jar, and solr users would get lots of fancy new stuff off-the-shelf. Thoughts? Ideas? Concerns? thanks ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene Spatial Future
Hi Ryan, On Apr 3, 2011, at 12:50 PM, Ryan McKinley wrote: 2. 3rd party tools -- In general people working on complex geographic problems use JTS and other LGPL tools. There is some great work happening at Apache SIS now, but it is a long way from being a viable ASL alternative. Thanks for actually recognizing the work going on in Apache SIS. So far, we've got a 0.1-incubating release that includes: 1. a working QuadTree implementation (which I've heard you say you think would be a good idea for Lucene/Solr). We think it's a good idea in general. 2. a method of loading/populating the QuadTree with GeoRSS. 3. a web service (REST-ful, but could use improvement) to query the QuadTree using a bounding box or point/radius and get back XML. 4. a demo (part of the web service WAR) that uses GMaps to show it off. There are a number of interesting ideas folks have had in the SIS community (including OGC-like layer services that connect to OODT, HBase, etc.) and implementation platforms for services (JAX-RS from CXF, etc) that I think will help make the project awesome. But, like all open source, there is tons of work to do. I can say is that we'd welcome you (and anyone else interested) working on geospatial stuff at Apache in SIS, if you're interested. Patches welcome. Contributions welcome. Collaborations welcome. From the original SIS proposal, accepted into the Incubator in February 2010 [1]: We propose to construct Apache SIS, an ASL 2.0 licensed toolkit that spatial information system builders or users can leverage to support the aforementioned activities, alleviating much of the software and potentially legal difficulties in implementing SIS/GIS systems. This project will look to expand on those concepts and serve as a place to store reference implementations of spatial algorithms, utilities, services, etc. as well as serve as a sandbox to explore new ideas. Further, the goal is to have Apache SIS grow into a thriving Apache top-level community, where a host of SIS/GIS related software (OGC datastores, REST-ful interfaces, data standards, etc.) can grow from and thrive under the Apache umbrella. One thing to note: we're not trying to build our system so that it depends on any underlying storage/search/indexing/etc platform (e.g., Lucene or Solr). We're trying to build it so that that part is swappable out. That doesn't preclude us from being interested in having such a plugin-though. Cheers, Chris [1] http://wiki.apache.org/incubator/SpatialProposal Within ASF, it is legally ok to have build dependencies on LGPL. The Lucene contrib bdb even includes a dependency on a GPL(ish)! However, these dependencies are not recommended and only happen if the community (and PMC) think the tradeoff is worthwhile. Given the primary concerns of the lucene community, I totally understand why a build dependency on LGPL may not be acceptable. As designed the proposed spatial API does not require JTS. All geometry has a 'simple' implementation what works for points and boxes. I even refactored the JTS dependencies to different packages that could be hosted elsewhere. This works, but it is far from ideal because it makes testing the JTS implementations much more difficult -- given that the community of developers working on the code is likely to use the complex implementations, this separation is unacceptable (for me anyway). If I am going to spend the time to make good tests, i want to make sure the classes I use are a 1st class citizens. Likewise, if the real work goes into testing the external packages, it stinks if it does not apply to the simple implementations / base framework. I don't think the spatial developer community should be split across two code bases and build systems. In the days of sub-projects, I would have proposed that option, but now I see two options: A. Work on spatial lucene outside of apache -- perhaps osgeo or even just github. (would need a different name) B. Allow JTS compile-time dependency in lucene, and move spatial contrib to a real module I think option A is better long term, but I feel like the kid saying if I can't have my way I'll take my ball (code) and go home -- i don't want this to sound like an ultimatum, but an honest discussion about what has the best chance of fostering a thriving development community. If we do elect for option A, I would also suggest we delete the spatial contrib (in 4.0) and have solr depend on the external .jar -- this way lucene users would have what they need directly with the external .jar, and solr users would get lots of fancy new stuff off-the-shelf. Thoughts? Ideas? Concerns? thanks ryan - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org ++
