Hi Ville, I'm a developer on GeoWave and thought I'd chime in on the topic of scale a bit. Andrea makes a great point - it's true the setup and overhead involved in leveraging key-value stores like HBase, Accumulo, and Cassandra may be more involved than necessary for anything below a certain scale. When dealing with IoT data sources with extreme volume and velocity it is very nice to leverage a standard "big data" ecosystem and is worth the investment - but it is not for everyone.
GeoWave was born from these "big data" use cases, but I thought I'd provide an update that GeoWave has recently expanded its support for key-value stores to include some very lightweight options (such as RocksDB and Redis). We maintain common APIs, features, and integrations (such as GeoServer of course) regardless of the key-value store. The idea is to just make it a matter of switching connection parameters to leverage a different backend store. Here's an article <https://www.eclipse.org/community/eclipse_newsletter/2018/december/geowave.php> I recently wrote that touches on some of the advantages we've found in leveraging GeoWave with the right key-value store for the right job/scale. Anyways, I definitely don't want to take away from the tremendous flexibility that SQL and traditional databases provide, but thought I'd interject with an option - particularly if you think you ultimately will need to scale to a "big data" ecosystem, but would rather defer worrying about that additional complexity at this stage. The buy in or cost can be very low using something like RocksDB in a purely local embedded datastore (and there are cloud options for RocksDB as well). Also, Redis as a service is extremely easy to setup. Then you could do what we've been doing to stick with a common API from GeoWave, and take the hit to setup a distributed architecture as it makes sense. Rich On Thu, Jan 24, 2019 at 2:38 AM Andrea Aime <andrea.a...@geo-solutions.it> wrote: > On Thu, Jan 24, 2019 at 7:49 AM Ville Koivisto < > ville.koivi...@kuntotekniikka.fi> wrote: > >> *Hi Andrea, * >> >> >> >> Your comment is appreciated. You also hit the nail on the head with your >> guess: we’re dealing with sensors attached to machinery along municipal >> road network. At first we’ll have, let’s say thousands of updates per >> minute. Postgis is already our current solution, we’ve just not yet >> configured it with Geoserver to handle near real time data (disabling >> caching for layers “real time layers” and so on, I guess). >> >> >> >> Still, we might later on incorporate larger data sources, and hit that >> very big data. >> > > That still depends on how big is big. Real time data tends to have a high > focus on current data and little attention to past data, if any. > That can still be addressed with a normal database using separate tables > for current and archived, or leveraging table partitioning, and > eventually using actual database server clustering. Eventually that breaks > too and you actually need to go fully distributed. > > Just saying, this is not a black or white situation, there are multiple > designs that can be used, you go for the simplest, it won't scale to very > large datasets, > you go for the big data one, and you'll likely find it has a lot of > overhead for the small datasets, both in terms of setup, maintenance and > cost. > > Cheers > Andrea > > == > > GeoServer Professional Services from the experts! Visit > http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf > Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa > (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 > http://www.geo-solutions.it http://twitter.com/geosolutions_it > ------------------------------------------------------- *Con riferimento > alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - > Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni > circostanza inerente alla presente email (il suo contenuto, gli eventuali > allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i > destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per > errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le > sarei comunque grato se potesse darmene notizia. This email is intended > only for the person or entity to which it is addressed and may contain > information that is privileged, confidential or otherwise protected from > disclosure. We remind that - as provided by European Regulation 2016/679 > “GDPR” - copying, dissemination or use of this e-mail or the information > herein by anyone other than the intended recipient is prohibited. If you > have received this email by mistake, please notify us immediately by > telephone or e-mail.* > _______________________________________________ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users >
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
