Re: [Geoserver-devel] upgrade postgresql JDBC driver to 42.1.1 (full binary xfer)

2017-05-30 Thread Dave Blasby 
Hi, Andrea,

if you aren't using prepared statements, it doesn't know that the data is
binary and it will "kick back" to the standard encoding (with a lot of
"\001"-like in the text).  It will work, but you are at approximately 300%
overhead.

As mentioned, above, I believe it works for PostgreSQL 8.2+ (December 2006).

I did take a look at TWKB, but for GeoGig purposes, the loss of precision
is a problem.  However, I am planning to look into it - it makes huge space
savings when you are looking at already-reduced-precision geometries.

I was also going to look at TWKB-like lossless encoding
(ie. Double#doubleToLongBits() and then using a TWKB-like int-diff
encoding).  However, it's likely better to just LZ4 the WKB.  When I do the
testing, I'll let you know what happens.

Either way, encoding TWKB as real "binary" (wrt Base64) will be another big
win.

Thanks,
Dave

On Tue, May 30, 2017 at 10:43 AM, Andrea Aime 
wrote:

> On Tue, May 30, 2017 at 7:01 PM, Dave Blasby 
> wrote:
>
>> Hi, Andrea,
>>
>> Only updating the driver version.
>>
>> I can work with the PostGIS maintainer to get it in - it's fairly easy
>> (just need to ensure that you are using prepared statements).
>>
>
> Maintainer eh? It used to be me and Justin.
> When you say "ensure that you are using prepared statements" is that only
> to get the actual binary transfer, or it would just
> break?
> And does that require some specific version of postgresql?
>
> The 25% less transfer is interesting, wondering if you have been looking
> at TWKB, that should provide greater savings (with
> some potential loss of precision).
>
> Cheers
> Andrea
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313 <+39%200584%20962313>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
> mob: +39  339 8844549 <+39%20339%20884%204549>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> ---
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Re: [Geoserver-devel] upgrade postgresql JDBC driver to 42.1.1 (full binary xfer)

2017-05-30 Thread Andrea Aime 
On Tue, May 30, 2017 at 7:01 PM, Dave Blasby 
wrote:

> Hi, Andrea,
>
> Only updating the driver version.
>
> I can work with the PostGIS maintainer to get it in - it's fairly easy
> (just need to ensure that you are using prepared statements).
>

Maintainer eh? It used to be me and Justin.
When you say "ensure that you are using prepared statements" is that only
to get the actual binary transfer, or it would just
break?
And does that require some specific version of postgresql?

The 25% less transfer is interesting, wondering if you have been looking at
TWKB, that should provide greater savings (with
some potential loss of precision).

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Re: [Geoserver-devel] upgrade postgresql JDBC driver to 42.1.1 (full binary xfer)

2017-05-30 Thread Dave Blasby 
Hi, Andrea,

Only updating the driver version.

I can work with the PostGIS maintainer to get it in - it's fairly easy
(just need to ensure that you are using prepared statements).  The
difficulty is in verifying it's working - you don't really have any access
to what's happening in the driver.  For GeoGig, I verified using wireshark
(you can also verify by stepping into the driver).

Thanks,
Dave

On Tue, May 30, 2017 at 1:03 AM, Andrea Aime 
wrote:

> So, are you proposing just an upgrade of the driver, or are you also going
> to change the postgis store code to leverage the
> binary transfer?
>
> Cheers
> Andrea
>
> On Mon, May 29, 2017 at 11:22 PM, Dave Blasby 
> wrote:
>
>> Hi,
>>
>> For GeoGig, we noticed that the latest Postgresql JDBC driver (42.1.1)
>> now allows full-binary data transfer.  This is better than 25% less
>> bandwidth than Base64 encoding!
>>
>> I'd like to update the pom.xml  from 9.4.1211 to 42.1.1.
>>
>> NOTE: PostGIS could be upgraded to use this as its a big win for a remote
>> (networked) database.
>> Note: you must use a prepared statement!
>>
>>
>> https://jdbc.postgresql.org/download.html
>>
>> From the pom.xml;
>> 9.4.1211
>>
>> Comments?
>>
>> Thanks,
>> Dave
>>
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Geoserver-devel mailing list
>> Geoserver-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>
>>
>
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313 <+39%200584%20962313>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
> mob: +39  339 8844549 <+39%20339%20884%204549>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> ---
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Re: [Geoserver-devel] upgrade postgresql JDBC driver to 42.1.1 (full binary xfer)

2017-05-30 Thread Andrea Aime 
So, are you proposing just an upgrade of the driver, or are you also going
to change the postgis store code to leverage the
binary transfer?

Cheers
Andrea

On Mon, May 29, 2017 at 11:22 PM, Dave Blasby 
wrote:

> Hi,
>
> For GeoGig, we noticed that the latest Postgresql JDBC driver (42.1.1) now
> allows full-binary data transfer.  This is better than 25% less bandwidth
> than Base64 encoding!
>
> I'd like to update the pom.xml  from 9.4.1211 to 42.1.1.
>
> NOTE: PostGIS could be upgraded to use this as its a big win for a remote
> (networked) database.
> Note: you must use a prepared statement!
>
>
> https://jdbc.postgresql.org/download.html
>
> From the pom.xml;
> 9.4.1211
>
> Comments?
>
> Thanks,
> Dave
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>
>


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel