Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
On 11/24/2011 02:36 AM, Kevin Grittner wrote: Oliver Jowett wrote: Can we get a mechanism for minor protocol changes in this future version? Something as simple as exchanging a list of protocol features during the initial handshake (then use only features that are present on both sides) would be enough. The difficulty of making any protocol changes at the moment is a big stumbling block. I've been thinking the same thing. Any new protocol should include a way for each side to publish a list of what it can accept from the other during initial handshaking. (You could probably retrofit that to the current protocol version) Perhaps. It would be great if both sides could recognize the case where the feature negotiation was absent and use a default feature list for the protocol available on the other end. What about a hand-shake protocol based on simple binary-protocol minor version instead of features. We keep the v3 protocol as is but can add cumulative conditionally enabled features when we bump the minor version. The hand shake requires that the server sends a parameter back with it's highest supported minor version: FE= StartupPacket =BE ParameterStatus(binary_minor = 23) And the client can send any number between 1=binary_minor back to enable newer protocol versions and/or limit what the server sends FE= Execute(SET binary_minor = 20) To keep full backwards compatibility: 1) if backend does not send a binary_minor parameter on connection the highest supported minor version is assumed to be 0 (current format) 2) the backend assumes the active minor version is 0 unless the SET binary_minor is received I think bumping a minor version is better than feature flags because: 1) the hand shake is simpler and faster 2) coding is easier as all previous features are known to be supported and active when implementing feature+1 I'm not exactly sure about the COPY BINARY feature Tom mentioned. But probably we could prefix the data with the int4 containing the minor version? -Mikko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
On 25 November 2011 07:54, Mikko Tiihonen mikko.tiiho...@nitorcreations.com wrote: =BE ParameterStatus(binary_minor = 23) FE= Execute(SET binary_minor = 20) Yeah this was almost exactly what I was thinking about how to retrofit it, except it might be clearer to have, say, supported_binary_minor (read-only, advertised by the server on startup) vs. binary_minor (read-write, defaults to 0) as otherwise you have special behavior for just one parameter where the advertised version doesn't actually match the currently-set version. Re list vs. always-incrementing minor version, you could just use an integer and set bits to represent features, which would keep it simple but also let clients be more selective about which features they implement (you could support feature 21 and 23 without supporting 22) Oliver -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
On Tue, Nov 22, 2011 at 6:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Oliver Jowett oli...@opencloud.com writes: On 23 November 2011 10:47, Mikko Tiihonen mikko.tiiho...@nitorcreations.com wrote: Here is a patch that adds a new flag to the protocol that is set when all elements of the array are of same fixed size. How does a client detect that this feature is supported? The only way that anything like this will go in is as part of a protocol version bump, so discoverability would reduce to knowing which protocol you're using. We should file this away as one of the things we might want to do whenever there's sufficient critical mass for a new wire protocol version. Note that COPY BINARY files would be affected too, and so we'd want to make sure that this sort of change is recognizable from a binary file's header. Wire format changes can only be made with a protocol version bump? Is this a new policy? In the past they were just made...for example the money type was bumped to 64 bits. In the past it was always buyer beware for binary format consumers. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
Merlin Moncure mmonc...@gmail.com writes: On Tue, Nov 22, 2011 at 6:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: The only way that anything like this will go in is as part of a protocol version bump, Wire format changes can only be made with a protocol version bump? Is this a new policy? In the past they were just made...for example the money type was bumped to 64 bits. In the past it was always buyer beware for binary format consumers. Well, (a) our standards have gone up over time, (b) binary protocol is getting more widely used (in part due to your own efforts), and (c) money is a third-class stepchild anyway. I don't think we can get away with changing the binary representation of such widely used types as int and float arrays, unless we have some pretty carefully thought through notion of how the client and server will negotiate what to do. Now it's possible we could do that without formally calling it a protocol version change, but I don't care at all for the idea of coming up with one-off hacks every time somebody decides that some feature is important enough that they have to have it Right Now instead of waiting for a sufficient accumulation of reasons to have a protocol flag day. I think but we made arrays a bit smaller! is a pretty lame response to have to give when somebody complains that Postgres 9.2 broke their client software. When we do it, I want to have a *long* list of good reasons. BTW, so far as the actual array format is concerned, I don't think the proposal is acceptable as-is: it renders the received array entirely unreadable unless the reader knows a-priori what the sender thought the typlen was. It would be a lot better if the fixed-length flag meant that the typlen is given once in the array header rather than once per element. I'm not thrilled by the no nulls restriction, either, though I admit I don't have a good idea about avoiding that offhand. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
On 24 November 2011 05:36, Tom Lane t...@sss.pgh.pa.us wrote: Now it's possible we could do that without formally calling it a protocol version change, but I don't care at all for the idea of coming up with one-off hacks every time somebody decides that some feature is important enough that they have to have it Right Now instead of waiting for a sufficient accumulation of reasons to have a protocol flag day. I think but we made arrays a bit smaller! is a pretty lame response to have to give when somebody complains that Postgres 9.2 broke their client software. When we do it, I want to have a *long* list of good reasons. Can we get a mechanism for minor protocol changes in this future version? Something as simple as exchanging a list of protocol features during the initial handshake (then use only features that are present on both sides) would be enough. The difficulty of making any protocol changes at the moment is a big stumbling block. (You could probably retrofit that to the current protocol version) Oliver -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
Oliver Jowett wrote: Can we get a mechanism for minor protocol changes in this future version? Something as simple as exchanging a list of protocol features during the initial handshake (then use only features that are present on both sides) would be enough. The difficulty of making any protocol changes at the moment is a big stumbling block. I've been thinking the same thing. Any new protocol should include a way for each side to publish a list of what it can accept from the other during initial handshaking. (You could probably retrofit that to the current protocol version) Perhaps. It would be great if both sides could recognize the case where the feature negotiation was absent and use a default feature list for the protocol available on the other end. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
On 23 November 2011 10:47, Mikko Tiihonen mikko.tiiho...@nitorcreations.com wrote: Here is a patch that adds a new flag to the protocol that is set when all elements of the array are of same fixed size. When the bit is set the 4 byte length is only sent once and not for each element. Another restriction is that the flag can only be set when there are no NULLs in the array. How does a client detect that this feature is supported? At a glance the JDBC patch doesn't use it on the send path, but presumably clients could use this when sending binary-format arrays to the server - but only if the server understood the format. (Ideally a pair of settings would be useful here - one to say the server understands the new format and another the client sets to say please use the new format that defaults to off - then you could avoid confusing old clients, too) Oliver -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays
Oliver Jowett oli...@opencloud.com writes: On 23 November 2011 10:47, Mikko Tiihonen mikko.tiiho...@nitorcreations.com wrote: Here is a patch that adds a new flag to the protocol that is set when all elements of the array are of same fixed size. How does a client detect that this feature is supported? The only way that anything like this will go in is as part of a protocol version bump, so discoverability would reduce to knowing which protocol you're using. We should file this away as one of the things we might want to do whenever there's sufficient critical mass for a new wire protocol version. Note that COPY BINARY files would be affected too, and so we'd want to make sure that this sort of change is recognizable from a binary file's header. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
