Re: [HACKERS] String encoding during connection "handshake"
On Wednesday 28 November 2007, Alvaro Herrera wrote: > [EMAIL PROTECTED] escribió: > > Martijn, > > > > :) don't take it personal, I am just trying to obtain confirmation that I > > > > understood well the problem. Afterall, it's just that C has a very > > outdated notion of "char"s (and no notion of Unicode). I was naively > > under the impression that "char"s have evolved in nowadays C. > > This is not the language's fault in any way. We support plenty of > encodings beyond UTF-8. Yes, you support (and worry about) encodings simply because of a C limitation dating from 1974, if I recall correctly... In Java, for example, a "char" is a very well defined datum, namely a Unicode point. While in C it can be some char or another (or an error!) depending on what encoding was used. The only definition that stands up is that a "char" is a byte. Its interpretation is unsure and unsafe (see my original problem). On Wednesday 28 November 2007, Martijn van Oosterhout wrote: > On Wed, Nov 28, 2007 at 05:54:05PM +0200, [EMAIL PROTECTED] wrote: > > Regarding the problem of "One True Encoding", the answer seems obvious to > > me: use only one encoding per database cluster, either UTF-8 or UTF-16 or > > another Unicode-aware scheme, whichever yields a statistically smaller > > database for the languages employed by the users in their data. This > > encoding should be a one time choice! De facto, this is already happening > > now, because one cannot change collation rules after a cluster has been > > created. > > Umm, each database in a cluster can have a different encoding, so there > is no such thing as the "cluster's encoding". I implied that a cluster should have a single encoding that covers the whole Unicode set. That would certainly satisfy everybody. > You can certainly argue > that it should be a one time choice, but I doubt you'll get people to > remove the possibilites we have now. If fact, if anything we'd probably > go the otherway, allow you to select the collation on a per > database/table/column level (SQL complaince requires this). The collation order is implemented in close relationship with the byte representation of strings, but conceptually depends on the locale solely and has nothing to do with the encoding. > This has nothing to do with C by the way. C has many features that > allow you to work with different encodings. It just doesn't force you > to use any particular one. Yes, my point exactly! C forces you to worry about encoding. I mean, if you're not an ASCII-only user ;) Think of it this way: if I give you a Java String you will perfectly know what I meant; if I send you a C char* you don't know what it is in the absence of extra information - you can even use it as a uint8*, as it is actually done in md5.c. I consider this matter closed from my point of view and I have modified the JDBC driver according to my needs. Thank you all for the help. ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] String encoding during connection "handshake"
On Wednesday 28 November 2007, Trevor Talbot wrote: > I'm not entirely sure how that's supposed to solve the client > authentication issue though. Demanding that clients present auth data > in UTF-8 is no different than demanding they present it in the > encoding it was entered in originally... Oh no, it's a big difference: PREDICTABILITY! Why must I guess the encoding used by the administrator? What if he's Chinese? Instead, I know the cluster's encoding, just as I know the server name and the TCP port. And the connection handshake carries on without misunderstandings (read wrong encoding). ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] String encoding during connection "handshake"
Martijn, :) don't take it personal, I am just trying to obtain confirmation that I understood well the problem. Afterall, it's just that C has a very outdated notion of "char"s (and no notion of Unicode). I was naively under the impression that "char"s have evolved in nowadays C. Regarding the problem of "One True Encoding", the answer seems obvious to me: use only one encoding per database cluster, either UTF-8 or UTF-16 or another Unicode-aware scheme, whichever yields a statistically smaller database for the languages employed by the users in their data. This encoding should be a one time choice! De facto, this is already happening now, because one cannot change collation rules after a cluster has been created. During the handshake, all clients should be assumed to serve data in the cluster's encoding. Have a nice day, too. On Wednesday 28 November 2007, Martijn van Oosterhout wrote: > On Wed, Nov 28, 2007 at 11:39:33AM +0200, [EMAIL PROTECTED] wrote: > > During the authentication phase, no such conversion takes place - you > > were right and I couldn't believe it! In the case when your database > > name, your user name or password contain non-ASCII characters, you're out > > of luck if the stored values were submitted in another encoding by the > > administrator. > > The problem is, what conversion. You don't know the encoding of the > server yet (because you havn't selected a DB) and you don't know the > encoding to the client. The only real possibility is to declare One > True Encoding and decree every username/password be in that. But you're > never going to get people to agree on that. > > > I assume that no names conversion takes place between client and cluster > > metadata when a role is created (CREATE ROLE... PASSWORD...) or when a > > database is created (CREATE DATABASE...). Or does it? In that case, the > > names are encoded in the encoding of the database that the administrator > > was connected to. > > Honestly, UNIX usernames/passwords have always worked like this so > we're not really doing anything wierd by doing it this way. Users need > to type the password in the same encoding it was added. It not usually > a big deal because people set their own passwords... > > Have a nice day, ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] PG 7.3 is five years old today
On Wednesday 28 November 2007, Gregory Stark wrote: > > This tells me that the v3 protocol appeared at 7.4, so there's no need to > > support v2 in future database versions (starting with 8.3?). It would > > simplify code in interfaces like JDBC too. > > I think the second half of this is correct. There would be no need to > support the old protocol in client interface drivers since the only > supported databases would all support the new protocol. > > Whether there's any need to support the old protocol in the server depends > on whether there are any clients out there which use it which is harder to > determine and not affected by whether Postgres 7.3 is still around. Actually it doesn't make sense to do it halfway (for example, why would you keep the v2 protocol in the database if it is not supported anymore by clients?!). Either you drop v2 support or you don't, if the community is keen on preserving compatibility between "any" client and "any" database. I for one am not keen on that. Just as I would drop support for Java older than 1.4 in the JDBC driver. Anyway, the decision is not mine. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] PG 7.3 is five years old today
I'm not a developper, but it occured to me that you should consider dropping the support for client-server wire protocol v2. I quote a comment I found in JDBC driver's code: // NOTE: To simplify this code, it is assumed that if we are // using the V3 protocol, then the database is at least 7.4. That // eliminates the need to check database versions and maintain // backward-compatible code here. // // Change by Chris Smith <[EMAIL PROTECTED]> This tells me that the v3 protocol appeared at 7.4, so there's no need to support v2 in future database versions (starting with 8.3?). It would simplify code in interfaces like JDBC too. > > I see that there are two or three minor bug fixes in the REL7_3_STABLE > > branch since 7.3.20. Rather than just leaving those to rot, maybe the > > actual policy should be "only one more update after 8.3 comes out". > > > > Comments, opinions? > > Release 7.3.21 with and EOL addendum :). E.g; this is the last release > of 7.3 and 7.3 is now considered unsupported. ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] String encoding during connection "handshake"
Ok, that's bad. I've also read crypt.c and md5.c. And what a nightmare is C compared to Java (granted, there's a difference in age of more than 20 years). My guess is that since the "char" type is one byte long, all "char *" expressions are actually pointers to array of bytes which are transmitted through the wire and/or stored in the database. When these strings arrive from or leave for a client that has another declared encoding than the database, the string of bytes is replaced with an equivalent one from a Unicode perspective. Am I right? During the authentication phase, no such conversion takes place - you were right and I couldn't believe it! In the case when your database name, your user name or password contain non-ASCII characters, you're out of luck if the stored values were submitted in another encoding by the administrator. I assume that no names conversion takes place between client and cluster metadata when a role is created (CREATE ROLE... PASSWORD...) or when a database is created (CREATE DATABASE...). Or does it? In that case, the names are encoded in the encoding of the database that the administrator was connected to. Thank you both. On Tuesday 27 November 2007, Usama Munir wrote: > Martin is actually right. No assumption is made about the encoding of the > password. The password is recieved as a set of bytes over the wire-level > protocol and then processed accordingly as per your pg_hba settings. please > refer to auth.c method recv_password_packet(Port *port). The comment on the > last line of the method might be of your intrest, and i quote > > "Return the received string, Note we do not attempt to do any character set > conversion on it; since we don't know the client's encoding, there woudn't > be much point" > > / Usama ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] String encoding during connection "handshake"
On Tuesday 27 November 2007, Martijn van Oosterhout wrote: > I was under the impression that the username/password, had no encoding, > they are Just a Bunch of Bits, i.e. byte[]. I cannot agree to that, simply because Postgres supports (or at least claims to) multi-byte characters. And user names, passwords and database names are character strings. > Looking at it another way, the encoding is part of the password. The > correctly entered password in the wrong encoding is also wrong, because > the matching is done at the byte level. I'm afraid that is true to some extent, that's why I'm asking in the first place. A user should be able to authenticate as long as he/she is able to write the password, regardless of the OS's locale setting. > This is all AIUI, Thanks fot the input, I'm waiting for others, too. Or point me to the relevant source files. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] String encoding during connection "handshake"
Hi all.
I have read the documentation, searched the mailing lists and inspected the
code JDBC driver code. I do need to address this question to actual
developers.
Simply put, what is the client encoding that the server assumes BEFORE the
client connection is established, that is, during the authentication phase? I
know there's a "client_encoding" setting on the server side that indicates
the encoding used in the communication stream, but its default value is the
database's encoding. Which is not known before the user gets authenticated
and the "logical" connection is actually made.
I'm asking this so that I can fix the JDBC driver that wrongly assumes that
user name, password and database name are made up of ASCII characters only.
This issue has come up before, but no action has been carried out. See
http://archives.postgresql.org/pgsql-jdbc/2007-10/msg00128.php
I also need a vital information regarding the MD5 hash that is computed is
some authentication scenarios. This hash is based on char[] (String) values
that must be converted into byte[] before being handed over to the hash
algorithm. What is the encoding used by the server to make this conversion
during the authentication phase (in order to verify the submitted password)?
I also saw that the JDBC driver sends right away this pair to the server:
{"client_encoding", "UNICODE"}. Does that mean that the client is requesting
the server to interpret the communication stream as encoded in "UTF8"?
Thanks.
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
