This is one of the "big issues" the DBI needs to address soon.
As I'm short on time right now I'm going to just throw open this topic and let you all chip in with ideas...
I'll try to pick up the pieces later and then, after reviewing here, post a summary of proposals to dbi-users.
So, what say you all?
Firstly, we need to identify what bits could potentially be in different character sets?
* row data (of char/varchar/text/... types) * column names * table names * database names
As far as I can see, any or all of the above could potentially be in a non-ascii character set. We need to be able to ask the database what character set each of those is in.
I'd like to ask the question of whether we need to bother with non UTF-8 character sets. The reason is that they're the only ones that have an extra flag inside Perl to make them work properly.
Actually, we may well wish to transform non-UTF-8 data into UTF-8 inside Perl in order to get character semantics on it.
There's probably a lot of speed loss waiting to happen in any discussion like this; we should probably try to ensure that people who continue to use plain ascii are not affected by the changes if possible.
I've added UTF-8 support (kind of) to DBD::Pg, which was a saviour for us. It's very simplistic and not a good model for further development, but filled our needs adequately. It allows the user to set a flag $dbh->{pg_enable_utf8} which makes DBD::Pg check for UTF-8 in returned string data and toggle Perl's UTF-8 flag if that happens to be the case. It doesn't even query the database to find out what the encoding might be. Postgres doesn't support (to my knowledge) character sets at any granularity other than a whole database.
You pointed out before that MySQL 4 can have different character sets per-column, which makes life more interesting; that really demans that the character set comes back as a piece of metadata for each item of data returned by the database... I don't know whether or not MySQL has a way of forcing the results returned by the database into a specific character set?
Sorry, but I have no idea about any other databases and how they handle character sets. It might be a good idea to get an idea of what sort of common denominator of features each database supports...
In summary, my needs are pretty simple: I need to get UTF-8 data out and have that marked as UTF-8 in perl by the driver. I don't want to have to follow each fetchrow_array() with a map { decode_utf8($_) } @results. I don't think that sending data in to the database has the same issues (well, it appears to work without problems for us).
-Dom
-- | Semantico: creators of major online resources | | URL: http://www.semantico.com/ | | Tel: +44 (1273) 722222 | | Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |
