Dear Perl gurus,
This is my first post. I'm using Perl with great joy, and I'd like to
express my
gratitude for all you are doing to keep Perl stable and fun to use.
I'd like to ask to object to re-releasing this version and discuss on
how to
make 4.043 backwards compatible instead.
This change will with 100% certainty corrupt all BLOB data written to
the
database when the developer did not read the release notes before
applying the
latest version of DBD::mysql (and changed its code consequently).
Knowing that sysadmins have the habit of not always reading the
release notes of
each updated package the likelihood that this will happen will
therefore high.
I myself wasn't even shown the release notes as it was a dependency
of an
updated package that I applied.
The exposure of this change is big as DBD::mysql affects multiple
applications
and many user bases.
I believe deliberately introducing industry wide database corruption is
something that will significantly harm peoples confidence in using Perl.
I believe that not providing backwards compatibility is not in line
with the
Perl policy that has been carefully put together by the community to
maintain
the quality of Perl as it is today.
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
I therefore believe the only solution is an upgrade that is by
default backwards
compatible, and where it is the user who decides when to start UTF8
encode the
input values of a SQL request instead.
If it is too time consuming or too difficult it should be considered
to park the
UTF8-encoding "fix" and release a version with the security fix first.
I have the following objections against this release:
1. the upgrade will corrupt more records than it fixes (it does more
harm than good)
2. the reason given for not providing backward compatibility
("because it was
hard to implement") is not plausible given the level of unwanted side
effects.
This especially knowing that there is already a mechanism in place
to signal
if its wants UTF8 encoding or not
(mysql_enable_utf8/mysql_enable_utf8mb4).
3. it costs more resources to coordinate/discuss a "way forward" or
options than
to implement a solution that addresses backwards compatibility
4. it is unreasonable to ask for changing existing source knowing
that depending
modules may not be actively maintained or proprietary
It can be argued that such module should always be maintained but
it does not
change the fact that a good running Perl program becomes unusable
5. it does not inform the user that after upgrading existing code
will start
write corrupt BLOB records
6. it does not inform the user about the fact that a code review of
all existing
code is necessary, and how it needs to be changed and tested
7. it does not give the user the option to decide how the BLOB's
should be
stored/encoded (opt in)
8. it does not provide backwards compatibility
By doing so it does not respect the Perl policy that has been
carefully put
together by the community to maintain the quality of Perl as it is
today.
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
9. it blocks users from using DBD::mysql upgrades as long as they
have not
rewritten their existing code
10. not all users from DBD::mysql can be warned beforehand about the
side
effects as it is not known which private parties have code that use
DBD::mysql
12. I believe development will go faster when support for backwards
compatibility is addressed
13. having to write 1 extra line for each SQL query value is a monks
job that
will make the module less attractive to use
About forking to DBD::mariadb?:
The primary reason to create such a module is when the communication
protocol of
Mariadb has become incompatible with Mysql.
To use this namespace to fix a bug in DBD::mysql does not meet that
criteria and
causes confusion for developers and unnecessary pollution of the DBD
namespace.
---
For people that do not know the impact of the change that is pending
to be
committed:
(see Github issue that includes 3 reports of companies that suffered
data loss
https://github.com/perl5-dbi/DBD-mysql/issues/117 )
Issue: some UTF8 characters are not properly displayed after retrieval
Cause: SQL query values are not UTF8 encoded when sent to the
database but they
are all decoded once retrieved.
Occurence: Only records with string data that can only be written
with UTF8. It
can be considered rare as people haven't reported this issue after 10
years of
usage.
Regional impact: Only affects countries which characters need UTF8
encoding and
only affects string values.
Steps to recover from it: Read string data unencoded and write it
encoded.
Changes of upgrade pending to be re-released:
SQL query values are both UTF8 encoded when sent to the database as
when its
retrieved (including BLOB fields).
BLOB fields will be excluded from encoding only if you specify its
data type.
Side effects from installing upgrade:
- BLOB data will be written after UTF8 encoding and will therefore be
corrupt
- no possibility to detect if a BLOB field is corrupt or not. Only
when known
when the INSERT/UPDATE took place, and when the upgrade was installed
- existing data will still display incorrect
Occurence: every INSERT/UPDATE statement will start writing corrupted
BLOB data
Regional impact: worldwide
Steps to recover from it corrupted BLOBs? You cannot. Your binary
blobs are
encoded as if they were UTF8 strings. Your binary data is
unrecoverable (as in
"gone forever").
If you are a dentist you have to ask your customers to come back to
make another
x-ray as the made photo's are gone.
What is asked from the developer to prevent this from happening?
- do not miss reading the release notes before upgrading
- review all source code (including written by other included
modules) and
specify the data type of each SQL parameter value
before: $dbh->do('INSERT INTO test (BLOB1,BLOB2,BLOB3,BLOB4)
VALUES(?,?,?,?)',undef,$col1,$col2,$col3);
after: $dbh->do('INSERT INTO test (BLOB1,BLOB2,BLOB3,BLOB4)
VALUES(?,?,?,?)');
$sth->bind_param(1, $file, SQL_BLOB);
$sth->bind_param(2, $file, SQL_BLOB);
$sth->bind_param(3, $file, SQL_BLOB);
...
One line more for each SQL statement. This will be a time consuming
monks task
during which the user will ask why this is necessary while it worked
before.
- upgrade scripts need to be written to UTF8 encode existing string data
- retest all source code
