Re: [rt-users] RT saves data in quoted-printable, why???
On Thu, 5 Mar 2015 18:50:57 +0100 Palle Girgensohn
* Ensure that all data containing non-ASCII is quoted-printable
encoded for PostgreSQL, instead of merely all data not claiming to be
text/plain
Why is this? It seems it is doing more harm than good -- it makes it
harder to use indexes and makes it harder to use it from the tables
directly? PostgreSQL is very capable of storing any kind of character
set? We use UTF-8 for everything, this fix breaks a lot of things for
us.
The commit in question is 3a9c38ed, which changes:
} elsif (!$RT::Handle-BinarySafeBLOBs
$MIMEType !~ /text\/plain/gi
!Encode::is_utf8( $Body, 1 ) ) {
$ContentEncoding = 'quoted-printable';
}
...to:
} elsif (!$RT::Handle-BinarySafeBLOBs
$Body =~ /\P{ASCII}/
!Encode::is_utf8( $Body, 1 ) ) {
$ContentEncoding = 'quoted-printable';
}
That is, any data which claimed to be text/plain would blindly be
attempted to be shoved into the database; this includes content which
contained _invalid_ UTF-8 byte sequences. The commit was in RT 4.0; RT
4.2 is much better about transcoding to real UTF-8, or marking the part
as application/octet-stream if it cannot be decoded. In that sense,
this logic is now less necessary. However, the commit was absolutely
necessary at the time to not _lose_ data. Erring on the side of data
preservation, at the expense of possibly-unnecessary encoding, seems
like not an unreasonable choice.
That being said, the Encode::is_utf8() check attempts to verify that we
only quoted-printable encode data whose bytes are not currently a
correctly-encoded UTF-8 byte stream. The bug now lies there, as after
the utf8-reckoning branch (2620658..af9fe7c), the $Body argument is
guaranteed to contain bytes, not characters. Per the Encode::is_utf8()
documentation:
[INTERNAL] Tests whether the UTF8 flag is turned on in the STRING. If
CHECK is true, also checks whether STRING contains well-formed UTF-8.
Returns true if successful, false otherwise.
The UTF8 flag is almost certainly off on $Body (it _may_ be on, and
still contain only codepoints 128, but this is unlikely), so the
well-formed-ness of the string (which is the relevant thing we wish to
confirm) is never checked.
I _believe_ that this code may be replaced by (untested):
} elsif (!$RT::Handle-BinarySafeBLOBs
!eval { Encode::decode( UTF-8, $Body, Encode::FB_CROAK); 1 }
) {
$ContentEncoding = 'quoted-printable';
}
I may push a branch later that makes this change, and see what breaks.
All of that aside, note that it makes it harder to use indexes makes
little sense -- there are no indexes on Content. The full-text index
uses its own tsvector, which it constructs after decoding the Content,
so the transfer encoding of the Content column is irrelevant to that.
- Alex
Re: [rt-users] Upgrade to 4.2.9 Lost Dashboards.
On Thu, 5 Mar 2015 14:52:39 -0500 Alex Vandiver ale...@bestpractical.com wrote: On Wed, 25 Feb 2015 11:52:37 + Sam Maher sam.ma...@lawsonlewisblakers.co.uk wrote: I have moved to a new server and upgraded to the latest version but all my privileged users except one have lost their dashboards. Has anyone else had this problem? Please specify what you mean by latest version. I missed that the subject specified you upgraded to 4.2.9. What version were you upgrading from? - Alex
Re: [rt-users] sphinx weirdness [explanation and possible solution]
On Wed, 25 Feb 2015 13:54:45 +0100 Arkadiusz Miśkiewicz ar...@maven.pl wrote: Ok, mysql is too smart! For sphinx to work mysql needs to first query AttachmentsIndex_3 and then make joins to it. Otherwise sphinx won't work. That's due to sphinx architecture. Here mysql is too smart and differently optimizes query thus breaking sphinx support in rt. Now I've tried FORCE INDEX and such but wasn't able to force mysql to first query AttachmentsIndex_3. There is STRAIGHT_JOIN that forces joins orders, so maybe that is some solution. This limitation is unfortunately documented, and not easily fixable: https://bestpractical.com/docs/rt/latest/full_text_indexing.html#Caveats1 STRAIGHT_JOIN also won't work since mysql still is able to make changes and optimizations to the query. Fortunately code below seems to be working - using UNION and separate, simple sphinx subquery: [snip] That query is incorrect; it unions Attachment ids (from the Sphinx results table) with Ticket ids (from the ti subquery on Tickets). Devs, could you please change querying code, so that sphinx will always get its own subquery? Please try the straight MySQL FTS, included in 4.2.10, instead. It is much faster, and not nearly as fiddly. - Alex
Re: [rt-users] Upgrade to 4.2.9 Lost Dashboards.
On Wed, 25 Feb 2015 11:52:37 + Sam Maher sam.ma...@lawsonlewisblakers.co.uk wrote: I have moved to a new server and upgraded to the latest version but all my privileged users except one have lost their dashboards. Has anyone else had this problem? Please specify what you mean by latest version. Are the dashboards listed in Home → All Dashboards? - Alex
Re: [rt-users] RT saves data in quoted-printable, why???
On Thu, 5 Mar 2015 14:31:56 -0500 Alex Vandiver
ale...@bestpractical.com wrote:
I _believe_ that this code may be replaced by (untested):
} elsif (!$RT::Handle-BinarySafeBLOBs
!eval { Encode::decode( UTF-8, $Body,
Encode::FB_CROAK); 1 } ) {
$ContentEncoding = 'quoted-printable';
}
I may push a branch later that makes this change, and see what breaks.
https://github.com/bestpractical/rt/compare/4.2-trunk...4.2%2Fless-qp
- Alex
Re: [rt-users] RT saves data in quoted-printable, why???
https://issues.bestpractical.com/Ticket/Display.html?id=29735 -- Zito
Re: [rt-users] relative date search
You can enter 14 days ago into the Created box, or Created 14 days ago into the Advanced screen of the search builder. On Fri, 6 Mar 2015 5:47 pm Eierschmalz, Bernhard bernhard.eierschm...@scheppach.com wrote: Hello, is it possible to make date-searches in RT with relative date? For example search all open tickets where creation-date is older than 14 days. best regards Bernhard
[rt-users] relative date search
Hello, is it possible to make date-searches in RT with relative date? For example search all open tickets where creation-date is older than 14 days. best regards Bernhard
Re: [rt-users] relative date search
Here's all of the acceptable syntax: https://metacpan.org/pod/Time::ParseDate#Relative-date-formats On Fri, 6 Mar 2015 5:48 pm Alex Peters a...@peters.net wrote: You can enter 14 days ago into the Created box, or Created 14 days ago into the Advanced screen of the search builder. On Fri, 6 Mar 2015 5:47 pm Eierschmalz, Bernhard bernhard.eierschm...@scheppach.com wrote: Hello, is it possible to make date-searches in RT with relative date? For example search all open tickets where creation-date is older than 14 days. best regards Bernhard
Re: [rt-users] RT saves data in quoted-printable, why???
On Fri, 6 Mar 2015 00:06:32 +0100 Václav Ovsík vaclav.ov...@i.cz wrote: https://issues.bestpractical.com/Ticket/Display.html?id=29735 Aha -- thanks for digging that out! I thought I vaguely recalled something in this area previously. https://issues.bestpractical.com/Ticket/Attachment/286095/157750/utf8-encoding.patch looks to be functionally fairly similar to the branch. There are a few other, orthogonal fixes in there that may still be interesting to tease out into their own commits. It looks like I see changes to: * Fix the computed max size of base64'd attachments; I'd need to squint at it harder, but seems eminently reasonable. * Attempt to gracefully deal with TruncateLongAttachments truncating mid-byte of UTF-8 data. As above; the decode/encode is an interesting trick to attempt to ensure that the byte stream is consistent. I'd like to test it a bit, but seems not unreasonable. * Choose base64 vs QP based on which is shorter; I'm less convinced by this, since it means that for large data, it gets QP'd, base64'd, and then one of those _again_ -- which isn't terribly efficient. I'm less convinced by the tradeoff of computation time to stored in-database size. If you're interested in reworking the patch into a 2-3 commit series, I'm happy to apply for 4.2-trunk. - Alex
[rt-users] RT saves data in quoted-printable, why???
http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/net/QuotedPrintableCodec.html in the releasen note for RT-4.2.2 [https://www.bestpractical.com/release-notes/rt/4.2.2] * Ensure that all data containing non-ASCII is quoted-printable encoded for PostgreSQL, instead of merely all data not claiming to be text/plain Why is this? It seems it is doing more harm than good -- it makes it harder to use indexes and makes it harder to use it from the tables directly? PostgreSQL is very capable of storing any kind of character set? We use UTF-8 for everything, this fix breaks a lot of things for us. I think this was a bad design decision, but maybe I can't see the entire picture? Best regards, Palle signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [rt-users] Login in unix
On Mon, Mar 02, 2015 at 12:44:16PM +1300, Aaron Guise wrote: You cannot. The users created in the Web UI are only for RT. To login to *NIX you need to create the user at the OS level. Of course you could do the user creation with a scrip once the user is created. Not sure I would though as personally I prefer to closely control whom gets access to my systems. you should consider investiguating some PAM/NSS solutions to use the RT table users to provide de passwd database. # aptitude search '~npam ~nsql' -F '%p %d' libpam-mysql Module PAM permettant l'authentification depu libpam-mysql:i386 Module PAM permettant l'authentification depu libpam-pgsql Module PAM pour l'authentification depuis un libpam-pgsql:i386 Module PAM pour l'authentification depuis un -- Marc Chantreux, Mes coordonnées: http://annuaire.unistra.fr/chercher?n=chantreux Direction Informatique, Université de Strasbourg (http://unistra.fr) Don't believe everything you read on the Internet -- Abraham Lincoln
