[Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
The following bug has been CLOSED
==
http://www.dbmail.org/mantis/bug_view_advanced_page.php?bug_id=166
==
Reported By:aaron
Assigned To:aaron
==
Project:DBMail
Bug ID: 166
Category: IMAP daemon
Reproducibility:always
Severity: major
Priority: normal
Status: closed
==
Date Submitted: 24-Jan-05 09:42 CET
Last Modified: 19-May-05 14:22 CEST
==
Summary:IMAP sort breakage in 2.0.3
Description:
Upgrading to DBMail 2.0.3 broke my IMAP client, TWIG. I typically use
reverse arrival date sorting (newest first). The sympton is that
absolutely nothing appears in the email list, nor is there a count of
messages.
Non-reversed arrival date sort works alright, although the change below
subtlely alters the order of emails in non-reversed.
Commenting out the following lines from imaputil.c, in build_imap_search,
lines 2225-2227, fixes it.
// } else if(sorted (strcasecmp(search_keys[*idx], us-ascii) ==
0)) {
// /* TODO */
// (*idx)++;
This appears to have been added by Paul between 2.0.2 and 2.0.3. I'll dig
up some logs if you'd like!
==
--
aaron - 18-Mar-05 17:26 CET
--
Resolved in 2.0.4. There's a long discussion on the mailing list about
c-client's ability to work around broken servers. Basically we outsmarted
it just enough that it thought we were working when we really weren't.
Bug History
Date Modified Username FieldChange
==
24-Jan-05 09:42aaron New Bug
24-Jan-05 09:43aaron Description Updated
18-Mar-05 17:26aaron Bugnote Added: 616
18-Mar-05 17:26aaron Assigned To = aaron
18-Mar-05 17:26aaron Resolution open = fixed
18-Mar-05 17:26aaron Status new = resolved
19-May-05 14:22paul Status resolved = closed
==
[Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
The following bug has been RESOLVED.
==
http://dbmail.org/mantis/bug_view_advanced_page.php?bug_id=166
==
Reported By:aaron
Assigned To:aaron
==
Project:DBMail
Bug ID: 166
Category: IMAP daemon
Reproducibility:always
Severity: major
Priority: normal
Status: resolved
Resolution: fixed
==
Date Submitted: 24-Jan-05 09:42 CET
Last Modified: 18-Mar-05 17:26 CET
==
Summary:IMAP sort breakage in 2.0.3
Description:
Upgrading to DBMail 2.0.3 broke my IMAP client, TWIG. I typically use
reverse arrival date sorting (newest first). The sympton is that
absolutely nothing appears in the email list, nor is there a count of
messages.
Non-reversed arrival date sort works alright, although the change below
subtlely alters the order of emails in non-reversed.
Commenting out the following lines from imaputil.c, in build_imap_search,
lines 2225-2227, fixes it.
// } else if(sorted (strcasecmp(search_keys[*idx], us-ascii) ==
0)) {
// /* TODO */
// (*idx)++;
This appears to have been added by Paul between 2.0.2 and 2.0.3. I'll dig
up some logs if you'd like!
==
--
aaron - 18-Mar-05 17:26 CET
--
Resolved in 2.0.4. There's a long discussion on the mailing list about
c-client's ability to work around broken servers. Basically we outsmarted
it just enough that it thought we were working when we really weren't.
Bug History
Date Modified Username FieldChange
==
24-Jan-05 09:42aaron New Bug
24-Jan-05 09:43aaron Description Updated
18-Mar-05 17:26aaron Bugnote Added: 616
18-Mar-05 17:26aaron Assigned To = aaron
18-Mar-05 17:26aaron Resolution open = fixed
18-Mar-05 17:26aaron Status new = resolved
==
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
All, Just as a note I never intended for this code to get back ported to 2.0, so I guess if you guys want to work it out that is cool, however my vote would be to remove it rather than fix it and have a patch on the side if someone wants to fool with it. Otherwise I think it is a waste of time to work on the code for 2.0 as it is limted by the parsing system and other issues people are improving on in 2.1. just my .02 -leif I'm putting the two strands of this thread back together ;-) ... On Tue, Jan 25, 2005, Leif Jackson [EMAIL PROTECTED] said: That US-ASCII isn't recognized at all means that a fallback is always used. I would be interested in seeing the command that SquirrelMail uses. Perhaps we'd be much better off replacing the tries-to-be-general _ic_sort with an actual hack -- one that recognizes specific SORT commands that we know we can implement and that we've seen used by clients in the wild. I thought about making it a specific hack but then I knew it would be a wast of time when we got to x.x version of dbmail with header cahcing where it can be done right. Anyway I agree with you guys, and say leave 2.0.x alone with the _ic_sort and just make a push to make it right for 2.1 ..etc. Tho latly I have been way to busy to help so you can ignore me if you want :) We're not getting to 2.1 anytime soon, and I don't want to leave this broken in 2.0 if we can make a sane set of workarounds. Which is to say, all of the strings that are recognized and then skipped because the code says TODO are essentially silent failures to perform the requested sort command. That's not OK. If we get a string that we don't have the SQL to handle, then we should give a suitable error message and allow the client to work around us. I tried that yesterday. Won't work even for ARRIVAL, because of the way the result set is constructed in db_search. But perhaps I was just being dense. Accualy paul you could do this with a search of the cmd string to set a key for reverse and then just tell my btree code not to re-order the mid's as it takes them in reverse to being with if I rember correctly. I'll start reading through this part of the code, as I've never done much work on the IMAP side of things. What I'd like to do is take a survey of the SORT command options, and then see what sort of SQL we can reasonably do, and then work backwards saying, We have SQL than can do X, Y and Z so we'll only accept SORT strings involving X, Y and Z. If the SORT string asks for Q, then we give an error. Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron Stone wrote: We're not getting to 2.1 anytime soon, Not true. cvs-head doesn't have many, if any, regressions at the moment. It already has fixed a significant number of bugs that are marked as wont-fix for 2.0 If we make a concerted effort, 2.1.0 can be released in no time at all. In fact, I'd very much like to release 2.1.0 before I start depending on the header tables. Ilja, I'll prepare cvs-head for 2.1.0 release (tag dbmail_2_1_0_release) today. I just committed all remaining pending changes before 2.1.0 and will start testing with different clients before I do the actual tagging. Can you release later this week if I give the go ahead signal? and I don't want to leave this broken in 2.0 if we can make a sane set of workarounds. Since we've fixed all the major bugs in 2.0 I think the time has come to at least start thinking about putting 2.0 in maintenance mode. All I have to do is convince a guy named Aaron :-) Which is to say, all of the strings that are recognized and then skipped because the code says TODO are essentially silent failures to perform the requested sort command. That's not OK. If we get a string that we don't have the SQL to handle, then we should give a suitable error message and allow the client to work around us. I do agree here. Silent failures are not good. But they've also been in there since 2.0.0. I'll start reading through this part of the code, as I've never done much work on the IMAP side of things. What I'd like to do is take a survey of the SORT command options, and then see what sort of SQL we can reasonably do, and then work backwards saying, We have SQL than can do X, Y and Z so we'll only accept SORT strings involving X, Y and Z. If the SORT string asks for Q, then we give an error. Even knowing what sort of sql we can do given the 2.0 database layout is not enough. I think Leif will agree the sort code was not designed with even a semblance of full compliance in mind. It also leans too heavily on the search code to extract strict sort functionality from it. I'd say: leave it be. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Wed, Jan 26, 2005, Paul J Stevens [EMAIL PROTECTED] said: Aaron Stone wrote: We're not getting to 2.1 anytime soon, Not true. cvs-head doesn't have many, if any, regressions at the moment. It already has fixed a significant number of bugs that are marked as wont-fix for 2.0 What wont-fix bugs in 2.0 are resolved in 2.1 without header caching? If we make a concerted effort, 2.1.0 can be released in no time at all. In fact, I'd very much like to release 2.1.0 before I start depending on the header tables. Where 2.1.0 involves mostly cleanups and rewrites of the imap server? I guess I'm a little out of touch with just what's happened so far in 2.1 :-\ If we're still keeping with the Linux kernel-style version numbers, then 2.1.0 is a release only developers should be touching. Even if 2.1.0 is rock solid, I fail to see how that benefits regular users -- the current release plan on the wiki (albeit from Nov. 2004) says that you want to do header caching in 2.1.1. So a user who things, gee 2.1.0 is rock solid, I'll just use that will have to go right back to 2.0.x if there's a security release or other bugfix release, since going to 2.1.x might involve database changes. [snip] and I don't want to leave this broken in 2.0 if we can make a sane set of workarounds. Since we've fixed all the major bugs in 2.0 I think the time has come to at least start thinking about putting 2.0 in maintenance mode. All I have to do is convince a guy named Aaron :-) Just a quick look at the 2.0.4 wiki gives me the impression that 2.0 is in a combined bugfix / code speedup mode. Basically, the idea is that most people are on 2.0 now, and things are getting smoother and faster with each release. That's a good thing in my book! Which is to say, all of the strings that are recognized and then skipped because the code says TODO are essentially silent failures to perform the requested sort command. That's not OK. If we get a string that we don't have the SQL to handle, then we should give a suitable error message and allow the client to work around us. I do agree here. Silent failures are not good. But they've also been in there since 2.0.0. So what? You uncovered how bad this bug is by silenty recognizing US-ASCII, which was the cornerstone of c-client's SORT request. If a client requests ISO-8859-1, we recognize and silenty fail on that, too. We just happen not to have received a bug report from someone whose client issues the command SORT (REVERSE ARRIVAL) ISO-8859-1 ALL. It's a bug and needs to be fixed. I'll start reading through this part of the code, as I've never done much work on the IMAP side of things. What I'd like to do is take a survey of the SORT command options, and then see what sort of SQL we can reasonably do, and then work backwards saying, We have SQL than can do X, Y and Z so we'll only accept SORT strings involving X, Y and Z. If the SORT string asks for Q, then we give an error. Even knowing what sort of sql we can do given the 2.0 database layout is not enough. I think Leif will agree the sort code was not designed with even a semblance of full compliance in mind. It also leans too heavily on the search code to extract strict sort functionality from it. I'd say: leave it be. If you're saying that there are no valid arguments to SORT, then let's take it out of the capabilities. It's just by luck that no client is asking for something it actually relies on DBMail returning sane values for. We're offering to do this SORT thing, but all we give back is garbage. Why offer in the first place? Aaron
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Wed, 26 Jan 2005 09:54:09 +0100, Paul J Stevens [EMAIL PROTECTED] wrote: Aaron Stone wrote: We're not getting to 2.1 anytime soon, Not true. cvs-head doesn't have many, if any, regressions at the moment. It already has fixed a significant number of bugs that are marked as wont-fix for 2.0 If we make a concerted effort, 2.1.0 can be released in no time at all. In fact, I'd very much like to release 2.1.0 before I start depending on the header tables. Ilja, I'll prepare cvs-head for 2.1.0 release (tag dbmail_2_1_0_release) today. I just committed all remaining pending changes before 2.1.0 and will start testing with different clients before I do the actual tagging. Can you release later this week if I give the go ahead signal? No problem. We'll label this as a development release I reckon? cheers, Ilja
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron Stone wrote: What wont-fix bugs in 2.0 are resolved in 2.1 without header caching? recent flag (bug #97) comes to mind. I know I said I would try to fix that one for 2.0, but that was before the sort breakage surfaced. Also, insertion will be more robust in terms of mime parsing because of gmime. Generally, there's not much difference between 2.0 and cvs-head yet from the client's point of view. Internally, things have changed quite a bit for imap and insertion. If we make a concerted effort, 2.1.0 can be released in no time at all. In fact, I'd very much like to release 2.1.0 before I start depending on the header tables. Where 2.1.0 involves mostly cleanups and rewrites of the imap server? I guess I'm a little out of touch with just what's happened so far in 2.1 :-\ Mostly imap stuff, yes. But like I said I'm also reshuffling the insertion code and I'm adding a lot of test-code. If we're still keeping with the Linux kernel-style version numbers, then 2.1.0 is a release only developers should be touching. Even if 2.1.0 is rock solid, I fail to see how that benefits regular users -- the current release plan on the wiki (albeit from Nov. 2004) says that you want to do header caching in 2.1.1. True. But before I move on the 2.1.1, I need a 2.1.0 release first right? So a user who things, gee 2.1.0 is rock solid, I'll just use that will have to go right back to 2.0.x if there's a security release or other bugfix release, since going to 2.1.x might involve database changes. The database changes I have in mind will be forward compatible. That is: 2.1.x code will use more tables, but 2.0.x will be able to use the same database without a problem. So people will be able to use both 2.0 and 2.1 connecting to the same database *at the same time*. I don't mind working on 2.0 when it comes to bugs. But with just two people actively working on the code at the moment, maintaining and actively moving forward two different branches just seems a bit much. Just a quick look at the 2.0.4 wiki gives me the impression that 2.0 is in a combined bugfix / code speedup mode. Basically, the idea is that most people are on 2.0 now, and things are getting smoother and faster with each release. That's a good thing in my book! agreed. 2.0 is definitely not abandon-ware. It's what all my clients use. I do agree here. Silent failures are not good. But they've also been in there since 2.0.0. So what? You uncovered how bad this bug is by silenty recognizing US-ASCII, which was the cornerstone of c-client's SORT request. If a client requests ISO-8859-1, we recognize and silenty fail on that, too. We just happen not to have received a bug report from someone whose client issues the command SORT (REVERSE ARRIVAL) ISO-8859-1 ALL. It's a bug and needs to be fixed. There's no discussion there. Just whether it should be targetted at 2.0. If you feel strongly about fixing it for 2.0, I say excellent. Perhaps you can find an elegant solution to this problem :-) If you're saying that there are no valid arguments to SORT, then let's take it out of the capabilities. It's just by luck that no client is asking for something it actually relies on DBMail returning sane values for. We're offering to do this SORT thing, but all we give back is garbage. Why offer in the first place? Because the SM users love it? -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Paul J Stevens wrote: Aaron Stone wrote: If we're still keeping with the Linux kernel-style version numbers, then 2.1.0 is a release only developers should be touching. Even if 2.1.0 is rock solid, I fail to see how that benefits regular users -- the current release plan on the wiki (albeit from Nov. 2004) says that you want to do header caching in 2.1.1. True. But before I move on the 2.1.1, I need a 2.1.0 release first right? Just to confirm (since I'm still unsure), you are saying that 2.1 is for development not production right?
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Matthew T. O'Connor wrote: Paul J Stevens wrote: Aaron Stone wrote: If we're still keeping with the Linux kernel-style version numbers, then 2.1.0 is a release only developers should be touching. Even if 2.1.0 is rock solid, I fail to see how that benefits regular users -- the current release plan on the wiki (albeit from Nov. 2004) says that you want to do header caching in 2.1.1. True. But before I move on the 2.1.1, I need a 2.1.0 release first right? Just to confirm (since I'm still unsure), you are saying that 2.1 is for development not production right? Absolutely!!! The whole 2.1.x series will be a development series, whereas 2.0.x will be for stable production systems. Until 2.2.0 happens at the end of the 2.1 drive. -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Paul J Stevens wrote: Matthew T. O'Connor wrote: Paul J Stevens wrote: True. But before I move on the 2.1.1, I need a 2.1.0 release first right? Just to confirm (since I'm still unsure), you are saying that 2.1 is for development not production right? Absolutely!!! The whole 2.1.x series will be a development series, whereas 2.0.x will be for stable production systems. Until 2.2.0 happens at the end of the 2.1 drive. Good, glad that has been made crystal clear :-) But I don't think you need a 2.1 release. 2.1 will just be CVS Head, it should probably be tagged 2.1 at some point, but we don't need (or want) a 2.1 CVS branch.
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Wed, Jan 26, 2005, Matthew T. O'Connor matthew@zeut.net said: Good, glad that has been made crystal clear :-) But I don't think you need a 2.1 release. 2.1 will just be CVS Head, it should probably be tagged 2.1 at some point, but we don't need (or want) a 2.1 CVS branch. I was going to suggest a dbmail_2_1_branch, until the mantra main development happens on HEAD started to ring between my ears. So we'll just tag the 2_1_x release points along the way, and eventually we'll be stable and *then* we'll branch off with dbmail_2_2_branch before continuing new work in HEAD again. Aaron
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Mon, Jan 24, 2005, Leif Jackson [EMAIL PROTECTED] said: I would say that as the _ic_sort was mostly just a stop gap untill we had header caching available, that it is obvious the whole sort part of dbmail should be reworked, however the current implementation is near optimal for at least the sort needed for squirrelmail, as you mentioned before I never intended this to be anything but for SM. If anyone has time to work on code I would urge we focus on header caching so as to be able to make the sort and other items work well for 2.1 and leave the sort code that was merged into 2.0 as is and make a not that it is intened for SM only, or add a option to enable the sort code and runtime...etc. Header caching isn't an option for 2.0, so we need to make it work reliably and as fast as possible within its database schema. I do know that SquirrelMail is more common than TWIG, but more importantly, SquirrelMail has its own IMAP client library written in PHP whereas TWIG uses PHP's copy of the c-client library. It would not surprise me if many more PHP email programs are not working with 2.0.3, either, though I haven't looked into it. What specifically makes the _ic_sort code work for SquirrelMail that can't be extended for other clients and/or for a more general case? And, more broadly, if _ic_sort is really _ic_sort_smhack, what can we do so that the hack is only used if we've really got SquirrelMail on the line? Aaron
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Mon, Jan 24, 2005, Paul J Stevens [EMAIL PROTECTED] said: Well, this whole affair started with me trying to speedup searches and trying to apply the lessons learned there to the sort code. But you're quite right, fixing sort is too bloody to do in 2.0. Bummer. Aaron, before I can start on the header_table, I will have to start reshuffling some of the insertion code. I'll keep it clean ans test-driven, but stuff will move to and be assimilated into dbmail-message.c. Thanks for the heads up. Good luck with your work on this! Aaron
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron Stone wrote: On Mon, Jan 24, 2005, Leif Jackson [EMAIL PROTECTED] said: I would say that as the _ic_sort was mostly just a stop gap untill we had header caching available, that it is obvious the whole sort part of dbmail should be reworked, however the current implementation is near optimal for at least the sort needed for squirrelmail, as you mentioned before I never intended this to be anything but for SM. If anyone has time to work on code I would urge we focus on header caching so as to be able to make the sort and other items work well for 2.1 and leave the sort code that was merged into 2.0 as is and make a not that it is intened for SM only, or add a option to enable the sort code and runtime...etc. Header caching isn't an option for 2.0, so we need to make it work reliably and as fast as possible within its database schema. I do know that SquirrelMail is more common than TWIG, but more importantly, SquirrelMail has its own IMAP client library written in PHP whereas TWIG uses PHP's copy of the c-client library. It would not surprise me if many more PHP email programs are not working with 2.0.3, either, though I haven't looked into it. I was pleasantly surprised by your command trace yesterday. Apparently c-client is smart. Smart enough to hide server-errors from the client. The fact that A01 SORT (REVERSE ARRIVAL) US-ASCII ALL failed is very important in this respect. The build_imap_search function is full of TODO statements for search keys that are silently ignored whereas US-ASCII and UTF-8 which are both explicitely required by the specs were/are totally absent and therefor trigger an error. What specifically makes the _ic_sort code work for SquirrelMail that can't be extended for other clients and/or for a more general case? Probably nothing for specific cases, but making it work for the general case is beyond me given the 2.0 code for reasons I outlined yesterday. And, more broadly, if _ic_sort is really _ic_sort_smhack, what can we do so that the hack is only used if we've really got SquirrelMail on the line? We haven't had many complaints so far, so perhaps do it right in 2.1 should indeed take priority. The correct way to solve this for now in 2.0 would be some switch in dbmail.conf. But that would open an other can of worms we've been talking about: capabilities. So to paraphrase: if no-one cares it's broken, don't fix it. Twig works (sort of) thanks to c-client, SM works thanks to _ic_sort,... -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Tue, Jan 25, 2005, Paul J Stevens [EMAIL PROTECTED] said:
I was pleasantly surprised by your command trace yesterday. Apparently
c-client is smart. Smart enough to hide server-errors from the client.
The fact that
A01 SORT (REVERSE ARRIVAL) US-ASCII ALL
failed is very important in this respect. The build_imap_search function
is full of TODO statements for search keys that are silently ignored
whereas US-ASCII and UTF-8 which are both explicitely required...
[snip]
Looks like SORT (ARRIVAL) US-ASCII ALL kinda works in 2.0.3, because for
ARRIVAL, there's:
if(sorted (strcasecmp(search_keys[*idx], arrival) == 0)) {
key.type = IST_SORT;
strncpy(key.search, order by pms.internal_date, MAX_SEARCH_LEN);
(*idx)++;
Whereas (REVERSE ARRIVAL) fails because REVERSE doesn't work. In 2.0.2,
US-ASCII itself threw an error, and so in both cases fallbacks were used.
So REVERSE ARRIVAL can be done just with a desc in the order by
clause. However REVERSE with any of the other sorts, working or not, won't
work.
That US-ASCII isn't recognized at all means that a fallback is always
used. I would be interested in seeing the command that SquirrelMail uses.
Perhaps we'd be much better off replacing the tries-to-be-general _ic_sort
with an actual hack -- one that recognizes specific SORT commands that we
know we can implement and that we've seen used by clients in the wild.
Aaron
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Tue, Jan 25, 2005, Paul J Stevens [EMAIL PROTECTED] said:
I was pleasantly surprised by your command trace yesterday. Apparently
c-client is smart. Smart enough to hide server-errors from the client.
The fact that
A01 SORT (REVERSE ARRIVAL) US-ASCII ALL
failed is very important in this respect. The build_imap_search function
is full of TODO statements for search keys that are silently ignored
whereas US-ASCII and UTF-8 which are both explicitely required...
[snip]
Looks like SORT (ARRIVAL) US-ASCII ALL kinda works in 2.0.3, because for
ARRIVAL, there's:
if(sorted (strcasecmp(search_keys[*idx], arrival) == 0)) {
key.type = IST_SORT;
strncpy(key.search, order by pms.internal_date, MAX_SEARCH_LEN);
(*idx)++;
Whereas (REVERSE ARRIVAL) fails because REVERSE doesn't work. In 2.0.2,
US-ASCII itself threw an error, and so in both cases fallbacks were used.
So REVERSE ARRIVAL can be done just with a desc in the order by
clause. However REVERSE with any of the other sorts, working or not, won't
work.
That US-ASCII isn't recognized at all means that a fallback is always
used. I would be interested in seeing the command that SquirrelMail uses.
Perhaps we'd be much better off replacing the tries-to-be-general _ic_sort
with an actual hack -- one that recognizes specific SORT commands that we
know we can implement and that we've seen used by clients in the wild.
I thought about making it a specific hack but then I knew it would be a
wast of time when we got to x.x version of dbmail with header cahcing
where it can be done right. Anyway I agree with you guys, and say leave
2.0.x alone with the _ic_sort and just make a push to make it right for
2.1 ..etc. Tho latly I have been way to busy to help so you can ignore me
if you want :)
-leif
Aaron
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron Stone wrote:
On Tue, Jan 25, 2005, Paul J Stevens [EMAIL PROTECTED] said:
I was pleasantly surprised by your command trace yesterday. Apparently
c-client is smart. Smart enough to hide server-errors from the client.
The fact that
A01 SORT (REVERSE ARRIVAL) US-ASCII ALL
failed is very important in this respect. The build_imap_search function
is full of TODO statements for search keys that are silently ignored
whereas US-ASCII and UTF-8 which are both explicitely required...
[snip]
Looks like SORT (ARRIVAL) US-ASCII ALL kinda works in 2.0.3, because for
ARRIVAL, there's:
if(sorted (strcasecmp(search_keys[*idx], arrival) == 0)) {
key.type = IST_SORT;
strncpy(key.search, order by pms.internal_date, MAX_SEARCH_LEN);
(*idx)++;
Whereas (REVERSE ARRIVAL) fails because REVERSE doesn't work. In 2.0.2,
US-ASCII itself threw an error, and so in both cases fallbacks were
used.
So REVERSE ARRIVAL can be done just with a desc in the order by
clause. However REVERSE with any of the other sorts, working or not,
won't
work.
I tried that yesterday. Won't work even for ARRIVAL, because of the way
the
result set is constructed in db_search. But perhaps I was just being
dense.
Accualy paul you could do this with a search of the cmd string to set a
key for reverse and then just tell my btree code not to re-order the mid's
as it takes them in reverse to being with if I rember correctly.
Thanks,
Leif
p.s. Until there is header caching anything but Date sort is painfully
slow becuase the code has to reparse all the messages headers with the
mime code. If we could somehow make the extensible table for header values
and keys and then a table of key refs we could make the mime parsers just
check for the value for that uniq key in the cache if it isn't there add
it and then just have a ref to the messages uniq id, then anytime the mime
parse header is called it could check for an already parsed header. This
is my .02 of how I would quickly whip it out, I wanted to get a little
crazy also and add memcached interface to the header caching code for
large sites like my customers where many many people will be grinding
throught the cached headers..etc.
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
I'm putting the two strands of this thread back together ;-) ... On Tue, Jan 25, 2005, Leif Jackson [EMAIL PROTECTED] said: That US-ASCII isn't recognized at all means that a fallback is always used. I would be interested in seeing the command that SquirrelMail uses. Perhaps we'd be much better off replacing the tries-to-be-general _ic_sort with an actual hack -- one that recognizes specific SORT commands that we know we can implement and that we've seen used by clients in the wild. I thought about making it a specific hack but then I knew it would be a wast of time when we got to x.x version of dbmail with header cahcing where it can be done right. Anyway I agree with you guys, and say leave 2.0.x alone with the _ic_sort and just make a push to make it right for 2.1 ..etc. Tho latly I have been way to busy to help so you can ignore me if you want :) We're not getting to 2.1 anytime soon, and I don't want to leave this broken in 2.0 if we can make a sane set of workarounds. Which is to say, all of the strings that are recognized and then skipped because the code says TODO are essentially silent failures to perform the requested sort command. That's not OK. If we get a string that we don't have the SQL to handle, then we should give a suitable error message and allow the client to work around us. I tried that yesterday. Won't work even for ARRIVAL, because of the way the result set is constructed in db_search. But perhaps I was just being dense. Accualy paul you could do this with a search of the cmd string to set a key for reverse and then just tell my btree code not to re-order the mid's as it takes them in reverse to being with if I rember correctly. I'll start reading through this part of the code, as I've never done much work on the IMAP side of things. What I'd like to do is take a survey of the SORT command options, and then see what sort of SQL we can reasonably do, and then work backwards saying, We have SQL than can do X, Y and Z so we'll only accept SORT strings involving X, Y and Z. If the SORT string asks for Q, then we give an error. Aaron
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron, I added those lines because charset us-ascii sort is required by the rfc. What is the exact sort
command used by twig? I can add that one to the test script and fix this. Also, please test the new
sort/search improvements (bug #149) with twig, to validate the changes *before* we release 2.0.4
Anyway, I think I will fix bug #97 in the 2.0 branch after all. The fixing of bugs #149 and #97 combined will
probably make dbmail's searching and sorting actually usable. At least that's my experience from running tests
on cvs-head.
[EMAIL PROTECTED] wrote:
==
http://www.dbmail.org/mantis/bug_view_advanced_page.php?bug_id=166
==
Reported By:aaron
Assigned To:
==
Project:DBMail
Bug ID: 166
Category: IMAP daemon
Reproducibility:always
Severity: major
Priority: normal
Status: new
==
Date Submitted: 24-Jan-05 09:42 CET
Last Modified: 24-Jan-05 09:43 CET
==
Summary:IMAP sort breakage in 2.0.3
Description:
Upgrading to DBMail 2.0.3 broke my IMAP client, TWIG. I typically use
reverse arrival date sorting (newest first). The sympton is that
absolutely nothing appears in the email list, nor is there a count of
messages.
Non-reversed arrival date sort works alright, although the change below
subtlely alters the order of emails in non-reversed.
Commenting out the following lines from imaputil.c, in build_imap_search,
lines 2225-2227, fixes it.
// } else if(sorted (strcasecmp(search_keys[*idx], us-ascii) ==
0)) {
// /* TODO */
// (*idx)++;
This appears to have been added by Paul between 2.0.2 and 2.0.3. I'll dig
up some logs if you'd like!
==
Bug History
Date Modified Username FieldChange
==
24-Jan-05 09:42aaron New Bug
24-Jan-05 09:43aaron Description Updated
==
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
--
Paul Stevens mailto:[EMAIL PROTECTED]
NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED]
The Netherlandshttp://www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron Stone wrote: On Mon, Jan 24, 2005, Paul J Stevens [EMAIL PROTECTED] said: Aaron, I added those lines because charset us-ascii sort is required by the rfc. What is the exact sort command used by twig? I can add that one to the test script and fix this. Also, please test the new sort/search improvements (bug #149) with twig, to validate the changes *before* we release 2.0.4 The command is... 0006 SORT (REVERSE ARRIVAL) US-ASCII ALL which soon after logs this: Jan 24 00:51:32 [dbmail/imap4d] imaputil.c,build_imap_search: unknown search key [US-ASCII] which is why I added the case for us-ascii... Followed by... 0007 FETCH 1:20 (UID ENVELOPE BODY.PEEK[HEADER.FIELDS (Newsgroups Content-MD5 Content-Disposition Content-Language Content-Location Followup-To References)] INTERNALDATE RFC822.SIZE FLAGS) So sort failed, and the client falls back to client-side sorting... In plain 2.0.3, the search key unknown message doesn't appear, and TWIG doesn't follow up with the message range request. Because of the tagged OK response. Incidentally, it's probably not TWIG itself that's confused, but rather the wu-imapd c-client that's built into PHP... And rightly so. The current SORT implementation is: - unfinished business - does not comply with the relevant internet draft. Reading that draft (draft-ietf-imapext-sort-1) I see too many issues to mention here. A couple of examples: The weighted search setup I added last week is actually in direct conflict with the draft because the order of the search keys *is* relevant for sorting. We'll be able to use weighted search to quickly retrieve matching message ids, but actually sorting those ids must depend on the original order of the search keys. However, since searching and sorting are currently non-orthogonally implemented that is not a simple matter. UTF-8 support is required, whereas dbmail has *no* support for specific charsets. Sorting by subject should use a specially converted form of the subjectlines involved. etc, etc... It was always announced by Leif as being aimed at squirrelmail users. Most issues with sorting will be much simpler to do when we start using the header table design. Until then: if it works your lucky. I'll look into it... -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron,
I've reverted _ic_sort behaviour to pre-2.0.3. Can you confirm this wrt twig?
This has given me something to think about:
- Leif interleaved the sort with the search code. There are some valid reason's for doing so, but we need to
treat the search arguments fundamentally differently from the sort keys, where they are currently part of the
same struct. Messy.
- Given that my understanding of the current sort code is less than perfect, I suggest that the sort keys
*not* be used for search commands, as is now (partly) the case. Instead, the search keys should be used for
determining the result set, and next the sort keys should be used for reordering the set. This, it seems,
would require the search run to use the sort keys for retrieving all the message attributes that will later on
be used for sorting.
Paul J Stevens wrote:
Aaron, I added those lines because charset us-ascii sort is required by
the rfc. What is the exact sort command used by twig? I can add that one
to the test script and fix this. Also, please test the new sort/search
improvements (bug #149) with twig, to validate the changes *before* we
release 2.0.4
Anyway, I think I will fix bug #97 in the 2.0 branch after all. The
fixing of bugs #149 and #97 combined will probably make dbmail's
searching and sorting actually usable. At least that's my experience
from running tests on cvs-head.
[EMAIL PROTECTED] wrote:
==
http://www.dbmail.org/mantis/bug_view_advanced_page.php?bug_id=166
==
Reported By:aaron
Assigned To:
==
Project:DBMail
Bug ID: 166
Category: IMAP daemon
Reproducibility:always
Severity: major
Priority: normal
Status: new
==
Date Submitted: 24-Jan-05 09:42 CET
Last Modified: 24-Jan-05 09:43 CET
==
Summary:IMAP sort breakage in 2.0.3
Description: Upgrading to DBMail 2.0.3 broke my IMAP client, TWIG. I
typically use
reverse arrival date sorting (newest first). The sympton is that
absolutely nothing appears in the email list, nor is there a count of
messages.
Non-reversed arrival date sort works alright, although the change below
subtlely alters the order of emails in non-reversed.
Commenting out the following lines from imaputil.c, in build_imap_search,
lines 2225-2227, fixes it.
// } else if(sorted (strcasecmp(search_keys[*idx], us-ascii) ==
0)) {
// /* TODO */
// (*idx)++;
This appears to have been added by Paul between 2.0.2 and 2.0.3. I'll dig
up some logs if you'd like!
==
Bug History
Date Modified Username Field
Change
==
24-Jan-05 09:42aaron New
Bug 24-Jan-05 09:43aaron
Description Updated
==
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
--
Paul Stevens mailto:[EMAIL PROTECTED]
NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED]
The Netherlandshttp://www.nfg.nl
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron, I've reverted _ic_sort behaviour to pre-2.0.3. Can you confirm this wrt twig? This has given me something to think about: - Leif interleaved the sort with the search code. There are some valid reason's for doing so, but we need to treat the search arguments fundamentally differently from the sort keys, where they are currently part of the same struct. Messy. - Given that my understanding of the current sort code is less than perfect, I suggest that the sort keys *not* be used for search commands, as is now (partly) the case. Instead, the search keys should be used for determining the result set, and next the sort keys should be used for reordering the set. This, it seems, would require the search run to use the sort keys for retrieving all the message attributes that will later on be used for sorting. I would say that as the _ic_sort was mostly just a stop gap untill we had header caching available, that it is obvious the whole sort part of dbmail should be reworked, however the current implementation is near optimal for at least the sort needed for squirrelmail, as you mentioned before I never intended this to be anything but for SM. If anyone has time to work on code I would urge we focus on header caching so as to be able to make the sort and other items work well for 2.1 and leave the sort code that was merged into 2.0 as is and make a not that it is intened for SM only, or add a option to enable the sort code and runtime...etc. Thanks, Leif
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Leif Jackson wrote: I would say that as the _ic_sort was mostly just a stop gap untill we had header caching available, that it is obvious the whole sort part of dbmail should be reworked, however the current implementation is near optimal for at least the sort needed for squirrelmail, as you mentioned before I never intended this to be anything but for SM. If anyone has time to work on code I would urge we focus on header caching so as to be able to make the sort and other items work well for 2.1 and leave the sort code that was merged into 2.0 as is and make a not that it is intened for SM only, or add a option to enable the sort code and runtime...etc. Well, this whole affair started with me trying to speedup searches and trying to apply the lessons learned there to the sort code. But you're quite right, fixing sort is too bloody to do in 2.0. Aaron, before I can start on the header_table, I will have to start reshuffling some of the insertion code. I'll keep it clean and test-driven, but stuff will move to and be assimilated into dbmail-message.c. -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl
