Re: Changing IMAP separator - does it break things?

[@lbutlr]

On 14 Oct 2020, at 12:37, @lbutlr  wrote:
> jeorachu

I swear I typed hierarchy.

-- 
"A synonym is a word you use when you can't spell the word you first
thought of." - Burt Bacharach

Re: Changing IMAP separator - does it break things?

[@lbutlr]

On 14 Oct 2020, at 05:39, Sami Ketola  wrote:
>> On 14. Oct 2020, at 14.20, @lbutlr  wrote:
>> 
>> On 14 Oct 2020, at 04:24, Sami Ketola  wrote:
>>> On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
 Those Mac clients who do not use nested folders do not seem to be
 affected. That is, I've already suggested to the Mac Mail user that he
 start Thunderbird, move all his mailboxes into the IMAP root folder, and
 then go on using Mac Mail. I hope that's what he has done.
>> 
>>> There is a bug in Mail.app that it always assumes / as hierarchy separator 
>>> and 
>>> completely ignores server provided separator.
>> 
>> Cite? Because no, that is not my experience at all.
>> 
>> From my maildir on my FreeBSD server:
>> 
>> # ls -lnd .root*
>> drwx--  5 89  89  512 Oct 14 05:16 .root
>> drwx--  5 89  89  512 Oct 14 02:02 .root.44
>> drwx--  5 89  89  512 Oct 14 02:02 .root.Cron
>> drwx--  5 89  89  512 Oct 14 05:16 .root.Hastur
>> … etc
>> 
>> Works fine in Mail.app on Mac OS and has for at least 16 years.
> 
> I just tested it with my Dovecot 2.2.36 setup. I did set up an account with 
> '.' as separator and created some folders.

Are you ignoring the advice in Dovecot to leave its jeorachu separator field 
alone?

There's a difference between the IMAP separator and the hierarchy separator, 
and you should not change (or set) the latter at all.

-- 
Oh, he's just like any other man, only more so.

Re: Changing IMAP separator - does it break things?

[Markus Winkler]

On 14.10.20 15:02, Victor Sudakov wrote:

I was asking because the section "Hierarchy separators" in
https://doc.dovecot.org/configuration_manual/namespace/ somewhat implies
that

However, changing the separator doesn’t change the on-disk “layout separator”.

That's why I thought that the separator is in fact virtual.


You're right - please see my former mail and the link you repeated above. 
There's a nice table which shows the possible combinations of hierarchy 
separator (called 'NS sep' there) and 'Layout sep' (structure on disk) and 
the respective results for 'Mailbox name' and the directory structure in 
the filesystem.



At present, if my folder layout in the mail client is

Inbox
   |
   +---Friends
  |
  +Paul
  |
  +Jessica





the folder "Jessica" on-disk is something like 
~/Maildir/.Friends.Jessica/{cur,new,tmp}/

-^^
Then you have configured a Layout sep '.' (mail_location = maildir:~/Maildir).


This says nothing about your hierarchy separator, which is controlled by 
the parameter 'separator ='.




Do you mean to say that if I change the IMAP separator to "/", Dovecot
will not translate "/" into "." and the folder "Jessica" will become
inaccessible?


If you currently have 'separator = .' then of course the hierarchy 
separator is '.'. If you change that to 'separator = /' normally the access 
to "Jessica" is still possible.


But (as you can see in the table mentioned above) the name of the mailbox 
(= folder) from the IMAP clients point of view is now 'Friends/Jessica'.


Before the change and with 'separator = .' the name of the mailbox was 
'Friends.Jessica'.


If you telnet to Dovecot and (after authentication) type something like this:

a LIST "" "*"

The server may responds with:

* LIST (\HasChildren) "." INBOX

Here you can see, that this name space has a '.' as the hierarchy separator.

If the response is this:

* LIST (\HasChildren) "/" INBOX

then we know that '/' is the hierarchy separator.


So, if you have an IMAP client which ist standard-compliant he will check 
what's the configured hierarchy separator and will use it for accessing the 
folder structure. And again: this has nothing to do with the on disk 
structure of the folders.


Regards,
Markus

Re: Changing IMAP separator - does it break things?

[Filip Hajný]

14. 10. 2020 v 13:39, Sami Ketola :
> 
>> On 14. Oct 2020, at 14.20, @lbutlr  wrote:
>> 
>> On 14 Oct 2020, at 04:24, Sami Ketola  wrote:
>>> On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
 Those Mac clients who do not use nested folders do not seem to be
 affected. That is, I've already suggested to the Mac Mail user that he
 start Thunderbird, move all his mailboxes into the IMAP root folder, and
 then go on using Mac Mail. I hope that's what he has done.
>> 
>>> There is a bug in Mail.app that it always assumes / as hierarchy separator 
>>> and 
>>> completely ignores server provided separator.
>> 
>> Cite? Because no, that is not my experience at all.
>> 
>> From my maildir on my FreeBSD server:
>> 
>> # ls -lnd .root*
>> drwx--  5 89  89  512 Oct 14 05:16 .root
>> drwx--  5 89  89  512 Oct 14 02:02 .root.44
>> drwx--  5 89  89  512 Oct 14 02:02 .root.Cron
>> drwx--  5 89  89  512 Oct 14 05:16 .root.Hastur
>> … etc
>> 
>> Works fine in Mail.app on Mac OS and has for at least 16 years.
> 
> I just tested it with my Dovecot 2.2.36 setup. I did set up an account with 
> '.' as separator and created some folders.
> First mail.app succeeded in creating test root folder but when I tried to 
> create subfolder 'test2' under 'test' Mail.app issued
> CREATE "test/test2" folder instead of "test.test2". And then failed to list 
> the created folder completely.
> 
> Everything worked if I did set separator to '/'

In my case, having the hierarchy separator set to '/', subfolders created using 
the Mail client in macOS Catalina resulted in '.folder\2esubfolder‘ on disk (2e 
corresponds to '.‘). I had to change the hierarchy separator to a dot.

So exactly the opposite!

-F

Re: Changing IMAP separator - does it break things?

[Filip Hajný]

14. 10. 2020 v 12:24, Sami Ketola :
> 
>> On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
>> 
>> Those Mac clients who do not use nested folders do not seem to be
>> affected. That is, I've already suggested to the Mac Mail user that he
>> start Thunderbird, move all his mailboxes into the IMAP root folder, and
>> then go on using Mac Mail. I hope that's what he has done.
> 
> There is a bug in Mail.app that it always assumes / as hierarchy separator 
> and 
> completely ignores server provided separator.
> 
> Not sure if anyone has created BUG to Apple about this. Atleast I have not.
> 
> Sami
> 

So hm, I actually re-read the original post, and this is pretty weird, because 
I had to change the Dovecot separator from '/' to '.' in order to make macOS 
clients work with subfolders properly.

I never had this problem myself because only use folders, not subfolders. And 
it was pointed to me by another user, who relies on subfolders and had problems 
for some time - *maybe* from since around the time she upgraded to Catalina.

-F

Re: Changing IMAP separator - does it break things?

[Victor Sudakov]

@lbutlr wrote:
> On 13 Oct 2020, at 20:36, Victor Sudakov  wrote:
> > I am using Dovecot 2.3.4.1 with maildirs. Just found out that MacOS Mail
> > clients cannot work with nested folders, presumably because the MacOS
> > Mail does not understand "." as the IMAP separator.
> 
> This is not correct. Mac Mail has no problems with nested folders and never 
> has had any problem with nested folders. I have always used the '.' separator 
> and I've had a 'Root' mail folder with subfolders since at least 2004.

I think I have already posted this link: 
https://discussions.apple.com/thread/8347018

Also this: http://hints.macworld.com/article.php?story=2007090423071517
It contains weird advice but at least proves that the problem exists for *some* 
people.
See also the comments therein.

> 
> > I don't want to break mail for 50 people just to please one Mac Mail
> > user. The users are advanced developers actively using mail sorting
> > rules, nested folders etc.
> 
> And what makes them think nested folders do not work?

I don't have the exact error message from him but it was pretty
obvious about a folder "foo/bar/bob" being non-existent or inaccessible.
And indeed, it's "~/Maildir/.foo.bar.bob" on the IMAP server's disk, there are 
no
subdirectories for mail folders.

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


signature.asc
Description: PGP signature

Re: Changing IMAP separator - does it break things?

[Victor Sudakov]

@lbutlr wrote:
> On 14 Oct 2020, at 04:24, Sami Ketola  wrote:
> > On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
> >> Those Mac clients who do not use nested folders do not seem to be
> >> affected. That is, I've already suggested to the Mac Mail user that he
> >> start Thunderbird, move all his mailboxes into the IMAP root folder, and
> >> then go on using Mac Mail. I hope that's what he has done.
> 
> > There is a bug in Mail.app that it always assumes / as hierarchy separator 
> > and 
> > completely ignores server provided separator.
> 
> Cite? Because no, that is not my experience at all.

Cannot site a bug, can only site a users's complaint:
https://discussions.apple.com/thread/8347018

> 
> From my maildir on my FreeBSD server:
> 
>  # ls -lnd .root*
> drwx--  5 89  89  512 Oct 14 05:16 .root
> drwx--  5 89  89  512 Oct 14 02:02 .root.44
> drwx--  5 89  89  512 Oct 14 02:02 .root.Cron
> drwx--  5 89  89  512 Oct 14 05:16 .root.Hastur
> … etc
> 
> Works fine in Mail.app on Mac OS and has for at least 16 years.

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


signature.asc
Description: PGP signature

Re: Changing IMAP separator - does it break things?

[Victor Sudakov]

Filip Hajný wrote:
> 14. 10. 2020 v 11:51, Victor Sudakov :
> > 
> >> I had to do it with 10-20 live users and it went fine, nobody noticed
> >> anything (I made the change in the night hours just to be sure
> >> though). I had to go down and fix some erratic folder names that
> >> resulted from the botched separator handling on some macOS Mail
> >> accounts.
> > 
> > Did it change anything on Dovecot disk/storage or is the IMAP separator just
> > "virtual" - ephemeral ?
> 
> Changing the separator won't fix anything that already exists. You should see 
> nonsense folders in the form of  '.folder\2esubfolder‘ there that cannot be 
> worked with in any way. I manually moved the contents of these to their 
> properly named ('.folder.subfolder‘) variants - on disk.
> 

I was asking because the section "Hierarchy separators" in
https://doc.dovecot.org/configuration_manual/namespace/ somewhat implies
that 

However, changing the separator doesn’t change the on-disk “layout separator”.

That's why I thought that the separator is in fact virtual.

At present, if my folder layout in the mail client is

Inbox
  |
  +---Friends
 |
 +Paul
 |
 +Jessica

the folder "Jessica" on-disk is something like 
~/Maildir/.Friends.Jessica/{cur,new,tmp}/

Do you mean to say that if I change the IMAP separator to "/", Dovecot
will not translate "/" into "." and the folder "Jessica" will become
inaccessible?



-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


signature.asc
Description: PGP signature

Re: Changing IMAP separator - does it break things?

[Sami Ketola]

> On 14. Oct 2020, at 14.20, @lbutlr  wrote:
> 
> On 14 Oct 2020, at 04:24, Sami Ketola  wrote:
>> On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
>>> Those Mac clients who do not use nested folders do not seem to be
>>> affected. That is, I've already suggested to the Mac Mail user that he
>>> start Thunderbird, move all his mailboxes into the IMAP root folder, and
>>> then go on using Mac Mail. I hope that's what he has done.
> 
>> There is a bug in Mail.app that it always assumes / as hierarchy separator 
>> and 
>> completely ignores server provided separator.
> 
> Cite? Because no, that is not my experience at all.
> 
> From my maildir on my FreeBSD server:
> 
> # ls -lnd .root*
> drwx--  5 89  89  512 Oct 14 05:16 .root
> drwx--  5 89  89  512 Oct 14 02:02 .root.44
> drwx--  5 89  89  512 Oct 14 02:02 .root.Cron
> drwx--  5 89  89  512 Oct 14 05:16 .root.Hastur
> … etc
> 
> Works fine in Mail.app on Mac OS and has for at least 16 years.

I just tested it with my Dovecot 2.2.36 setup. I did set up an account with '.' 
as separator and created some folders.
First mail.app succeeded in creating test root folder but when I tried to 
create subfolder 'test2' under 'test' Mail.app issued
CREATE "test/test2" folder instead of "test.test2". And then failed to list the 
created folder completely.

Everything worked if I did set separator to '/'

Sami

Re: Changing IMAP separator - does it break things?

[@lbutlr]

On 14 Oct 2020, at 04:24, Sami Ketola  wrote:
> On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
>> Those Mac clients who do not use nested folders do not seem to be
>> affected. That is, I've already suggested to the Mac Mail user that he
>> start Thunderbird, move all his mailboxes into the IMAP root folder, and
>> then go on using Mac Mail. I hope that's what he has done.

> There is a bug in Mail.app that it always assumes / as hierarchy separator 
> and 
> completely ignores server provided separator.

Cite? Because no, that is not my experience at all.

From my maildir on my FreeBSD server:

 # ls -lnd .root*
drwx--  5 89  89  512 Oct 14 05:16 .root
drwx--  5 89  89  512 Oct 14 02:02 .root.44
drwx--  5 89  89  512 Oct 14 02:02 .root.Cron
drwx--  5 89  89  512 Oct 14 05:16 .root.Hastur
… etc

Works fine in Mail.app on Mac OS and has for at least 16 years.

-- 
I'm not old, I'm chronologically challenged.

Re: Changing IMAP separator - does it break things?

[@lbutlr]

On 13 Oct 2020, at 20:36, Victor Sudakov  wrote:
> I am using Dovecot 2.3.4.1 with maildirs. Just found out that MacOS Mail
> clients cannot work with nested folders, presumably because the MacOS
> Mail does not understand "." as the IMAP separator.

This is not correct. Mac Mail has no problems with nested folders and never has 
had any problem with nested folders. I have always used the '.' separator and 
I've had a 'Root' mail folder with subfolders since at least 2004.

> I don't want to break mail for 50 people just to please one Mac Mail
> user. The users are advanced developers actively using mail sorting
> rules, nested folders etc.

And what makes them think nested folders do not work?

-- 
Alice: If you would just shut up for about for two seconds, this sex dream
   would pass the Bechdel test.

Re: Changing IMAP separator - does it break things?

[Markus Winkler]

On 14.10.20 11:57, Victor Sudakov wrote:

Besides that: changing the hierarchy separator is generally possible. But
you schould try and check that with a test system to prevent problems with
your productive environment and all the other users.



I was hoping someone had already done that and I could skip this step :-)


No, I didn't. ;-) I almost always use '/' as the separator so I didn't 
observed any problem with Mac Mail users.


Regards,
Markus

Re: Changing IMAP separator - does it break things?

[Markus Winkler]

Hi Victor,

On 14.10.20 11:51, Victor Sudakov wrote:

Did it change anything on Dovecot disk/storage or is the IMAP separator just
"virtual" - ephemeral ?


just for the for the sake of completeness: there is a difference between 
the hierarchy separator (for the mailbox name) and the layout separator, 
which is responsible for the folder names on the disk - they are completely 
independent.


Here you can see the details, if you not already know this page:

https://doc.dovecot.org/configuration_manual/namespace/

Regards,
Markus

Re: Changing IMAP separator - does it break things?

[Sami Ketola]

> On 14. Oct 2020, at 12.51, Victor Sudakov  wrote:
> 
> Those Mac clients who do not use nested folders do not seem to be
> affected. That is, I've already suggested to the Mac Mail user that he
> start Thunderbird, move all his mailboxes into the IMAP root folder, and
> then go on using Mac Mail. I hope that's what he has done.

There is a bug in Mail.app that it always assumes / as hierarchy separator and 
completely ignores server provided separator.

Not sure if anyone has created BUG to Apple about this. Atleast I have not.

Sami

Re: Changing IMAP separator - does it break things?

[Victor Sudakov]

Filip Hajný wrote:
> > 
> >> I had to do it with 10-20 live users and it went fine, nobody noticed
> >> anything (I made the change in the night hours just to be sure
> >> though). I had to go down and fix some erratic folder names that
> >> resulted from the botched separator handling on some macOS Mail
> >> accounts.
> > 
> > Did it change anything on Dovecot disk/storage or is the IMAP separator just
> > "virtual" - ephemeral ?
> 
> Changing the separator won't fix anything that already exists. You should see 
> nonsense folders in the form of  '.folder\2esubfolder‘ there that cannot be 
> worked with in any way. I manually moved the contents of these to their 
> properly named ('.folder.subfolder‘) variants - on disk.

Thanks for the warning Filip!

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


signature.asc
Description: PGP signature

Re: Changing IMAP separator - does it break things?

[Victor Sudakov]

Markus Winkler wrote:
> 
> On 14.10.20 04:36, Victor Sudakov wrote:
> > Do you think I can change the hierarchy separator in the "inbox"
> > namespace without breaking other clients and the mail layout on disk,
> > losing mail etc?
> 
> I personally would try to avoid such a change (really IMHO), especially if
> it's "only":
> 
> > I don't want to break mail for 50 people just to please one Mac Mail
> > user.
> 
> _one_ user. Could Thunderbird or another IMAP client be an option for this
> single user? And perhaps Apple can tell some details when their Mail client
> will be standards-compliant?

Looking at https://discussions.apple.com/thread/8347018 I don't hold my breath.

> 
> Besides that: changing the hierarchy separator is generally possible. But
> you schould try and check that with a test system to prevent problems with
> your productive environment and all the other users.
> 

I was hoping someone had already done that and I could skip this step :-)

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


signature.asc
Description: PGP signature

Re: Changing IMAP separator - does it break things?

[Filip Hajný]

14. 10. 2020 v 11:51, Victor Sudakov :
> 
>> I had to do it with 10-20 live users and it went fine, nobody noticed
>> anything (I made the change in the night hours just to be sure
>> though). I had to go down and fix some erratic folder names that
>> resulted from the botched separator handling on some macOS Mail
>> accounts.
> 
> Did it change anything on Dovecot disk/storage or is the IMAP separator just
> "virtual" - ephemeral ?

Changing the separator won't fix anything that already exists. You should see 
nonsense folders in the form of  '.folder\2esubfolder‘ there that cannot be 
worked with in any way. I manually moved the contents of these to their 
properly named ('.folder.subfolder‘) variants - on disk.

-F

Re: Changing IMAP separator - does it break things?

[Victor Sudakov]

Filip Hajný wrote:
> > 
> > On 14.10.20 04:36, Victor Sudakov wrote:
> >> Do you think I can change the hierarchy separator in the "inbox"
> >> namespace without breaking other clients and the mail layout on disk,
> >> losing mail etc?
> > 
> > I personally would try to avoid such a change (really IMHO), especially if 
> > it's "only":
> > 
> >> I don't want to break mail for 50 people just to please one Mac Mail
> >> user.
> > 
> > _one_ user. Could Thunderbird or another IMAP client be an option for this 
> > single user? And perhaps Apple can tell some details when their Mail client 
> > will be standards-compliant?
> > 
> > Besides that: changing the hierarchy separator is generally possible. But 
> > you schould try and check that with a test system to prevent problems with 
> > your productive environment and all the other users.
> > 
> 
> I had to do it with 10-20 live users and it went fine, nobody noticed
> anything (I made the change in the night hours just to be sure
> though). I had to go down and fix some erratic folder names that
> resulted from the botched separator handling on some macOS Mail
> accounts.

Did it change anything on Dovecot disk/storage or is the IMAP separator just
"virtual" - ephemeral ?

> 
> The funny thing is that I set up a vanilla Dovecot server elsewhere
> and subfolders worked fine there on the very same macOS client.
> Couldn’t find any significant difference in the configs that would
> explain the problem, also the relative lack of similar reports out
> there seemed mind boggling.

Those Mac clients who do not use nested folders do not seem to be
affected. That is, I've already suggested to the Mac Mail user that he
start Thunderbird, move all his mailboxes into the IMAP root folder, and
then go on using Mac Mail. I hope that's what he has done.

> 
> (I’m using the Dovecot-based Mailcow mail stack for production, which
> makes troubleshooting a little more challenging.)

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


signature.asc
Description: PGP signature

Re: Changing IMAP separator - does it break things?

[Filip Hajný]

14. 10. 2020 v 11:28, Markus Winkler :
> 
> Hi Victor,
> 
> On 14.10.20 04:36, Victor Sudakov wrote:
>> Do you think I can change the hierarchy separator in the "inbox"
>> namespace without breaking other clients and the mail layout on disk,
>> losing mail etc?
> 
> I personally would try to avoid such a change (really IMHO), especially if 
> it's "only":
> 
>> I don't want to break mail for 50 people just to please one Mac Mail
>> user.
> 
> _one_ user. Could Thunderbird or another IMAP client be an option for this 
> single user? And perhaps Apple can tell some details when their Mail client 
> will be standards-compliant?
> 
> Besides that: changing the hierarchy separator is generally possible. But you 
> schould try and check that with a test system to prevent problems with your 
> productive environment and all the other users.
> 

I had to do it with 10-20 live users and it went fine, nobody noticed anything 
(I made the change in the night hours just to be sure though). I had to go down 
and fix some erratic folder names that resulted from the botched separator 
handling on some macOS Mail accounts.

The funny thing is that I set up a vanilla Dovecot server elsewhere and 
subfolders worked fine there on the very same macOS client. Couldn’t find any 
significant difference in the configs that would explain the problem, also the 
relative lack of similar reports out there seemed mind boggling.

(I’m using the Dovecot-based Mailcow mail stack for production, which makes 
troubleshooting a little more challenging.)

-F

Re: Changing IMAP separator - does it break things?

[Markus Winkler]

Hi Victor,

On 14.10.20 04:36, Victor Sudakov wrote:

Do you think I can change the hierarchy separator in the "inbox"
namespace without breaking other clients and the mail layout on disk,
losing mail etc?


I personally would try to avoid such a change (really IMHO), especially if 
it's "only":



I don't want to break mail for 50 people just to please one Mac Mail
user.


_one_ user. Could Thunderbird or another IMAP client be an option for this 
single user? And perhaps Apple can tell some details when their Mail client 
will be standards-compliant?


Besides that: changing the hierarchy separator is generally possible. But 
you schould try and check that with a test system to prevent problems with 
your productive environment and all the other users.


Regards,
Markus