Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Sure Adam, this makes perfect sense.

And having properly reviewed the "debian-release" mailing list
description, I can see that this is made quite clear:

> Coordinating Debian releases
> Coordination of Debian releases issues such as testing migrations,
> transitions and removals. This list should not be considered a
> discussion list; discussions related to releases issues should be
> held on more appropriate lists such as debian-devel, debian-legal or
> debian-project.

Best regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288


The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.

-Original Message-
From: Adam D. Barratt via dovecot 
Reply-To: Adam D. Barratt 
To: arnoldo...@parallaxict.com
Cc: t...@sirainen.com, debian-rele...@lists.debian.org, Dovecot Mailing
List 
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Fri, 12 Jul 2019 13:43:34 +0100

On Wed, 2019-07-10 at 12:26 +0100, Arnold Opio Oree wrote:
> Understood Adam,
> 
> My thinking is that this is a package released with Debian 10. And so
> has everything to do with the release.

For clarity here - debian-release is neither a support forum nor a
discussion list, nor a means of reporting issues in software contained
within a Debian release. Debian has support fora and a bug tracking
system to handle such reports and issues, as per 
https://www.debian.org

/support . Having a single list that discusses any issue in any piece
of software shipped by Debian cannot possibly scale.

Rather, this list is the contact point and team alias for the Release
Team, who oversee and manage the release process. It is up to
individual package maintainers to triage and deal with issues reported
against their packages and then to liaise with us if required.

Regards,

Adam


-- 

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Adam D. Barratt via dovecot]

On Wed, 2019-07-10 at 12:26 +0100, Arnold Opio Oree wrote:
> Understood Adam,
> 
> My thinking is that this is a package released with Debian 10. And so
> has everything to do with the release.

For clarity here - debian-release is neither a support forum nor a
discussion list, nor a means of reporting issues in software contained
within a Debian release. Debian has support fora and a bug tracking
system to handle such reports and issues, as per https://www.debian.org
/support . Having a single list that discusses any issue in any piece
of software shipped by Debian cannot possibly scale.

Rather, this list is the contact point and team alias for the Release
Team, who oversee and manage the release process. It is up to
individual package maintainers to triage and deal with issues reported
against their packages and then to liaise with us if required.

Regards,

Adam

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Hello Aki,

Thank you for your assurance.

It is really important to me to see that things are done well in the open 
source community. As there are many vested interests that would seek to have it 
otherwise.

And there are certain parts of the world, such as the place of my birth, 
Africa. Where Open Source really does hold the key to freedom from the worst 
forms of oppression and suffering known to man.

Sorry Adam, this will be my last cc on this topic. Just thought it was 
important to get this point to the Debian Release Management Team.

Warmest regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.




Understood Adam,

My thinking is that this is a package released with Debian 10. And so
has everything to do with the release.

I am new to the community, and have not yet studied the guidelines.

Regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.

-Original Message-
From: Adam D. Barratt 
To: arnoldo...@parallaxict.com
Cc: t...@sirainen.com, Dovecot Mailing List , 
debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 12:24:44 +0100

On 2019-07-10 12:01, Arnold Opio Oree wrote:

Please stop CCing this thread to the Debian Release Management mailing 
list.

If there are issues with the Dovecot packages in Debian then they
should 
be reported as bugs against the packages.

Thanks.

Adam
(Debian Release Team member)


-Original Message-
From: Aki Tuomi 
To: arnoldo...@parallaxict.com, t...@sirainen.com, Dovecot Mailing List

Cc: debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 14:13:53 +0300

Your report is most appreciated, though it lacked quite a lot of detail
that could've expedited the investigation why it happened. With the
further information we can hopefully figure out why you are
experiencing
the problems you reported. There is no offense intended.

Aki

On 10.7.2019 14.01, Arnold Opio Oree via dovecot wrote:
> Hello Timo,
> 
> I have scheduled some time to carry through the test requested by
> Aki.
> 
> I will add responding to you as part of that process.
> 
> Not sure if I have said or done anything to offend you, as your tone
> appears quite abrasive, even where you do not have an actual response
> to an observation.
> 
> It is my view that it is a privilege to provide an open source
> platform, and therefore a responsibility wherever feasible not only
> to properly maintain and support it, but also to properly document
> it; both of which from my experience and findings to date, Dovecot
> have failed to do.
> 
> Kind regards,
> 
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
> 
> arnoldo...@parallaxdt.com
> 
> 
> http://www.parallaxdt.com
> 
> 
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
> 
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
> 
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately,
> disregard
> its contents and disclose them to no other persons.
> 
> 
> -Original Message-
> From: Timo Sirainen via dovecot <
> dovecot@dovecot.org
> >
> Reply-To: Timo Sirainen <
> t...@sirainen.com
> >
> To: Arnold Opio Oree <
> arnold.o...@parallaxict.com
> >
> Cc: Dovecot Mailing List <
> dovecot@dovecot.org
> >
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive - BUG REPORTS!
> Date: Wed, 10 Jul 2019 11:09:57 +0300
> 
> On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <
> dovecot@dovecot.org
> > wrote:
> > > Dovecot Team,
> > > 
> > > I'd like to report a number of bugs, that are to my view all
> > > critical.
> 
> It would help to get your doveconf -n, example command lines c

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Understood Adam,
My thinking is that this is a package released with Debian 10. And so
has everything to do with the release.
I am new to the community, and have not yet studied the guidelines.
Regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.
-Original Message-
From: Adam D. Barratt 
To: arnoldo...@parallaxict.com
Cc: t...@sirainen.com, Dovecot Mailing List , 
debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 12:24:44 +0100

On 2019-07-10 12:01, Arnold Opio Oree wrote:
Please stop CCing this thread to the Debian Release Management mailing
list.
If there are issues with the Dovecot packages in Debian then they
should be reported as bugs against the packages.
Thanks.
Adam(Debian Release Team member)

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Adam D. Barratt via dovecot]

On 2019-07-10 12:01, Arnold Opio Oree wrote:

Please stop CCing this thread to the Debian Release Management mailing 
list.


If there are issues with the Dovecot packages in Debian then they should 
be reported as bugs against the packages.


Thanks.

Adam
(Debian Release Team member)

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Aki Tuomi via dovecot]

Your report is most appreciated, though it lacked quite a lot of detail
that could've expedited the investigation why it happened. With the
further information we can hopefully figure out why you are experiencing
the problems you reported. There is no offense intended.

Aki

On 10.7.2019 14.01, Arnold Opio Oree via dovecot wrote:
> Hello Timo,
>
> I have scheduled some time to carry through the test requested by Aki.
>
> I will add responding to you as part of that process.
>
> Not sure if I have said or done anything to offend you, as your tone
> appears quite abrasive, even where you do not have an actual response
> to an observation.
>
> It is my view that it is a privilege to provide an open source
> platform, and therefore a responsibility wherever feasible not only to 
> properly maintain and support it, but also to properly document it; both of 
> which from my experience and findings to date, Dovecot have failed to do.
>
> Kind regards,
>
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
>
> arnoldo...@parallaxdt.com
>
> http://www.parallaxdt.com
>
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
>
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
>
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately, disregard
> its contents and disclose them to no other persons.
>
>
> -Original Message-
> From: Timo Sirainen via dovecot 
> Reply-To: Timo Sirainen 
> To: Arnold Opio Oree 
> Cc: Dovecot Mailing List 
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive - BUG REPORTS!
> Date: Wed, 10 Jul 2019 11:09:57 +0300
>
> On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <
> dovecot@dovecot.org> wrote:
>>> Dovecot Team,
>>>
>>> I'd like to report a number of bugs, that are to my view all
>>> critical.
> It would help to get your doveconf -n, example command lines causing
> the problems and the error messages it outputs or what the wrong
> behavior looks like in filesystem. It's now rather difficult to guess
> what exactly you tried and what happened.
>
> Also what kind of output does readpst make? I'm not sure why you're
> using DIRNAMEs here.
>
>>> doveadm-sync -1/general
>>>
>>> 1) If DIRNAMEs are not different between command line and
>>> mail_location doveadm sync will fail, saying that the source and
>>> destination directories are the same
> This sounds very strange. I'm not sure what exactly you did, and I
> couldn't reproduce with a small test.
>
>>> 2) The -n / -N flags do not work, and a sync will fail strangely if
>>> location is specified in the namespace definition
> Again, sounds strange.
>
>>> 3) Adds mbox to path name under mailbox directory (where syncing
>> from
>>> an mbox source)
> Probably with different parameters you could avoid it.
>
>>> 4) Not having the mailboxes at source named the same as those at
>>> destination causes errors and partial sync 
>>>
>>> 5) Not having the target mailboxes formatted to receive the sync
>>> (//DIRNAME/) will cause sync errors.
> I don't understand these. Target mailboxes are supposed to be empty
> initially, and after the initial sync they should be in the expected
> format. Why would they be different?
>
>>> doveadm-sync
>>>
>>> 1) With large synchronizations UIDs are corrupted where multiple
>>> syncs are executed and the program can no longer synchronize
> What exactly is the error message?
>
>>> dovecot
>>>
>>> 1) Panics and fails to expand ~ to user home: observed cases are
>>> where multiple namespaces are being used
> Panic message and more details would also help.
>
>>> With regards to the last error that I requested help on i.e.
>>> \Noselect. This has been resolved more-or-less by the workarounds
>>> that I have implemented for the bugs reported above.
>>>
>>> I have seen a number of threads whilst researching the \Noselect
>>> issue where people have been very confused. My finding was that
>>> \Noselect is a function of the IMAP specification server-side
>>> implementation RFC3501 (
>>> https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
>>> server was returning directories with \Noselect because the
>> mailboxes
>>> were malformed on account of dovadm-sync errors. In order to fix
>> this
>>> I formed a bash command to transverse the mailbox hierarchy and
>&

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Hello Timo,

I have scheduled some time to carry through the test requested by Aki.

I will add responding to you as part of that process.

Not sure if I have said or done anything to offend you, as your tone
appears quite abrasive, even where you do not have an actual response
to an observation.

It is my view that it is a privilege to provide an open source
platform, and therefore a responsibility wherever feasible not only to properly 
maintain and support it, but also to properly document it; both of which from 
my experience and findings to date, Dovecot have failed to do.

Kind regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.


-Original Message-
From: Timo Sirainen via dovecot 
Reply-To: Timo Sirainen 
To: Arnold Opio Oree 
Cc: Dovecot Mailing List 
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 11:09:57 +0300

On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <
dovecot@dovecot.org> wrote:
> > Dovecot Team,
> > 
> > I'd like to report a number of bugs, that are to my view all
> > critical.

It would help to get your doveconf -n, example command lines causing
the problems and the error messages it outputs or what the wrong
behavior looks like in filesystem. It's now rather difficult to guess
what exactly you tried and what happened.

Also what kind of output does readpst make? I'm not sure why you're
using DIRNAMEs here.

> > doveadm-sync -1/general
> > 
> > 1) If DIRNAMEs are not different between command line and
> > mail_location doveadm sync will fail, saying that the source and
> > destination directories are the same

This sounds very strange. I'm not sure what exactly you did, and I
couldn't reproduce with a small test.

> > 2) The -n / -N flags do not work, and a sync will fail strangely if
> > location is specified in the namespace definition

Again, sounds strange.

> > 3) Adds mbox to path name under mailbox directory (where syncing
> from
> > an mbox source)

Probably with different parameters you could avoid it.

> > 4) Not having the mailboxes at source named the same as those at
> > destination causes errors and partial sync 
> > 
> > 5) Not having the target mailboxes formatted to receive the sync
> > (//DIRNAME/) will cause sync errors.

I don't understand these. Target mailboxes are supposed to be empty
initially, and after the initial sync they should be in the expected
format. Why would they be different?

> > doveadm-sync
> > 
> > 1) With large synchronizations UIDs are corrupted where multiple
> > syncs are executed and the program can no longer synchronize

What exactly is the error message?

> > dovecot
> > 
> > 1) Panics and fails to expand ~ to user home: observed cases are
> > where multiple namespaces are being used

Panic message and more details would also help.

> > With regards to the last error that I requested help on i.e.
> > \Noselect. This has been resolved more-or-less by the workarounds
> > that I have implemented for the bugs reported above.
> > 
> > I have seen a number of threads whilst researching the \Noselect
> > issue where people have been very confused. My finding was that
> > \Noselect is a function of the IMAP specification server-side
> > implementation RFC3501 (
> > https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
> > server was returning directories with \Noselect because the
> mailboxes
> > were malformed on account of dovadm-sync errors. In order to fix
> this
> > I formed a bash command to transverse the mailbox hierarchy and
> > create the missing folders critical to the sdbox format, namely
> > DIRNAME.

Nowadays we have also an option to disable creation of \Noselect
folders, because they confuse people. Using mail_location = ...:NO-
NOSELECT - It won't affect existing folders immediately though.

-----Original Message-
From: Arnold Opio Oree 
Reply-To: arnoldo...@parallaxict.com
To: Aki Tuomi , Dovecot Mailing List <
dovecot@dovecot.org>
Cc: debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Mon, 08 Jul 2019 13:21:51 +0100

Hello Aki,

Thanks for looking into these.

I will as requested attempt the relevant procedures under Dovecot
2.3.6.

To make the test fair, I will need to fork the rel

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Timo Sirainen via dovecot]

On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot  
wrote:
> 
> Dovecot Team,
> 
> I'd like to report a number of bugs, that are to my view all critical.

It would help to get your doveconf -n, example command lines causing the 
problems and the error messages it outputs or what the wrong behavior looks 
like in filesystem. It's now rather difficult to guess what exactly you tried 
and what happened.

Also what kind of output does readpst make? I'm not sure why you're using 
DIRNAMEs here.

> doveadm-sync -1/general
> 
> 1) If DIRNAMEs are not different between command line and mail_location 
> doveadm sync will fail, saying that the source and destination directories 
> are the same

This sounds very strange. I'm not sure what exactly you did, and I couldn't 
reproduce with a small test.

> 2) The -n / -N flags do not work, and a sync will fail strangely if location 
> is specified in the namespace definition

Again, sounds strange.

> 3) Adds mbox to path name under mailbox directory (where syncing from an mbox 
> source)

Probably with different parameters you could avoid it.

> 4) Not having the mailboxes at source named the same as those at destination 
> causes errors and partial sync 
> 
> 5) Not having the target mailboxes formatted to receive the sync 
> (//DIRNAME/) will cause sync errors.

I don't understand these. Target mailboxes are supposed to be empty initially, 
and after the initial sync they should be in the expected format. Why would 
they be different?

> doveadm-sync
> 
> 1) With large synchronizations UIDs are corrupted where multiple syncs are 
> executed and the program can no longer synchronize

What exactly is the error message?

> dovecot
> 
> 1) Panics and fails to expand ~ to user home: observed cases are where 
> multiple namespaces are being used

Panic message and more details would also help.

> With regards to the last error that I requested help on i.e. \Noselect. This 
> has been resolved more-or-less by the workarounds that I have implemented for 
> the bugs reported above.
> 
> I have seen a number of threads whilst researching the \Noselect issue where 
> people have been very confused. My finding was that \Noselect is a function 
> of the IMAP specification server-side implementation RFC3501 ( 
> https://tools.ietf.org/html/rfc3501#section-6.3.
>  
> 6).
>  And for me the server was returning directories with \Noselect because the 
> mailboxes were malformed on account of dovadm-sync errors. In order to fix 
> this I formed a bash command to transverse the mailbox hierarchy and create 
> the missing folders critical to the sdbox format, namely DIRNAME.

Nowadays we have also an option to disable creation of \Noselect folders, 
because they confuse people. Using mail_location = ...:NO-NOSELECT - It won't 
affect existing folders immediately though.

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Hello Aki,

Thanks for looking into these.

I will as requested attempt the relevant procedures under Dovecot
2.3.6.

To make the test fair, I will need to fork the relevant production
groupware stack (which is now stable and in operation, with our
enterprise (email) data successfully migrated from Microsoft Exchange)
to a new staging server; given that the current staging server is now
of a materially different configuration to the production server (where
the most controlled observations of these bugs were made).

Kindly give me some time, as I have an urgent internal openstack
deployment project to kick-off, that has been delayed by an overrun of
this internal groupware stack deployment project.

Best regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.


-Original Message-
From: Aki Tuomi via dovecot 
Reply-To: Aki Tuomi 
To: arnoldo...@parallaxict.com, Arnold Opio Oree <
arnold.o...@parallaxict.com>, Arnold Opio Oree via dovecot <
dovecot@dovecot.org>
Cc: debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Mon, 8 Jul 2019 08:48:08 +0300 (EEST)

Hi!

Thank you for reporting these. We will look into them. In the mean
time, can you see if any of these are fixed in 2.3.6?

Aki
> On 07/07/2019 18:12 Arnold Opio Oree via dovecot  > wrote:
> 
> 
> Dovecot Team,
> 
> I'd like to report a number of bugs, that are to my view all
> critical.
> 
> System: Replicated on multiple Debian 10 (Buster) systems
> Dovecot Version(s): 2.3.4.1
> 
> doveadm-sync -1/general
> 
> 1) If DIRNAMEs are not different between command line and
> mail_location doveadm sync will fail, saying that the source and
> destination directories are the same
> 
> 2) The -n / -N flags do not work, and a sync will fail strangely if
> location is specified in the namespace definition
> 
> 3) Adds mbox to path name under mailbox directory (where syncing from
> an mbox source)
> 
> 4) Not having the mailboxes at source named the same as those at
> destination causes errors and partial sync 
> 
> 5) Not having the target mailboxes formatted to receive the sync
> (//DIRNAME/) will cause sync errors.
> 
> doveadm-sync
> 
> 1) With large synchronizations UIDs are corrupted where multiple
> syncs are executed and the program can no longer synchronize
> 
> dovecot
> 
> 1) Panics and fails to expand ~ to user home: observed cases are
> where multiple namespaces are being used
> 
> Please let me know if you need me to elaborate or to provide any
> further information that you may need to replicate the bugs, or if I
> can help in any other way.
> 
> With regards to the last error that I requested help on i.e.
> \Noselect. This has been resolved more-or-less by the workarounds
> that I have implemented for the bugs reported above.
> 
> I have seen a number of threads whilst researching the \Noselect
> issue where people have been very confused. My finding was that
> \Noselect is a function of the IMAP specification server-side
> implementation RFC3501 ( 
> https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
> server was returning directories with \Noselect because the mailboxes
> were malformed on account of dovadm-sync errors. In order to fix this
> I formed a bash command to transverse the mailbox hierarchy and
> create the missing folders critical to the sdbox format, namely
> DIRNAME.
> 
> Kind regards,
> 
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
> 
> arnoldo...@parallaxdt.com
> 
> http://www.parallaxdt.com
> 
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
> 
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
> 
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately,
> disregard its contents and disclose them to no other persons.
> 
> -Original Message-
> From: Arnold Opio Oree via dovecot < dovecot@dovecot.org>
> Reply-To: arnoldo...@parallaxict.com, Arnold Opio Oree < 
> arnold.o...@parallaxict.com>
> To: dovecot@dovecot.org
> Cc: r...@sys4.de, aki.tu...@open-xchange.com
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive.
> Date: Thu, 04 Jul 2019

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Aki Tuomi via dovecot]

 
 
  
   Hi!
  
  
   
  
  
   Thank you for reporting these. We will look into them. In the mean time, can you see if any of these are fixed in 2.3.6?
  
  
   
  
  
   Aki
  
  
   
On 07/07/2019 18:12 Arnold Opio Oree via dovecot  wrote:
   
   

   
   

   
   
Dovecot Team,
   
   

   
   
I'd like to report a number of bugs, that are to my view all critical.
   
   

   
   
System: Replicated on multiple Debian 10 (Buster) systems
   
   
Dovecot Version(s): 2.3.4.1
   
   

   
   
doveadm-sync -1/general
   
   

   
   
1) If DIRNAMEs are not different between command line and mail_location doveadm sync will fail, saying that the source and destination directories are the same
   
   

   
   
2) The -n / -N flags do not work, and a sync will fail strangely if location is specified in the namespace definition
   
   

   
   
3) Adds mbox to path name under mailbox directory (where syncing from an mbox source)
   
   

   
   
4) Not having the mailboxes at source named the same as those at destination causes errors and partial sync 
   
   

   
   
5) Not having the target mailboxes formatted to receive the sync (//DIRNAME/) will cause sync errors.
   
   

   
   
doveadm-sync
   
   

   
   
1) With large synchronizations UIDs are corrupted where multiple syncs are executed and the program can no longer synchronize
   
   

   
   
dovecot
   
   

   
   
1) Panics and fails to expand ~ to user home: observed cases are where multiple namespaces are being used
   
   

   
   
Please let me know if you need me to elaborate or to provide any further information that you may need to replicate the bugs, or if I can help in any other way.
   
   

   
   
With regards to the last error that I requested help on i.e. \Noselect. This has been resolved more-or-less by the workarounds that I have implemented for the bugs reported above.
   
   

   
   
I have seen a number of threads whilst researching the \Noselect issue where people have been very confused. My finding was that \Noselect is a function of the IMAP specification server-side implementation RFC3501 (
https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the server was returning directories with \Noselect because the mailboxes were malformed on account of dovadm-sync errors. In order to fix this I formed a bash command to transverse the mailbox hierarchy and create the missing folders critical to the sdbox format, namely DIRNAME.
   
   

   
   
Kind regards,
   
   

   
   
Arnold Opio Oree
   
   

 Chief Executive Officer


 Parallax Digital Technologies


 


 arnoldo...@parallaxdt.com


 


 http://www.parallaxdt.com


 


 tel : +44 (0) 333 577 8587


 fax : +44 (0) 20 8711 2477


 


 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288


 


 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons.

   
   

   
   
-Original Message-
   
   
From: Arnold Opio Oree via dovecot <
dovecot@dovecot.org>
   
   
Reply-To: 
arnoldo...@parallaxict.com, Arnold Opio Oree <
arnold.o...@parallaxict.com>
   
   
To: 
dovecot@dovecot.org
   
   
Cc: 
r...@sys4.de, 
aki.tu...@open-xchange.com
   
   
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive.
   
   
Date: Thu, 04 Jul 2019 14:52:28 +0100
   
   

   
   

   
   
Hi all,
   
   

   
   
The guidance provided so far has been really helpful, and has helped a great deal to bringing down wasted energy on finding and executing a viable path. I am now at the final due action to complete our Dovecot application to our use-case, but am stuck on an issue that I cannot find any easily accessible documentation on.
   
   

   
   
Generally this is what has been done:
   
   

   
   
1. Uploaded the enterprise data PST to the target groupware server.
   
   
2. Prepared the server by changing the mailbox format to sdbox and the the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/
   
   
3. Converted the pst (on-server) to a recursive mbox hierarchy using readpst
   
   
4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and to copy it into the enterprise archive user's mailboxes
   
   
4.i. The biggest issue I faced at this point was doveadm-sync saying that the source and destination pointed to the same location, whereas they clearly did not. 
   
   
4.i.a. I resolved this by removing the location= setting from 

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Dovecot Team,

I'd like to report a number of bugs, that are to my view all critical.

System: Replicated on multiple Debian 10 (Buster) systems
Dovecot Version(s): 2.3.4.1

doveadm-sync -1/general

1) If DIRNAMEs are not different between command line and mail_location
doveadm sync will fail, saying that the source and destination
directories are the same

2) The -n / -N flags do not work, and a sync will fail strangely if
location is specified in the namespace definition

3) Adds mbox to path name under mailbox directory (where syncing from
an mbox source)

4) Not having the mailboxes at source named the same as those at
destination causes errors and partial sync 

5) Not having the target mailboxes  formatted to receive the sync
(//DIRNAME/) will cause sync errors.

doveadm-sync

1) With large synchronizations UIDs are corrupted where multiple syncs
are executed and the program can no longer synchronize

dovecot

1) Panics and fails to expand ~ to user home: observed cases are where
multiple namespaces are being used

Please let me know if you need me to elaborate or to provide any
further information that you may need to replicate the bugs, or if I
can help in any other way.

With regards to the last error that I requested help on i.e.
\Noselect.  This has been resolved more-or-less by the workarounds that
I have implemented for the bugs reported above.

I have seen a number of threads whilst researching the \Noselect issue
where people have been very confused. My finding was that \Noselect is
a function of the IMAP specification server-side implementation RFC3501
(https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
server was returning directories with \Noselect because the mailboxes
were malformed on account of dovadm-sync errors. In order to fix this I
formed a bash command to transverse the mailbox hierarchy and create
the missing folders critical to the sdbox format, namely DIRNAME.

Kind regards,

Arnold Opio Oree

-Original Message-
From: Arnold Opio Oree via dovecot 
Reply-To: arnoldo...@parallaxict.com, Arnold Opio Oree <
arnold.o...@parallaxict.com>
To: dovecot@dovecot.org
Cc: r...@sys4.de, aki.tu...@open-xchange.com
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive.
Date: Thu, 04 Jul 2019 14:52:28 +0100


Hi all,

The guidance provided so far has been really helpful, and has helped a
great deal to bringing down wasted energy on finding and executing a
viable path. I am now at the final due action to complete our Dovecot
application to our use-case, but am stuck on an issue that I cannot
find any easily accessible documentation on.

Generally this is what has been done:

1. Uploaded the enterprise data PST to the target groupware server.
2. Prepared the server by changing the mailbox format to sdbox and the
the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/
3. Converted the pst (on-server) to a recursive mbox hierarchy using
readpst
4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and
to copy it into the enterprise archive user's mailboxes
4.i. The biggest issue I faced at this point was doveadm-sync saying
that the source and destination pointed to the same location, whereas
they clearly did not. 
4.i.a. I resolved this by removing the location= setting from the
target namespace, and allowing it to default to mail_location =
setting, and then using a completely different DIRNAME for the import
doveadm-sync execution (which was the desired final DIRNAME); I then
once the sync had been successful, changed the mail_location DIRNAME so
that it pointed to the imported mail DIRNAME; and hence the imported
email data was in the live mailboxes
4.i.b. doveadm-import failed several times, and was throwing quite
inexplicable errors, so I moved onto doveadm-sync
4.i.c. I also had to make sure that the source and destination folder
names matched, otherwise doveadm-syc threw very many errors and only
partially imported the data
4.i.d. An issue which I decided just to live with is that an mbox
DIRNAME was added to each mailbox as well as the DIRNAME specified so
the path to mail is mbox/dbox-Mails. My thought is that with the data
live on an IMAP server it will be possible to do a dysync through TCP
to correct this problem.

The final issue that I am facing now, is that when readpst finds empty
folders in the source pst hierarchy, it does not create an mbox file in
the mbox hierarchy folder space. This causes doveadm-sync to not create
the target data required for its mailbox structure i.e. DIRNAME sub-
folder and index file (with our configuration). At this point either
doveadm-sync or the dovecot process makes these empty folders not
selectable.

The question now is how would I go about making all of these folders
selectable, e.g. with an internal or external command line tool to
change flags / create necessary sdbox mailbox constituent data?

Many thanks,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital 

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Dovecot Team,
I'd like to report a number of bugs, that are to my view all critical.
System: Replicated on multiple Debian 10 (Buster) systemsDovecot
Version(s): 2.3.4.1
doveadm-sync -1/general
1) If DIRNAMEs are not different between command line and mail_location
doveadm sync will fail, saying that the source and destination
directories are the same
2) The -n / -N flags do not work, and a sync will fail strangely if
location is specified in the namespace definition
3) Adds mbox to path name under mailbox directory (where syncing from
an mbox source)
4) Not having the mailboxes at source named the same as those at
destination causes errors and partial sync 
5) Not having the target mailboxes  formatted to receive the sync
(//DIRNAME/) will cause sync errors.
doveadm-sync
1) With large synchronizations UIDs are corrupted where multiple syncs
are executed and the program can no longer synchronize
dovecot
1) Panics and fails to expand ~ to user home: observed cases are where
multiple namespaces are being used
Please let me know if you need me to elaborate or to provide any
further information that you may need to replicate the bugs, or if I
can help in any other way.
With regards to the last error that I requested help on i.e.
\Noselect.  This has been resolved more-or-less by the workarounds that
I have implemented for the bugs reported above.
I have seen a number of threads whilst researching the \Noselect issue
where people have been very confused. My finding was that \Noselect is
a function of the IMAP specification server-side implementation RFC3501
(https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
server was returning directories with \Noselect because the mailboxes
were malformed on account of dovadm-sync errors. In order to fix this I
formed a bash command to transverse the mailbox hierarchy and create
the missing folders critical to the sdbox format, namely DIRNAME.
Kind regards,
Arnold Opio Oree
-Original Message-From: Arnold Opio Oree via dovecot <
dovecot@dovecot.org>Reply-To: arnoldo...@parallaxict.com, Arnold Opio
Oree To: dovecot@dovecot.orgCc: r...@sys4.de
, aki.tuomi@open-xchange.comSubject: Re: Applying Dovecot for a large /
deep folder-hierarchy archive.Date: Thu, 04 Jul 2019 14:52:28 +0100

Hi all,
The guidance provided so far has been really helpful, and has helped a
great deal to bringing down wasted energy on finding and executing a
viable path. I am now at the final due action to complete our Dovecot
application to our use-case, but am stuck on an issue that I cannot
find any easily accessible documentation on.
Generally this is what has been done:
1. Uploaded the enterprise data PST to the target groupware server.2.
Prepared the server by changing the mailbox format to sdbox and the the
Dovecot mail location to mail_location=/var/vmail/domain/user/mail/3.
Converted the pst (on-server) to a recursive mbox hierarchy using
readpst4. Executed doveadm-sync to convert mbox hierarchy data into
sdbox and to copy it into the enterprise archive user's mailboxes4.i.
The biggest issue I faced at this point was doveadm-sync saying that
the source and destination pointed to the same location, whereas they
clearly did not. 4.i.a. I resolved this by removing the location=
setting from the target namespace, and allowing it to default to
mail_location = setting, and then using a completely different DIRNAME
for the import doveadm-sync execution (which was the desired final
DIRNAME); I then once the sync had been successful, changed the
mail_location DIRNAME so that it pointed to the imported mail DIRNAME;
and hence the imported email data was in the live mailboxes4.i.b.
doveadm-import failed several times, and was throwing quite
inexplicable errors, so I moved onto doveadm-sync4.i.c. I also had to
make sure that the source and destination folder names matched,
otherwise doveadm-syc threw very many errors and only partially
imported the data4.i.d. An issue which I decided just to live with is
that an mbox DIRNAME was added to each mailbox as well as the DIRNAME
specified so the path to mail is mbox/dbox-Mails. My thought is that
with the data live on an IMAP server it will be possible to do a dysync
through TCP to correct this problem.
The final issue that I am facing now, is that when readpst finds empty
folders in the source pst hierarchy, it does not create an mbox file in
the mbox hierarchy folder space. This causes doveadm-sync to not create
the target data required for its mailbox structure i.e. DIRNAME sub-
folder and index file (with our configuration). At this point either
doveadm-sync or the dovecot process makes these empty folders not
selectable.
The question now is how would I go about making all of these folders
selectable, e.g. with an internal or external command line tool to
change flags / create necessary sdbox mailbox constituent data?
Many thanks,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com