Re: Running a script with cyradm throwing ReadLine errors
Dear ellie, > I did a bit of reading, and apparently Term::ReadLine is a
stub module that just loads "an implementation", which in your case
wants to be Term::ReadLine::Gnu. My guess is that, when you uninstall
Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles
because it's missing an implementation, and consequently the fallback
code I pointed out previously is used instead. So, from this I'm
concluding that the "correct setup" from above is adequate for the
Cyrus::IMAP::DummyReadline interface, but is not sufficient for a real
ReadLine implementation. Sounds like we've found our bug!
the more I thought about it, the clearer it got. I do not think any more
that the *real* issue is which stub Term::ReadLine uses.
Different stubs might react differently when fed with undefined file
handles, but this is only a distracting secondary issue. The real
culprit is how the run function is implemented.
Let's consider the original code for that function again:
# trivial; wrapper for _run with correct setup
sub run {
my $cyradm;
_run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
}
How should *__DATA__ have become a handle to the desired file (which
should be executed) in any way? There is absolutely no parameter
parsing, and after having researched what special meaning __DATA__ has,
it became also clear that *__DATA__ isn't mysteriously assigned a
reasonable value before run() is called.
So I made some very trivial changes. The function now reads:
# trivial; wrapper for _run with correct setup
sub run {
my ($cyradm, $fh);
my $file = shift;
defined $file || die "No filename given, aborting.\n";
open($fh, $file) || die "Could not open file '$file', aborting.\n";
_run(\$cyradm, [*STDIN, *STDOUT, *STDERR], $fh);
}
Now the whole thing works as expected, regardless of what stub modules
are installed for Term::ReadLine.
We could improve that code further; for example, it lacks a check if
there is the right number of parameters (additional parameters are
currently just ignored). Personally, I wouldn't need detailed checks; I
just want it to execute that script file, avoiding ugly error messages
from Perl itself relating to undefined values and so on.
At a first glance, I couldn't see how the new code could be incompatible
to the existing version. At least, there are no other calls to run() in
that module (only to _run() which I didn't alter). I am quite sure that
you have a bunch regression tests for all your modules, so let's see
what they reveal.
I am looking forward to your comments ...
Thank you very much again!
Regards,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Running a script with cyradm throwing ReadLine errors
ason was that I did my first tests with an *empty* script.
However, now that my script is meaningful, I noticed that it did not get
executed at all even if Term::ReadLine::Gnu was not installed. In other
words, when I uninstalled Term::ReadLine::Gnu again and ran
perl -MCyrus::IMAP::Shell -e 'run("./000")'
the script "000" was *not* executed when the original version of
Cyrus::IMAP::Shell was in place. I didn't notice this from the beginning
on because I did my first tests with an empty script, and no errors were
thrown.
When I changed the module's code and assigned *__DATA__ a handle to the
file desired (as shown in my previous post), the script was executed.
To put it all together:
---
- Cyrus::IMAP::DummyReadLine is never used if the Perl distribution is
recent (i.e. already includes Term::ReadLine). This applies whether
Term::ReadLine::Gnu is installed or not.
- _run() is always called with undef as third parameter. If
Term::ReadLine::Gnu is installed, errors are thrown if we execute perl
-MCyrus::IMAP::Shell -e 'run("./000")' in a terminal. No errors are
thrown if Term::ReadLine::Gnu is not installed. The errors
Term::ReadLine::Gnu throws are clearly due to the fact that the third
argument to _run() is undef (the error thrown reads "Bad filehandle:
__DATA__ at ...").
- With the original version of Cyrus::IMAP::Shell, even when
Term::ReadLine::Gnu is not installed, the script is not executed
when we issue perl -MCyrus::IMAP::Shell -e 'run("")' in a terminal.
- If we assign *__DATA__ the filehandle to the desired script in the
run() function before calling _run(), the desired script does get
executed when we issue perl -MCyrus::IMAP::Shell -e 'run("./000")'. This
works whether Term::ReadLine::Gnu is installed or not.
Hence, the problem is clearly related to Cyrus::IMAP::Shell. It does
either just not execute the script given (if Term::ReadLine::Gnu is not
installed), or it makes Term::ReadLine::Gnu throw errors (if it is
installed) due to the uninitialized file handle.
It would be very nice if somebody could fix that bug or correct
Cyrus::IMAP::Shell's man page accordingly.
Fortunately, there is a workaround. We could make cyradm execute a
script directly with no problem:
cat 000 | cyradm
Shame on me that I didn't see this earlier - it would have solved my
problem and would have saved me a whole day or two. I guess
Cyrus::IMAP::Shell's man page had distracted me too much...
However, I'd still be strongly interested in a bug fix for the module.
In fact, I already have a clean solution in mind, but I'd like to test
it before posting it here. I'll report back in a few hours.
Thank you very much again!
Regards,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Running a script with cyradm throwing ReadLine errors
Dear ellie,
On 19.12.2018 01:38, ellie timoney wrote:
>> Then I have replaced the following code in Cyrus::IMAP::Shell
>
> That's very interesting. Does the same modified code continue to work if you
> uninstall Term::Readline::Gnu again? That is to say, does the non-gnu
> version break with that addition, or continue to work?
I have just done that test: Yes, the same modified code continues to
work even if Term::ReadLine::Gnu is uninstalled, i.e. my "patch" does
not break the non-gnu version.
>> In other words, I just have made sure that this mysterious *__DATA__
>> variable is reasonably defined in every case before _run is called.
>
> I had a look in Shell.pm and found this comment near the top:
>
>> # run(*FH|'FH')
>> # read commands from the filehandle and pass to exec(); defaults to
>> # __DATA__
I also had seen this comment, but couldn't make any sense from it.
> So maybe that explains where the expectation for __DATA__ is coming from...
> so:
>
>> # trivial; wrapper for _run with correct setup
>
> I wonder if the "correct setup" is not correct enough!
There are many aspects I didn't understand yet. To me, it seems that
_run is called with a bunch of uninitialized parameters. For example,
where are $cyradm and *__DATA__ initialized? I am currently lacking the
time to do my homework (i.e. to completely understand how this is
supposed to work under normal circumstances), so I don't want to let
other persons waste their time for explaining it to me ...
However, despite the fact that I haven't grasped the overall concept
yet, there is obviously a bug with parsing the command line.
>> I have no idea why the "buggy" command line / argument parsing does not
>> strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet
>> how *__DATA__ is supposed to be assigned a reasonable value to during
>> the normal course of execution. I currently can only speculate that
>> Term::ReadLine:: does this for us, while
>> Term::ReadLine::Gnu doesn't.
>
> I did a bit of reading, and apparently Term::ReadLine is a stub module that
> just loads "an implementation", which in your case wants to be
> Term::ReadLine::Gnu. My guess is that, when you uninstall
> Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles because
> it's missing an implementation, and consequently the fallback code I pointed
> out previously is used instead. So, from this I'm concluding that the
> "correct setup" from above is adequate for the Cyrus::IMAP::DummyReadline
> interface, but is not sufficient for a real ReadLine implementation. Sounds
> like we've found our bug!
I have come to a similar conclusion, and "not sufficient" in this case
probably means that *__DATA__ is not initialized (or assigned to)
correctly. I still have no idea which part of the program is responsible
to assign it the desired file descriptor under normal circumstances.
Possibly Cyrus::IMAP::DummyReadLine does initialize *__DATA__ correctly
(because that module knows who it belongs to :-) and what is needed
later), while Term::ReadLine::Gnu can't know about *__DATA__'s existence
at all. But this is just a completely uneducated guess.
> I'll have a bit of a play with it and see if I can find/fix the discrepancy
> between the interfaces :)
I'll try to free some time and eventually have a look into
Cyrus::IMAP::DummyReadLine. I think we'll have to find out where
*__DATA__ is normally initialized, and move that initialization to
another place so that it happens regardless of the actual ReadLine "plugin".
> Cheers,
Again, thank you very much for all your help and your support!
Binarus
> ellie
>
> On Wed, Dec 19, 2018, at 5:00 AM, Binarus wrote:
>> Dear ellie,
>>
>> On 17.12.2018 23:57, ellie timoney wrote:
>>> Hi Binarus,
>>>
>>>> Could anybody please tell me what I might do wrong here?
>>>
>>> This kind of smells like maybe your system has two versions of perl
>>> installed (or two versions of Term::ReadLine, or maybe even two versions of
>>> Cyrus::IMAP::Shell), and they're getting in each other's way?
>>>
>>> I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and
>>> this caught my eye:
>>>
>>>> # ugh. ugh. suck. aieee.
>>>> my $use_rl = 'Cyrus::IMAP::DummyReadline';
>>>> {
>>>> if (eval { require Term::ReadLine; }) {
>>>> $use_rl = 'Term::ReadLine';
>>>> }
>>>> }
>>
>> I have done some further investigations (very roughly because I don't
>> have the time at the moment). It seems that the code which pars
Re: Running a script with cyradm throwing ReadLine errors
Dear ellie,
On 17.12.2018 23:57, ellie timoney wrote:
> Hi Binarus,
>
>> Could anybody please tell me what I might do wrong here?
>
> This kind of smells like maybe your system has two versions of perl installed
> (or two versions of Term::ReadLine, or maybe even two versions of
> Cyrus::IMAP::Shell), and they're getting in each other's way?
>
> I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and
> this caught my eye:
>
>> # ugh. ugh. suck. aieee.
>> my $use_rl = 'Cyrus::IMAP::DummyReadline';
>> {
>> if (eval { require Term::ReadLine; }) {
>> $use_rl = 'Term::ReadLine';
>> }
>> }
I have done some further investigations (very roughly because I don't
have the time at the moment). It seems that the code which parses the
command line and the run parameters in Cyrus::IMAP::Shell is buggy (or
at least not prepared to handle Term::ReadLine::Gnu).
As a proof, I have reinstalled Term::ReadLine:Gnu and verified that the
problem was showing again.
Then I have replaced the following code in Cyrus::IMAP::Shell
# trivial; wrapper for _run with correct setup
sub run {
my $cyradm;
_run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
}
by the following code:
# trivial; wrapper for _run with correct setup
sub run {
my $cyradm;
open(*__DATA__, "./000");
_run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
}
In other words, I just have made sure that this mysterious *__DATA__
variable is reasonably defined in every case before _run is called.
Now the command
perl -MCyrus::IMAP::Shell -e 'run("000")'
executed without any error message.
To verify that the script worked as intended, I added a few lines to it:
connect -noauthenticate localhost
auth cyrus
lm
When run as shown above, it did exactly what it was supposed to. It
asked for the password and then listed all mailboxes and their subfolders.
So now I have at least a system where I can have Term::ReadLine::Gnu
installed (and thus can have a history and command editing capabilities
in cyradm) _and_ can execute a script, although the script's filename is
hardcoded.
Probably it would be absolutely trivial for the authors of
Cyrus::IMAP::Shell to fix this issue. It would be very nice if somebody
could care about it. Perhaps it's already fixed in the newer versions? I
am still on 2.5.10.
I have no idea why the "buggy" command line / argument parsing does not
strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet
how *__DATA__ is supposed to be assigned a reasonable value to during
the normal course of execution. I currently can only speculate that
Term::ReadLine:: does this for us, while
Term::ReadLine::Gnu doesn't.
Regards,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Question for upgrading
On 15.12.2018 17:05, Nic Bernstein wrote: > On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: >> >> I was trying to upgrade part of our Cyrus imap installation, >> concretely that one consisting in still 2.3. I was planning to set up >> Cyrus 3.0. I have seen all works properly except for the unexpunge >> command because as someone stated here, a reconstruct -V max was >> needed.The problem is that this reconstruct command, takes ages and >> I'm not able to keep the service offline so many time. So I have been >> thinking in the following scenario : >> > > Egoitz, > As long as you've followed all of the various steps needed to account > for version changes, as outlined in the Release Notes for /all/ > intermediary releases, then the last step should be the updating of the > indexes. Rather than running "reconstruct -V max" on all mailboxes at > once, simply run it on subsets. We've done this on large installations > without ill effect. You can have the service on line during the > reconstruct, and, as you note, have most all functionality present. In my case, before the reconstruct had finished, I had several problems which might be not acceptable for large organizations. For example, users could not move messages between folders in their mailbox. I would consider this quite basic functionality, because deleting a message (with most clients) also means moving it (to trash). Functionality was back not before the reconstruct had finished completely. If interested, please see the respective thread from yesterday / today for details (I don't want to clutter the list by repeating all that stuff here). Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Question for upgrading
On 13.12.2018 16:52, Egoitz Aurrekoetxea wrote: > I was trying to upgrade part of our Cyrus imap installation, concretely > that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I > have seen all works properly except for the unexpunge command because as > someone stated here, a reconstruct -V max was needed.The problem is that > this reconstruct command, takes ages and I'm not able to keep the > service offline so many time. Eventually you are meaning me by "someone", so I kick in here: The reconstruct (for me) was only needed because I had transferred the mailboxes using cyradm and the xfer command. As ellie has explained me in detail, xfer is an IMAP extension which transfers the data on a lower level than the standard IMAP commands. Therefore, there is a trade-off: If you transfer the data at "low-level" (e.g. xfer, file system), the transfer will be fast, and (at least in case of xfer) everything (flags, permissions etc.) will be transferred precisely, but you probably will have to reconstruct afterwards. If you transfer the data at "higher" level, i.e. using standard imap commands, for example using imapsync, the transfer will be slower, and you eventually lose flags or permissions (note that I personally never had any problem with imapsync, but can't say anything about other software). But you don't need to reconstruct afterwards. Could you eventually give us some figures? In my case, I transferred between 10 and 20 GB of data from 2.4.16 to 2.5.10 (this is not that much data, but it contained a complicated shared folder (public) namespace with thousands of deeply nested folders). However, reconstruct -V max took less than 10 minutes afterwards. Given that the source and the destination server were both running in VMs on the *same* physical hardware which is over 7 years old and even runs from normal HDD storage (it's a dual Xeon 5620 system, no SSDs), this is very fast. So I am curious what the size of your data is. You could do the transfer and let the reconstruct run over the weekend when your users are not at work. Based on my figures, you should be able to transfer and reconstruct several TB of messages during one weekend ... Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Documentation for Cyrus::IMAP::Admin and friends
Dear ellie, you are really a champion! On 18.12.2018 00:39, ellie timoney wrote: > Hi Binarus, > > Looks like the documentation for the authenticate() function is pretty > incomplete... > >> Well, this man page "documents" the method by exactly one line >> (identically in two places): >> >> $client->authenticate; > > From my read of the documentation, this is enough. It will figure out on its > own which SASL options are appropriate, and then prompt you (on the > controlling terminal) for the password if/as needed. This nicely avoids > implicitly instructing users to hardcode their credentials in a script, so in > this sense it's probably good that it's underdocumented (maybe deliberately > so). This is great, and now I am embarrassed that I did not just try it. Querying the password at runtime is by far better than hardcoding the password and the mechanism in the script. Obviously, sometimes I am trying to read too much or am thinking too complicated instead of just trying ... > Looking in the source of Cyrus::IMAP, the accepted arguments are: > >> -mechanism, -service, -authz, -user, -minssf, -maxssf, -password, -tlskey, >> -notls, -cafile, -capath > > If unspecified, "-service" defaults to "imap", "-maxssf" defaults to 1, > "user" defaults to the current unix user, and the rest default to zero or > blank. Thank you very much for this insight. This is important and really interesting ... > Note that "user" is the user to authenticate as (i.e. whose credentials are > to be checked), whereas "authz" is who to authoriZe as (i.e. which identity > to assume, once successfully authenticated). authz only works for SASL > mechanisms that allow proxy authentication... the imtest(1) man page says > "e.g. PLAIN, DIGEST-MD5", I'm not sure if there are others. I think I saw something like that during my research. For example, imapsync provides --proxyauth1 and --proxyauth2 command line parameters. >> The same applies to CPAN: The modules cannot be found there. > > The Cyrus::* perl modules are tightly coupled at build time to the installed > cyrus version, so it doesn't make any sense to distribute them independently > via CPAN. I see. This makes sense. >> Where can I find reasonable (in the sense: may be short and bad, but >> must be *complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP >> so that I can write my own helper scripts in the future? Do I really >> have to unpack Cyrus imapd's sources and read the module source code to >> get some ideas? > > Sorta, but you don't need the Cyrus imapd sources -- if you have the perl > module, you have the perl module's source. Look for the "Cyrus/IMAP.pm" file > somewhere on your system (something like "sudo find / -name IMAP.pm" should > do the trick) and there it is. Again, thank you very much. You are right, getting it from the Perl installation is by far easier. Before actually opening it, I'll copy it to another place and work with the copy (in case I accidentally alter it while looking at it). > Cheers, I can't tell how grateful I am for your support (and for providing Cyrus imapd in general). IMHO, this is still the best IMAP server for small and large installations! Thank you very much again, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?
Dear ellie,
thanks again for answering my questions!
On 17.12.2018 23:38, ellie timoney wrote:
> Hi Binarus,
>
>> Could anybody please shortly explain why? What exactly are the
>> techniques and mechanisms cyradm's xfer uses to do its thing? I had been
>> quite sure that it just uses the IMAP protocol, but there seems to be
>> more to it ...
>
> You're right, but missing a detail. The XFER command is an IMAP extension,
> for use in cyrus clusters ("murders"), where mailboxes need to be identical
> between servers. I don't know much about the details, but it makes sense to
> me that it would copy the mailboxes across at a "deeper" level, to make sure
> internal identifiers and such match.
OK, got it. I am glad that I at least didn't misunderstand cyradm
completely. I originally thought that xfer was just a command which
cyradm internally would translate into standard IMAP commands, which is
wrong as I have understood now.
> Something like imapsync would probably be using the CREATE and
> APPEND commands instead, which let the destination server choose
> its own internal identifiers, and which will probably be different from the
> source ones. But they would be stored in the exact method the destination
> server prefers (such as using the newest version of the mailbox index
> format...)
Then it actually was the right idea to try cyradm xfer instead of
imapsync. After all, we want our mailboxes, folders and messages
transferred to the new server without anything being altered as far as
possible, don't we? Now that I can use scripts with cyradm (thanks to
your answer to one of my other posts), this will be a pleasure next time.
>> Now that I have used cyradm's xfer command to relocate the mailboxes /
>> messages / folders / subfolders, I surprisingly had to run the "heaviest"
>> form of reconstruct (-V max).
>
> That's not the "heaviest" form of reconstruct, it's just updating the mailbox
> indexes to the "max"imum (-V)ersion supported by the server. This ends up
> adding a bunch of extra information to the indexes that the newer server
> version can make use of. Generally if the index version is too old for what
> it's trying to do, it'll fall back to calculating missing values the long
> way, or provide less info, or just refuse the requested operation. The
> assertion failure you saw is clearly a bug, but this smells familiar, so
> maybe it's already been fixed. The latest version of the 2.5 release is
> 2.5.12, and is nearly 2 years newer than 2.5.10, for what it's worth.
OK, understood. In my case, it decided to refuse the operation ...
somehow. Obviously, it didn't tell the client that it refused, but
probably believed that the operation would be successful, sent an OK to
the client, began the operation (the message indeed disappeared from
Thunderbird's message list), noticed that it couldn't perform it and
then reverted it (the message re-appeared in Thunderbird's message list
about one second after it had disappeared).
This is no real issue to me even if the log entries are due to a bug, as
long as we can solve that problem by running the reconstruct. I just
wanted to know why we have to reconstruct; thanks again for the insight!
Regarding the version: In general, I don't have any problem with
compiling software myself. Unfortunately, I then am responsible for the
security updates as well, which I don't want, because I am not a full
time administrator (I have to care for apache2, ntp, cyrus, sendmail,
mysql and subversion servers, having only a few hours per week besides
my normal job for doing this). Therefore, I'd like the Linux
distribution / package manager to care for the security fixes, which
means that I have to use the distribution's stock packages.
However, this is an exceptional case because I will eventually have to
upgrade anyway due to the problem with the shared folder namespace
showing twice (described in one of my other posts). This is a thing
which our users (including myself) probably can't accept, so if
upgrading is the only remedy, I'll eventually do that. But I'll first
wait for some answers to that other post.
> Hope this helps!
It really did :-)
Thank you very much again!
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Running a script with cyradm throwing ReadLine errors
Dear ellie, thank you very much for your help! On 17.12.2018 23:57, ellie timoney wrote: > Hi Binarus, > >> Could anybody please tell me what I might do wrong here? > > This kind of smells like maybe your system has two versions of perl installed > (or two versions of Term::ReadLine, or maybe even two versions of > Cyrus::IMAP::Shell), and they're getting in each other's way? Since this is a fresh installation of Debian stretch, and since I didn't compile or install anything by hand yet, and since the Debian package management is usually very reliable, I am quite sure that this is not the problem. > Which... fills me with confidence. Looks like a workaround for missing > (broken?) Term::Readline but that comment isn't super enlightening. I wonder > if it will Just Work if you uninstall Term::Readline? This idea is very interesting, and you are absolutely right! While I didn't want to remove Term::ReadLine itself (because it is a core module and the usual module uninstall tools have difficulties with uninstalling it), I removed Term::ReadLine:Gnu (which I had additionally installed) instead. This made the error go away, and it seems that I can execute scripts now. So you have provided the solution and solved the problem. However, there is a downside. I am using cyradm quite often, mainly for setting permissions in a large shared folder (i.e. public) hierarchy. For this reason, I really need the nice feature which bash and many sorts of other shells provide: Hit the "Cursor-Up" key and have the shell repeat the previous command; the ability to edit the command line is often associated with this. Obviously, we can't have this feature in cyradm when only Term::ReadLine is installed. When this is the case, I even can't use "Cursor-Left" or "Cursor-Right" keys because they only produce weird character sequences instead of moving the cursor. This was the reason why I installed Term::ReadLine::Gnu in addition to Term::ReadLine. When Term::ReadLine:Gnu is installed, the command history feature in cyradm works as expected, and I can edit the command line (including using cursor keys) in a reasonable manner. Now it looks that I can either run scripts with cyradm _or_ can have its command line history and editing, but not both features at the same time. I think I could live with that, but of course I would be grateful if somebody would share a method to enable both features. Perhaps there is another module which I could use as a replacement for Term::ReadLine::Gnu and which does not break scripting? Thank you very much again, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Shared folders (AKA public) namespace showing twice
Dear all,
hopefully, this is my last post for today.
I just have put a new server running 2.5.10 live and have noticed that the
Shared Folders are appearing twice. Please note that I have renamed the "Shared
Folders" prefix to "public" (see below). When a user who has access to the
Shared Folders logs in (e.g. via Thunderbird), that user's mailbox folder
hierarchy shows up as follows:
Inbox
Drafts
Sent
Junk
Trash
public <--- expected ("Shared folders" namespace)
Folder1
Folder2
public <--- This is my problem
Folder3
...
As you can see, the "virtual" public folder contains all shared (public)
subfolders (nothing is missing, the permissions and all IMAP operations are
working as expected), but there is an additional subfolder "public" which never
has been actively created and which was empty every time we examined it.
I originally thought that this was an artifact which had been created when
moving all data from the old server to the new one, and deleted it. Upon
deletion, Thunderbird did not show any error and moved that folder into
"Trash". From then on, it was not visible any more at its original place. But
guess what: It was there again as soon as Thunderbird had been restarted.
To make sure that this was not due to a Thunderbird bug, I connected to the
server in question as administrator (cyrus in this case) via cyradm and listed
all mailboxes and subfolders (lm command). Indeed, there was *no* mailbox or
folder which had "public" as its name.
I investigated the issue further with the help of telnet, which revealed
something interesting (I logged in as a normal non-admin user who has access to
the shared (public) folders; I have left out the prologue and the LOGIN dialog
for brevity and clarity):
root@morn:/etc# telnet localhost imap
[...]
A1 NAMESPACE
* NAMESPACE (("" ".")) (("other." ".")) (("public." "."))
A1 OK Completed
A1 LIST "" "%"
* LIST (\Noinferiors \HasNoChildren) "." INBOX
* LIST (\HasNoChildren) "." Drafts
* LIST (\HasNoChildren) "." Junk
* LIST (\HasNoChildren) "." Sent
* LIST (\HasChildren) "." Trash
* LIST (\Noselect \HasChildren) "." public.public
A1 OK Completed (0.003 secs 7 calls)
The NAMESPACE command's return value is as expected, but the LIST command
returns "public.public" at the place where it (IMHO) just should return
"public". I believe that this is the reason for our problem, and that this
can't be correct.
This might or might not be related to the following bug report / discussion
from 2005: https://github.com/cyrusimap/cyrus-imapd/issues/744
In every case, listing "public.public" is wrong, isn't it?
I can further tell that we did not have that problem with our old server
running 2.4.16 with the *literally same* configuration files (imapd.conf and
cyrus.conf).
These are the snippets from imapd.conf which are probably relevant:
altnamespace: yes
userprefix: other
sharedprefix: public
unixhierarchysep: no
autocreate_quota: 0
Could somebody please tell me if the behavior I have observed is a bug or if I
am doing something wrong? If it's a bug, is there something I can do about it
(besides compiling the newest version of Cyrus imapd myself or changing the
Linux distro)?
Thank you very much in advance,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Documentation for Cyrus::IMAP::Admin and friends
Dear all,
as described in my previous messages, I recently had a hard time relocating
(i.e. moving) mailboxes and the public namespace from 2.4.16 to a new server
running 2.5.10. I would have happily written a Perl script which surely had
solved much of my problems, but I couldn't do that due to the lack of
documentation for the respective Perl modules.
This is best explained by example:
I have found a Perl script on the 'net which looks trustworthy at the first
sight and which contains (among others) the following lines of code:
use Cyrus::IMAP::Admin;
# Connect to Cyrus
$imap = Cyrus::IMAP::Admin->new("my_server") || die "Unable to connect to
my_server";
$imap->authenticate(-user => "foo",
-mechanism => "LOGIN",
-password => "password",
);
I wanted to understand exactly what the script does and thus tried to find
Cyrus::IMAP::Admin's documentation. So I issued
man Cyrus::IMAP::Admin
and found that some methods are documented, but not the authenticate() method
which is needed first and one of the most important ones. However, the man page
states (in the section about the "new()" method):
"Instantiates a cyradm object. This is in fact an Cyrus::IMAP object with a
few additional methods, so all Cyrus::IMAP methods are available if needed.
(In particular, you will always want to use the "authenticate" method.)"
This tricked me into believing that the "authenticate" method would be part of
Cyrus::IMAP and would be explained in that module's documentation, so I issued
man Cyrus::IMAP
Well, this man page "documents" the method by exactly one line (identically in
two places):
$client->authenticate;
It does not explain anything about it; notably, it does not mention any
parameters. But from the example script mentioned above, I knew that there was
more to it. So I read the rest of that man page and found:
"The Cyrus::IMAP module provides an interface to the Cyrus imclient library."
So I installed the development files for Cyrus imapd and issued
man imclient
Now, finally, this is a man page a C programmer probably can live with. But
when looking more thoroughly into it, I saw that there is no "password"
parameter to the authenticate() function although the Perl module's
authenticate() method has one:
int imclient_authenticate (struct imclient *imclient, struct sasl_client
**availmech, const char *service, const char *user, int protallowed);
Plus, I could not find any hints regarding the relationship between the
parameters of the C functions and those of the Perl module methods.
I then headed over to cyrusimap.org and tried to find documentation for the
Perl modules. Surprisingly, I couldn't. Not a single word. They don't seem to
exist. The tools and helper programs are documented, but the Perl modules are
not even mentioned.
The same applies to CPAN: The modules cannot be found there.
To make a long story short:
Where can I find reasonable (in the sense: may be short and bad, but must be
*complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP so that I can
write my own helper scripts in the future? Do I really have to unpack Cyrus
imapd's sources and read the module source code to get some ideas?
Thank you very much in advance,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?
Dear all, yesterday, I have moved a bunch of user mailboxes and public folders from a server running 2.4.16 to another server running 2.5.10, using cyradm's xfer command (see my previous messages). After having finished the migration, I noticed a weird behavior in Thunderbird (which is our standard email client): When trying to move a message from the Inbox to the Junk or Trash folder, the message disappeared from the Inbox for a short time, then reappeared. The log files on the server were showing the dreaded entries: Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error: assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(>guid) "Fortunately", a lot of other users are affected by such log entries and weird email client behavior as well, so finding the solution on the 'net was not too difficult: Running reconstruct did not lead to anywhere, but running reconstruct -V max was the solution. This lets me scratch my head: In the past, I have upgraded Cyrus imapd at least three times, each time using imapsync (instead of cyradm's xfer command) to move the mailboxes and the public namespace (including all messages and subfolders) from the old server to the new one. In none of these cases, it was necessary to reconstruct anything afterwards. This would have been illogical anyway: Each time, the new server had been setup from scratch, and no mailboxes / messages / folders / subfolders had been moved directly via file transfer from the old to the new server. Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max). Could anybody please shortly explain why? What exactly are the techniques and mechanisms cyradm's xfer uses to do its thing? I had been quite sure that it just uses the IMAP protocol, but there seems to be more to it ... Thank you very much for any insight, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Feature request: Add wildcard capabilities to cyradm's xfer command or make it handle public namespaces reasonably
Dear all, yesterday, I have retired a server running 2.4.16 and have transferred the folder structure and all messages to a new server running 2.5.10. In the past, I have used imapsync to do that sort of transfers, which worked reliably. But this time, I decided to prefer an "original vendor tool"; hence, I used cyradm and its xfer command. After having fiddled around with the configuration files on the old and the new server, I could move the user mailboxes without any problem (all mailboxes whose name begins with "user."). This process went smoothly and worked reasonably fast, and (the most important thing for me) all subfolders in each mailbox have been moved recursively. The problem: I don't have too many users / mailboxes, but a huge and well-crafted (in terms of folder structure and permissions) public folder (namespace) which is an important collaboration tool. It did not seem possible to relocate the public folder in a reasonable manner using cyradm's xfer command. Obviously, it is able to relocate a single folder from one public namespace to another, but it does not relocate this folder's subfolders. Unfortunately, in contrast to some other of cyradm's commands, xfer does not seem to support wildcards either. This was the moment when I thought about stopping using cyradm for this task and using imapsync again. But I am a bit stubborn sometimes, so I went on and ended up with manually moving several thousands of folders (actually, it was semi-automatic: from within cyradm, I issued an lm command, copied a part of the listing from the terminal window into a text editor, generated the respective commands there (using macros within the editor), pasted the commands back into the terminal window, waited until it had finished, and restarted that process). This is ugly and silly. Hence the question: Could we extend cyradm's xfer command so that we could give wildcards to it, like, for example, cyradm's lm command? Or (more elegant and desirable) so that it recursively includes all subfolders when it relocates a folder, even when this folder is in the public namespace? Thank you very much for any thoughts, Binarus P.S. Please see my next messages for reasons why I didn't write a Perl script using Cyrus::IMAP:Admin for this task. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Running a script with cyradm throwing ReadLine errors
Dear all,
I was just trying to explore cyradm a little bit further and hence was
experimenting with its scripting capabilities. Having cyradm run a script
should be pretty easy. man cyradm tells us:
perl -MCyrus::IMAP::Shell -e 'run("myscript")'
So I created the simplest possible script (that means an empty one) and tried
to run it:
touch 000
chmod a+x 000 (just in case ...)
perl -MCyrus::IMAP::Shell -e 'run("000")'
The only thing I got was:
Use of uninitialized value within @layers in string eq at
/usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Term/ReadLine/Gnu.pm line 280.
Bad filehandle: __DATA__ at
/usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Term/ReadLine/Gnu.pm line 769.
Putting something meaningful into the script did not change the situation.
I have googled and read documentation (mainly on cyrusimapd.org) for several
hours, but could not find the reason for the problem.
I even have put allowplaintext=yes into imapd.conf and restarted imapd (knowing
that this probably wasn't very smart, but the term "layers" in the error
message made me mistrustful because there are authentication "layers", and I
don't have any problems with Term::ReadLine::Gnu in general). As expected, this
didn't change the situation either.
This happened with 2.4.16 as well as with 2.5.10.
Could anybody please tell me what I might do wrong here?
Thank you very much in advance,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Why Cyrus?
On 17.01.2018 11:56, Paul van der Vlis wrote: > A new customer asks me to build a new mailserver environment with > Dovecot. I normally use Cyrus. > > My question: What's better in Cyrus? Please note that what I am reporting here has happened some years ago, so the situation may be different now. Nevertheless, it may show you a problem which you probably would not have thought of. Some years ago, there was some sort of Dovecot hype, so we decided to give Dovecot a try as IMAP server for our small company environment. I found the setup to be more complicated than Cyrus' (maybe just because I were used to the latter). We were using Thunderbird as IMAP client software, and we were using a large public IMAP structure. So everybody had access to their personal mailbox with all privileges, and had access to the public namespace with certain privileges. Most users had only read and "add" access to the public namespace (i.e. could read and add messages and folders, but could not delete them), but a few users were allowed to delete as well. Now there was the following problem: When you are running Thunderbird in default configuration, and you try to delete a message, then it technically *moves* that message from the folder where it currently resides into the *trash folder of the mailbox of the user who is deleting that message*. Thus, when deleting a message from the public tree, Thunderbird tries to move that message from the respective folder in the public namespace to a user's personal trash folder. In other words, it tries to move that message from one namespace to another one (i.e. from the public namespace to the normal user mailbox namespace). This is a thing Dovecot was not able to do at that point in time. Every time we wanted to delete a message from the public tree (namespace), we got an error message saying that it couldn't move messages between different namespaces (please forgive me that I really can't remember the exact wording, it *really* happened several years ago). Given Thunderbird's market share, this was ridiculous. Being able to smoothly delete messages from the public namespace with *any* email client is a sine qua non. Therefore, after playing around for some days and trying to solve that problem, we dumped Dovecot and have never tried it again since then. Please note that we of course know that 1) We could Thunderbird have messages delete immediately (i.e. finally) instead of moving them to trash, and that this would have solved that problem. However, moving messages and folders to trash instead of immediately deleting them has its advantages, and it has its reasons that this is the default behavior for most major email clients. 2) There are other email clients (as far as I can remember, Outlook was one of them) where deleting messages and folders from the public namespace and putting them into trash was no problem (such clients technically do not move the message or folder, but "simulate" moving by first *copying* the message or folder into trash and then deleting it *immediately and finally* from its original location). However, we expect our IMAP server to be compatible with *all* reasonable clients and all basic IMAP commands, including moving messages. In fact, we have changed our email client software several times since then (currently using Thunderbird again) while our IMAP server software always stayed the same: Cyrus IMAPd. *Summary / What to learn from that experience:* Surprisingly, we can definitely confirm that several years ago Dovecot was seriously misbehaving with one of the most common email clients, while it worked with other ones. The misbehavior did not concern some exotic action or software, but a very common action (delete messages and folders from the public namespace) with a very wide-spread client software (Thunderbird). I do not know if this is still the case, but this experience shows that you should test *every possible client software* you (the customer) could use with Dovecot, and that you should set up all IMAP features and test if any actions you will carry out in the future will work. For example, you should set up some normal user mailboxes, set up a public tree (namespace) with several nested subfolders, assign different access rights to them, set up a shared namespace (shared mailboxes - do not confuse them with public folders), try to move and copy folders and messages between them, and try to delete messages and folders from any of them. Do these tests with as many different email clients as you can. For us, none of those tests failed when Cyrus IMAPd was the IMAP server. Just my two cents ... Regards, Binarus P.S. The problem described above technically is not limited to deleting messages; as explained above it will occur every time a message will be *moved* (in an IMAP sense) between different namespaces, meaning that it might occur during the most basic actions (like moving a me
Re: Mailbox administration in cyrus
On 21.06.2017 15:06, Dr. Peer-Joachim Koch wrote: > Hi, > > is there an esay way to remove ALL mailboxes of an user ? > > I did not find any info about something like > > dm -r user.OLDUSER > > removing every folder within OLDUSER and OLDUSER as well. > > How can I remove an old user with all mailboxes ? Try something like dm user.OLDUSER* DISCLAIMER: I don't have access to a cyrus imapd at the moment, and I can't remember for sure. This is just a quick (but maybe wrong) answer to eventually put you on the right track. In every case, before actually doing dm <pattern*>, try the same with some other harmless command (e.g. lm <pattern*>) to check if the wildcard pattern really does what you want. Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Problems with paragraph characters in SASL passwords?
On 27.05.2017 15:48, Adam Tauno Williams wrote:
> On Sat, 2017-05-27 at 10:30 -0300, Patrick Boutilier wrote:
>>> I am very happy with Cyrus imapd since many years. I am using it to
>>> host all IMAP mail boxes of my company. I am using SASL and its
>>> tools (mainly saslpasswd2) for password management. The primary
>>> IMAP client in the company is Thunderbird.
>>> As soon as the password contained a paragraph character ("§"),
>>> Cyrus / SASL refused the connection due to a wrong password even
>> Works for me from a telnet to port 143 then issuing:
>> . login
>> replacing user and password with correct values.
>> But it does fail in Thunderbird.
>
> Yep, I have experienced this type of issue numerous times. A variety
> of clients fail to correctly encode the authentication credentials -
> particularly if you are using a chat-expect authorization like PLAIN or
> LOGIN. To have something that always works it is best to keep
> usernames and passwords to ASCII/UTF-7.
>
> This is not a SASL bug.
>
> This is an every-client-rolled-their-own issue. :(
>
Patrick and Adam,
thank you very much for taking the time and verifying the issue. And
some years ago, they stated "There will be no new Thunderbird release
because Thunderbird is ready" ...
Regards,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Problems with paragraph characters in SASL passwords?
Dear all,
I am very happy with Cyrus imapd since many years. I am using it to host
all IMAP mail boxes of my company. I am using SASL and its tools (mainly
saslpasswd2) for password management. The primary IMAP client in the
company is Thunderbird.
Recently, I have decided to replace all IMAP passwords by longer ones.
While this worked in the vast majority of cases, there were several
mailboxes where Cyrus / SASL refused the connection with the new
password. I have lost several hours of debugging this until the
following turned out:
As soon as the password contained a paragraph character ("§"), Cyrus /
SASL refused the connection due to a wrong password even if the password
was entered correctly into Thunderbird's password dialog. This happened
with Thunderbird 52.1.1 and Cyrus imapd 2.4.16 (as shipped with Debian
wheezy).
My question is: Is there a known problem with paragraph characters in
SASL / Cyrus passwords, or does Thunderbird cause that problem (for
certain reasons, I haven't been able to test other clients yet, and
googling for some hours also did not lead to anything)?
Thank you very much in advance,
Binarus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
blacklisted. In contrast, SpamAssassin returns some sort of spam score which is computed according to some very intelligent magic, to long, long training (hopefully) and to all sorts of other rules of arbitrary complexity. If you rely on that, you again get into the problem that you have to check you junk folder regularly. So the different approaches are (a) manually blacklisting domains which have been proven to send spam (done by humans, our approach) versus automatic computing some spam score according to sophisticated rules and relying on that in further processing the message (done by software, SpamAssassin's approach). > You made an assertion that there was nothing else you could do on the server > aside from your current approach, which is just not the case. The claim made > to bolster this argument is technically untrue. For all I know you might have > been unaware that you can run SpamAssassin as a filter in the MTA. (There are > a surprising number of mail administrators that know literally nothing about > how messages get moved around.) I know about that. But I also know that you need a lot of expertise to make it work the right way, and you can't get around the obligation to check your spam folders on a regular basis if you process your messages according to the spam score. Amusing: AFAIK, you can include SPF and DKIM tests into SpamAssassin's rules ... > You have missed my point. If you get SPAM at a rate of x/day, then waiting to > look when you have 365x to wade through is your own decision — so don’t whine > about it. You could easily decide to look at 7x, or 28x, or whatever. You > could look when you get to 1000, however long that takes. My point is that > you make the decision about when to process your false positives, and the > number of messages you must consider is a direct result of your own > scheduling. It's not really my decision. There is just no time for it during normal working days. Or, in other words, during working days, my time is too expensive to waste it that way. > I don’t think you actually know how many legitimate messages you are > rejecting, because if you could find/count them you would just accept them. We've got an idea about that since there are the subject, envelope-froms and Froms, and the envelope-tos and Tos in the server's log for all messages (rejected or accepted). > I also don’t know what kind of business your users conduct, so I don’t know > if there is a large opportunity cost associated with rejecting a legitimate > message. The policy you promote may work very well for you, but I would not > consider it responsible for the vast majority of mail administrators. I agree. It's just an approach which works great for us. At least as long as the big freemail providers don't implement SPF or DKIM the right way, our policy will not be an option for most other companies, let alone private people. Nevertheless we are hoping that more and more small companies will employ more rigorous policies in the future. That seems to be the only way to force the big providers to use those technologies. > It’s possible to reduce spam by 100% by just turning off the server, though > the utility of that approach is suspect. Agreed. And finally: I am really hoping that the discussion about the sense and nonsense of SPF, DKIM and DMARC stops now. I am assuming that we all agree that setting up an SPF or DKIM record for lists.cmu.edu and signing the messages won't harm anybody and on the other side would make life easier at least for a few people who are minded like us. If the administrator is willing to setup such technologies and gets an OK from his management, we will be grateful. If not, we will find a way around the problem at our side. In my first post, I probably should just have asked just for what I proposed, without giving a reason for it. On the other hand, I probably should give reasons when asking somebody to do something for me ... Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
ear, you deserve > whatever the volume in which you are mired. I don't know what sort of job you have, but I can tell you that you are lucky if you can take the time for doing this on a daily (or weekly or monthly) basis (I usually have done it during Xmas holidays). Furthermore, are you god who can decide what we deserve? I think you have got something quite wrong: Nobody deserves any SPAM message. > I have not seen you indicate how you address all of the legitimate messages > you are rejecting because their senders don’t implement SPF or DKIM. Do you > have good statistics to indicate how many fewer *ham* messages you’re > delivering each day? That would be an excellent data point if it existed. We address them by returning a DSN which contains a polite text explaining the problem and asking the sender if he could tell his administrator / provider about it. I don't have figures regarding the other clients, but at least in my case the number of ham messages which I did not receive due to our new policies is negligible. At least, (nearly) no more having trouble with SPAM (working time, dangers of phishing / viruses, legal aspects of unread messages) is more than worth "losing" a negligible number of ham messages. Peace, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
> If you want to see flame wars even more pointless and/or entertaining than > this one, check out the mailing lists for DMARC. ;-) They make these recent > exchanges seem quaint by comparison. I am sorry that this thread is not useful to you. I don't consider it a flame war. Every party (except the one who called us "phenomenally stupid") had a reasoning which at least is worth thinking about. > FWIW, mailing lists and DMARC make a particularly noxious couple, as almost > all mailing lists will break DMARC, and thus lead to all sorts of rejections. > That very subject is the topic of the most vitriolic flame wars on the DMARC > lists. Maybe. We are currently not interested in DMARC. > At the risk of perpetuating this severely off-topic thread, IMHO if "Binarus" > is able to eliminate "90% solely by checking for SPF and DKIM" then one must > question just what the rest of their anti-Spam measures were doing? The answer is quite easy: Until now, there just haven't been any measures against SPAM on the server side. Instead, users have used Thunderbird's junk filter (which works great IMHO). So, before checking SPF / DKIM, the clients actually have received every message which hit the server, except the messages which were addressed to non-existent recipients. We know quite well how many SPAM got to the clients before and after implementing the SPF / DKIM checks. The problem with letting the clients doing the SPAM handling (explained by the example of my personal account): Once per year, I had to go through the JUNK folder to see if there were false positives in that folder. Some weeks ago, this ended in manually searching through about 12000 spam message and thereby finding about 10 important *ham* messages. Given some court decisions here in Germany, it could eventually be dangerous to not handle a message in a timely manner or even to never know about that message if the sender can prove that your server has accepted the message. Therefore, there is no way around checking your SPAM folder if you let your MUA sort out the SPAM. Some weeks ago, for a reason I still don't know, the SPAM volume hitting our clients suddenly doubled (or tripled, I don't have the exact figures). Therefore, we have decided to change our SPAM handling. Nobody is keen on scanning 10 SPAM messages at the end of the year (my mailbox is not the worst one) ... By the way, I am now finishing my working day, having got exactly 4 SPAM messages (two weeks ago: between 200 und 300 per day). Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
On 05.04.2016 09:34, lst_hoe02--- via Info-cyrus wrote: > The "we generally have to reject all messages which are not secured by SPF or > DKIM" mean you want to force others to use non standard headers so in fact > you are breaking SMTP RFC. I think we don't. At least SPF works without additional headers in the messages. Furthermore, I still can't see how we would break RFCs even if we would "force" people to use the DKIM header (in fact, we are not forcing anybody to do so, because we let messages pass which have at least *one* of SPF or DKIM passed): The RFCs nowhere say that every MTA MUST accept ANY message regardless of the sender, connecting server etc. On the contrary, the RFCs explicitly name mechanisms (e.g. DSNs) which should be used if a message cannot be delivered to its recipient, and people are rejecting messages (and returning appropriate DSNs) according to their own policies for decades now. If you are saying that not accepting *all* messages means breaking the RFCs, I disagree. What I exceptionally like about the way we have implemented the SPF and DKIM checks is that the sender gets informed about the problem because he will receive an appropriate DSN containing a polite message which explains the problem. In summary, I am convinced that our MTA's behavior conforms with the RFCs. > It is your server so your rules, but don't complain if other do not agree > with you. I promise I won't :-) Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
On 05.04.2016 14:15, Alvin Starr via Info-cyrus wrote: > > I kind of have to agree with Andreas to some extent on this. > SPF/DKIM does not help on incoming spam filtering all that much just because > so few people use it and the default action is to accept mail that has no > SPF/DKIM tagging. Our default action is to reject all messages which do not pass either the SPF or the DKIM test. > > It is great however for controlling how other people abuse your email address. > SPF can stop people from sending mail as you from systems that are not your > own. Not really, AFAIK. Even if you add the SPF record to your domain's DNS, a spammer of course can still use @ as envelope sender or From: header. It is the receiving part who checks if the connecting MTA (i.e. the "sending server") is allowed to send messages for (the check is done by querying the name server for for the SPF record and then checking if the sending (connecting) server one of the servers the SPF record allows). In other words, if no SPF checks are done by the *receiving* MTAs, fake messages will make their way through the net without problems. > I would argue that anybody operating a mail server should use SPF/DKIM just > to make sure they are not helping the spammers. I strongly agree. > Sadly putting these tools in place is not trivial and it will only be when > postfix, sendmail, qmail and others include SPF/DKIM setups as part of the > default install can things really start to change. Actually, I have been surprised how ridiculously easy I could setup the *sending* part of SPF. Using SPF as a sender means adding one TXT record (whose syntax can't be simpler) to your DNS records; this could be done within minutes (no more true if you want your MTA to forward messages from other domains; that's a special case). DKIM is slightly more complicated since it needs additional software which must be interfaced to the MTA. I used opendkim and liked it very much, though. Checking SPF and DKIM (the *receiving* part) was much more complicated in our case, though. So I would recommend everybody who wants to improve email security to start with the sending part. If you don't forward messages for other domains, just start with adding the SPF record to your name server (and end that record with "-all" in every case, despite other examples which could be found on the net). Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
On 05.04.2016 09:42, lst_hoe02--- via Info-cyrus wrote: > > As stated the spam actually reaching our inboxes after around 90% cutoff is > valid DKIM/SPF signed as it is mostly from the big free providers like > Outlook.com, Google and Yahoo. Some other big share is from professional spam > farms with always alternating IP and Domains ranges from all over the world > with also valid DKIM/SPF. Next big share is from educational servers also > mostly valid DKIM/SPF. The tiny rest with around 10% is in fact not DKIM/SPF > signed. > From the valid e-mail around 20% looks like having a valid SPF/DKIM, mostly > professional newsletters not personal mail from customers. > > So No, SPF/DKIM is no useful spam fighting tool at least not in our corner of > the world. > We seem to be located in the same country (Germany), nevertheless the situation is completely different for us. As I have already reported, we have cut off SPAM by 90% solely by checking for SPF and DKIM, and it looks like we could cut down it by another order of magnitude if we are blacklisting domains which have sent SPF- or DKIM-"signed" SPAM (doing so for a few days, but no exact figures yet). I admit that our situation is somewhat special because we are purely B2B, and I absolutely don't care about a freemail provider being blacklisted. I can't even remember the last time when we got a valid message which has been sent from a freemailer account. Actually, if everybody did SPF or DKIM tests, this finally would force the providers to implement DKIM or SPF the right way. For example, using an individual DKIM signature for every sender of a domain is ridiculously easy (at least when using the opendkim daemon). That would be a great progress because then you could blacklist individual senders instead of the provider. Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
On 04.04.2016 23:02, Vincent Fox via Info-cyrus wrote: > I'll admit I am testing SPF as a greylisting measure. > Your IP gets hardfail, you get 5min deferral. > > I don't delude myself it does anything other than catch maybe > 5-10% of spammers that don't bother with retries. More often it > seems to catch people like a major network backbone operation > that OUGHT to know better, that has no SPF and acted like it > was going to require committees and 2 months for the > brain surgery. > > YMMV indeed. > Well, that seems to be the case. I have no reason to boast here; it is indeed true that we cut down the number of spam messages by 90% solely by rejecting all messages without one of SPF or DKIM. Since a few days, we are blacklisting the domains which have sent SPAM, and now it looks like we could cut down the SPAM an additional order of magnitude by doing so (no exact figures yet). Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
On 04.04.2016 21:50, Joseph Brennan via Info-cyrus wrote: > >> But with SPF or DKIM, you can immediately blacklist any sender >> domain after having received SPAM from that domain. > > It would never be a phished stolen account, so that would be safe. > You are right. It is the only logical thing to accept emails from stolen or phished accounts for the sole reason that they have been stolen or phished. Joking apart: After having repaired the problem, the victim (i.e. the legitimate, white-hat real owner of the account) hopefully sees the DSNs, and, if his message is important, might call us and ask what has happened. Even more, the DSN from our MTA eventually might let the "real owners" know that somebody is doing damage to them. Did you think about that? By the way, there are countries where you are liable if you send viruses, and in those countries, people might be even more grateful if they receive a DSN after a spammer has abused their account. Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
> > You are for sure aware that neither SPF nor DKIM are able or designed to > fight Spam. I know that a lot of people are stressing this. But it is not my opinion nor experience (see below). > In fact more than half of the Spam reaching our inboxes are valid according > DKIM/SPF so we even might reduce spam by rejecting DKIM/SPF signed mail. In our case, we have cut down SPAM by approximately 90% alone by doing SPF and DKIM checks with incoming messages. For example, my own corporate mailbox approximately got over 200 SPAM messages per day before rejecting messages without DKIM or SPF, and now I am getting somewhere between 10 and 30. That's what I meant by "extremely effective in our case". > DKIM/SPF does only include that the sending server is mandated by DNS to send > mail for the given domain and this is easily done with all modern spammer > tools. Well, most spammer tools might do this. But the spammer then first has to get a domain and then has to set up the DNS entries, which obviously is too complicated for most spammers. Furthermore, I am constantly seeing messages trying to get into the server which originate from dynamic IP addresses. The majority of spam messages seems still to be sent directly (i.e. without passing a "smarthost") to our receiving MTA by PCs which have been infected with a trojan horse; it seems that the sending MTA more often than not is part of this trojan and thus sends the messages from a dynamic IP address. I am convinced that it is impossible for a spammer to continuously update his SPF entries for all devices he has under control with the dynamic IP addresses of these devices. [N.B. Of course, we are rejecting messages event if they pass SPF, but the SPF entry has something like +all or ~all in it.] Now to the most important part of SPF and DKIM (I am stressing this because I am convinced that many people really believe that you can't fight SPAM with SPF or DKIM): As you correctly have stated, if a message passes the SPF or DKIM test, it can be taken for sure that the *owner* (or some person which has been authorized by the owner) of the (pretended) sender's domain actually has authorized that message (at least indirectly). In other words, if a message which passes the SPF or DKIM test contains SPAM, the owner of the (pretended) sender's domain either has allowed somebody to use the domain for sending spam, or he obviously is not in control of his staff or his mail or DNS server. In either case, you could (and should) blacklist this sender domain. This is the key aspect: Without SPF and DKIM, you can *not* blacklist a sender domain after receiving SPAM from that domain, because you could be sure that the sender domain has been faked by the spammer, and if you would have blacklisted it, you would not get legitimate emails from there any more (imagine the spammer had used someb...@ibm.com as sender's address). But with SPF or DKIM, you can immediately blacklist any sender domain after having received SPAM from that domain. You now know *for sure* that the spammer did not abuse / fake the sender's address (letting apart such things like a hacked mail relay etc.), but that the domain owner has authorized the SPAM, thus you are sure that you do not want to get any more messages from that domain. Combine SPF / DKIM with domain blacklisting, and then you *have* an efficient spam fighting tool. Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
On 04.04.2016 18:12, Sebastian Hagedorn via Info-cyrus wrote: > Personally, I think that's a phenomenally stupid approach. As long as you > can't show me an RFC that says you MUST or even SHOULD use SPF or DKIM, > you're breaking SMTP. I think it's a phenomenally intelligent approach. I can't see in which way SMTP is broken by using DKIM or SPF. The DKIM signature is in an additional header (additional headers *are* allowed by the RFCs), and signing and checking usually is done by milters (I am sure that you know them). If a message is rejected by the receiving MTA due to failing SPF or DKIM, the sender will get a DSN (which is perfectly in conformance with the RFCs). By the way, many people use all sorts of mail filtering and DSNs (and do so since 20 years and more) without an RFC saying they SHOULD or MUST do so. Are all people which use any sort of mail filter breaking SMTP as well? Could you please give an example of an SMTP RFC which is violated by SPF or DKIM? Regards, Binarus > >> Due to the exponential increase of spam, we generally have to reject all >> messages which are not secured by SPF or DKIM, and we know a lot of other >> people who do the same (by the way, this has proven to be extremely >> effective in our case). When our MTA encounters such a message, it >> rejects it and returns a bounce message to the pretended sender, >> notifying him about the problem. > -- > Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 > Regionales Rechenzentrum (RRZK) > Universität zu Köln / Cologne University - Tel. +49-221-470-89578 > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
Dave, On 04.04.2016 16:32, Dave McMurtrie wrote: > I completely agree. I'll run this up the management chain and see if I > can get approval. Really, the ideal solution would be to set up a list > server in the cyrusimap.org domain and handle it there because CMU > management doesn't care what we do in that domain. I'd love to do that, > but I'm hesitant to foist that change on Cyrus users since info-cyrus > has been on lists.andrew for so many years now. > > Your input is appreciated, though. Well, not being an expert in that area, my 2 cents: I think I wouldn't move to another server, too (never touch a running system). But eventually you could forward all messages from lists.andrew to cyrusimap.org which then could sign and send them? That way you could keep the current server (nearly unaltered) for mailing list management, processing the received messages and sending messages. The only change would be to not directly send messages, but to forward them. Before sending, cyrusimap.org should rewrite the envelope-from and from, making them something like "cyrus-imapd-l...@cyrusimap.org". The receiving MTAs could then get the public DKIM key from cyrusimap.org and check if the signature is valid, i.e. if the message actually has been sent by cyrusimap.org. Or, even easier: Just add an appropriate SPF record to the DNS configuration of andrew.cmu.edu, and we could test what happens. Adding such record should get immediate approval by your management since it does not affect other DNS records or the mailing list server in any way. In other words, you would just have one more TXT record in your DNS which will not interfere with any other system component in any way. I strongly assume that this already would be sufficient. Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Request: Please sign this list's messages via DKIM or SPF
Dave, On 04.04.2016 13:22, Dave McMurtrie wrote: >> the messages which are being sent from this mailing list's server don't seem >> to be protected by SPF or signed by DKIM. Are there plans to implement at >> least one of these in the near future? >> > > We currently have no plans to implement either, but I can put it on our > list of things to do. > Thank you very much for considering. Due to the exponential increase of spam, we generally have to reject all messages which are not secured by SPF or DKIM, and we know a lot of other people who do the same (by the way, this has proven to be extremely effective in our case). When our MTA encounters such a message, it rejects it and returns a bounce message to the pretended sender, notifying him about the problem. Of course, we absolutely do not want those DSNs to get onto the mailing lists, so we have to implement exception rules for every mailing list which does not use SPF or DKIM. This generally works, but it requires some manual effort, and it only works until a mailing list server changes ... Given that, I think that other people would be grateful if you would implement one of those techniques, too, and implementing is very easy :-) Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Request: Please sign this list's messages via DKIM or SPF
Dear list administrator, the messages which are being sent from this mailing list's server don't seem to be protected by SPF or signed by DKIM. Are there plans to implement at least one of these in the near future? Regards, Binarus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
