On Jun 25, 2012, at 12:27 PM, Jared Johnson wrote:
>> I am to the point where I need to deploy. I was hoping that I could get my
>> changes merged into the main branch, but that appears impossible. I hate
>> the prospect of forking, but I see few alternatives.
>>
>> I hereby propose the creation of qpsmtpd-dev.
>>
>> The primary aim is merging the forks maintained by myself, Steve, Chris,
>> Jared, and others. I am willing to do much of the work to accomplish this.
>> I believe there is much to be gained by doing so, not the least of which
>> is saving others the pain of learning they are wasting their time trying
>> to improve QP in any way that doesn't fit with the unstated goals and
>> vision of the project. A secondary goal is to provide a development branch
>> of QP in which ideas can be rapidly tested, deployed, and when desired,
>> backported to the main branch.
>
> I like the idea, although I'm not absolutely sure that it would accomplish
> everything it would mean to.
Not trying is guaranteed not to.
> A primary example to me would be the
> divergent methods used by various forks for supporting "per-recipient
> configuration". That thread demonstrates how each of our various methods
> made some sort of sense to the others, but were also completely
> unacceptable to the others for various reasons.
Simply because each entire enchilada is insufficient/unusable for every other
persons doesn't mean they can't each benefit from each other. Perhaps there
isn't one per-recipient solution to rule them all, but I'm willing to bet that
if each fork was properly coded in paragraph sized methods, that there would be
a significant amount of overlap that could and should be abstracted into a
shared class. Then every fork would have less custom code to manage.
A classic example: every plugin that uses DNS loads its own resolver, fetches
results itself, and then does whatever processing it does with the results.
Having a shared DNS class that lets plugins do all but the local processing is
low hanging fruit.
> The only solutions I can
> think of would be (1) maintain more than one 'aggressive development'
> fork, perhaps eventually letting the forks die that don't get used or
> hacked on much;
I think that's a perfect use for branches. :-)
> (2) somehow support multiple options for how to accomplish
> the same thing, selected within the config; (3) pick a single option that
> is "blessed" by upstream -- forcing some to remain quite forked if the
> single option is not useful for them.
We could, for example, have a 3 branches within -dev that each implement
per-recpient handling in one fashion or another. Then it would be easy to see
how each is implemented and compare. Creating a "per-recipient" base branch,
that the other 3 share from could immediately reduce the divergence. When
someone wants to implement one of the 3 types, they can clone the dev-fork and
just merge in the branch that does what they want. This is easy enough that we
can even document the process.
> At any rate, if a branch like this exists and someone else had the time
> and motivation to do the leg work of applying enhancements from the
> various forks to the core branch, I would most definitely ask my employers
> to put most of our codebase at your disposal to do this. Even if the
> result wasn't something we could use, and we had to remain forked... at
> least it would be useful to _someone_ :)
https://github.com/qpsmtpd-dev/qpsmtpd-dev
It has been created. You have commit, as do Ask, Robert, and a couple others.
The policy for gaining write access is liberal. The tolerance for change is
great. Please use branches. :-)
My goal is to get you (and others) running qpsmtpd-dev. It doesn't look like
your fork on GitHub is up-to-date. If you get it updated, I should be able to
diff against it and 0.84, and get a pretty good idea what you've changed, and
then perhaps introduce changes to qpsmtpd-dev that minimize the distance
between your fork and -dev. Small enough that you can reliably "git merge
my-recipient-hacks" against master, and be off and running.
As to the state of qpsmtpd-dev.
All the pending pull requests awaiting "review" to the main repo are there.
Many more logging tweaks have been made. I have greatly simplified several of
the plugins that did deferred rejection, in favor of naughty. Most all the
plugins use is_immune().
Between karma, naughty, and is_immune(), the -dev version is way more efficient
than 0.84. Having the DNS plugins process to completion immediately, so that
subsequent plugins can skip processing further improves efficiency. Disabling
user validity tests for naughty connections also enhances security, as it
reduces spammers ability to enumerate valid email addresses. Add in a trained
dspam and the FPs that sneak past spamassassin are nabbed.
Besides writing unit tests, writing the log processors has done the most to
gain a level of consistency and reliability across the plugins. The log
watching scripts add an immense amount of visibility into QP. While watching
the output of logs/summarize.pl, any plugins that aren't working properly
really stand out. That has pushed a number of 'why didn't plugin X do Y?'
issues up to the surface.
Speaking of the log processing plugins, I have written 3 now (log/watch.pl,
log/summarize.pl, and log/logs2sql.pl) and they work for me. But they need to
be refactored some, cleaned up, and some of the hard coded settings moved into
a plugin registry file (a list of plugins, short names, and aliases). Something
along the lines of /etc/services for QP plugins. Something log processors can
references and report uniformly.
QP-dev is "run ready" in the state that's on GitHub. It's not quite a "drop in
qmail-smtpd" replacement, but it's much closer. I plan to deploy on a few
client machines this week. I've been running QP on my server for so long that
I'm sure deploying anew will uncover opportunities to lubricate the process.
Once those are worked in, I plan to tag a dev release.
Then, a long overdue perltidy. :-)
Matt
