Re: Apache James 3.4 install

[Jerry Malcolm]

Mark,

Once you get the build working, there are a couple of logging 
enhancements I STRONGLY recommend you make.

1) 
mailet\base\src\main\java\org\apache\mailet\base\MailetPipelineLogging.java

    This class is called around the invocation of all mailets. It has a 
"logBegin..." and a "logEnd..." method for each mailet. The problem is 
that it tells you that you (for example) "entered the SpamAssassin" 
mailet, and the later that you "ended the SpamAssassin mailet".  The 
problem is that if you get a hundred emails every few minutes, you have 
a hundred log entries telling you that one of those 100 entered and 
exited the mailet... not "WHICH" email entered/exited.  In order to do 
any sort of significant debug, you need to be able to follow a 
particular email through the pipeline.  Easy change.... add 
mail.getName() to both the bgn/end log entries (plus the additional "{}" 
to the first parameter as well):

    public static void logBeginOfMailetProcess(Mailet mailet, Mail mail) {
        LOGGER.debug("Mail: {} Entering mailet: {} ({})", 
mail.getState(), getMailetInfo(mailet), mail.getName());
    }
    public static void logEndOfMailetProcess(Mailet mailet, Mail mail) {
        LOGGER.debug("Mail: {} End of mailet: {} ({})", 
mail.getState(), getMailetInfo(mailet), mail.getName());
    }

Note: these mailet log entries were not showing up anywhere until I 
added another line which had been omitted in log4j.properties in the 
version I had:

    log4j.logger.org.apache.mailet.base=DEBUG, MAILETCONTAINER

Now, you can grep the mailetcontainer log for mail "name" which is a 
unique id string and now easily trace the path of a particular email 
through the pipeline and see where it ended up or where it came to an 
unexpected abrupt end.

2) Second change I recommend:  I was having problems with mail not 
showing up in folders.  I found it has to do with the 'email storing' 
mailets not always obtaining the correct account name. There are three 
mailets in question: ToSenderFolder, ToRecipientFolder, and 
LocalDelivery.  They are almost identical. But the problem was occurring 
for me in all three.  I'll explain what I did for the 
ToSenderFolder.java class.  Propagate to others...

    Open--- 
server\mailet\mailets\src\main\java\org\apache\james\transport\mailets\ToSenderFolder.java. 
  You'll see a line:

              LOGGER.error("Local delivery with ToSenderFolder mailet 
for mail {} with sender {} in folder {}", mail.getName(), sender, folder);
    First, but not critical.. that should not be LOGGER.error.  It 
should be LOGGER.info or LOGGER.debug.  It's not an error condition.

    a) move or copy that line ABOVE the mailboxAppender line since if 
there's a failure, you'll never get to the logger line.

    b) add output of the account (username) it derived to the log 
statement:

            LOGGER.info("Local delivery with ToSenderFolder mailet for 
mail {} with sender {}; username {} in folder {}", mail.getName(), 
sender, username, folder);

    Now you have a fighting chance at figuring out what might be 
happening to your email.  In my case, I found that the username it was 
deriving often had random capital letters in it which makes it not match 
my all-lowercase JAMES_USER account name. This may not have a thing to 
do with your original problem.  But this was definitely a problem for 
me.  My fix was to simply always lowercase the username.  Works for me 
since I can enforce all lowercase usernames.  But not the correct fix 
for the product.

Logging is everything... I'm convinced that to quickly isolate problems 
it is imperative to have the necessary log data.  So I'm on a mission to 
enhance all of the JAMES logging.  I'm going to be opening a 
contribution to JAMES for these soon.  But until then, they are pretty 
trivial changes to make (assuming, of course, we get your build  issues 
resolved...)

Jerry


On 10/25/2019 1:30 PM, Jerry Malcolm wrote:
Is this with the 'master' branch?  Or did you go back to 3.3?

It looks like a java compiler error.  But it looks like it never got 
into a compile.  The error "invalid flag: --release" seems to be a 
compiler invocation error.  One thing... if you are on master, I 
believe I saw a discussion that master now REQUIRES Java 11 to 
compile, but Java 8 to run... But posts today seem to point to Java 11 
being required to run as well soon.   If you are building master but 
not compiling with Java 11, that could be the issue.  3.3 does not 
require Java 11.  And for all intents and purposes, I think 3.3 would 
suffice for your needs, at least for startup.   I suspect it'll be 
much easier to move to 3.4+ (i.e. master) after all of the initial 
startup issues are resolved.


On 10/25/2019 1:10 PM, Mark Gordon wrote:
Thanks for all your replies.

I cloned the repository and I get this error when I try to compile.
https://james.apache.org/server/3/dev-build.html

In the JAMES_SRC_HOME top level directory (where the parent pom.xml
resides), invoke maven with 'mvn' with any of the following command line
arguments:

    - clean - deletes the target directory, making the system ready 
for a
    clean build.
    - compile - compiles the source code.
    - test - run unit tests for James.
    - *package - generates all the James distributions, packed. From the
    root directory, cd to 'server/container/spring/target' to have 
the build
    distribution. Notice, for the latest trunk(revision 1430655+), a 
specific
    profile argument need to be set: '-Pwith-assembly'. The location 
of final
    distributions is also changed to 
'JAMES_SRC_HOME/server/app/target'.*
    - javadocs:javadocs - builds the James javadocs.
    - site - builds the entirety of the James website.


I did:

clone
mvn clean
mvn compile


[INFO] Apache James :: Server :: Web Admin server integration tests 
SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time:  16.635 s
[INFO] Finished at: 2019-10-25T18:04:55Z
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project james-server-util: Fatal error compiling:
invalid flag: --release -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with 
the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR]   mvn <goals> -rf :james-server-util

On Fri, Oct 25, 2019 at 10:11 AM Jerry Malcolm <techst...@malcolms.com>
wrote:

Mark,

I originally cloned 3.3 branch when I started my migration back in
July.  Then I saw official 3.4 was released.  I was surprised that 
there
wasn't an official 3.4 branch in git.  But when I asked, I was told 
that
'master' branch was the one to use.   I haven't checked recently.  But
from what you are saying, I gather that there is still no 3.4 branch in
git (??).  Master will give you the latest code.  But the way I
understand it, master changes every few days as new commits are
finalized, which is going to be a bear to try to replicate 
problems.  It
appears that the last 'official' version in git is the 3.3 branch.
Might be best if you went back to 3.3 for now.

I have been a JAMES user for about 16-17 years.  I have done a few
migrations over that period.  My migration from ver 3.0.0beta5 (circa
2014) to the latest version has been painful at best.  You can read the
'diary' of my migration saga in this users group and the server-dev
group starting 3 months ago.  I have encountered several 'out of the
box' problems during my migration attempt which have cost me many days.
Until now, I have been the lone voice in the wilderness pointing this
out.  I don't want to scare you... but I started playing with migration
in late July.  I went live with the updated JAMES this past Monday
evening.  Even after going live, I found and reported a problem with 
the
three mailets that put mail in folders a couple of days ago. I don't
know if I'm an isolated special case that causes this particular 
problem
or not.  But mail was periodically being discarded, sometimes with an
exception in the logs and sometimes with no exception.  I hacked the
code to get it to work with my situation.  But it needs to be addressed
by someone who understands the 'right' way to do it.

All this to say, I feel your pain.  I love JAMES.  But if I wasn't
simply bullheaded enough to remain, I would have abandoned JAMES 
after a
few weeks of trying to get this migration to work last summer.  I know
just enough about the internal workings of JAMES to be dangerous.  I've
found a few problems. I've used duct-tape and baling wire and now have
everything working.   But I don't know the right way to fix them.  I
love the fact that the JAMES team is diligently working on new
enhancements (I've got a few requests in myself...).  But you are now a
2nd situation approaching abandoning JAMES because you can't get it to
work out of the box.  I believe any 'out of the box' problem should be
viewed as category-1 crit-sit ahead of any enhancement work. That's
where you gain or lose a user.

With what I have learned during my trek, I'm convinced that I can help
you get up and running.  But if we could get one other person from the
JAMES development team that really understands the inner workings to
join me and agree to be a quick-response when out-of-the-box problems
are reported.  I believe it could avoid having people abandon JAMES 
as a
solution.  We MUST get out-of-the-box to be a good experience for 
everyone.

So, I'm willing to work with you.  All of this is very fresh in my
mind.  Let me know more details about what you are seeing, and we'll 
get
you up and running.

Jerry

On 10/25/2019 4:55 AM, Mark Gordon wrote:
Jerry,

Where are you getting the updated 3.4 source or binary?
I cloned the git repository.  That seems like it is 3.5 (the next
version)
I did not see an active branch for 3.4 with bug fixes.

I have used james in the past and I don't remember having this much
trouble
getting it going. I tasked one of my employees with getting james 
going
and
he told me it did not work and reverted to using postfix. I wanted 
to use
james because we are a java shop and will need to write maillets for
processing the email. I wasted a full day messing around with this and
got
nowhere.

-Mark





On Thu, Oct 24, 2019 at 10:46 PM Jerry Malcolm 
<techst...@malcolms.com>
wrote:

Mark,

I'm running 3.4 (+plus a few fixes/tweaks I've had to add...). I'm 
going
to be really uneducated here.... If I am running ActiveMQ or 
RabbitMQ,
it's totally under the covers.  I'm not even sure what it used for,
unless it's for internal passing of the email.   I have done 
nothing to
configure it.  I assume I've had no problems with it either.  
Basically
no clue about anything related to ActiveMQ.  Do you have a special
requirement to get involved with configuring ActiveMQ?

I'm not using POP.  Just IMAP and SMTP.

If all of the logs are set to debug AND if all of the object paths 
that
are writing logs have a definition in log4j.properties, you should 
have
a pretty good trace of the mail all the way from entering the 
pipeline
until JAMES is 'done' with it.  Most of this should be in
mailetcontainer.log.  If you are not seeing a detailed trace of
entering/exiting each mailet through the pipeline, we need to start
there.  Without some log/trace info, it's going to too much 
conjecture
as to what is happening with the mail item in the pipeline.

Are you familiar with the matcher/mailet philosophy and the
mailetcontainer.xml pipeline definition?  I can bring you up to speed
quickly on that if necessary.  Knowing that will help determine the
problem.

Can you give me a bit more detail defining mail 'getting lost'? Is 
this
only on outbound, only on inbound, or both?  And if outbound,
should/does the email get written to the sent folder by James (as
opposed to Thunderbird writing a 'copy')?  Does the email not get
delivered externally?  If inbound, do you see the mail coming in 
(in the
logs), but it just doesn't show up in a destination folder?

Jerry

On 10/24/2019 7:15 PM, Mark Gordon wrote:
James,

Are you running 3.4?  I don;t think the DNS issue is the 
problem.  It
is
just really odd.  The log looks correct.  Even when I kick it up to
TRACE
for most packages.  THe email just disappears,

Pop and iMAP seem to be working.  SMTP accept the email but then it
just
disappears.

I just checked out the GIT repository for the project. I am going to
try
to build. the package.

If you are not using ActiveMQ are you using RabbitMQ? how is that
configured?

Thanks
Mark





On Thu, Oct 24, 2019 at 4:32 PM Jerry Malcolm 
<techst...@malcolms.com>
wrote:

I'm not using ActiveMQ.  So I can't make a recommendation there. I
haven't had any problems with DNS.  What do the error messages say?

On 10/24/2019 5:42 PM, Mark Gordon wrote:
Thanks for the response James.

Should I revert back to a previous version?
Is this the first version using ActiveMQ?   SHould I be using an
alternative to ActiveMQ?

I am getting DNS errors when it is trying to resolve localhost.
  Looks
like it is also struggling with the hostname.

-Mark












On Thu, Oct 24, 2019 at 2:50 PM Jerry Malcolm <
techst...@malcolms.com>
wrote:

Mark,

I'm right with you.... It turns out that log4j.properties has
several
omissions.  Several were fixed a month or so ago.  Not sure if 
your
version picked up the changes.  But I've added even more 
additions
to
get the logging info I have needed in my migration journey.

Here is my list of logger control lines from my current
conf/log4j.properties.  Change all of the INFOs to DEBUGs.  There
might
be even more log data if you change everything to TRACE 
instead of
DEBUG.  Just don't run it too long in that state unless you 
have a
huge
harddrive.

Look for mailet log entries in mailetcontainer.log, specifically
from
ToSenderFolder, ToRecipientFolder, and LocalDelivery mailets.
Hopefully
with the heavy logging, it'll show you the folder it puts mail 
into.

One thing... if you find ANY exception dumps in the logs with the
ReadOnly exception on folders, let's talk.  I've learned a lot 
about
that problem in the past couple of days.  And if you get that
exception
your mail will NOT be stored anywhere.  But I think I've found 
a fix
for
that if you are getting it.

Let me know.

--Jerry

Log4j.properties (not the entire file... just the bottom part 
of the
file)
log4j.logger.etm.core.monitor.EtmMonitor= INFO, CONS, FILE
log4j.logger.org.apache.james.dnsservice=INFO, DNSSERVICE
log4j.logger.org.apache.james.domainlist=INFO, DOMAINLIST
log4j.logger.org.apache.james.fetchmail=INFO, FETCHMAIL
log4j.logger.org.apache.james.imapserver=INFO, IMAPSERVER
log4j.logger.org.apache.james.lmtpserver=INFO, LMTPSERVER
log4j.logger.org.apache.james.mailboxmanager=INFO, MAILBOXMANAGER
log4j.logger.org.apache.james.mailetcontainer=INFO, 
MAILETCONTAINER
log4j.logger.org.apache.james.mailetcontext=INFO, MAILETCONTAINER
log4j.logger.org.apache.james.mailprocessor=INFO, MAILETCONTAINER
log4j.logger.org.apache.james.mailqueuefactory=INFO,
MAILQUEUEFACTORY
log4j.logger.org.apache.james.mailrepositorystore=INFO,
MAILREPOSITORYSTORE
log4j.logger.org.apache.james.mailspooler=INFO, MAILETCONTAINER
log4j.logger.org.apache.james.pop3server=INFO, POP3SERVER
log4j.logger.org.apache.james.protocols.api=INFO, SMTPSERVER
log4j.logger.org.apache.james.protocols.imap=INFO, IMAPSERVER
log4j.logger.org.apache.james.protocols.smtp=INFO, SMTPSERVER
log4j.logger.org.apache.james.smtpserver=INFO, SMTPSERVER
log4j.logger.org.apache.james.spamassassin=INFO, MAILETCONTAINER
log4j.logger.org.apache.james.transport=INFO, MAILETCONTAINER
log4j.logger.org.apache.james.usersrepository=INFO, 
USERSREPOSITORY
log4j.logger.org.apache.james.virtualusertable=INFO,
VIRTUALUSERTABLE
log4j.logger.org.apache.mailet.base=INFO, MAILETCONTAINER

On 10/24/2019 4:16 PM, Mark Gordon wrote:
I have a virtual ubuntu 18.04 install with postgres 10.  I am
trying
to
get
apache james 3.4 to work as a simple smtp server and pop3 
server.

I unzip'ed the install and edited the james.

I was getting an error on startup so I did this:

mailetcontainer.xml

Was:
             <mailet matcher="All" class="WithPriority">
                 <value>8</value>
             </mailet>
Changed:
             <mailet matcher="All" class="WithPriority">
<priority>8</priority>
             </mailet>



I am running smtp on two ports
smtpserver.xml

<bind>0.0.0.0:25,0.0.0.0:27</bind>

SEtting auth required to true
<authRequired>true</authRequired>

My ISP is blocking 25 so I am sending on port 27.

I am adding my domain via james-cli.
<domainlist 
class="org.apache.james.domainlist.jpa.JPADomainList">
             <!--
<autodetect>true</autodetect>
<autodetectIP>true</autodetectIP>
-->
</domainlist>

I used james-cli to add a domain and user with a password.  I
checked
the
derby database and both were added correctly.

I have my mx record set at my DNS provider.   I am using
thunderbird
and
I
have my smtp server set to this system

I send an email from gmail to the user on this system.

INFO  21:06:39,470 | 
org.apache.james.smtpserver.SendMailHandler |
Successfully spooled mail
Mail1571951199365-0550fcd4-2530-4595-a4bd-8e2900e90728 from
MaybeSender{mailAddress=Optional[m...@yyyyyyy.com]} on
mail-vs1-f46.google.com/209.85.217.46 for [m...@xxxxxxxx.com]

XXXXX and YYYYY are changed...

I then pop my mail from thunderbird and it connects OK I know 
this
because
if I enter the wrong password I get an error.  and POP always 
says
that
no
new emails.

it is the same when i send from thunderbird to a gmail account
using
james
as my smtp server.I get this in the log:

INFO  21:12:10,931 |
org.apache.james.domainlist.lib.AbstractDomainList |
Local host is: email.
INFO  21:12:11,071 | 
org.apache.james.smtpserver.SendMailHandler |
Successfully spooled mail
Mail1571951531021-8f8e30e7-ed3d-4520-8649-b4203ce5b408 from
MaybeSender{mailAddress=Optional[m...@xxxxx.com]} on
63.228.129.34/63.228.129.34 for [m...@yyyyyy.com]

but I never get the email.

Where can I go see the spooled email?  I changed the logging to
DEBUG
but I
am still not getting much.  POP is not showing much of 
anything....
actually nothing.

HELP!!!!!!!!

Thanks,
Mark



























---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: 
server-user-h...@james.apache.org


--------------------------------------------------------------------- 

To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org