Finally got there, without TLS for now, that is tomorrows job!
I am posting my findings in case they can help someone in the future.
As expected I was really not understanding how the parameters were used
and therefore had them all wrong.
ptload does do recursive searches and the behaviour for the first (with
attributeDescription: 1.1) seems to be completely OK, the problem is if
the settings for the other searches are wrong, it doesn't progress to them.
It looks like saslauthd caches successfully authentications because I
only see it in the traffic the first time I make a successful connection
for a user, I didn't round to working out how to clear the cache, but as
far as I can tell this step is runs after ptload gets the authorization
details, so if ptloader doesn't work, no authentication.
Users are in ou=SBSUsers,ou=Users,ou=MyBusiness,dc=MyCompany,dc=com
Groups are in ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=com
Groups and users have 'sAMAccountName' which is single word, the part
before @, for a group the 'name' attribute is the same single word, but
for users it is firstname lastname (this is just how I put users and
groups in the directory). To avoid confusion for anyone reading my conf
later I am using 'sAMAccountName' for users and 'name' for groups
although 'sAMAccountName' will also work for groups and may be more
reliable for some setups.
Active directory stores group members in the 'member' attribute of each
group as dn's. There is no attribute that matches sAMAccountNames to
groups. Also there is no attribute of a user which lists the groups it
is in, that is to say, I can't see one in the LDAP attributes and whilst
memberOf can be used from powershell I suspect it is doing something
other than matching an ldap attribute, because it returned nothing for me.
The token %u passes the full username including any domain from the
client, %U strips the domain off if present - in my setup we only login
with the part before the @ so these work the same, but hopefully I have
used %U everywhere to ensure compatability with 'sAMAccountName'
The token I had failed to note earlier is %D which passes the dn that is
found in the previous search, we need this to filter for groups by their
'member' attribute.
So I need to search for a user in the user base using a filter, but use
the default user attribute instead of setting one (seems to default to
(objectClass=*)).
I need to filter groups using the group base, there is no attribute
option for the group. I think this just makes sure the member search
only looks in groups, in my case I have used the filter to restrict to
distribution groups by specifying not some security group types I have
identified in my directory, they may not be correct or exhaustive for
other directories. This is all used in the member search and is not a
search in its own right, at least as far as I can tell.
I also need to search for members in the groups and I use the filter
method. With filter the attribute needs to return a single value - the
group name. It seems that with the attribute method the attribute can
hold multiple values so presumably you would use this if your user has
an attribute with a list of groups it is a member of? So I need to
match my user in the 'member' attribute of the groups (limited by the
group base and filter), which means I need to supply the users dn
instead of the login name, i.e. %D instead of %U. And what I want back
is the groups name or sAMAccountName.
So when ptload finds the user, gets the users dn, and searches for
members of the groups by dn it returns all the (distribution) groups
that the user is a member of, and then I guess exits successfully
passing control to saslauthd to authenticate.
So, the section of my imapd.conf that does this, with our actual company
name and bind credentials altered, but the ou's left the same otherwise,
looks like this:
##
## Other LDAP items
## This is for AUTHORIZATION, we use saslauthd for AUTHENTICATION
##
# These may or may not be required, but since the idea is to keep all
# the user and group information in active directory we must surely need
# to tell cyrus how to find it?
#
# First we need to tell it to use ptloader for authorization
auth_mech: pts
#
# And tell ptloader to use LDAP
pts_module: ldap
ptloader_sock: /var/lib/cyrus/ptclient/ptsock
#
# The defaults for the cache settings should be fine
# db type defaults to twoskip, db_path to configdirectory/ptscache.db
# other settings are only for kerberos module
ptscache_db: twoskip
ptscache_db_path: /var/lib/cyrus/ptclient/ptscache.db
#
# General settings
# Probably also useful to tell cyrus where the LDAP is
#ldap_uri: ldaps:/temp-std.MyCompany.local:636
ldap_uri: ldap://temp-std.MyCompany.local:389
ldap_bind_dn:
cn=auser,ou=SBSUsers,ou=Users,ou=MyBusiness,dc=MyCompany,dc=local
ldap_password: somethingsecret
# Don't attempt SASL for authorization, it is used for authentication
already
ldap_sasl: 0
# For ldaps we will need version 3
ldap_version: 3
# Make sure cyrus can find the CA file to accept LDAP servers certificate
#ldap_ca_dir: /etc/ssl/certs/
# And ensure that we check the certificate
#ldap_verify_peer: 1
# Might also be worth specifying the ciphers we want
#ldap_ciphers: TLSv1.3:TLSv1.2:+TLSv1:+HIGH:!aNULL:@STRENGTH
# Set a limit on number of record for single query
ldap_size_limit: 100
#
# User lookups
# Set a default search base although it looks like we can set separately
for users and groups
# This filter works to make sure the account is a user and not disabled
ldap_base: ou=MyBusiness,dc=MyCompany,dc=local
ldap_scope: sub
ldap_filter:
(&(objectClass=person)(sAMAccountName=%u)(!(userAccountControl=514)))
# But lets have a simpler testing filter
#ldap_filter: (sAMAccountName=%U)
#
#
# Groups - we will need these for shared folder ACIs
# Set a filter to identify a group, this one ensures it is a
distribution group
# and not a security group
ldap_group_base: ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
ldap_group_scope: sub
ldap_group_filter:
(&(objectClass=group)(!(groupType=2147483656))(!(groupType=2147483652)))
#
# Method to extract members from the group, this is poorly documented
but after much trial an
# error, the 'member' attribute in AD groups contains distinguished
names (DNs) so need to
# use a filter to return all of the group names that contain the DN for
our supplied username
# in their 'member' attribute'. %D is the token for the user dn
# The attribute for the group name we set to 'name' although
'sAMAccountName' returns the
# same value it is a bit confusing when debugging.
ldap_member_base: ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
ldap_member_scope: sub
ldap_member_method: filter
ldap_member_filter: (member=%D)
ldap_member_attribute: name
And with this configuration I can now authenticate users and ptdump now
has information to display, which includes the groups for all users.
I still need to test with TLS re-enabled at each level (between cyrus
and ldap server, and between client and cyrus) but I do not forsee this
as an issue.
The only issue I see is that ptload has converted the group names to all
lowercase and most of them had capitalisation, I assume this will
confuse cyrus since the shared folders I will import have capitals but
I'm sure I can rename them in cyradm if this is an issue and rejig the
ACLS to suit.
I hope this info helps someone else.
On 17/06/2021 13:31, Jim Wallis wrote:
I feel like I am getting somewhere.
If I change the search base back to dc=MyCompany,dc=local in
imapd.conf (matching saslauthd), when I run imtest with pts enabled I
do get searchResRef stanzas back from the server. These represent
other branches of the directory.
Lightweight Directory Access Protocol
LDAPMessage searchResEntry(2) "CN=Test
User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local" [1 result]
messageID: 2
protocolOp: searchResEntry (4)
searchResEntry
objectName: CN=Test
User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local
attributes: 0 items
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResRef(2)
messageID: 2
protocolOp: searchResRef (19)
searchResRef: 1 item
LDAPURL:
ldap://ForestDnsZones.MyCompany.local/DC=ForestDnsZones,DC=MyCompany,DC=local
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResRef(2)
messageID: 2
protocolOp: searchResRef (19)
searchResRef: 1 item
LDAPURL:
ldap://DomainDnsZones.MyCompany.local/DC=DomainDnsZones,DC=MyCompany,DC=local
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResRef(2)
messageID: 2
protocolOp: searchResRef (19)
searchResRef: 1 item
LDAPURL:
ldap://MyCompany.local/CN=Configuration,DC=MyCompany,DC=local
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResDone(2) success [1 result]
messageID: 2
protocolOp: searchResDone (5)
searchResDone
resultCode: success (0)
matchedDN:
errorMessage:
[Response To: 679124]
[Time: 0.000733000 seconds]
When I look at the searches (and wireshark hung so I had to start a
new session to get these) I see:
Pts enabled:
Lightweight Directory Access Protocol
LDAPMessage searchRequest(2) "dc=MyCompany,dc=local" wholeSubtree
messageID: 2
protocolOp: searchRequest (3)
searchRequest
baseObject: dc=MyCompany,dc=local
scope: wholeSubtree (2)
derefAliases: neverDerefAliases (0)
sizeLimit: 1
timeLimit: 5
typesOnly: False
Filter: (sAMAccountName=testuser)
filter: equalityMatch (3)
attributes: 1 item
AttributeDescription: 1.1
[Response In: 588]
PTS disabled:
Actually, this is coming through, saslauthd must have cached the last
search and is authenticating without contacting the server, but I'm
pretty sure the only difference in the search is that saslauthd is
looking for attribute "dn" where ptload is looking for "1.1"
Whatever ldap_member_attribute I set, the search always has
AttributeDescription: 1.1 so is it an issue with the attribute that
ptload is requesting? I need to go through the man page for imapd.conf
again and see if i have missed something that defines an attribute
which is overriding ldap_user_attribute, ldap_group_attribute and
ldap_member_attribute.
Thanks
Jim
On 17/06/2021 11:29, Jim Wallis wrote:
Hi Andrew,
If I am understanding the packet logs correctly, the LDAP server is
responding (only inserting the ldap parts of the log entry):
The search:
Lightweight Directory Access Protocol
LDAPMessage searchRequest(2)
"ou=MyBusiness,dc=MyCompany,dc=local" wholeSubtree
messageID: 2
protocolOp: searchRequest (3)
searchRequest
baseObject: ou=MyBusiness,dc=MyComapny,dc=local
scope: wholeSubtree (2)
derefAliases: neverDerefAliases (0)
sizeLimit: 1
timeLimit: 5
typesOnly: False
Filter: (sAMAccountName=testuser)
filter: equalityMatch (3)
equalityMatch
attributeDesc: sAMAccountName
assertionValue: testuser
attributes: 1 item
AttributeDescription: 1.1
[Response In: 557126]
The results:
Lightweight Directory Access Protocol
LDAPMessage searchResEntry(2) "CN=Test
User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local" [1 result]
messageID: 2
protocolOp: searchResEntry (4)
searchResEntry
objectName: CN=Test
User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local
attributes: 0 items
[Response To: 557125]
[Time: 0.047382000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResDone(2) success [1 result]
messageID: 2
protocolOp: searchResDone (5)
searchResDone
resultCode: success (0)
matchedDN:
errorMessage:
[Response To: 557125]
[Time: 0.047382000 seconds]
I did change my ldap base from dc=MyCompany,dc=local when I noticed
it seemed to be querying additional parts of the directory, although
I am quite sure all users are unique. It is now:
ldap_base: ou=MyBusiness,dc=MyCompany,dc=local
With the group and member bases set to DistributionGroups:
ldap_group_base:
ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
ldap_member_base:
ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
The LDAP is still mostly laid out as per the MS Small Business Server
2003 way of setting up the schema (but on 2012 R2), I have no idea if
it is normal for an AD installation. The distribution groups ou is
one I added to try to keep my distribution groups away from the
security groups.
Anyway, going back to the results, I don't understand this at low
level but you are asking about searchresref and the server seems to
be giving searchResEntry - is this as simple as MS implementing
non-standard methods?
But, in searchResEntry, whilst it finds the objectName, which is a
DN, it is returning 0 attributes, so it can't be as simple as
ptloader doesn't understand the response, the response also appears
incomplete? (I have ldap_member_attribute: set and have tried all
sorts of attributes that I know are in the directory, including
sAMAccoutName which we can see it recognises in the search itself).
I already set the debug option on ptloader (forgot to say in first
post) in cyrus.conf:
ptloader cmd="ptloader -d1" listen="var/lib/cyrus/ptclient/ptsock"
And have tried increasing from 1 but as others have reported it does
not seem to affect verbosity. The debug output goes in syslog similar
to the lines I had before although now that I at least have the bind
working I now get:
cyrus/ptloader[63125]: searching ou=MyBusiness,dc=MyCompany,dc=local
with (sAMAccountName=testuser) failed
Now, what is interesting and needs more investigation later, is that
if I disable pts again and use the same imtest command, it first
preforms a bind using the saslauthd user (so I know sasluthd is
making the bind), the results do include searchResRef entries
(several, I think because I am searching a at dc level?) and cyrus
goes on to make a subsequent bind as testuser, using testuser's
password (in the pts examples the initial bind is by the cyrus user,
and there is no subsequent bind for the test user).
I need to make a site visit now so will dig into this some more later.
thanks for your help so far!
Jim
On 16/06/2021 18:45, AndrewHardy via Info wrote:
Hi Jim,
It sounds like you’ve spotted a few issues and have fixed them. I’m
quite interested to know what your imapd configuration is for pts
and also how the configuration knows what OU the user you’re testing
resides in? I haven't personally set up LDAP auth with Cyrus
although it’s been something I’ve been wanting to do for some time
(so showing a bit of interest in this thread).
When you mentioned nothing seems to get returned to ptloader, is it
possible to confirm if there was a ldap response from the user
search in the ldap packet trace?
Was the result count returned and what was the format of
searchresref? If that is returned on the wire (confirming user was
found after the initial search) then I guess next step is to debug
ptloader to see what that’s expecting to receive and whether the
ldap response data gets there ? I’m assuming the search does find
the user of interest and the result count is 1 and the distinguished
name of the object is returned over the wire to the mail server? If
the user is searched but not found, that’s likely a place to
investigate e.g where in the tree is it trying to find the user -
does the attribute you’re checking for and the user searched match up?
For ptloader, there is a debug option. May be worth enabling that
although not sure what that technically offers in terms of
troubleshooting info. Im a bit unsure on how to attach strace in
this context, which process ptloader operates under or whether it’s
it’s own seperate process with its own pid. If it’s the later it’s
probably going to be a bit easier. Bit of trial and error most likely.
Regards
Andrew
Sent from my iPhone
On 17/06/2021, at 03:48, Jim Wallis <[email protected]> wrote:
Hi Andrew,
This has been very helpful. I didn't think of using wireshark
before, it has enabled me to at least sort out a typo in my bind DN
, but I am still not much further on.
With auth_mech: pts commented out, and using an imtest line like:
imtest -u test -a test -w passfortest localhost
I can see from the wireshark capture that Saslauthd is binding as
the test user and imtest authenticates OK
with auth_mech: pts uncommented I need to provide bind credentials
in imapd.conf (I use the ones for my cyrus user so I can tell what
is being passed). This cyrus user binds successefully and looks up
the test user, but nothing seems to get returned to ptloader, so I
probably need to spend more time thinking about what attribute I
need to return (I have tried several likely candidates). It doesn't
look as though saslauthd is used at all when I have pts on. Imtest
responds not authenticated.
More work to do... I guess I need to learn how to use strace.
Thanks
Jim
On 15/06/2021 20:30, AndrewHardy via Info wrote:
Hi Jim,
I’m not sure specifically what your issue is here but I’ll help
provide some options for debugging ldap (specific to the windows
logging question). I’d suggest dropping the TLS 636 LDAPS and just
start with standard LDAP 389, removing TLS complexity whilst you
validate a working configuration.
I’d suggest also just running wireshark on the domain controller
where you expect ldap connections to reach and capture the 389
traffic with a filter of your source mail server (or component
doing the authentication). You’d be able to assess the ldap flow;
whether the server sees it or not (and what the response was from
the server, if any).
You could also open another terminal, open saslauthd with strace
and dump the authentication process to file to analyse what may be
happening. If you can’t wireshark the target authentication
service, you could perhaps tcpdump on the ldap client side, drop
that into a .pcap file then open this in wireshark on your desktop
for further analysis.
I’m hoping those potential options can help verify what is/isn’t
happening. An absence of traffic leaving the server would imply
something on the client side, ruling out the ldap server at fault
- at least initially.
Hope this helps simplify troubleshooting and helps confirm what
is/isn’t happening at least from an ldap perspective. Unsure of
enabling ldap debugging in windows, I’ve searched for that for
years and never really come up with a good solution. Wireshark
and/or tcpdump has helped me solve all problems I’ve had with
ldap/authentication although ldaps can be a bit more problematic
when just using packet tracing software. Can still come in useful
though.
Regards
Andrew
Sent from my iPhone
On 16/06/2021, at 06:30, [email protected] wrote:
It seemed like it would be a good idea to use the existing Active
Directory DC as the LDAP source for all mail users.
I got cyrus (3.2.6 from Buster backports) running using saslauthd
for authentication against the directory, and test users could
authenticate OK and see their mailbox in Thunderbird. relevant
entires in imapd.conf:
allowplaintext: yes
sasl_mech_list: PLAIN
sasl_pwcheck_method: saslauthd
Because I want to use some groups for shared folders and
distribution lists, I also want to authorize users and groups
against the directory so enabled:
auth_mech: pts
pts_module: ldap
ptloader_sock: var/lib/cyrus/ptclient/ptsock
ldap_uri: ldaps:/companydomaincontroller:636
ldap_bind_dn: a cn that works in other binds
ldap_password: the password for above
ldap_sasl: 0
ldap_version: 3
ldap_ca_dir: /etc/ssl/certs (which is where the ca cert that
works with saslauthd resides)
ldap_verify_peer: 1
ldap_base: base that other binds can successfully find users from
ldap_scope: sub
ldap_filter: (sAMAccountName=%u) (I have tried more complex
filters and decided on this as one that should work)
ldap_user_attribute: mail
ldap_size_limit: 1024
I have commented out the group member stuff for now, need to walk
before I can run!
With these options enabled, no one can authenticate, even though
my understanding is that authentication is distinct from
authorization. Thunderbird users can no longer login and imtest
for user cyrus (which is in the directory) gives the same output
up to a line :
C: A01 AUTHENTICATE PLAIN {a hash}
S: A01 NO authentication failure
Authentication failed. generic failure
Security strength factor: 256
(the SSF is reported the same, but I have for now only asked for
level 0)
In /var/log/syslog I find the following lines:
cyrus/ptloader[8230]: ldap_initialize failed (ldaps:/DC:636)
cyrus/imap[8229]: timeout_select exiting. r = 1; errno = 0
cyrus/imap[8229]: timeout_select: sock = 11, rp = 0x7ffecbb6ad30,
wp = 0x0, sec = 30
cyrus/imap[8229]: timeout_select exiting. r = 1; errno = 0
cyrus/imap[8229]: ptload read data back
cyrus/imap[8229]: ptload(): bad response from ptloader server:
ptsmodule_connect() failed
cyrus/imap[8229]: No data available at all from ptload()
cyrus/imap[8229]: ptload completely failed: unable to canonify
identifer: cyrus
cyrus/imap[8229]: SASL bad userid authenticated
cyrus/imap[8229]: badlogin: localhost [::1] PLAIN (-notset-)
[SASL(-6): can't request information until later in exchange:
Information that was requested is not yet available.]
I have been searching for answers for days and at one point found
a reference that claimed ptsloader is not enabled by default in
Debian, so I have downloaded the source package and compiled
cyrus-imaps using a configure script based on the Debain default
config options with some extra options: --with-auth=pts
--with-pts=ldap and --with-ldap
This has made no difference.
I have also downlaoded the 3.4.1 source package from experimental
and compiled with the same options ands still no difference to
behaviour so suspect this is a red herring?
So why is ptloader not retrieving any data?
The ldap_bind credentials I have given it work fine with
saslauthd or postfix or ldapsearch.
Presumably then my filter and user attributes are bad? But I
can't see why.
What is the order of operations within cyrus?
I assume that it authenticates first using saslauthd, and then
uses the same username to check authorization in pts, but
enabling PTS seems to prevent authentication. Although in syslog
it is suggesting that SASL has authenticated, but with a bad userid?
The windows DC uses a directory migrated from an older one on a
small business server where microsoft recommended using an
internal .local domain which has always been a bit of a headache
for me. This means that our search base is a DC=local, as are the
bind DN and userPrincipleName, but the mail and proxyAddresses
email addresses are all .com
Is the problem related to this? do I need to enable virtual
domains and/or cross realm authentication for ptloader to get a
response from the server?
Is there any way to call ptloader outside of master to try to
work out exactly what is being passed and what result it achieves?
Also, how does ptdump work? I get no indication that it has done
anything, is this simply because ptloader has never yet obtained
any data for it to dump?
As for checking the LDAP server logs, if anyone knows how I can
do this on windows 2012 please advise! Looking at directory
services in event viewer I see very few entries and nothing
relating to communication from my mail server. I assume I need to
enable a different log level but I can't find out how or what.
All the examples I can find are based on openldap installations,
is what I am trying to do possible, or are the Active Directory
schema completely incompatible with ptloader?
Jim Wallis
*Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions
<https://cyrus.topicbox.com/groups/info> + participants
<https://cyrus.topicbox.com/groups/info/members> + delivery options
<https://cyrus.topicbox.com/groups/info/subscription> Permalink
<https://cyrus.topicbox.com/groups/info/T1c604a219c5fa805-Mcd1753c617bd13026db356bb>
------------------------------------------
Cyrus: Info
Permalink:
https://cyrus.topicbox.com/groups/info/T1c604a219c5fa805-M37210cd1ad4e36ee341a3eeb
Delivery options: https://cyrus.topicbox.com/groups/info/subscription
