Re: using nscd (ldap) makes passwd/group disappearing while installing ports

2012-02-01 Thread Stefan Esser
Am 01.02.2012 01:03, schrieb Benjamin Lee:
> What's going on is:
> 
> 1) The port checks if the group exists
> 2) nscd caches that the group does not exist in its negative cache
> 3) pw(8) creates the group then checks if it exists
> 4) nscd returns the negative cache entry (group does not exist)
> 
> This causes pw(8) to error since it expects the group that it just
> created to exist.

I had suggested before, that nscd be changed to only rely on cached
negative entries, if they have repeatedly been seen in the underlying
files/data-bases.

E.g. only consider negative entries valid after they have been
repeatedly stored into the cache (I'd think 3 times are a reasonable
number). That way, the first lookup of an account or group will lead to
an entry with count=1, which is found during cache lookup but which is
ignored due to a too low "verification count". If the negative lookup is
repeated 3 times within some reasonable time window (say 60 seconds),
then it is to be considered verified.

This will make the above sequence of queries and modifications of the
passwd or group databases work with caching.

This concept does not protect against scenarios where the negative cache
entry is made active by several queries (e.g. manual checking for the
presence of an account or group before the software installation repeats
these tests). That is the reason to ask for more than 2 negative replies
(3, perhaps better 5) before a negative cache entry is trusted. The main
purpose of the cache (reduced latency for positive queries, limited load
due to negative queries) will still be maintained.

>> I also have this error very often when rebuilding/updating or even
>> installing cups when "nscd" is enabled. A simple restart of nscd helps
>> in most cases, most times I need to disable "cache" tag in
>> /etc/nsswitch.conf, then everything runs smooth.
>>
>> Well, this behaviour is since a couple of years now, occurs sporadic. I
>> have had in FreeBSD 7, 8, 9 and I see it in 10. What is it?
>>
>> I like the cache facility, since in domains with a lot of users
>> searching LDAP takes some time and caching help keeping traffic and
>> latency short. But the namservice caching mechanism seems to be
>> unreliable. What is up there?
> 
> You should put "files" before "cache" in /etc/nsswitch.conf, e.g.:
> 
> group: files cache ldap
> passwd: files cache ldap
> 
> The problem is that tools that modify the passwd and group files, like
> pw(8), don't invalidate nscd's negative cache entries when making
> changes.

You point out an alternative to making negative entries trusted only
after they have been repeatedly entered into the cache: Tools that are
used to modify the passwd/group databases might signal their changes to
nscd. They could either purge the modified caches for the current or for
all users, or they could just clear the single affected entry. In each
case, nscd needs to re-fetch the modified (and possibly all other
cached) entries. A simple implementation of such invalidation could be
to invoke "nscd -I" after each modification performed by "pw".

Still I think that the reduced trust in negative entries that have not
been repeatedly tested is the best solution.

I had looked into implementing such a logic in the cache, a few months
back. It took more effort than I had hoped due to the way the cache is
implemented, but I still think it should be possible without major
changes to nscd.

Regards, STefan
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: using nscd (ldap) makes passwd/group disappearing while installing ports

2012-02-01 Thread Daniel O'Connor

On 01/02/2012, at 19:25, O. Hartmann wrote:
>> The problem is that tools that modify the passwd and group files, like
>> pw(8), don't invalidate nscd's negative cache entries when making
>> changes.
>> 
>> 
> 
> Thank you for the explanation.

How feasible would it be for pw to try and notify nscd? Or for nscd to monitor 
the passwd & group files?

Either would be somewhat racy though..

--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C






___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: using nscd (ldap) makes passwd/group disappearing while installing ports

2012-02-01 Thread O. Hartmann
On 02/01/12 01:03, Benjamin Lee wrote:
> On 01/31/2012 03:03 PM, O. Hartmann wrote:
>> I'm using on a couple of servers the nameservice cache dameon nscd and
>> cache "group", "passwd" and "sudoers". Backend is LDAP, but local files
>> should searched first. then ldap. cache is searched the very first even
>> before files.
>>
>> Well, I'd expect that if a group is present, like "cups" or "dhcp" and
>> reside in the local file (/etc/group or /etc/passwd), they are cached.
>>
>> Installing net/isc-dhcp42-server fails with this error:
>>
>>
>> gmake[1]: Leaving directory
>> `/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2/server'
>> gmake[1]: Entering directory
>> `/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2'
>> gmake[1]: Nothing to be done for `all-am'.
>> gmake[1]: Leaving directory
>> `/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2'
>> ===>  Installing for isc-dhcp42-server-4.2.3_2
>> ===>   Generating temporary packing list
>> ===> Creating users and/or groups.
>> Creating group `dhcpd' with gid `136'.
>> pw: group disappeared during update
>> *** Error code 70
>>
>> Stop in /usr/ports/net/isc-dhcp42-server.
>> *** Error code 1
>>
>> Stop in /usr/ports/net/isc-dhcp42-server.
> 
> What's going on is:
> 
> 1) The port checks if the group exists
> 2) nscd caches that the group does not exist in its negative cache
> 3) pw(8) creates the group then checks if it exists
> 4) nscd returns the negative cache entry (group does not exist)
> 
> This causes pw(8) to error since it expects the group that it just
> created to exist.
> 
>> I also have this error very often when rebuilding/updating or even
>> installing cups when "nscd" is enabled. A simple restart of nscd helps
>> in most cases, most times I need to disable "cache" tag in
>> /etc/nsswitch.conf, then everything runs smooth.
>>
>> Well, this behaviour is since a couple of years now, occurs sporadic. I
>> have had in FreeBSD 7, 8, 9 and I see it in 10. What is it?
>>
>> I like the cache facility, since in domains with a lot of users
>> searching LDAP takes some time and caching help keeping traffic and
>> latency short. But the namservice caching mechanism seems to be
>> unreliable. What is up there?
> 
> You should put "files" before "cache" in /etc/nsswitch.conf, e.g.:
> 
> group: files cache ldap
> passwd: files cache ldap
> 
> The problem is that tools that modify the passwd and group files, like
> pw(8), don't invalidate nscd's negative cache entries when making
> changes.
> 
> 

Thank you for the explanation.

Cheers,
Oliver



signature.asc
Description: OpenPGP digital signature


Re: using nscd (ldap) makes passwd/group disappearing while installing ports

2012-01-31 Thread Benjamin Lee
On 01/31/2012 03:03 PM, O. Hartmann wrote:
> I'm using on a couple of servers the nameservice cache dameon nscd and
> cache "group", "passwd" and "sudoers". Backend is LDAP, but local files
> should searched first. then ldap. cache is searched the very first even
> before files.
> 
> Well, I'd expect that if a group is present, like "cups" or "dhcp" and
> reside in the local file (/etc/group or /etc/passwd), they are cached.
> 
> Installing net/isc-dhcp42-server fails with this error:
> 
> 
> gmake[1]: Leaving directory
> `/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2/server'
> gmake[1]: Entering directory
> `/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2'
> gmake[1]: Nothing to be done for `all-am'.
> gmake[1]: Leaving directory
> `/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2'
> ===>  Installing for isc-dhcp42-server-4.2.3_2
> ===>   Generating temporary packing list
> ===> Creating users and/or groups.
> Creating group `dhcpd' with gid `136'.
> pw: group disappeared during update
> *** Error code 70
> 
> Stop in /usr/ports/net/isc-dhcp42-server.
> *** Error code 1
> 
> Stop in /usr/ports/net/isc-dhcp42-server.

What's going on is:

1) The port checks if the group exists
2) nscd caches that the group does not exist in its negative cache
3) pw(8) creates the group then checks if it exists
4) nscd returns the negative cache entry (group does not exist)

This causes pw(8) to error since it expects the group that it just
created to exist.

> I also have this error very often when rebuilding/updating or even
> installing cups when "nscd" is enabled. A simple restart of nscd helps
> in most cases, most times I need to disable "cache" tag in
> /etc/nsswitch.conf, then everything runs smooth.
> 
> Well, this behaviour is since a couple of years now, occurs sporadic. I
> have had in FreeBSD 7, 8, 9 and I see it in 10. What is it?
> 
> I like the cache facility, since in domains with a lot of users
> searching LDAP takes some time and caching help keeping traffic and
> latency short. But the namservice caching mechanism seems to be
> unreliable. What is up there?

You should put "files" before "cache" in /etc/nsswitch.conf, e.g.:

group: files cache ldap
passwd: files cache ldap

The problem is that tools that modify the passwd and group files, like
pw(8), don't invalidate nscd's negative cache entries when making
changes.


-- 
Benjamin Lee
http://www.b1c1l1.com/



signature.asc
Description: OpenPGP digital signature


using nscd (ldap) makes passwd/group disappearing while installing ports

2012-01-31 Thread O. Hartmann
I'm using on a couple of servers the nameservice cache dameon nscd and
cache "group", "passwd" and "sudoers". Backend is LDAP, but local files
should searched first. then ldap. cache is searched the very first even
before files.

Well, I'd expect that if a group is present, like "cups" or "dhcp" and
reside in the local file (/etc/group or /etc/passwd), they are cached.

Installing net/isc-dhcp42-server fails with this error:


gmake[1]: Leaving directory
`/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2/server'
gmake[1]: Entering directory
`/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2'
gmake[1]: Nothing to be done for `all-am'.
gmake[1]: Leaving directory
`/usr/ports/net/isc-dhcp42-server/work/dhcp-4.2.3-P2'
===>  Installing for isc-dhcp42-server-4.2.3_2
===>   Generating temporary packing list
===> Creating users and/or groups.
Creating group `dhcpd' with gid `136'.
pw: group disappeared during update
*** Error code 70

Stop in /usr/ports/net/isc-dhcp42-server.
*** Error code 1

Stop in /usr/ports/net/isc-dhcp42-server.



I also have this error very often when rebuilding/updating or even
installing cups when "nscd" is enabled. A simple restart of nscd helps
in most cases, most times I need to disable "cache" tag in
/etc/nsswitch.conf, then everything runs smooth.

Well, this behaviour is since a couple of years now, occurs sporadic. I
have had in FreeBSD 7, 8, 9 and I see it in 10. What is it?

I like the cache facility, since in domains with a lot of users
searching LDAP takes some time and caching help keeping traffic and
latency short. But the namservice caching mechanism seems to be
unreliable. What is up there?



signature.asc
Description: OpenPGP digital signature