Re: get rid of getpwent?
On Feb 13 21:41, Achim Gratz wrote: > Corinna Vinschen writes: > > Oh, hmm. Well, it might be possible, but somehow I'm not excited by the > > idea. While it looks like getpwent is mostly used for this purpose, you > > don't really know it. I think I'll try to implement it fully and then > > let the admin decide what to allow. > > Configurable would be OK of course. I'm not sure when I'll get around > to testing the new snapshot, but I can certainly try how long it takes > when I do "~" in the shell. IIRC, mkpasswd took several minutes to > pull down the domain accounts (I've then deleted all but the handful > that was used for network shares). Keep in mind, the search in getpwent/getgrent is not yet implemented. You only get what the application cached so far. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpu2h73m6efr.pgp Description: PGP signature
Re: get rid of getpwent?
Corinna Vinschen writes: > Oh, hmm. Well, it might be possible, but somehow I'm not excited by the > idea. While it looks like getpwent is mostly used for this purpose, you > don't really know it. I think I'll try to implement it fully and then > let the admin decide what to allow. Configurable would be OK of course. I'm not sure when I'll get around to testing the new snapshot, but I can certainly try how long it takes when I do "~" in the shell. IIRC, mkpasswd took several minutes to pull down the domain accounts (I've then deleted all but the handful that was used for network shares). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent?
On Feb 13 18:50, Achim Gratz wrote: > Corinna Vinschen writes: > > Caching is wonderful for the usual requests for single entries from the > > DB, and for this we have already two caches, the LSA cache and Cygwin's > > own cache. But caching doesn't help at all when enumerating. > > Would it be possible to only look (for user name completion purposes) at > the current user plus whatever is in %SystemDrive%\Users plus whatever > is found in /etc/passwd? That way no beans are spilled about domain > users that couldn't be gleaned from the local file system and in almost > all cases that's the list one would want to complete from anyway. Oh, hmm. Well, it might be possible, but somehow I'm not excited by the idea. While it looks like getpwent is mostly used for this purpose, you don't really know it. I think I'll try to implement it fully and then let the admin decide what to allow. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpr9XkfoL4BV.pgp Description: PGP signature
Re: get rid of getpwent?
Greetings, Achim Gratz! > Corinna Vinschen writes: >> Caching is wonderful for the usual requests for single entries from the >> DB, and for this we have already two caches, the LSA cache and Cygwin's >> own cache. But caching doesn't help at all when enumerating. > Would it be possible to only look (for user name completion purposes) at > the current user plus whatever is in %SystemDrive%\Users Assuming, the mentioned directory at all exists... And contain relevant data... And the data is full... and not redundant... Too many assumptions to work from. > plus whatever is found in /etc/passwd? > That way no beans are spilled about domain users that couldn't be gleaned > from the local file system and in almost all cases that's the list one would > want to complete from anyway. You have to know, how easy it is to confuse a user, who expect a file system being solid source of security-related information. Starting from profiles of nonexistent accounts, and down to relocated profiles and complete absence of the directory in question at all. -- WBR, Andrey Repin (anrdae...@yandex.ru) 13.02.2014, <22:34> Sorry for my terrible english...
Re: get rid of getpwent?
Corinna Vinschen writes: > Caching is wonderful for the usual requests for single entries from the > DB, and for this we have already two caches, the LSA cache and Cygwin's > own cache. But caching doesn't help at all when enumerating. Would it be possible to only look (for user name completion purposes) at the current user plus whatever is in %SystemDrive%\Users plus whatever is found in /etc/passwd? That way no beans are spilled about domain users that couldn't be gleaned from the local file system and in almost all cases that's the list one would want to complete from anyway. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf Blofeld V1.15B11: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 13 10:43, Christopher Faylor wrote: > On Thu, Feb 13, 2014 at 03:44:19PM +0100, Corinna Vinschen wrote: > >Yes, I think so too. I have some preliminary code (actually, just > >empty function shells right now) which are supposed to implement > >full enumerating. > > > >However, system admins might not exactly approve. I discussed this > >with our Linux folks, and I learned that NSS backends like SSSD or > >winbind default to NOT allowing enumerating, but giving the admin a > >choice to enable it. > > > >So I think for our case a configuration option in /etc/nsswitch.conf > >to limit the scope of the enumeration might be feasible. > > Or, nscd.conf which has stuff like: > > enable-cachepasswd yes > positive-time-to-live passwd 600 > negative-time-to-live passwd 20 > suggested-size passwd 211 > check-files passwd yes > persistent passwd yes > shared passwd yes > max-db-size passwd 33554432 > auto-propagate passwd yes I know that nsswitch.conf is not quite the right place for the configuration variables, but I was reluctant to introduce YA file to read at startup. If nobody cares, we can also go with a limited nscd.conf approach for the configuration variables. > I understand why a sysadmin might not want you to be able to enumerate > user names but that really isn't, IMO, a reason not to implement the > functionality (not that you are proposing this). You obviously can't > assume that people won't exercise the capability if it is available. > > Security through obscurity...? Nah. Nah. But restricting the capability for pure networking reasons is on order, IMHO. Assuming that Cygwin has been setup by an admin and the /etc files are not writable by the ordinary user, of course. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpWTkVwtudZc.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Thu, Feb 13, 2014 at 03:44:19PM +0100, Corinna Vinschen wrote: >On Feb 13 09:35, Christopher Faylor wrote: >> On Thu, Feb 13, 2014 at 11:00:25AM +0100, Corinna Vinschen wrote: >> >On Feb 12 16:37, Christopher Faylor wrote: >> >> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote: >> >> >There's only one tiny problem. Whatever I think about the full >> >> >enumerate being right or wrong, I have this vague feeling that I'd like >> >> >to have this implemented fully at one point. My cat disapproves, but we >> >> >can't agree on everything, I guess. Another configuration option in >> >> >/etc/nsswitch.conf might comfort her. >> >> >> >> I don't know if this has been mentioned but would a cache help here, >> >> i.e., nscd? I think that's how Linux deals with this type of situation. >> > >> >Caching is wonderful for the usual requests for single entries from the >> >DB, and for this we have already two caches, the LSA cache and Cygwin's >> >own cache. But caching doesn't help at all when enumerating. >> > >> >There's also the problem to rely on an external program. >> >> But that's no different than Linux. I've never looked at the code but >> apparently libc has hooks for talking to nscd. We could do the same >> with cygserver. >> >> >If it turns out that the current implementation is too slow, I'm >> >prepared to add caching to cygserver to have a system-wide caching >> >server, but Cygwin shouldn't *require* that cygserver runs. And either >> >way, it still wouldn't help when enumerating all accounts. >> >> nscd does more than just keep information around in memory. As I said, >> it's how Linux deals with this situation. I know because I didn't install >> nscd when setting up a minimal Fedora 20 server at work and was met with >> awful lags and timeouts in services which tried to read from our nis. >> So Fedora doesn't require nscd but it sure does help. >> >> But, even after having set it up, I still have to remember not to do >> ls ~cg because it just takes forever. So, if it is possible to >> enumerate users then I think you just do it and let people learn the >> cost. > >Yes, I think so too. I have some preliminary code (actually, just >empty function shells right now) which are supposed to implement >full enumerating. > >However, system admins might not exactly approve. I discussed this >with our Linux folks, and I learned that NSS backends like SSSD or >winbind default to NOT allowing enumerating, but giving the admin a >choice to enable it. > >So I think for our case a configuration option in /etc/nsswitch.conf >to limit the scope of the enumeration might be feasible. Or, nscd.conf which has stuff like: enable-cachepasswd yes positive-time-to-live passwd 600 negative-time-to-live passwd 20 suggested-size passwd 211 check-files passwd yes persistent passwd yes shared passwd yes max-db-size passwd 33554432 auto-propagate passwd yes I understand why a sysadmin might not want you to be able to enumerate user names but that really isn't, IMO, a reason not to implement the functionality (not that you are proposing this). You obviously can't assume that people won't exercise the capability if it is available. Security through obscurity...? Nah. cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 13 09:35, Christopher Faylor wrote: > On Thu, Feb 13, 2014 at 11:00:25AM +0100, Corinna Vinschen wrote: > >On Feb 12 16:37, Christopher Faylor wrote: > >> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote: > >> >There's only one tiny problem. Whatever I think about the full > >> >enumerate being right or wrong, I have this vague feeling that I'd like > >> >to have this implemented fully at one point. My cat disapproves, but we > >> >can't agree on everything, I guess. Another configuration option in > >> >/etc/nsswitch.conf might comfort her. > >> > >> I don't know if this has been mentioned but would a cache help here, > >> i.e., nscd? I think that's how Linux deals with this type of situation. > > > >Caching is wonderful for the usual requests for single entries from the > >DB, and for this we have already two caches, the LSA cache and Cygwin's > >own cache. But caching doesn't help at all when enumerating. > > > >There's also the problem to rely on an external program. > > But that's no different than Linux. I've never looked at the code but > apparently libc has hooks for talking to nscd. We could do the same > with cygserver. > > >If it turns out that the current implementation is too slow, I'm > >prepared to add caching to cygserver to have a system-wide caching > >server, but Cygwin shouldn't *require* that cygserver runs. And either > >way, it still wouldn't help when enumerating all accounts. > > nscd does more than just keep information around in memory. As I said, > it's how Linux deals with this situation. I know because I didn't install > nscd when setting up a minimal Fedora 20 server at work and was met with > awful lags and timeouts in services which tried to read from our nis. > So Fedora doesn't require nscd but it sure does help. > > But, even after having set it up, I still have to remember not to do > ls ~cg because it just takes forever. So, if it is possible to > enumerate users then I think you just do it and let people learn the > cost. Yes, I think so too. I have some preliminary code (actually, just empty function shells right now) which are supposed to implement full enumerating. However, system admins might not exactly approve. I discussed this with our Linux folks, and I learned that NSS backends like SSSD or winbind default to NOT allowing enumerating, but giving the admin a choice to enable it. So I think for our case a configuration option in /etc/nsswitch.conf to limit the scope of the enumeration might be feasible. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpcj0Sq6Ff6H.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Thu, Feb 13, 2014 at 11:00:25AM +0100, Corinna Vinschen wrote: >On Feb 12 16:37, Christopher Faylor wrote: >> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote: >> >There's only one tiny problem. Whatever I think about the full >> >enumerate being right or wrong, I have this vague feeling that I'd like >> >to have this implemented fully at one point. My cat disapproves, but we >> >can't agree on everything, I guess. Another configuration option in >> >/etc/nsswitch.conf might comfort her. >> >> I don't know if this has been mentioned but would a cache help here, >> i.e., nscd? I think that's how Linux deals with this type of situation. > >Caching is wonderful for the usual requests for single entries from the >DB, and for this we have already two caches, the LSA cache and Cygwin's >own cache. But caching doesn't help at all when enumerating. > >There's also the problem to rely on an external program. But that's no different than Linux. I've never looked at the code but apparently libc has hooks for talking to nscd. We could do the same with cygserver. >If it turns out that the current implementation is too slow, I'm >prepared to add caching to cygserver to have a system-wide caching >server, but Cygwin shouldn't *require* that cygserver runs. And either >way, it still wouldn't help when enumerating all accounts. nscd does more than just keep information around in memory. As I said, it's how Linux deals with this situation. I know because I didn't install nscd when setting up a minimal Fedora 20 server at work and was met with awful lags and timeouts in services which tried to read from our nis. So Fedora doesn't require nscd but it sure does help. But, even after having set it up, I still have to remember not to do ls ~cg because it just takes forever. So, if it is possible to enumerate users then I think you just do it and let people learn the cost. I also would be pretty reluctant about deprecating a standard linux function since there is nothing here that I can see which makes this a distinctly linux problem unless you actually can't enumerate users. cgf cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Corinna Vinschen! >> >There's only one tiny problem. Whatever I think about the full >> >enumerate being right or wrong, I have this vague feeling that I'd like >> >to have this implemented fully at one point. My cat disapproves, but we >> >can't agree on everything, I guess. Another configuration option in >> >/etc/nsswitch.conf might comfort her. >> >> I don't know if this has been mentioned but would a cache help here, >> i.e., nscd? I think that's how Linux deals with this type of situation. > Caching is wonderful for the usual requests for single entries from the > DB, and for this we have already two caches, the LSA cache and Cygwin's > own cache. But caching doesn't help at all when enumerating. > There's also the problem to rely on an external program. If it turns > out that the current implementation is too slow, I'm prepared to add > caching to cygserver to have a system-wide caching server, but Cygwin > shouldn't *require* that cygserver runs. And either way, it still > wouldn't help when enumerating all accounts. Another rock in the direction of caching. Caching security-relevant data 'en masse' is usually a not-so-good idea, and should be considered very carefully. Even for Linux, common recommendation is to disable nscd to improve security and reliability of the accounting data. -- WBR, Andrey Repin (anrdae...@yandex.ru) 13.02.2014, <15:36> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 12 16:37, Christopher Faylor wrote: > On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote: > >There's only one tiny problem. Whatever I think about the full > >enumerate being right or wrong, I have this vague feeling that I'd like > >to have this implemented fully at one point. My cat disapproves, but we > >can't agree on everything, I guess. Another configuration option in > >/etc/nsswitch.conf might comfort her. > > I don't know if this has been mentioned but would a cache help here, > i.e., nscd? I think that's how Linux deals with this type of situation. Caching is wonderful for the usual requests for single entries from the DB, and for this we have already two caches, the LSA cache and Cygwin's own cache. But caching doesn't help at all when enumerating. There's also the problem to rely on an external program. If it turns out that the current implementation is too slow, I'm prepared to add caching to cygserver to have a system-wide caching server, but Cygwin shouldn't *require* that cygserver runs. And either way, it still wouldn't help when enumerating all accounts. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpRCgAsi59wB.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote: >On Feb 12 11:16, Ken Brown wrote: >> On 2/12/2014 4:08 AM, Corinna Vinschen wrote: >> >On Feb 11 19:06, Eric Blake wrote: >> >>On 02/11/2014 05:06 PM, Warren Young wrote: >> >>>On 2/11/2014 16:25, David Stacey wrote: >> getpwent() is called in three different places. >> >>> >> >>>To those of you who have investigated these code paths: do any of them >> >>>look like they couldn't be replaced by getpwnam() or other calls that >> >>>would let cygwin1.dll do single-record AD/SAM lookups, rather than >> >>>whole-table/tree scans? >> >>> >> >>>That is, do any of these programs really need to visit every record in >> >>>/etc/passwd? >> >> >> >>libreadline wants to know how to tab-complete ~foo; to do that, it has >> >>to find all usernames beginning with foo. How would you do that without >> >>visiting every single record? >> > >> >This seems to be the major usage of getpwent these days. The question >> >is, how bad is it if only a handful entries, or even only a single one >> >(of oneself) show up? >> > >> >Either way, implementing a full getpwent requires to return the local >> >users, the users of the primary domain, and the users of all trusted >> >domains. I know of domains with 200K users and there are probably >> >bigger ones. How long should a search take when a user presses >> >after the ~? And then, shall the process running the getpwent actually >> >cache all of them? This seems really excessive. >> >> What about the following compromise: If /etc/passwd exists, then >> getpwent behaves as it does currently. > >This part is relatively easy to implement. > >> Otherwise, it returns a >> handful of entries, or possibly just the current user. > >The handful entries would be the ones the process has cached at that >point in time. The tricky part is that getpwent would have to keep >track which entries from the file are in the cache so that those are not >accidentally enumerated twice. > >> This gives >> users a choice. If tab-completion in this situation is important to >> them, they can keep their /etc/passwd file. > >There's only one tiny problem. Whatever I think about the full >enumerate being right or wrong, I have this vague feeling that I'd like >to have this implemented fully at one point. My cat disapproves, but we >can't agree on everything, I guess. Another configuration option in >/etc/nsswitch.conf might comfort her. I don't know if this has been mentioned but would a cache help here, i.e., nscd? I think that's how Linux deals with this type of situation. cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 12 11:16, Ken Brown wrote: > On 2/12/2014 4:08 AM, Corinna Vinschen wrote: > >On Feb 11 19:06, Eric Blake wrote: > >>On 02/11/2014 05:06 PM, Warren Young wrote: > >>>On 2/11/2014 16:25, David Stacey wrote: > getpwent() is called in three different places. > >>> > >>>To those of you who have investigated these code paths: do any of them > >>>look like they couldn't be replaced by getpwnam() or other calls that > >>>would let cygwin1.dll do single-record AD/SAM lookups, rather than > >>>whole-table/tree scans? > >>> > >>>That is, do any of these programs really need to visit every record in > >>>/etc/passwd? > >> > >>libreadline wants to know how to tab-complete ~foo; to do that, it has > >>to find all usernames beginning with foo. How would you do that without > >>visiting every single record? > > > >This seems to be the major usage of getpwent these days. The question > >is, how bad is it if only a handful entries, or even only a single one > >(of oneself) show up? > > > >Either way, implementing a full getpwent requires to return the local > >users, the users of the primary domain, and the users of all trusted > >domains. I know of domains with 200K users and there are probably > >bigger ones. How long should a search take when a user presses > >after the ~? And then, shall the process running the getpwent actually > >cache all of them? This seems really excessive. > > What about the following compromise: If /etc/passwd exists, then > getpwent behaves as it does currently. This part is relatively easy to implement. > Otherwise, it returns a > handful of entries, or possibly just the current user. The handful entries would be the ones the process has cached at that point in time. The tricky part is that getpwent would have to keep track which entries from the file are in the cache so that those are not accidentally enumerated twice. > This gives > users a choice. If tab-completion in this situation is important to > them, they can keep their /etc/passwd file. There's only one tiny problem. Whatever I think about the full enumerate being right or wrong, I have this vague feeling that I'd like to have this implemented fully at one point. My cat disapproves, but we can't agree on everything, I guess. Another configuration option in /etc/nsswitch.conf might comfort her. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpg8HlgfmOmb.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Ken Brown! > What about the following compromise: If /etc/passwd exists, then > getpwent behaves as it does currently. Otherwise, it returns a handful > of entries, or possibly just the current user. This gives users a > choice. If tab-completion in this situation is important to them, they > can keep their /etc/passwd file. Makes sense. You get your current user name substituted in all cases, if you want more - you fill in the list of the names you want to see. -- WBR, Andrey Repin (anrdae...@yandex.ru) 12.02.2014, <21:03> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Wed, 12 Feb 2014, Ken Brown wrote: On 2/12/2014 4:08 AM, Corinna Vinschen wrote: On Feb 11 19:06, Eric Blake wrote: On 02/11/2014 05:06 PM, Warren Young wrote: On 2/11/2014 16:25, David Stacey wrote: getpwent() is called in three different places. To those of you who have investigated these code paths: do any of them look like they couldn't be replaced by getpwnam() or other calls that would let cygwin1.dll do single-record AD/SAM lookups, rather than whole-table/tree scans? That is, do any of these programs really need to visit every record in /etc/passwd? libreadline wants to know how to tab-complete ~foo; to do that, it has to find all usernames beginning with foo. How would you do that without visiting every single record? This seems to be the major usage of getpwent these days. The question is, how bad is it if only a handful entries, or even only a single one (of oneself) show up? Either way, implementing a full getpwent requires to return the local users, the users of the primary domain, and the users of all trusted domains. I know of domains with 200K users and there are probably bigger ones. How long should a search take when a user presses after the ~? And then, shall the process running the getpwent actually cache all of them? This seems really excessive. What about the following compromise: If /etc/passwd exists, then getpwent behaves as it does currently. Otherwise, it returns a handful of entries, or possibly just the current user. This gives users a choice. If tab-completion in this situation is important to them, they can keep their /etc/passwd file. Works for me! And I'd vote "just the current user" in lieu of "a handful" because even a handful has a huge overhead associated with it. Richard -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/12/2014 4:08 AM, Corinna Vinschen wrote: On Feb 11 19:06, Eric Blake wrote: On 02/11/2014 05:06 PM, Warren Young wrote: On 2/11/2014 16:25, David Stacey wrote: getpwent() is called in three different places. To those of you who have investigated these code paths: do any of them look like they couldn't be replaced by getpwnam() or other calls that would let cygwin1.dll do single-record AD/SAM lookups, rather than whole-table/tree scans? That is, do any of these programs really need to visit every record in /etc/passwd? libreadline wants to know how to tab-complete ~foo; to do that, it has to find all usernames beginning with foo. How would you do that without visiting every single record? This seems to be the major usage of getpwent these days. The question is, how bad is it if only a handful entries, or even only a single one (of oneself) show up? Either way, implementing a full getpwent requires to return the local users, the users of the primary domain, and the users of all trusted domains. I know of domains with 200K users and there are probably bigger ones. How long should a search take when a user presses after the ~? And then, shall the process running the getpwent actually cache all of them? This seems really excessive. What about the following compromise: If /etc/passwd exists, then getpwent behaves as it does currently. Otherwise, it returns a handful of entries, or possibly just the current user. This gives users a choice. If tab-completion in this situation is important to them, they can keep their /etc/passwd file. Ken -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Wed, 12 Feb 2014, Corinna Vinschen wrote: On Feb 11 19:06, Eric Blake wrote: On 02/11/2014 05:06 PM, Warren Young wrote: On 2/11/2014 16:25, David Stacey wrote: getpwent() is called in three different places. To those of you who have investigated these code paths: do any of them look like they couldn't be replaced by getpwnam() or other calls that would let cygwin1.dll do single-record AD/SAM lookups, rather than whole-table/tree scans? That is, do any of these programs really need to visit every record in /etc/passwd? libreadline wants to know how to tab-complete ~foo; to do that, it has to find all usernames beginning with foo. How would you do that without visiting every single record? This seems to be the major usage of getpwent these days. The question is, how bad is it if only a handful entries, or even only a single one (of oneself) show up? Either way, implementing a full getpwent requires to return the local users, the users of the primary domain, and the users of all trusted domains. I know of domains with 200K users and there are probably bigger ones. How long should a search take when a user presses after the ~? And then, shall the process running the getpwent actually cache all of them? This seems really excessive. Not only is it excessive, it's a massive security hole. ...I don't know why the present crop of geniuses don't know one of the most fundamental rules of security: don't give up usernames. (Yet, they do it all the time.) I vote get rid of the damned thing - not that my vote counts or that this is the place for it anyway! Richard -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Corinna Vinschen! > Either way, implementing a full getpwent requires to return the local > users, the users of the primary domain, and the users of all trusted > domains. I know of domains with 200K users and there are probably > bigger ones. How long should a search take when a user presses > after the ~? And then, shall the process running the getpwent actually > cache all of them? This seems really excessive. IMO, such practice should be actually discouraged. I mean, listing ALL users of ALL... well, you got it. For the reasons you mentioned above - the possible results of such operation are largely unpredictable. -- WBR, Andrey Repin (anrdae...@yandex.ru) 12.02.2014, <15:49> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 11 19:06, Eric Blake wrote: > On 02/11/2014 05:06 PM, Warren Young wrote: > > On 2/11/2014 16:25, David Stacey wrote: > >> getpwent() is called in three different places. > > > > To those of you who have investigated these code paths: do any of them > > look like they couldn't be replaced by getpwnam() or other calls that > > would let cygwin1.dll do single-record AD/SAM lookups, rather than > > whole-table/tree scans? > > > > That is, do any of these programs really need to visit every record in > > /etc/passwd? > > libreadline wants to know how to tab-complete ~foo; to do that, it has > to find all usernames beginning with foo. How would you do that without > visiting every single record? This seems to be the major usage of getpwent these days. The question is, how bad is it if only a handful entries, or even only a single one (of oneself) show up? Either way, implementing a full getpwent requires to return the local users, the users of the primary domain, and the users of all trusted domains. I know of domains with 200K users and there are probably bigger ones. How long should a search take when a user presses after the ~? And then, shall the process running the getpwent actually cache all of them? This seems really excessive. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpcPzVinfHBS.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Warren Young! >> problem was an assumption made in the 'checkfile' perl script: it was >> assumed that cygwin1.dll is the first DLL listed by objdump. > Details, details. :) That's where is the devil, we know, right? -- WBR, Andrey Repin (anrdae...@yandex.ru) 12.02.2014, <08:23> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 02/11/2014 05:06 PM, Warren Young wrote: > On 2/11/2014 16:25, David Stacey wrote: >> getpwent() is called in three different places. > > To those of you who have investigated these code paths: do any of them > look like they couldn't be replaced by getpwnam() or other calls that > would let cygwin1.dll do single-record AD/SAM lookups, rather than > whole-table/tree scans? > > That is, do any of these programs really need to visit every record in > /etc/passwd? libreadline wants to know how to tab-complete ~foo; to do that, it has to find all usernames beginning with foo. How would you do that without visiting every single record? -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/11/2014 16:25, David Stacey wrote: getpwent() is called in three different places. To those of you who have investigated these code paths: do any of them look like they couldn't be replaced by getpwnam() or other calls that would let cygwin1.dll do single-record AD/SAM lookups, rather than whole-table/tree scans? That is, do any of these programs really need to visit every record in /etc/passwd? problem was an assumption made in the 'checkfile' perl script: it was assumed that cygwin1.dll is the first DLL listed by objdump. Details, details. :) -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 11/02/2014 02:25, Andrey Repin wrote:
Greetings, David Stacey!
Greetings, Andrey Repin! (I've wanted to type that for such a long time...)
I don't have my "almost everything" Cygwin install here to run it
against, so unless someone beats me to it, I won't be posting results
for many hours at least.
Delighted to oblige. I ran your perl script on all executables and DLLs
in /bin. Results attached.
Curious that mc is missing from 32-bit list.
You are quite correct. I've looked at the source code for Midnight
Commander, and getpwent() is called in three different places. The
problem was an assumption made in the 'checkfile' perl script: it was
assumed that cygwin1.dll is the first DLL listed by objdump. For mc.exe,
this isn't the case - cyggcc_s-1.dll is listed first, with cygwin1.dll
second.
I made a quick amendment to said perl script to cater for this, and the
new results are attached. The differences between the two lists are
mainly due to packages not being available for 64-bit Cygwin (e.g.
xemacs, ytree). Guile is worthy of a mention: cygguile-12.dll is in the
32-bit list, but there is no equivalent in the 64-bit list. This is
probably because the 64-bit version of guile is at a later version
compared to its 32-bit cousin.
Hopefully third time lucky!
Dave.
PS: For reference, the revised perl script is as follows:
#!/usr/bin/perl -w
my ($exe, $symbol) = @ARGV;
my $in_cygdll = 0;
die "usage: $0 exename symbol\n" unless length($symbol);
open my $dump, '-|', "objdump -p '$exe'" or die "Can't dump $exe: $!\n";
while (<$dump>) {
if (m/DLL Name: cygwin1.dll/) {
$in_cygdll = 1;
}
elsif (m/DLL Name: /) {
if ($in_cygdll) {
last; # Last cygwin1.dll symbol found; on to another DLL
}
}
elsif ($in_cygdll) {
my @parts = split;
if (@parts == 3 and $parts[2] eq $symbol) {
print "$exe\n";
last;
}
}
}
/bin/ali.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/emacs-X11.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/lefty.exe
/bin/mc.exe
/bin/nano.exe
/bin/otp.exe
/bin/pure-pwconvert.exe
/bin/tcsh.exe
/bin/vim.exe
/bin/xdvi-xaw.exe
/bin/xemacs-21.4.22.exe
/bin/ytree.exe
/bin/cygedit-0.dll
/bin/cyggtk-1-2-0.dll
/bin/cyggtk-x11-2.0-0.dll
/bin/cygguile-12.dll
/bin/cygguile-17.dll
/bin/cyggutils-1.dll
/bin/cygoctave-1.dll
/bin/cygperl5_14.dll
/bin/cygreadline5.dll
/bin/cygreadline6.dll
/bin/cygreadline7.dll
/bin/libpython2.7.dll
/bin/libpython3.2m.dll
/bin/libzsh-5.0.2.dll
/bin/ali.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/emacs-X11.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/lefty.exe
/bin/mc.exe
/bin/nano.exe
/bin/otp.exe
/bin/tcsh.exe
/bin/vim.exe
/bin/xdvi-xaw.exe
/bin/cygedit-0.dll
/bin/cyggtk-1-2-0.dll
/bin/cyggtk-x11-2.0-0.dll
/bin/cygguile-17.dll
/bin/cyggutils-1.dll
/bin/cygoctave-1.dll
/bin/cygperl5_14.dll
/bin/cygreadline7.dll
/bin/libpython2.7.dll
/bin/libpython3.2m.dll
/bin/libzsh-5.0.2.dll
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, David Stacey! >> I don't have my "almost everything" Cygwin install here to run it >> against, so unless someone beats me to it, I won't be posting results >> for many hours at least. > Delighted to oblige. I ran your perl script on all executables and DLLs > in /bin. Results attached. Curious that mc is missing from 32-bit list. -- WBR, Andrey Repin (anrdae...@yandex.ru) 11.02.2014, <06:24> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 10/02/2014 19:49, Warren Young wrote: I don't have my "almost everything" Cygwin install here to run it against, so unless someone beats me to it, I won't be posting results for many hours at least. Delighted to oblige. I ran your perl script on all executables and DLLs in /bin. Results attached. Dave. /bin/ali.exe /bin/emacs-nox.exe /bin/emacs-w32.exe /bin/fish.exe /bin/gdb.exe /bin/gvim.exe /bin/joe.exe /bin/lefty.exe /bin/nano.exe /bin/pure-pwconvert.exe /bin/vim.exe /bin/ytree.exe /bin/cygedit-0.dll /bin/cygreadline7.dll /bin/libpython2.7.dll /bin/libpython3.2m.dll /bin/libzsh-5.0.2.dll /bin/ali.exe /bin/emacs-nox.exe /bin/emacs-w32.exe /bin/fish.exe /bin/gdb.exe /bin/gvim.exe /bin/joe.exe /bin/lefty.exe /bin/mc.exe /bin/nano.exe /bin/vim.exe /bin/cygreadline7.dll /bin/libpython2.7.dll /bin/libpython3.2m.dll /bin/libzsh-5.0.2.dll -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/10/2014 04:16, Peter Rosin wrote:
On 2014-02-10 10:02, Warren Young wrote:
there *has* to be a better way than strings(1) to extract an EXE's list of DLL
imports.
objdump -x /bin/foo.exe
Thank you!
-x turns on 6 other flags, the only one of which that really matters
here is -p. The output is complex enough that I decided to write a
better parser than a grep call. Here's my new checkfile script:
#!/usr/bin/perl -w
my ($exe, $symbol) = @ARGV;
my $in_cygdll = 0;
die "usage: $0 exename symbol\n" unless length($symbol);
open my $dump, '-|', "objdump -p '$exe'" or die "Can't dump $exe: $!\n";
while (<$dump>) {
if (m/DLL Name: cygwin1.dll/) {
$in_cygdll = 1;
}
elsif (m/DLL Name: /) {
last; # Last cygwin1.dll symbol found; on to another DLL
}
elsif ($in_cygdll) {
my @parts = split;
if (@parts == 3 and $parts[2] eq $symbol) {
print "$exe\n";
last;
}
}
}
Run it like before, except that it takes the import name to search for
as a second parameter now:
$ find /bin -name \*.exe -exec ./checkfile {} getpwent \;
I don't have my "almost everything" Cygwin install here to run it
against, so unless someone beats me to it, I won't be posting results
for many hours at least.
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2014-02-10 10:02, Warren Young wrote: > On Feb 9, 2014, at 9:37 AM, David Stacey wrote: > >> On 09/02/2014 15:45, Warren Young wrote: >>> Results: >>> >>> /bin/cppcheck.exe >> >> As far as I can tell, cppcheck doesn't actually call getpwent() at all; this >> is a false positive turned up by strings(1). > > Yeah, there *has* to be a better way than strings(1) to extract an EXE's list > of DLL imports. The linker does it somehow... objdump -x /bin/foo.exe Cheers, Peter -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 9, 2014, at 9:37 AM, David Stacey wrote: > On 09/02/2014 15:45, Warren Young wrote: >> Results: >> >> /bin/cppcheck.exe > > As far as I can tell, cppcheck doesn't actually call getpwent() at all; this > is a false positive turned up by strings(1). Yeah, there *has* to be a better way than strings(1) to extract an EXE's list of DLL imports. The linker does it somehow... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 09/02/2014 15:45, Warren Young wrote: Results: /bin/cppcheck.exe This intrigued me. cppcheck is a static analyser, so what's it doing with getpwent()? I had a nosy in the source code, and it appears that cppcheck has a rule checking for POSIX calls that are not re-entrant. If a call to getpwent() is found in your source code, cppcheck will recommend that you replace this with getpwent_r(). As far as I can tell, cppcheck doesn't actually call getpwent() at all; this is a false positive turned up by strings(1). Dave. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/9/2014 11:16 AM, Corinna Vinschen wrote:
On Feb 9 17:10, Corinna Vinschen wrote:
On Feb 9 08:45, Warren Young wrote:
On Feb 7, 2014, at 10:51 AM, Warren Young wrote:
Here's a better check that doesn't give false positives:
$ cat < checkfile
#!/bin/sh
if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
END
$ find /bin -name \*.exe -exec ./checkfile {} \;
The strings(1) call got left out of that test. Here’s a working variant, which
tests for getpwent() calls instead:
$ cat < checkfile
if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
END
$ find /bin -name \*.exe -exec sh checkfile {} \;
Alas, Vim *does* call this API. Results:
It does so to evaluate a ~user in a path and I guess that's what other
apps do as well. This is kind of weird since it would obviously not be
necessary to enumerate accounts for this functionality:
if (~ in path)
if (~ is solo)
getpwuid(getuid ())
else
getpwname (username after the ~);
Huh, even though tcsh calls getpwent as well, this works fine:
$ cd ~foo
with foo being some not yet cached user name.
However, with my current code this:
$ cd ~
only returns the cached user entries.
emacs uses getpwent (on systems that provide it) for a similar reason:
to provide tab completion on commands that expect a user name.
Ken
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 9 17:10, Corinna Vinschen wrote:
> On Feb 9 08:45, Warren Young wrote:
> > On Feb 7, 2014, at 10:51 AM, Warren Young wrote:
> >
> > > Here's a better check that doesn't give false positives:
> > >
> > >$ cat < checkfile
> > >#!/bin/sh
> > >if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
> > >END
> > >$ find /bin -name \*.exe -exec ./checkfile {} \;
> >
> > The strings(1) call got left out of that test. Here’s a working variant,
> > which tests for getpwent() calls instead:
> >
> > $ cat < checkfile
> > if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
> > END
> > $ find /bin -name \*.exe -exec sh checkfile {} \;
> >
> > Alas, Vim *does* call this API. Results:
>
> It does so to evaluate a ~user in a path and I guess that's what other
> apps do as well. This is kind of weird since it would obviously not be
> necessary to enumerate accounts for this functionality:
>
> if (~ in path)
> if (~ is solo)
> getpwuid(getuid ())
> else
> getpwname (username after the ~);
Huh, even though tcsh calls getpwent as well, this works fine:
$ cd ~foo
with foo being some not yet cached user name.
However, with my current code this:
$ cd ~
only returns the cached user entries.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
pgpMJuuIR35HF.pgp
Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 9 08:45, Warren Young wrote:
> On Feb 7, 2014, at 10:51 AM, Warren Young wrote:
>
> > Here's a better check that doesn't give false positives:
> >
> >$ cat < checkfile
> >#!/bin/sh
> >if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
> >END
> >$ find /bin -name \*.exe -exec ./checkfile {} \;
>
> The strings(1) call got left out of that test. Here’s a working variant,
> which tests for getpwent() calls instead:
>
> $ cat < checkfile
> if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
> END
> $ find /bin -name \*.exe -exec sh checkfile {} \;
>
> Alas, Vim *does* call this API. Results:
It does so to evaluate a ~user in a path and I guess that's what other
apps do as well. This is kind of weird since it would obviously not be
necessary to enumerate accounts for this functionality:
if (~ in path)
if (~ is solo)
getpwuid(getuid ())
else
getpwname (username after the ~);
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
pgpHtUIp7VOqL.pgp
Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 7, 2014, at 10:51 AM, Warren Young wrote:
> Here's a better check that doesn't give false positives:
>
>$ cat < checkfile
>#!/bin/sh
>if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
>END
>$ find /bin -name \*.exe -exec ./checkfile {} \;
The strings(1) call got left out of that test. Here’s a working variant, which
tests for getpwent() calls instead:
$ cat < checkfile
if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
END
$ find /bin -name \*.exe -exec sh checkfile {} \;
Alas, Vim *does* call this API. Results:
/bin/ali.exe
/bin/biber.exe
/bin/cppcheck.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/emacs-X11.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/mc.exe
/bin/nano.exe
/bin/otp.exe
/bin/pure-pwconvert.exe
/bin/tcsh.exe
/bin/vim.exe
/bin/xdvi-xaw.exe
/bin/xemacs-21.4.22.exe
/bin/xgettext.exe
/bin/ytree.exe
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
RE: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
> and I fail to see how this is related to the on-the-fly generation of passwd > and group entries Well, if a cygwin app was run under such an account, it might be affected, that's all... Anton Lavrentiev Contractor NIH/NLM/NCBI -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 8, 2014, at 8:19 AM, Warren Young wrote: > On Feb 7, 2014, at 5:40 PM, Andrey Repin wrote: > >> In either case, repeatedly requesting the same record in a short amount of >> time will only test the system level cache. > > If that were true, moving the requested record around in /etc/passwd wouldn’t > cause a 6x spread in test results. That’s O(N) type behavior, whereas a cache > approaches O(1). A clarification: I agree that /etc/passwd is being cached for the entire test. My 12-line file is a bit over 1K, so it fits easily into a 4K RAM page. Also, I repeated my tests several times, so even the initial disk read is cached. The fact that we see a 6x speed difference depending on where in the file — or RAM page, if you prefer — the target record is tells us that lookups are dominated by parsing time. Parsing overhead shouldn’t exist for SAM or AD. They’re “pre-parsed.” The equivalent overhead in SAM/AD terms is tree and index traversal, which should happen a lot faster than O(N). -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 7, 2014, at 5:40 PM, Andrey Repin wrote: >>> I thought the point of the programme /was/ to call getpwnam() a million >>> times. Precisely. > In either case, repeatedly requesting the same record in a short amount of > time will only test the system level cache. If that were true, moving the requested record around in /etc/passwd wouldn’t cause a 6x spread in test results. That’s O(N) type behavior, whereas a cache approaches O(1). If SAM uses a tree-structured DB, as I suspect, it will be some logarithmic function, like O(log2(N)). -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 7 21:49, Lavrentiev, Anton (NIH/NLM/NCBI) [C] wrote: > > I think SAM/AD will be mostly quicker > > I do not want to be a party pooper here, but have you checked how > the AD approach will work from the unmanaged Windows service accounts? No, and I fail to see how this is related to the on-the-fly generation of passwd and group entries. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpOnX5YPI2p5.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Larry Hall (Cygwin)!
This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
#include
#include
#include
int main(int argc, const char* argv[])
{
int i;
const char* user = argv[1];
if (!user) {
printf("usage: %s username\n", argv[0]);
exit(1);
}
for (i = 0; i < 100; ++i) {
struct passwd* pw = getpwnam(user);
if (!pw) {
printf("User %s doesn't exist!\n", user);
exit(2);
}
else if (i == 0) {
printf("User %s is UID %d\n", user, pw->pw_uid);
}
}
}
So, each getpwnam() call takes 7.1 microseconds on average.
>>>
>>> I think you forgot to put an "exit(0);" after the last printf(). Without
>>> it, you're checking for the same user a million times, which is certainly
>>> going to take a little time. ;-)
>>>
>>
>> I thought the point of the programme /was/ to call getpwnam() a million
>> times. Time this as accurately as you can. Then, with a quick division, you
>> get the time for one call.
> Hm, I missed that he summarized with _microseconds_, even though I quoted
> that too in my response. :-(
> My average was much closer to 2 microseconds per call but that could be
> machine differences.
> In any case, sorry for the noise.
In either case, repeatedly requesting the same record in a short amount of
time will only test the system level cache.
--
WBR,
Andrey Repin (anrdae...@yandex.ru) 08.02.2014, <04:39>
Sorry for my terrible english...
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/7/2014 5:45 PM, David Stacey wrote:
On 07/02/14 21:44, Larry Hall (Cygwin) wrote:
On 2/7/2014 3:09 PM, Warren Young wrote:
This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
#include
#include
#include
int main(int argc, const char* argv[])
{
int i;
const char* user = argv[1];
if (!user) {
printf("usage: %s username\n", argv[0]);
exit(1);
}
for (i = 0; i < 100; ++i) {
struct passwd* pw = getpwnam(user);
if (!pw) {
printf("User %s doesn't exist!\n", user);
exit(2);
}
else if (i == 0) {
printf("User %s is UID %d\n", user, pw->pw_uid);
}
}
}
So, each getpwnam() call takes 7.1 microseconds on average.
I think you forgot to put an "exit(0);" after the last printf(). Without
it, you're checking for the same user a million times, which is certainly
going to take a little time. ;-)
I thought the point of the programme /was/ to call getpwnam() a million
times. Time this as accurately as you can. Then, with a quick division, you
get the time for one call.
Hm, I missed that he summarized with _microseconds_, even though I quoted
that too in my response. :-(
My average was much closer to 2 microseconds per call but that could be
machine differences.
In any case, sorry for the noise.
--
Larry
_
A: Yes.
> Q: Are you sure?
>> A: Because it reverses the logical flow of conversation.
>>> Q: Why is top posting annoying in email?
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 07/02/14 21:44, Larry Hall (Cygwin) wrote:
On 2/7/2014 3:09 PM, Warren Young wrote:
This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
#include
#include
#include
int main(int argc, const char* argv[])
{
int i;
const char* user = argv[1];
if (!user) {
printf("usage: %s username\n", argv[0]);
exit(1);
}
for (i = 0; i < 100; ++i) {
struct passwd* pw = getpwnam(user);
if (!pw) {
printf("User %s doesn't exist!\n", user);
exit(2);
}
else if (i == 0) {
printf("User %s is UID %d\n", user, pw->pw_uid);
}
}
}
So, each getpwnam() call takes 7.1 microseconds on average.
I think you forgot to put an "exit(0);" after the last printf(). Without
it, you're checking for the same user a million times, which is certainly
going to take a little time. ;-)
I thought the point of the programme /was/ to call getpwnam() a million
times. Time this as accurately as you can. Then, with a quick division,
you get the time for one call.
Dave.
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
RE: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
> I think SAM/AD will be mostly quicker I do not want to be a party pooper here, but have you checked how the AD approach will work from the unmanaged Windows service accounts? We've been experiencing rather nasty effects of the M$ design that when a host changes its password (it is required to, every so many days), it is no longer considered an "authorized" agent (rather, anonymous). Accessing AD anonymously (esp. from system-managed service account) is limited; like when you request a list, you get only first 100 (who at M$ had invented this?!) entries. Which means that if your code is scanning, it won't find more than 100 users (and they are alphabetized, so the "excess" users will simply disappear from view). That creates false-positive nonexistent users / groups. The only remedy is to restart the host... P.S. I'm not an AD person, and some of the info from the above comes from our sysadmins (how they see things unfolding). Anton Lavrentiev Contractor NIH/NLM/NCBI
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/7/2014 3:09 PM, Warren Young wrote:
This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
#include
#include
#include
int main(int argc, const char* argv[])
{
int i;
const char* user = argv[1];
if (!user) {
printf("usage: %s username\n", argv[0]);
exit(1);
}
for (i = 0; i < 100; ++i) {
struct passwd* pw = getpwnam(user);
if (!pw) {
printf("User %s doesn't exist!\n", user);
exit(2);
}
else if (i == 0) {
printf("User %s is UID %d\n", user, pw->pw_uid);
}
}
}
So, each getpwnam() call takes 7.1 microseconds on average.
I think you forgot to put an "exit(0);" after the last printf(). Without
it, you're checking for the same user a million times, which is certainly
going to take a little time. ;-)
--
Larry
_
A: Yes.
> Q: Are you sure?
>> A: Because it reverses the logical flow of conversation.
>>> Q: Why is top posting annoying in email?
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 7 13:09, Warren Young wrote: > On 2/7/2014 02:49, Corinna Vinschen wrote: > >On Feb 6 14:43, Warren Young wrote: > >>On 2/6/2014 07:13, Corinna Vinschen wrote: > > > >it would, of course, be possible to implement Cygwin > >command line tools along the lines of useradd/usermod/groupdel. For AD, > >they would just have to use LDAP, > > If by "use LDAP" you mean the ldap_* functions in the OpenLDAP > library, I can't recommend it. (See my other post on LDAP books.) You can also use the calls from wldap32.dll which is available anyway. > Such programs need not be portable. Never said so. > I don't see why such programs shouldn't be written straight to the > Windows API, even though this is naughty on Cygwin. The Win32 > security API fills the same role as libldap does on a Linux box > configured for LDAP. The underlying protocol is LDAP, so why not use it, given that lots of changes to AD cannot be done using the "High Level" Net API anyway. > You're right that such programs are probably going to be necessary, > if Cygwin moves to SAM/AD as primary. Windows Home edition user > management probably won't be powerful enough to do what Cygwin > needs, if SAM is Cygwin's Single Point of Truth on such systems. Again, it isn't. We will keep the passwd and group files for users who are more comfortable with them. We will also have an nsswitch.conf file for configuration. I attached my local sandbox version below. > I want the mkpasswd and mkgroup utilities to remain available They will, with slight changes. The default values generated for uid/gid numbers should preferredly reflect the automatism when reading from SAM/AD. > Corinna, an earlier post of yours suggested that /etc/foo was being > kept as primary for speed reasons, but are you comparing to SAM or > to AD? And have you tested it lately? Did I really write something about speed? I think SAM/AD will be mostly quicker but they will be especially less hassle and allow centralized maintainance, which is a real boon for admins. Also, the new Cygwin will only read and cache the requested entries from the passwd/group files, not the entire file. Also, if it turns out that AD is too slow for some reason or in some environment, we should consider to use cygserver as a centralized local cache. But this is something for later. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat # /etc/nsswitch.conf # #This file is read once by the first process in a Cygwin process tree. #To pick up changes, restart all Cygwin processes. # # passwd: # group: # #"files"only use /etc/passwd or /etc/group file. #"db" only use SAM/AD retrieval. #"files db" both, files preferred. This is the default. # #"db files" does not make any sense # passwd: files db group: files db # # Configuration of "db" style passwd/group handling: # # db_prefix: # #"auto" If "auto", prepend domain to account name if the account # is not a member of the machine's primary domain. Prepend # just the separator char if the account is a well-known # or builtin group. # #"primary" "primary" is like "auto", but prepend domain to account name # as well, if the account is a member of the machine's primary # domain. # #"always" If set to "always", always prepend domain, even for # well-known and builtin accounts. # db_prefix: auto # # db_cache: # #"yes" If yes, cache once retrieved DB values in local process, # hand cache down to child processes. # #"no" If no, fetch passwd or group entries anew, every time an # entry is requested. Default is "no". # db_cache: no # # db_separator: # # Set separator character between domain and account name to # the ASCII char X. Default is '+'. # db_separator: + pgpkZ0sx639tr.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 7 13:25, Warren Young wrote: > On 2/7/2014 13:09, Warren Young wrote: > > > >I want getpwent() and friends to remain available, but to switch to > >AD/SAM as primary, like OS X does all the time, > > I just realized that this means getpwent() turns into an AD database > linear scan at AD sites. > > Hmmm... So you see the problem. Also: http://cygwin.com/ml/cygwin/2014-02/msg00146.html Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpMZBww5Za8l.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/7/2014 13:09, Warren Young wrote:
I want getpwent() and friends to remain available, but to switch to
AD/SAM as primary, like OS X does all the time,
I just realized that this means getpwent() turns into an AD database
linear scan at AD sites.
Hmmm...
I think I'm still in favor of the idea. It seems like a "Doctor, it
hurts when I do a complete linear AD scan!" issue. ("So don't do that,
then!")
This could make Cygwin a positive force for getting these old programs
updated to use modern APIs.
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/7/2014 02:49, Corinna Vinschen wrote:
On Feb 6 14:43, Warren Young wrote:
On 2/6/2014 07:13, Corinna Vinschen wrote:
it would, of course, be possible to implement Cygwin
command line tools along the lines of useradd/usermod/groupdel. For AD,
they would just have to use LDAP,
If by "use LDAP" you mean the ldap_* functions in the OpenLDAP library,
I can't recommend it. (See my other post on LDAP books.)
Such programs need not be portable. It doesn't look like it will be
helpful to start with the ones from Linux's shadow-utils, since those
modify /etc/foo directly. They don't even attempt to abstract away
/etc/foo, NIS and LDAP, so they're no use to us except as design
exemplars. Cygwin equivalents should use the same command line format
and flags, they should use /etc/skel, etc.
I don't see why such programs shouldn't be written straight to the
Windows API, even though this is naughty on Cygwin. The Win32 security
API fills the same role as libldap does on a Linux box configured for LDAP.
You're right that such programs are probably going to be necessary, if
Cygwin moves to SAM/AD as primary. Windows Home edition user management
probably won't be powerful enough to do what Cygwin needs, if SAM is
Cygwin's Single Point of Truth on such systems.
I don't see a reason for this to change, given that so many other
POSIX systems share aspects of this behavior.
Sorry, I lost you there. What exactly of the current behaviour do
you want to keep?
I want getpwent() and friends to remain available, but to switch to
AD/SAM as primary, like OS X does all the time, or like Linux does when
LDAP is enabled in NSS.[*]
I want the mkpasswd and mkgroup utilities to remain available to
generate these files on systems where there are programs that insist on
scanning them directly. Such files will be understood to be potentially
stale caches of records that really live in SAM/AD. Plus, these files
are useful as debug dumps, and for casual grepping.
Corinna, an earlier post of yours suggested that /etc/foo was being kept
as primary for speed reasons, but are you comparing to SAM or to AD?
And have you tested it lately?
I don't think it's fair or even useful to compare the speed of /etc/foo
to AD. Sites using AD have implicitly said they're willing to pay a
speed hit to get certain benefits. The fact that AD is slower is like
pointing out that container ships are slower than a Maserati.
I'd expect a SAM lookup to be *faster* than any /etc/foo lookup, even
with very small /etc/foo files.
This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
#include
#include
#include
int main(int argc, const char* argv[])
{
int i;
const char* user = argv[1];
if (!user) {
printf("usage: %s username\n", argv[0]);
exit(1);
}
for (i = 0; i < 100; ++i) {
struct passwd* pw = getpwnam(user);
if (!pw) {
printf("User %s doesn't exist!\n", user);
exit(2);
}
else if (i == 0) {
printf("User %s is UID %d\n", user, pw->pw_uid);
}
}
}
So, each getpwnam() call takes 7.1 microseconds on average.
It used to take 5.5 seconds, until I moved my user record down to the
end of the /etc/passwd file, just to make it a worst-case test. 3 lines
of movement added ~1.6 us to the call time. Then I moved it to the top
of the file, and the program ran in 1.2 seconds. 6x delta between best
and worst case, in such a small file!
If you send me the Win32 code for a SAM equivalent, I'll run it here,
for comparison. Please keep the unnecessary overhead, like the
printf()s and logic inside the main work loop, so we can null out the
difference in the comparison.
Also, the usual problem: What exactly
do you write to passwd/group? If you write all users of a big corp,
the files get really big.
If Cygwin moves to SAM/AD as primary, then this doesn't matter.
The disk space hit is small in modern terms. Didn't someone calculate
135 bytes on average per /etc/passwd line recently? So, a massive
100,000 user corporation's /etc/passwd would be 13.5 MB. Big, but not
TeX big. Not even Vim big, these days. Sigh.
In such a system, Joe User probably isn't allowed to pull all 100,000
records anyway, for security reasons. Won't mkpasswd skip all those?
The speed hit doesn't really matter, since it will only be paid by
programs that insist on scanning these files directly, if AD/SAM are
primary. That's going to be things like misguided Perl scripts that
parse the file directly, instead of using Perl's built-in getpwent() and
such.
(I assume getpwnam() consults SAM/AD in Cygwin, now or in Cygwin.next.)
It checks for the file first, then it asks SAM/AD.
I realize getting rid of /etc/foo checks in these functions is a very
large change, from a
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 7 10:51, Warren Young wrote:
> On 2/7/2014 06:53, David Stacey wrote:
> >On 07/02/2014 09:49, Corinna Vinschen wrote:
> >>On Feb 6 14:43, Warren Young wrote:
> >>>I know a guy who currently has all of
> >>>Cygwin downloaded and ready to re-install, to test this.:)
> >>Try this: strings -f/bin/*.exe/bin/*.dll | grep getgrent
> >
> >Let me save you the trouble of reinstalling it
>
> Thank you!
>
> > Results attached.
>
> The cppcheck, cygqscintilla*.dll and nedit results are bogus. (i.e.
> They contain these strings for other reasons.) Here's a better
> check that doesn't give false positives:
>
> $ cat < checkfile
> #!/bin/sh
> if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
> END
> $ find /bin -name \*.exe -exec ./checkfile {} \;
>
> cygperl*.dll includes the functions only because it wraps them. I
> assume Perl's configuration script is smart enough to cope if
> they're missing.
Just to clarify: The entry points won't go away, but they might be
*sort of* non-functional. They will, for a start, only enumerate
the accounts which are cached in the process, which may be only
two (current user and her primary group), or none at all.
> Emacs...why am I not surprised that it uses these old functions?
>
> Vim does not, therefore Vim is the superior text editor. (I kid.)
You dont ;)
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
pgpStwzxII2ID.pgp
Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Warren Young! >> LDAP IS simple. > Anything tied to a PKI is going to be pretty complex, no matter how > simple the underlying tech is. > Then there's the fact that LDAP derives from X.500, a prototypically > overengineered OSI emission. DC=my,DC=sub,DC=domain,DC=com. P'tui! Well, well, mom said spitting is not nice. I agree that certain parts of specification look weird for untrained eye, but being properly explained, they make sense. Not to mention, this all could be wrapped up by a proper interface. >> It's a shame it is so little known outside specific circles. > I blame the books. I concur. Was a long time since I started grasping it, but then it was breakthrough. Much like regexps. > Several years ago, I bought most of what was available, since we needed > to do user authentication against an existing AD system in a custom > software system, to avoid parallel user management. > The most useful book I found was "LDAP: Programming Directory-Enabled > Applications with Lightweight Directory Access Protocol", by Howes and > Smith, since it covers things from the C API level upward. Most other > LDAP books either don't cover the programming level at all, or mention > it briefly on their way down from the directory service level. I find most books problematic to understand, because they talk too much about implementation, and too little about actual application of the system in real world. As an example: I have little knowledge in TCP/IP internal structure, this doesn't stop me from troubleshooting network issues... because it is not really necessary in 90% of real situations. The 90% of the rest is covered by a quick googling and little extrapolation. The 1% that is left falls under "Professionalism: An ability to make right decisions in a situation that never happen again in your life." Same applies to LDAP. People spoil their \m/ \m/ about how awesome they are using such awesome technology, but offering little real help. > It's not even the case that all the teaching material has moved online. > The OpenLDAP programming manual is a pile of man pages.[2] Man pages > are great, but go toss a neophyte into that pile and see if she doesn't > come out welted, weary and wincing. Same for ISC-DHCP... :/ The WORST documentation support for (probably) the MOST used DHCP server. > Given this, it's no wonder those who can write LDAP based software are > seen as a kind of elite. Acquiring a working knowledge of LDAP > programming is akin to a baleen whale feeding itself. Programming anything is really simple. Or I find it really simple. If you know how to use it, you can code it. If you don't know how to use it... You are coding a monster noone will be able to use. -- WBR, Andrey Repin (anrdae...@yandex.ru) 07.02.2014, <22:02> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/7/2014 06:53, David Stacey wrote:
On 07/02/2014 09:49, Corinna Vinschen wrote:
On Feb 6 14:43, Warren Young wrote:
I know a guy who currently has all of
Cygwin downloaded and ready to re-install, to test this.:)
Try this: strings -f/bin/*.exe/bin/*.dll | grep getgrent
Let me save you the trouble of reinstalling it
Thank you!
> Results attached.
The cppcheck, cygqscintilla*.dll and nedit results are bogus. (i.e. They
contain these strings for other reasons.) Here's a better check that
doesn't give false positives:
$ cat < checkfile
#!/bin/sh
if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
END
$ find /bin -name \*.exe -exec ./checkfile {} \;
cygperl*.dll includes the functions only because it wraps them. I
assume Perl's configuration script is smart enough to cope if they're
missing.
I assume the shells (bash, mc, and tcsh) also have alternate code paths
for systems where these functions don't exist.
ytree's use of these functions is obvious. I don't know that I'd bet on
it having an alternate code path, though.
I have no guess why socat needs these functions, but given that it was
recently updated, I will guess the socat devs aren't interested in
moving to more modern APIs. Programs like this, where user management
isn't any kind of real focus, are where you expect to find old, easy
APIs used in preference to more modern, more complicated APIs.
cygwin1.dll is obvious. :)
Emacs...why am I not surprised that it uses these old functions?
Vim does not, therefore Vim is the superior text editor. (I kid.)
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/7/2014 05:49, Andrey Repin wrote: LDAP IS simple. Anything tied to a PKI is going to be pretty complex, no matter how simple the underlying tech is. Then there's the fact that LDAP derives from X.500, a prototypically overengineered OSI emission. DC=my,DC=sub,DC=domain,DC=com. P'tui! It's a shame it is so little known outside specific circles. I blame the books. Several years ago, I bought most of what was available, since we needed to do user authentication against an existing AD system in a custom software system, to avoid parallel user management. The most useful book I found was "LDAP: Programming Directory-Enabled Applications with Lightweight Directory Access Protocol", by Howes and Smith, since it covers things from the C API level upward. Most other LDAP books either don't cover the programming level at all, or mention it briefly on their way down from the directory service level. The thing is, the book was written way back in 1997, before OpenLDAP was forked from the umich LDAP reference implementation. A huge number of the things the book says no longer applies because the API has changed out from under it. Many of the examples that do still work give deprecation warnings, or need to be compiled with -D flags to re-enable APIs that are marked for deprecation. I did a search on Amazon, sorting all LDAP books by release date. The newest one that covers programing looks to be Howes & Smith's other book, "Understanding and Deploying LDAP Directory Services" (coauthored with Good) which came out in 2003.[1] The book primarily covers administration of the Netscape (!) Directory Server. It does cover LDAP programming in a couple of chapters spanning 150 pages, but most of the examples are in Java. There is a touch of C and Perl, only. Even if you count all the code samples together and ignore the language and API differences, you can't call this anything but a whirlwind overview of LDAP programming. "Mastering OpenLDAP" is billed as "for developers", but there isn't a line of C code in the book. (Or any other programming language, really.) It's a sysadmin book. It's not even the case that all the teaching material has moved online. The OpenLDAP programming manual is a pile of man pages.[2] Man pages are great, but go toss a neophyte into that pile and see if she doesn't come out welted, weary and wincing. Given this, it's no wonder those who can write LDAP based software are seen as a kind of elite. Acquiring a working knowledge of LDAP programming is akin to a baleen whale feeding itself. -- [1] Amazon claims it was updated in 2013, but it's just a paperback re-issue of their 2003 hardback title. [2] http://goo.gl/gE2e6P -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 07/02/2014 09:49, Corinna Vinschen wrote:
On Feb 6 14:43, Warren Young wrote:
On 2/6/2014 07:13, Corinna Vinschen wrote:
Btw., it would be a good idea to get rid of calls to getpwent/getgrent
in future. They*probably* won't do anymore what they were supposed to
do if you don't have passwd/group files.
There must be a way to list an executable's DLL imports, and thereby
do a survey on Cygwin to see which executables currently import
those functions. If so, I know a guy who currently has all of
Cygwin downloaded and ready to re-install, to test this.:)
Try this: strings -f/bin/*.exe/bin/*.dll | grep getgrent
Let me save you the trouble of reinstalling it - one of my PCs has a
complete install of both architectures. Results attached.
Cheers,
Dave.
/bin/bash.exe: _getgrent32
/bin/biber.exe: getgrent
/bin/biber.exe: _getgrent32
/bin/biber.exe: __imp___getgrent32
/bin/biber.exe: _getgrent
/bin/cppcheck.exe: fgetgrent
/bin/cppcheck.exe: getgrent
/bin/cygdrop.exe: _getgrent32
/bin/emacs-nox.exe: _getgrent32
/bin/emacs-w32.exe: _getgrent32
/bin/emacs-X11.exe: _getgrent32
/bin/filan.exe: _getgrent32
/bin/mc.exe: _getgrent32
/bin/nedit.exe: library functions:"<((?# arithmetic
functions)abs|atan2|cos|exp|int|log|rand|sin|sqrt|srand|time|(?# conversion
functions)chr|gmtime|hex|localtime|oct|ord|vec|(?# structure
conversion)pack|unpack|(?# string
functions)chomp|chop|crypt|eval(?=\s*[^{])|index|lc|lcfirst|length|quotemeta|rindex|substr|uc|ucfirst|(?#
array and hash
functions)delete|each|exists|grep|join|keys|map|pop|push|reverse|scalar|shift|sort|splice|split|unshift|values|(?#
search and replace functions)pos|study|(?# file
operations)chmod|chown|link|lstat|mkdir|readlink|rename|rmdir|stat|symlink|truncate|unlink|utime|(?#
input/output)binmode|close|eof|fcntl|fileno|flock|getc|ioctl|open|pipe|print|printf|read|readline|readpipe|seek|select|sprintf|sysopen|sysread|sysseek|syswrite|tell|(?#
formats)formline|write|(?# tying variables)tie|tied|untie|(?# directory
reading routines)closedir|opendir|readdir|rewinddir|seekdir|telldir|(?# system
interaction)alarm|chdir|chroot|die|exec|exit|fork|getlogin|getpgrp|getppid|getpriority|glob|kill|setpgrp|setpriority|sleep|syscall|system|times|umask|wait|waitpid|warn|(?#
networking)accept|bind|connect|getpeername|getsockname|getsockopt|listen|recv|send|setsockopt|shutdown|socket|socketpair|(?#
system V
ipc)msgctl|msgget|msgrcv|msgsnd|semctl|semget|semop|shmctl|shmget|shmread|shmwrite|(?#
miscellaneous)defined|do|dump|eval(?=\s*\{)|local|my|ref|reset|undef|(?#
informations from system
databases)endpwent|getpwent|getpwnam|getpwuid|setpwent|endgrent|getgrent|getgrgid|getgrnam|setgrent|endnetent|getnetbyaddr|getnetbyname|getnetent|setnetent|endhostend|gethostbyaddr|gethostbyname|gethostent|sethostent|endservent|getservbyname|getservbyport|getservent|setservent|endprotoent|getprotobyname|getprotobynumber|getprotoent|setprotoent)>":::Subroutine::
/bin/procan.exe: _getgrent32
/bin/sh.exe: _getgrent32
/bin/socat.exe: _getgrent32
/bin/tcsh.exe: _getgrent32
/bin/xgettext.exe: getgrent
/bin/ytree.exe: _getgrent32
/bin/cygguile-12.dll: _getgrent32
/bin/cygguile-12.dll: _getgrent32
/bin/cygguile-17.dll: _getgrent32
/bin/cygobt-2.dll: _getgrent32
/bin/cygoctave-1.dll: _ZN12octave_group8getgrentERSs
/bin/cygoctave-1.dll: _ZN12octave_group8getgrentEv
/bin/cygoctave-1.dll: _getgrent32
/bin/cygperl5_14.dll: getgrent
/bin/cygperl5_14.dll: _getgrent32
/bin/cygqscintilla2-9.dll: NULL __FILE__ __LINE__ __PACKAGE__ __DATA__ __END__
AUTOLOAD BEGIN CORE DESTROY END EQ GE GT INIT LE LT NE CHECK abs accept alarm
and atan2 bind binmode bless caller chdir chmod chomp chop chown chr chroot
close closedir cmp connect continue cos crypt dbmclose dbmopen defined delete
die do dump each else elsif endgrent endhostent endnetent endprotoent endpwent
endservent eof eq eval exec exists exit exp fcntl fileno flock for foreach fork
format formline ge getc getgrent getgrgid getgrnam gethostbyaddr gethostbyname
gethostent getlogin getnetbyaddr getnetbyname getnetent getpeername getpgrp
getppid getpriority getprotobyname getprotobynumber getprotoent getpwent
getpwnam getpwuid getservbyname getservbyport getservent getsockname getsockopt
glob gmtime goto grep gt hex if index int ioctl join keys kill last lc lcfirst
le length link listen local localtime lock log lstat lt m map mkdir msgctl
msgget msgrcv msgsnd my ne next no not oct open opendir or ord our pack package
pipe pop pos print printf prototype push q qq qr quotemeta qu qw qx rand read
readdir readline readlink readpipe recv redo ref rename require reset return
reverse rewinddir rindex rmdir s scalar seek seekdir select semctl semget semop
send setgrent sethostent setnetent setpgrp setpriority setprotoent setpwent
setservent setsockopt shift shmctl shmget shmread shmwrite shutdown sin sleep
socket socketpair sort splice split sprintf sqrt srand stat study sub substr
symlink syscall sysopen sysread sys
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
Greetings, Corinna Vinschen! >> In some of these systems, you can edit /etc/foo and run a command to >> manually sync that content back to the "real" user info DB. (e.g. >> the BSDs) In others, direct edits to these files are ignored, but >> the OS syncs a subset of changes to the user info DB to these files, >> for the benefit of getpwent() and friends. (e.g. Mac OS X.) > That won't make much sense on Cygwin. The idea is to use the existing > OS tools to maintain the user/group databases in the first place. > Having said that, it would, of course, be possible to implement Cygwin > command line tools along the lines of useradd/usermod/groupdel. For AD, > they would just have to use LDAP, as Cygwin will do to read the account > info. It's pretty simple, as far as you view LDAP as simple, LDAP IS simple. It's a shame it is so little known outside specific circles. > However, I found that one functionality has to go completely. It will > not be possible anymore to ssh into your machine and add yourself to > arbitrary groups by adding your user name to groups in /etc/group. This > is just not feasible anymore. Erm. I call it a security improvement. To put it mildly. -- WBR, Andrey Repin (anrdae...@yandex.ru) 07.02.2014, <16:43> Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On Feb 6 14:43, Warren Young wrote: > On 2/6/2014 07:13, Corinna Vinschen wrote: > > > >Btw., it would be a good idea to get rid of calls to getpwent/getgrent > >in future. They *probably* won't do anymore what they were supposed to > >do if you don't have passwd/group files. > > There must be a way to list an executable's DLL imports, and thereby > do a survey on Cygwin to see which executables currently import > those functions. If so, I know a guy who currently has all of > Cygwin downloaded and ready to re-install, to test this. :) Try this: strings -f /bin/*.exe /bin/*.dll | grep getgrent > > A full implementation would > >require to enumerate the local SAM and, at least, the primary domain > >accounts at runtime. That would be possible, but it comes at a hefty > >price in terms of performance. > > Linux and Cygwin are pretty much the last ones standing when it > comes to preferentially reading plain text files in /etc for user > info. Big iron Unix, the BSDs, and Mac OS X now all treat these > files as secondary to some behind-the-scenes database. Which would be SAM and AD for us. > In some of these systems, you can edit /etc/foo and run a command to > manually sync that content back to the "real" user info DB. (e.g. > the BSDs) In others, direct edits to these files are ignored, but > the OS syncs a subset of changes to the user info DB to these files, > for the benefit of getpwent() and friends. (e.g. Mac OS X.) That won't make much sense on Cygwin. The idea is to use the existing OS tools to maintain the user/group databases in the first place. Having said that, it would, of course, be possible to implement Cygwin command line tools along the lines of useradd/usermod/groupdel. For AD, they would just have to use LDAP, as Cygwin will do to read the account info. It's pretty simple, as far as you view LDAP as simple, > In Cygwin, we have a kind of hybrid of these, owing to the fact that > the integration between Cygwin and Windows is pretty much one-way. > We have mkpasswd/group, which treats the DB as primary, like OS X, > but which must be run manually to sync changes, like the BSDs. That won't be necessary anymore. In fact, I would prefer if we could do without the passswd and group files, but we have to keep them for backward compatibility at least. However, I found that one functionality has to go completely. It will not be possible anymore to ssh into your machine and add yourself to arbitrary groups by adding your user name to groups in /etc/group. This is just not feasible anymore. > I don't see a reason for this to change, given that so many other > POSIX systems share aspects of this behavior. Sorry, I lost you there. What exactly of the current behaviour do you want to keep? > It would be nicer if Cygwin behaved more like OS X in this regard. > That is, for mkpasswd/group to be run automatically when the SAM/AD > changes. I don't see Microsoft doing that for us, though. Cygwin will directly read from SAM/AD. What's the point of automating writing passwd and group files? Also, the usual problem: What exactly do you write to passwd/group? If you write all users of a big corp, the files get really big. So far, every exec'ing process is reading the files anew. This is *so* 1990ths. > The only way I can think of for Cygwin to do that for itself would > be to run mkpasswd/group from setup.exe, in the same way that it > runs autorebase. Sorry, but that sounds rather backward to me. And again, what exactly do you write to the files? Only local accunts? Or local and primary domain? Or local, primary domain and all trusted domains? And what do you do with local changes to the files? mkpasswd/mkgroup would overwrite all changes, which does not sound like a terribly good idea to me. > I realize the current recommended practice is to keep /etc/foo as > small as possible, but shouldn't an AD/SAM DB lookup be faster than > a linear scan of a large /etc/foo file? Why lament the fact that > getpwent() is slow, when getpwnam() is its logical replacement, and > presumably much faster? I don't lament the fact. The problem is that a SAM/AD scan would have to be implemented in Cygwin in the first place. And then again, what do you suppose Cygwin should do in big AD forests? Enumerate all accounts of all trusted domains? As of today, this is an unsolved problem, Corinna > (I assume getpwnam() consults SAM/AD in Cygwin, now or in Cygwin.next.) It checks for the file first, then it asks SAM/AD. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpfZHcITav69.pgp Description: PGP signature
Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)
On 2/6/2014 07:13, Corinna Vinschen wrote: Btw., it would be a good idea to get rid of calls to getpwent/getgrent in future. They *probably* won't do anymore what they were supposed to do if you don't have passwd/group files. There must be a way to list an executable's DLL imports, and thereby do a survey on Cygwin to see which executables currently import those functions. If so, I know a guy who currently has all of Cygwin downloaded and ready to re-install, to test this. :) I tried futzing with objdump, nm, and dumpbin, and couldn't get anything useful. Mainly what I got was either irrelevant or complaints of the "no string table" sort. > A full implementation would require to enumerate the local SAM and, at least, the primary domain accounts at runtime. That would be possible, but it comes at a hefty price in terms of performance. Linux and Cygwin are pretty much the last ones standing when it comes to preferentially reading plain text files in /etc for user info. Big iron Unix, the BSDs, and Mac OS X now all treat these files as secondary to some behind-the-scenes database. In some of these systems, you can edit /etc/foo and run a command to manually sync that content back to the "real" user info DB. (e.g. the BSDs) In others, direct edits to these files are ignored, but the OS syncs a subset of changes to the user info DB to these files, for the benefit of getpwent() and friends. (e.g. Mac OS X.) In Cygwin, we have a kind of hybrid of these, owing to the fact that the integration between Cygwin and Windows is pretty much one-way. We have mkpasswd/group, which treats the DB as primary, like OS X, but which must be run manually to sync changes, like the BSDs. I don't see a reason for this to change, given that so many other POSIX systems share aspects of this behavior. It would be nicer if Cygwin behaved more like OS X in this regard. That is, for mkpasswd/group to be run automatically when the SAM/AD changes. I don't see Microsoft doing that for us, though. The only way I can think of for Cygwin to do that for itself would be to run mkpasswd/group from setup.exe, in the same way that it runs autorebase. I realize the current recommended practice is to keep /etc/foo as small as possible, but shouldn't an AD/SAM DB lookup be faster than a linear scan of a large /etc/foo file? Why lament the fact that getpwent() is slow, when getpwnam() is its logical replacement, and presumably much faster? (I assume getpwnam() consults SAM/AD in Cygwin, now or in Cygwin.next.) Here's the Mac OS X passwd(5) man page: http://goo.gl/AwIHku It's relevant here because Mac OS X uses OpenDirectory, an LDAP directory server. In that way, it is not unlike future-Cygwin+AD. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
