Re: [389-users] PAM Pass through authentication only one threaded
On 11/04/2013 10:46 AM, Jan Tomasek wrote: On 11/04/2013 05:22 PM, Rich Megginson wrote: On 11/04/2013 09:08 AM, Jan Tomasek wrote: On 11/04/2013 05:04 PM, Rich Megginson wrote: Does the script open a connection to the same server it is being called from? Yes. So this is a case of self-deadlock? I don't understand. What is it exactly that you expect will happen? If there is one connection it works. If there is 29 parallel bind requests it works. If there is 30 and more it immediately hang. I'm seeking why 29 is ok and 30 is bad. In other words deadlock happens only if I run 30+ parallel connections. The server by default has 30 worker threads. This is the (hidden) nsslapd-threadnumber attribute in cn=config. 30 is probably much too large for most deployments, which can get by with 2 threads per processor. I do this: for i in `seq 1 30` do time ldapsearch -LLL -H ldaps://xxx.cesnet.cz -x \ -b dc=perun-shadow,dc=cesnet,dc=cz \ -D "uid=semik$i,ou=People,dc=perun-shadow,dc=cesnet,dc=cz" \ -w 'zadek' -s base dn & done and 389 is immediately deadlocked. That should not happen I think. The script itself binds anonymously so it should not go into PAM. It doesn't, but because all of the other worker threads in the server are waiting on the lock, there is no worker thread to service the request. That script was just proof of concept. Maybe I should explain what I want to do in other thread. Yes, please. -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Re: [389-users] PAM Pass through authentication only one threaded
On 11/04/2013 05:22 PM, Rich Megginson wrote: > On 11/04/2013 09:08 AM, Jan Tomasek wrote: >> On 11/04/2013 05:04 PM, Rich Megginson wrote: >> >>> Does the script open a connection to the same server it is being called >>> from? >> >> Yes. > > So this is a case of self-deadlock? I don't understand. What is it > exactly that you expect will happen? If there is one connection it works. If there is 29 parallel bind requests it works. If there is 30 and more it immediately hang. I'm seeking why 29 is ok and 30 is bad. In other words deadlock happens only if I run 30+ parallel connections. I do this: > for i in `seq 1 30` > do > time ldapsearch -LLL -H ldaps://xxx.cesnet.cz -x \ > -b dc=perun-shadow,dc=cesnet,dc=cz \ > -D "uid=semik$i,ou=People,dc=perun-shadow,dc=cesnet,dc=cz" \ > -w 'zadek' -s base dn & > done and 389 is immediately deadlocked. That should not happen I think. The script itself binds anonymously so it should not go into PAM. That script was just proof of concept. Maybe I should explain what I want to do in other thread. -- --- Jan Tomasek aka Semik http://www.tomasek.cz/ signature.asc Description: OpenPGP digital signature -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Re: [389-users] PAM Pass through authentication only one threaded
On 11/04/2013 08:05 AM, Jan Tomasek wrote: Hi Rich, On 11/01/2013 04:28 PM, Rich Megginson wrote: If there is 30 or more parallel connections 389 hangs for ever. Very often killing process ldapsearch process does not help. Server is very often unable to restart so I have to kill it with -9. Please provide a stack trace - http://port389.org/wiki/FAQ#Debugging_Hangs stack trace is attached. I'm running: [root@pdap ~]# yum list installed |grep 389 389-admin.x86_64 1.1.29-1.el6 @epel 389-admin-console.noarch 1.1.8-1.el6 @epel 389-admin-console-doc.noarch 1.1.8-1.el6 @epel 389-adminutil.x86_64 1.1.15-1.el6 @epel 389-console.noarch 1.1.7-1.el6 @epel 389-ds.noarch 1.2.2-1.el6 @epel 389-ds-base.x86_64 1.2.11.15-22.el6_4 @updates 389-ds-base-libs.x86_641.2.11.15-22.el6_4 @updates 389-ds-console.noarch 1.2.6-1.el6 @epel 389-ds-console-doc.noarch 1.2.6-1.el6 @epel 389-dsgw.x86_641.1.10-1.el6 @epel Thank you Looks like you do not have the right debuginfo packages installed. But there may be enough information in the stack trace anyway. Your pam script is waiting: Thread 11 (Thread 0x7f60c97ea700 (LWP 10146)): #0 0x7f60f0b4209d in waitpid () from /lib64/libpthread.so.0 No symbol table info available. #1 0x7f60e033f2c1 in ?? () from /lib/security/pam_script.so No symbol table info available. #2 0x7f60e033f68a in pam_sm_authenticate () from /lib/security/pam_script.so No symbol table info available. #3 0x7f60e8aa2cee in ?? () from /lib64/libpam.so.0 No symbol table info available. #4 0x7f60e8aa2600 in pam_authenticate () from /lib64/libpam.so.0 No symbol table info available. #5 0x7f60e8cb1e24 in ?? () from /usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so No symbol table info available. #6 0x7f60e8cb22ca in pam_passthru_do_pam_auth () from /usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so No symbol table info available. It holds the lock in pam_passthru_do_pam_auth which causes all of the other threads to block. Why is the script not exiting? -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Re: [389-users] PAM Pass through authentication only one threaded
On 11/01/2013 08:49 AM, Jan Tomasek wrote: Hi Rich, On 11/01/2013 02:22 PM, Rich Megginson wrote: All ldapsearch scripts are executed in background = in parallel way. But server process them in serial way. I can tell that by increasing time needed to process ldapsearches. Increment around 2sec is caused by pam_unix delay because of wrong password. Is 389 bind process really serialized? Or have I just overlooked some limit? PAM is not thread safe, in our experience, so we have to serialize calls into PAM. thank you for confirmation of my observation. In fact I'm able to put my 389 server into deadlock. I've written simple auth script for libpam-script [1] It's purpose is to check pasword of user in other than main entry, attached. Content of /etc/pam.d/ldapserver: auth required/lib/security/pam_script.so onerr=fail dir=/usr/share/libpam-script accountrequired/lib/security/pam_script.so onerr=fail dir=/usr/share/libpam-script [root@pdap 8445]# ls -l /usr/share/libpam-script total 8 lrwxrwxrwx 1 root root 11 Oct 31 17:52 pam_script_acct -> perlauth.pl lrwxrwxrwx 1 root root 11 Oct 31 17:52 pam_script_auth -> perlauth.pl -rwxr-xr-x 1 root root 2450 Oct 31 19:45 perlauth.pl It works fine in it's serialized way - until there is maximum 29 parallel connections. If there is 30 or more parallel connections 389 hangs for ever. Very often killing process ldapsearch process does not help. Server is very often unable to restart so I have to kill it with -9. Please provide a stack trace - http://port389.org/wiki/FAQ#Debugging_Hangs My question is if there is any limit related to number of parallel bind operations. I guess there is something to related to 30 or more likely to 60 - my plugin itself open next connection to the same LDAP server. Thanks [1] http://sourceforge.net/projects/pam-script/-- --- Jan Tomasek aka Semik http://www.tomasek.cz/ -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Re: [389-users] PAM Pass through authentication only one threaded
Hi Rich,
On 11/01/2013 02:22 PM, Rich Megginson wrote:
All ldapsearch scripts are executed in background = in parallel way.
But server process them in serial way. I can tell that by increasing
time needed to process ldapsearches. Increment around 2sec is caused
by pam_unix delay because of wrong password.
Is 389 bind process really serialized? Or have I just overlooked some
limit?
PAM is not thread safe, in our experience, so we have to serialize calls
into PAM.
thank you for confirmation of my observation.
In fact I'm able to put my 389 server into deadlock.
I've written simple auth script for libpam-script [1] It's purpose is to
check pasword of user in other than main entry, attached.
Content of /etc/pam.d/ldapserver:
auth required/lib/security/pam_script.so onerr=fail
dir=/usr/share/libpam-script
accountrequired/lib/security/pam_script.so onerr=fail
dir=/usr/share/libpam-script
[root@pdap 8445]# ls -l /usr/share/libpam-script
total 8
lrwxrwxrwx 1 root root 11 Oct 31 17:52 pam_script_acct -> perlauth.pl
lrwxrwxrwx 1 root root 11 Oct 31 17:52 pam_script_auth -> perlauth.pl
-rwxr-xr-x 1 root root 2450 Oct 31 19:45 perlauth.pl
It works fine in it's serialized way - until there is maximum 29
parallel connections.
If there is 30 or more parallel connections 389 hangs for ever. Very
often killing process ldapsearch process does not help. Server is very
often unable to restart so I have to kill it with -9.
My question is if there is any limit related to number of parallel bind
operations. I guess there is something to related to 30 or more likely
to 60 - my plugin itself open next connection to the same LDAP server.
Thanks
[1] http://sourceforge.net/projects/pam-script/--
---
Jan Tomasek aka Semik
http://www.tomasek.cz/
#!/usr/bin/perl -w
use strict;
use Net::LDAPS;
use Net::LDAP::Constant qw(LDAP_SUCCESS);
use Sys::Syslog qw(:standard :macros);
use Net::LDAP::Util qw(escape_filter_value);
my $prg_name = $0;
$prg_name =~ s/.*\///;
my $ldap_host = 'localhost';
my $ldap_port = 636;
my $pam_user = 'PAM_USER';
my $pam_type = 'PAM_TYPE';
my $pam_password = 'PAM_AUTHTOK';
sub syslog_escape {
my $str = shift;
my @chr = split(//, $str);
for(my $i=0; $i<@chr; $i++) {
if (ord($chr[$i])>127) {
$chr[$i] = sprintf('\0x%X', ord($chr[$i]));
};
};
return join('', @chr);
};
sub logger {
my $priority = shift;
my $msg = shift;
openlog($prg_name, 'pid', LOG_LOCAL0);
syslog($priority, syslog_escape($msg));
closelog;
};
sub local_die {
logger(LOG_ERR, @_);
die;
};
sub log_pam_env {
my @out;
foreach my $key (keys %ENV) {
next unless ($key =~ /^PAM_/);
if ($key eq 'PAM_AUTHTOK') {
if (exists $ENV{$key}) {
if ($ENV{$pam_password} eq '') {
push @out, "$key=";
} else {
push @out, "$key=*hidden*" ;
};
};
} else {
push @out, "$key=".$ENV{$key};
};
};
logger(LOG_ERR, 'PAM env: '.join(' ', @out));
};
# Log all PAM_* env variables we got from LDAP server
log_pam_env();
my $ldaps = Net::LDAPS->new($ldap_host,
port => $ldap_port) or die "$@";
my $conn = $ldaps->bind;# TODO an anonymous bind
unless ($ENV{$pam_user}) {
local_die "Missing $pam_user in environment";
};
unless ($ENV{$pam_type}) {
local_die "Missing $pam_type in environment";
};
unless ($ENV{$pam_password}) {
if ($ENV{$pam_type} eq 'auth') {
local_die "Missing $pam_password in environment";
};
};
my $filter = '(objectClass=appPassword)';
if ($ENV{$pam_type} eq 'auth') {
$filter = "(&$filter(altUserPassword=".escape_filter_value($ENV{$pam_password})."))";
};
logger(LOG_ERR, $filter);
my $mesg = $ldaps->search( # perform a search
base => $ENV{$pam_user},
filter => $filter,
);
if ($mesg->code == LDAP_SUCCESS) {
foreach my $entry ($mesg->entries) {
# todo pridat popisku hesla ktera matchla
logger(LOG_ERR, 'Matched: '.$entry->dn);
exit 0;
};
local_die('Invalid password.');
} else {
local_die $mesg->error;
};
# K tomuhle bychom se nikdy nemeli dostat
local_die('This should not happen.');
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
Re: [389-users] PAM Pass through authentication only one threaded
On 11/01/2013 03:47 AM, Jan Tomasek wrote: Hello, I'm experimenting with PAM through authentication and it looks that 389 process parallel requests in serial way. To demonstrate this behavior I use simple testing script: for i in `seq 1 10` do time ldapsearch -LLL -H ldaps://xxx.cesnet.cz -x \ -b dc=perun-shadow,dc=cesnet,dc=cz \ -D "uid=semik$i,ou=People,dc=perun-shadow,dc=cesnet,dc=cz" \ -w 'zadek' -s base dn & done here is part of the output I get: bind DN [uid=semik6,ou=People,dc=perun-shadow,dc=cesnet,dc=cz] real0m2.127s bind DN [uid=semik10,ou=People,dc=perun-shadow,dc=cesnet,dc=cz] real0m4.392s bind DN [uid=semik1,ou=People,dc=perun-shadow,dc=cesnet,dc=cz] real0m6.405s bind DN [uid=semik5,ou=People,dc=perun-shadow,dc=cesnet,dc=cz] real0m8.699s bind DN [uid=semik2,ou=People,dc=perun-shadow,dc=cesnet,dc=cz] real0m10.926s ... All ldapsearch scripts are executed in background = in parallel way. But server process them in serial way. I can tell that by increasing time needed to process ldapsearches. Increment around 2sec is caused by pam_unix delay because of wrong password. Is 389 bind process really serialized? Or have I just overlooked some limit? PAM is not thread safe, in our experience, so we have to serialize calls into PAM. Configuration of PAM plugin: dn: cn=PAM Pass Through Auth,cn=plugins,cn=config objectClass: top objectClass: nsSlapdPlugin objectClass: extensibleObject objectClass: pamConfig cn: PAM Pass Through Auth nsslapd-pluginPath: libpam-passthru-plugin nsslapd-pluginInitfunc: pam_passthruauth_init nsslapd-pluginType: preoperation nsslapd-pluginEnabled: on nsslapd-pluginloadglobal: true nsslapd-plugin-depends-on-type: database pamMissingSuffix: ALLOW pamExcludeSuffix: cn=config pamIDMapMethod: RDN pamIDAttr: notUsedWithRDNMethod pamFallback: FALSE pamSecure: TRUE pamService: sshd nsslapd-pluginId: pam_passthruauth nsslapd-pluginVersion: 1.2.11.15 nsslapd-pluginVendor: 389 Project nsslapd-pluginDescription: PAM pass through authentication plugin nsslapd-pluginarg0: pamIncludeSuffix nsslapd-pluginarg1: dc=perun-shadow,dc=cesnet,dc=cz modifiersName: cn=directory manager modifyTimestamp: 20131101085721Z Thank you for any suggestions! -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
