Re: kerberos issues with 10.0_BETA post openssl update

[Brett Lymn]

On Mon, Sep 25, 2023 at 02:12:11PM +1300, Mark Davies wrote:
> 
> 
> Core was generated by `saslauthd'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  quote_string (s=0x73756372616d  0x73756372616d>,

It may just be coincidence but that address looks like a bunch of ascii
 sucrad or usrcda if the bytes are swapped.

It may be nothing though...

-- 
Brett Lymn
--
Sent from my NetBSD device.

"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 25/09/23 14:12, Mark Davies wrote:
was also able to confirm that rolling back just pam_krb5.so.4 to a 
version prior to Taylor's June change fixed it, so presumably there is 
something in that change that is not quite right.


I should add, my gut feeling here is that its some sort of use after 
free that we are getting away with most of the time but not always, but 
I can't spot anything.


cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 4/09/23 16:46, Mark Davies wrote:

1.  pam_ksu not working


Taylor fixed this.


2.  ktutil causes kadmind to segfault.


Christos fixed this.

3.   pam_krb5 will seemingly randomly die while validating perfectly 
valid username/password pairs.


Both dovecot's auth and saslauthd (configured to do pam, and pam to do 
pam_krb5) would get segmentation faults processing some connections 
while others (giving the same credentials) would succeed.


So with 1 and 2 fixed I rolled mail server forward again to see the 
state of issue 3 and confirmed that the problem still persists.  However 
was also able to confirm that rolling back just pam_krb5.so.4 to a 
version prior to Taylor's June change fixed it, so presumably there is 
something in that change that is not quite right.


Unfortunately, prior to putting the working pam_krb5 in, I wasn't able 
to persuade dovecot to give coredumps along with its segfaults.  I do 
however have a coredump for saslauthd from 3 weeks ago and two more from 
this morning:



Core was generated by `saslauthd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  quote_string (s=0x73756372616d address 0x73756372616d>,
out=out@entry=0x7f7fff06fbd0 "", idx=0, len=len@entry=256, 
display=display@entry=0)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/principal.c:418

(gdb) where
#0  quote_string (s=0x73756372616d address 0x73756372616d>,
out=out@entry=0x7f7fff06fbd0 "", idx=0, len=len@entry=256, 
display=display@entry=0)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/principal.c:418
#1  0x736565442cc0 in unparse_name_fixed 
(context=context@entry=0x736565752000, principal=0x7365656dd5a0,
name=name@entry=0x7f7fff06fbd0 "", len=len@entry=256, 
flags=flags@entry=0)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/principal.c:457
#2  0x736565443569 in krb5_unparse_name_fixed 
(context=context@entry=0x736565752000,
principal=, name=name@entry=0x7f7fff06fbd0 "", 
len=len@entry=256)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/principal.c:507
#3  0x7365654429ec in krb5_error_from_rd_error 
(context=context@entry=0x736565752000,

error=error@entry=0x7365657b7da0, creds=creds@entry=0x7365657b7c08)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/rd_error.c:86
#4  0x73656542cf22 in krb5_init_creds_step 
(context=context@entry=0x736565752000,
ctx=ctx@entry=0x7365657b7c00, in=in@entry=0x7f7fff070640, 
out=out@entry=0x7f7fff070650,

hostinfo=hostinfo@entry=0x0, flags=flags@entry=0x7f7fff070634)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/init_creds_pw.c:2334
#5  0x73656542de98 in krb5_init_creds_get 
(context=context@entry=0x736565752000, ctx=0x7365657b7c00)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/init_creds_pw.c:2634
#6  0x73656542b963 in krb5_get_init_creds_password 
(context=0x736565752000, creds=0x7f7fff071110,
client=0x7365656ddb20, password=0x7365657ea110 "tclubhideout99v", 
prompter=0x0, data=0x7365657f2000,

start_time=0, in_tkt_service=, options=0x736565789180)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/init_creds_pw.c:2728
#7  0x73656020279b in pam_sm_authenticate () from 
/usr/lib/security/pam_krb5.so.4
#8  0x736563804cee in openpam_dispatch 
(pamh=pamh@entry=0x7365657f2000, primitive=primitive@entry=0,
flags=-2147483648) at 
/src/work/10/src/external/bsd/openpam/dist/lib/libpam/openpam_dispatch.c:125
#9  0x736563803e66 in pam_authenticate (pamh=0x7365657f2000, 
flags=)
at 
/src/work/10/src/external/bsd/openpam/dist/lib/libpam/pam_authenticate.c:69

#10 0x00019e203ca9 in ?? ()
#11 0x00019e2083cc in ?? ()
#12 0x00019e20758d in ?? ()
#13 0x00019e207c8c in ?? ()
#14 0x00019e20a1ab in ?? ()
#15 0x00019e202edd in ?? ()
#16 0x7f7f3840bbb8 in ?? () from /usr/libexec/ld.elf_so
#17 0x0003 in ?? ()
#18 0x7f7fff0729f0 in ?? ()
#19 0x7f7fff072a08 in ?? ()
#20 0x7f7fff072a0b in ?? ()
#21 0x in ?? ()



Core was generated by `saslauthd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x796d85d9c091 in strlen () from /usr/lib/libc.so.12
(gdb) where
#0  0x796d85d9c091 in strlen () from /usr/lib/libc.so.12
#1  0x796d85cbbb4b in _strdup (str=0x736d6c616572 access memory at address 0x736d6c616572>)

at /src/work/10/src/lib/libc/string/strdup.c:60
#2  0x796d88081c17 in der_copy_general_string (from=, 
to=0x796d88a61390)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/asn1/der_copy.c:46
#3  0x796d8804a104 in copy_PrincipalName 
(from=from@entry=0x796d887d49a0, to=to@entry=0x796d88746220)

at asn1_krb5_asn1.c:1019
#4  0x796d8804a4c5 in copy_Principal 
(from=from@entry=0x796d887d49a0, to=to@entry=0x796d88746220)

at asn1_krb5_asn1.c:1160
#5  0x796d88443cb3 in krb5_copy_principal 

Re: kerberos issues with 10.0_BETA post openssl update

[Ken Hornstein]

>so in actual usage pretty well everything is going to use
>aes256-cts-hmac-sha1-96 (unless you have a really old client out there) 
>but the KDC is still going to create or update keys of all three types, 
>and that is whats failing here.

My apologies; going back I realize I conflated the client issues with
your kadmind segfault and I was thinking your CLIENTS were segfaulting.

I see later on you just transitioned to AES enctypes, which is probably
for the best anyway.  It sounds like someone could still explicitly
use kadmin to ask for arcfour and cause a denial-of-service attack
against kadmind though.

--Ken

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 9/09/23 13:24, Mark Davies wrote:

And yes I could probably explicitly add

   default_etypes = aes256-cts-hmac-sha1-96

to krb5.conf to drop the two obsolete types but then I'd have to notice 
and change it again if at some point in the future heimdal changed its 
defaults to something new.


For the record the above didn't work.  The correct way to set the 
default keys is to add for example the following to kdc.conf (or krb5.conf)



[kadmin]
default_keys = aes256-cts-hmac-sha1-96:pw-salt 
aes256-cts-hmac-sha384-192:pw-salt



With this added you don't get the segfault as it doesn't try to do 
arcfour-hmac-md5 so that is a workaround - and one I'll probably go with 
anyway (ignoring what I said before) as it lets me add additional modern 
keytypes that heimdal doesn't look like it will be defaulting till 8.

(see https://github.com/heimdal/heimdal/issues/988)

cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 9/09/23 06:39, Ken Hornstein wrote:

I don't know if you have control over this, but ... RC4?  In 2023?  Yikes.
Kerberos clients do send a list of the supported crypto algorithms to the
KDC as part of the AS-REQ message so this would indicate to me that one
of three things is happening:

1) Your client is configured to only allow RC4 as a cipher (I've seen
this happen when people misguidedly configure this in the Kerberos
configuration files; normally this should never be done except it very
unusual circumstances)
2) Your KDC does not support crypto algorithms other than RC4,
which ... double yikes if that's the case.
3) Your KDC DOES support more modern crypto algorithms but you haven't
changed your password in approximately forever.

If this is 1 or 3 it should be easy to fix and probably would be a good
idea to do so.



Yes clients send lists of supported crypto algorithms and KDC's have 
lists of keys of different etypes that can be used for a given principle 
and they negotiate which one they are going to use between what the 
client and server support.


By default heimdal makes keys of the following three etypes:
aes256-cts-hmac-sha1-96
des3-cbc-sha1
arcfour-hmac-md5

so in actual usage pretty well everything is going to use
aes256-cts-hmac-sha1-96 (unless you have a really old client out there) 
but the KDC is still going to create or update keys of all three types, 
and that is whats failing here.


And yes I could probably explicitly add

  default_etypes = aes256-cts-hmac-sha1-96

to krb5.conf to drop the two obsolete types but then I'd have to notice 
and change it again if at some point in the future heimdal changed its 
defaults to something new.


cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Ken Hornstein]

>> This looks like a jump to null in the RC4 logic using EVP_md4().
>> 
>> For EVP_rc4 we have a hack in Heimdal to do
>> 
>>  EVP_CIPHER_fetch(NULL, "rc4", "provider=legacy")

I don't know if you have control over this, but ... RC4?  In 2023?  Yikes.
Kerberos clients do send a list of the supported crypto algorithms to the
KDC as part of the AS-REQ message so this would indicate to me that one
of three things is happening:

1) Your client is configured to only allow RC4 as a cipher (I've seen
   this happen when people misguidedly configure this in the Kerberos
   configuration files; normally this should never be done except it very
   unusual circumstances)
2) Your KDC does not support crypto algorithms other than RC4,
   which ... double yikes if that's the case.
3) Your KDC DOES support more modern crypto algorithms but you haven't
   changed your password in approximately forever.

If this is 1 or 3 it should be easy to fix and probably would be a good
idea to do so.

--Ken

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 6/09/23 11:56, Taylor R Campbell wrote:

This looks like a jump to null in the RC4 logic using EVP_md4().

For EVP_rc4 we have a hack in Heimdal to do

EVP_CIPHER_fetch(NULL, "rc4", "provider=legacy")

but I'm not sure it actually works -- I can't get it to do anything in
a test program without also calling OSSL_PROVIDER_load("legacy"), at
which point it becomes unnecessary -- and we don't do it for MD4.

So if we can convince Heimdal to call OSSL_PROVIDER_load("legacy") at
some point on startup, I bet that will fix it.

It looks like the EVP_CIPHER_fetch hack (or EVP_MD_fetch hack) is also
a memory leak, according to
:

These functions usually have the name APINAME_fetch, where APINAME
is the name of the operation.  For example EVP_MD_fetch(3) can be
used to explicitly fetch a digest algorithm implementation.  The
user is responsible for freeing the object returned from the
APINAME_fetch function using APINAME_free when it is no longer
needed.

So I'm not sure we should be using it at all.


I've logged a PR for this: lib/57610

cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 6/09/23 22:46, Taylor R Campbell wrote:

Here's a workaround you could test with no code changes that shouldn't
break other applications: move /root/.k5login to /etc/k5login.d/root,
and set

[libdefaults]
kuserok = USER-K5LOGIN SYSTEM-K5LOGIN SIMPLE DENY

in /etc/krb5.conf.  Still worth finding a code fix for pam_ksu, but
you can try this workaround in the mean time.



Just to confirm that the workaround does work.

mark

Re: kerberos issues with 10.0_BETA post openssl update

[Taylor R Campbell]

> Date: Wed, 6 Sep 2023 10:39:34 +
> From: Taylor R Campbell 
> 
> A possible workaround is to set:
> 
>   [libdefaults]
>   k5login_directory = /root
> 
> However, that applies to _all_ kuserok checks for _all_ users, not
> just the pam_ksu one ror root, so it will probably break other things.
> I'm not sure there is a way in the config file to specify it just for
> pam_ksu or just for root.

Here's a workaround you could test with no code changes that shouldn't
break other applications: move /root/.k5login to /etc/k5login.d/root,
and set

[libdefaults]
kuserok = USER-K5LOGIN SYSTEM-K5LOGIN SIMPLE DENY

in /etc/krb5.conf.  Still worth finding a code fix for pam_ksu, but
you can try this workaround in the mean time.

Re: kerberos issues with 10.0_BETA post openssl update

[Taylor R Campbell]

> Date: Wed, 6 Sep 2023 16:41:00 +1200
> From: Mark Davies 
> 
> OK, so revision 1.10 of pam_ksu.c adds a call to 
> krb5_set_home_dir_access(NULL, FALSE);
> which causes the subsequent call to krb5_kuserok() to return false when 
> previously it would return true causing the whole pam_ksu to bail.
> 
> krb5_kuserok() is presuambly now returning false because if it can't 
> access the homedir it can't read /root/.k5login to see that 
> mark/r...@ecs.vuw.ac.nz is allowed.

The reason for revision 1.10 is that pam_ksu had a gaping security
hole without it, allowing the calling user to totally control the krb5
context by specifying ~/.krb5/config in the _calling user's_ home
directory and thereby spoof authentication decisions:

https://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2023-005.txt.asc

I verified that the hole was there without the change, and I verified
that the change plugged the hole.

However, I think you're right that the change causes it to take this
path in krb5_kuserok to block access to the _target user's_ home
directory too:

profile_dir = k5login_dir;
if (profile_dir == NULL) {
/* Don't deadlock with gssd or anything of the sort */
if (!_krb5_homedir_access(context))
return KRB5_PLUGIN_NO_HANDLE;

if (rk_getpwnam_r(luser, , pwbuf, sizeof(pwbuf), ) != 0) {
krb5_set_error_message(context, errno, "User unknown 
(getpwnam_r())");
return KRB5_PLUGIN_NO_HANDLE;
}
if (pwd == NULL) {
krb5_set_error_message(context, errno, "User unknown (getpwnam())");
return KRB5_PLUGIN_NO_HANDLE;
}
profile_dir = pwd->pw_dir;
}

A possible workaround is to set:

[libdefaults]
k5login_directory = /root

However, that applies to _all_ kuserok checks for _all_ users, not
just the pam_ksu one ror root, so it will probably break other things.
I'm not sure there is a way in the config file to specify it just for
pam_ksu or just for root.

Perhaps it would be appropriate to add these lines in pam_ksu.c (or
possibly just the first one):

goto out;
}
+   krb5_set_home_dir_access(context, TRUE);
PAM_LOG("kuserok: %s -> %s", su_principal_name, user);
rv = krb5_kuserok(context, su_principal, user);
+   krb5_set_home_dir_access(context, FALSE);

At that point, the config file should have been parsed already, so the
calling user's ~/.krb5/config can't hurt anything.  But I haven't
audited this path.  So I don't know if it's safe.

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 6/09/23 12:09, Mark Davies wrote:
The problem with that one is that su doesn't actually die, the pam_ksu 
just errors in some way so that pam abandons it and moves on to other 
authentication types, and I can't ktrace it as su is a suid program so 
I'll probably have to stuff some more debugging into pam_ksu.c to try 
and narrow it down.


OK, so revision 1.10 of pam_ksu.c adds a call to 
krb5_set_home_dir_access(NULL, FALSE);
which causes the subsequent call to krb5_kuserok() to return false when 
previously it would return true causing the whole pam_ksu to bail.



krb5_kuserok() is presuambly now returning false because if it can't 
access the homedir it can't read /root/.k5login to see that 
mark/r...@ecs.vuw.ac.nz is allowed.


cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 6/09/23 11:56, Taylor R Campbell wrote:

as to the su issue, I think that is a separate problem related to
revision 1.10 of pam_ksu.c.  If I revert that then su works.


Got a stack trace for that?


The problem with that one is that su doesn't actually die, the pam_ksu 
just errors in some way so that pam abandons it and moves on to other 
authentication types, and I can't ktrace it as su is a suid program so 
I'll probably have to stuff some more debugging into pam_ksu.c to try 
and narrow it down.


mark

Re: kerberos issues with 10.0_BETA post openssl update

[Taylor R Campbell]

> Date: Wed, 6 Sep 2023 09:54:16 +1200
> From: Mark Davies 
> 
> OK, found a simpler reproducible crash.  Run "kadmin -l" on a kdc and 
> try to change a password.
> 
> xen2# kadmin -l
> kadmin> passwd ecsproctor
> ecsproc...@ecs.vuw.ac.nz's Password:
> Verifying - ecsproc...@ecs.vuw.ac.nz's Password:
> Segmentation fault (core dumped)
> 
> Here is the backtrace
> 
> Core was generated by `kadmin'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x in ?? ()
> (gdb) where
> #0  0x in ?? ()
> #1  0x7f11ca0423d4 in ARCFOUR_string_to_key (context=0x7f11cafc7000, 
> enctype=KRB5_ENCTYPE_ARCFOUR_HMAC_MD5,
>  password=..., salt=..., opaque=..., key=0x7f11caf514d8)
>  at 
> /src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt-arcfour.c:83

This looks like a jump to null in the RC4 logic using EVP_md4().

For EVP_rc4 we have a hack in Heimdal to do

EVP_CIPHER_fetch(NULL, "rc4", "provider=legacy")

but I'm not sure it actually works -- I can't get it to do anything in
a test program without also calling OSSL_PROVIDER_load("legacy"), at
which point it becomes unnecessary -- and we don't do it for MD4.

So if we can convince Heimdal to call OSSL_PROVIDER_load("legacy") at
some point on startup, I bet that will fix it.

It looks like the EVP_CIPHER_fetch hack (or EVP_MD_fetch hack) is also
a memory leak, according to
:

   These functions usually have the name APINAME_fetch, where APINAME
   is the name of the operation.  For example EVP_MD_fetch(3) can be
   used to explicitly fetch a digest algorithm implementation.  The
   user is responsible for freeing the object returned from the
   APINAME_fetch function using APINAME_free when it is no longer
   needed.

So I'm not sure we should be using it at all.

> as to the su issue, I think that is a separate problem related to 
> revision 1.10 of pam_ksu.c.  If I revert that then su works.

Got a stack trace for that?

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 5/09/23 23:11, Markus Kilbinger wrote:

In a preexisting krb-installation on an aarch64 machine (odroid c2)
kadmin still dumps core when trying to add a new principal (even after
recent source changes):


Same here (other than and64)


xen2# kadmin -l add -r host/xx
Max ticket life [unlimited]:
Max renewable life [unlimited]:
Principal expiration time [never]:
Password expiration time [never]:
Attributes []:
Policy [default]:
Segmentation fault (core dumped)

xen2# gdb /usr/sbin/kadmin ./kadmin.core
GNU gdb (GDB) 11.0.50.20200914-git
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kadmin...
Reading symbols from /usr/libdata/debug//usr/sbin/kadmin.debug...
[New process 27647]
Core was generated by `kadmin'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x in ?? ()
(gdb) where
#0  0x in ?? ()
#1  0x70e63fc423d4 in ARCFOUR_string_to_key (context=0x70e640ec9000, 
enctype=KRB5_ENCTYPE_ARCFOUR_HMAC_MD5,

password=..., salt=..., opaque=..., key=0x70e640ed33d8)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt-arcfour.c:83
#2  0x70e63fc41531 in krb5_string_to_key_data_salt 
(context=context@entry=0x70e640ec9000,
enctype=KRB5_ENCTYPE_ARCFOUR_HMAC_MD5, password=..., salt=..., 
key=0x70e640ed33d8)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt.c:173
#3  0x70e63fc41662 in krb5_string_to_key_salt 
(context=context@entry=0x70e640ec9000, enctype=,
password=password@entry=0x7f7fff0650c0 "tmOogi:Mar", salt=..., 
key=)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt.c:225
#4  0x70e640811d25 in hdb_generate_key_set_password_with_ks_tuple 
(principal=,
ks_tuple=, n_ks_tuple=, 
num_keys=, keys=,

password=, context=)
at /src/work/10/src/crypto/external/bsd/heimdal/dist/lib/hdb/keys.c:800
#5  hdb_generate_key_set_password_with_ks_tuple (context=0x70e640ec9000, 
principal=,
password=password@entry=0x7f7fff0650c0 "tmOogi:Mar", 
ks_tuple=, n_ks_tuple=n_ks_tuple@entry=0,

keys=keys@entry=0x7f7fff064e10, num_keys=num_keys@entry=0x7f7fff064e18)
at /local/SAVE/10_64.base.debug/usr/include/krb5/hdb-protos.h:346
#6  0x70e640c0ef50 in _kadm5_set_keys 
(context=context@entry=0x70e640ec9180, ent=ent@entry=0x7f7fff064e68,
n_ks_tuple=n_ks_tuple@entry=0, ks_tuple=, 
password=password@entry=0x7f7fff0650c0 "tmOogi:Mar")
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/kadm5/set_keys.c:55
#7  0x70e640c09e70 in kadm5_s_create_principal 
(server_handle=0x70e640ec9180, princ=,
mask=, n_ks_tuple=0, ks_tuple=, 
password=0x7f7fff0650c0 "tmOogi:Mar")
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/kadm5/create_s.c:208
#8  0x0d40880d in add_one_principal (name=, 
rand_key=1, rand_password=0, use_defaults=0,
password=0x7f7fff0650c0 "tmOogi:Mar", policy=, 
key_data=key_data@entry=0x0,
max_ticket_life=0x0, max_renewable_life=0x0, attributes=0x0, 
expiration=0x0, pw_expiration=0x0)

at /src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/ank.c:154
#9  0x0d408bea in add_new_key (opt=opt@entry=0x7f7fff065600, 
argc=1, argv=)

at /src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/ank.c:263
#10 0x0d407920 in add_wrap (argc=, 
argv=0x7f7fff0659b8) at kadmin-commands.c:222

#11 0x0d40f441 in main (argc=3, argv=)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/kadmin.c:281

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 4/09/23 22:30, Taylor R Campbell wrote:

Can you install the debug set on one of the affected systems where you
can reproduce a problem to get more information out of the stack
traces?


OK, found a simpler reproducible crash.  Run "kadmin -l" on a kdc and 
try to change a password.


xen2# kadmin -l
kadmin> passwd ecsproctor
ecsproc...@ecs.vuw.ac.nz's Password:
Verifying - ecsproc...@ecs.vuw.ac.nz's Password:
Segmentation fault (core dumped)

Here is the backtrace

Core was generated by `kadmin'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x in ?? ()
(gdb) where
#0  0x in ?? ()
#1  0x7f11ca0423d4 in ARCFOUR_string_to_key (context=0x7f11cafc7000, 
enctype=KRB5_ENCTYPE_ARCFOUR_HMAC_MD5,

password=..., salt=..., opaque=..., key=0x7f11caf514d8)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt-arcfour.c:83
#2  0x7f11ca041531 in krb5_string_to_key_data_salt 
(context=context@entry=0x7f11cafc7000,
enctype=KRB5_ENCTYPE_ARCFOUR_HMAC_MD5, password=..., salt=..., 
key=0x7f11caf514d8)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt.c:173
#3  0x7f11ca041662 in krb5_string_to_key_salt 
(context=context@entry=0x7f11cafc7000, enctype=,
password=password@entry=0x7f7fff434b30 "FooBar123", salt=..., 
key=)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/krb5/salt.c:225
#4  0x7f11cac11d25 in hdb_generate_key_set_password_with_ks_tuple 
(principal=,
ks_tuple=, n_ks_tuple=, 
num_keys=, keys=,

password=, context=)
at /src/work/10/src/crypto/external/bsd/heimdal/dist/lib/hdb/keys.c:800
#5  hdb_generate_key_set_password_with_ks_tuple (context=0x7f11cafc7000, 
principal=,
password=password@entry=0x7f7fff434b30 "FooBar123", 
ks_tuple=ks_tuple@entry=0x0,
n_ks_tuple=n_ks_tuple@entry=0, keys=keys@entry=0x7f7fff434990, 
num_keys=num_keys@entry=0x7f7fff434998)

at /local/SAVE/10_64.base.debug/usr/include/krb5/hdb-protos.h:346
#6  0x7f11cb00ef50 in _kadm5_set_keys 
(context=context@entry=0x7f11cafc7180, ent=ent@entry=0x7f7fff434a28,
n_ks_tuple=n_ks_tuple@entry=0, ks_tuple=ks_tuple@entry=0x0, 
password=password@entry=0x7f7fff434b30 "FooBar123")
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/kadm5/set_keys.c:55
#7  0x7f11cb007813 in change (server_handle=0x7f11cafc7180, 
princ=, keepold=0, n_ks_tuple=0,

ks_tuple=0x0, password=0x7f7fff434b30 "FooBar123", cond=cond@entry=0)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/kadm5/chpass_s.c:95
#8  0x7f11cb007969 in kadm5_s_chpass_principal 
(server_handle=, princ=,
keepold=, n_ks_tuple=, 
ks_tuple=, password=)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/lib/kadm5/chpass_s.c:195
#9  0x0001d240945d in set_password (keepold=, 
password=,
principal=0x7f11cafb53c0) at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/cpw.c:113

#10 do_cpw_entry (data=, principal=0x7f11cafb53c0)
at /src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/cpw.c:141
#11 do_cpw_entry (principal=0x7f11cafb53c0, data=)
at /src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/cpw.c:130
#12 0x0001d240ed3b in foreach_principal (exp_str=, 
func=func@entry=0x1d2409340 ,
funcname=funcname@entry=0x1d240f97c "cpw", 
data=data@entry=0x7f7fff434ca0)

at /src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/util.c:626
#13 0x0001d240966f in cpw_entry (opt=opt@entry=0x7f7fff434d70, 
argc=1, argv=0x7f11cb268bc8)

at /src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/cpw.c:187
#14 0x0001d2407703 in passwd_wrap (argc=, 
argv=0x7f11cb268bc0) at kadmin-commands.c:263
#15 0x7f11ca401f48 in sl_command_loop (cmds=cmds@entry=0x1d2614d80 
,

prompt=prompt@entry=0x1d2410ef8 "kadmin> ", data=data@entry=0x0)
at /src/work/10/src/crypto/external/bsd/heimdal/dist/lib/sl/sl.c:333
#16 0x0001d240f2a5 in main (argc=, argv=)
at 
/src/work/10/src/crypto/external/bsd/heimdal/dist/kadmin/kadmin.c:290





as to the su issue, I think that is a separate problem related to 
revision 1.10 of pam_ksu.c.  If I revert that then su works.



cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Markus Kilbinger]

In a preexisting krb-installation on an aarch64 machine (odroid c2)
kadmin still dumps core when trying to add a new principal (even after
recent source changes):

# kadmin -l add -r host/test
Max ticket life [1 day]:
Max renewable life [1 week]:
Principal expiration time [never]:
Password expiration time [never]:
Attributes []:
Policy [default]:
Memory fault (core dumped)

# gdb /usr/sbin/kadmin kadmin.core
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kadmin...
(No debugging symbols found in /usr/sbin/kadmin)
[New process 591]
Core was generated by `kadmin'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x in ?? ()
(gdb) bt
#0  0x in ?? ()
#1  0xf37a50b81320 in ?? () from /usr/lib/libkrb5.so.28
#2  0xf37a50b803a8 in krb5_string_to_key_data_salt_opaque () from
/usr/lib/libkrb5.so.28
#3  0xf37a50b8045c in krb5_string_to_key_data_salt () from
/usr/lib/libkrb5.so.28
#4  0xf37a50b805dc in krb5_string_to_key_salt () from /usr/lib/libkrb5.so.28
#5  0xf37a50c61fe0 in hdb_generate_key_set_password_with_ks_tuple
() from /usr/lib/libhdb.so.16
#6  0xf37a50cae978 in _kadm5_set_keys () from /usr/lib/libkadm5srv.so.16
#7  0xf37a50ca9e40 in kadm5_s_create_principal () from
/usr/lib/libkadm5srv.so.16
#8  0x0fdf85c4 in add_one_principal ()
#9  0x0fdf8930 in add_new_key ()
#10 0x0fdf7944 in add_wrap ()
#11 0x0fdff0b8 in main ()

Am Di., 5. Sept. 2023 um 00:10 Uhr schrieb Mark Davies :
>
>
>
> On 4/09/23 22:30, Taylor R Campbell wrote:
> > Can you share `ldd /usr/libexec/kadmind' on the machine where it's
> > crashing?  Wondering whether it's mixing shlib versions in one address
> > space, like https://gnats.netbsd.org/57603 (though that's not the same
> > issue because you're only using things in base).
>
>
> /usr/libexec/kadmind:
>  -lgssapi.11 => /usr/lib/libgssapi.so.11
>  -lkrb5.27 => /usr/lib/libkrb5.so.27
>  -lhx509.6 => /usr/lib/libhx509.so.6
>  -lasn1.10 => /usr/lib/libasn1.so.10
>  -lcom_err.8 => /usr/lib/libcom_err.so.8
>  -lc.12 => /usr/lib/libc.so.12
>  -lroken.20 => /usr/lib/libroken.so.20
>  -lutil.7 => /usr/lib/libutil.so.7
>  -lcrypt.1 => /usr/lib/libcrypt.so.1
>  -lcrypto.15 => /usr/lib/libcrypto.so.15
>  -lwind.1 => /usr/lib/libwind.so.1
>  -lheimbase.2 => /usr/lib/libheimbase.so.2
>  -lsqlite3.1 => /usr/lib/libsqlite3.so.1
>  -lm.0 => /usr/lib/libm.so.0
>  -lheimntlm.5 => /usr/lib/libheimntlm.so.5
>  -lkadm5srv.15 => /usr/lib/libkadm5srv.so.15
>  -lhdb.15 => /usr/lib/libhdb.so.15
>
> > Can you install the debug set on one of the affected systems where you
> > can reproduce a problem to get more information out of the stack
> > traces?
>
> I'll rebuild a tree with MKDEBUG and MKDEBUGLIB set now, and see what I see.
>
> cheers
> mark
>

Re: kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

On 4/09/23 22:30, Taylor R Campbell wrote:

Can you share `ldd /usr/libexec/kadmind' on the machine where it's
crashing?  Wondering whether it's mixing shlib versions in one address
space, like https://gnats.netbsd.org/57603 (though that's not the same
issue because you're only using things in base).



/usr/libexec/kadmind:
-lgssapi.11 => /usr/lib/libgssapi.so.11
-lkrb5.27 => /usr/lib/libkrb5.so.27
-lhx509.6 => /usr/lib/libhx509.so.6
-lasn1.10 => /usr/lib/libasn1.so.10
-lcom_err.8 => /usr/lib/libcom_err.so.8
-lc.12 => /usr/lib/libc.so.12
-lroken.20 => /usr/lib/libroken.so.20
-lutil.7 => /usr/lib/libutil.so.7
-lcrypt.1 => /usr/lib/libcrypt.so.1
-lcrypto.15 => /usr/lib/libcrypto.so.15
-lwind.1 => /usr/lib/libwind.so.1
-lheimbase.2 => /usr/lib/libheimbase.so.2
-lsqlite3.1 => /usr/lib/libsqlite3.so.1
-lm.0 => /usr/lib/libm.so.0
-lheimntlm.5 => /usr/lib/libheimntlm.so.5
-lkadm5srv.15 => /usr/lib/libkadm5srv.so.15
-lhdb.15 => /usr/lib/libhdb.so.15


Can you install the debug set on one of the affected systems where you
can reproduce a problem to get more information out of the stack
traces?


I'll rebuild a tree with MKDEBUG and MKDEBUGLIB set now, and see what I see.

cheers
mark

Re: kerberos issues with 10.0_BETA post openssl update

[Christos Zoulas]

In article <20230904103054.553e160...@jupiter.mumble.net>,
Taylor R Campbell   wrote:
>> Date: Mon, 4 Sep 2023 16:46:35 +1200
>> From: Mark Davies 
>> 
>> Having updated from a 10.0_BETA built in march to one built couple
>> of weeks ago (post the openssl3 merge) I'm now seeing various
>> kerberos issues that I wasn't seeing before.
>
>Can you share `ldd /usr/libexec/kadmind' on the machine where it's
>crashing?  Wondering whether it's mixing shlib versions in one address
>space, like https://gnats.netbsd.org/57603 (though that's not the same
>issue because you're only using things in base).
>
>Can you install the debug set on one of the affected systems where you
>can reproduce a problem to get more information out of the stack
>traces?

You must be mixing the old and new versions of the OpenSSL libraries.
Kerberos works fine for me in HEAD.

christos

Re: kerberos issues with 10.0_BETA post openssl update

[Taylor R Campbell]

> Date: Mon, 4 Sep 2023 16:46:35 +1200
> From: Mark Davies 
> 
> Having updated from a 10.0_BETA built in march to one built couple
> of weeks ago (post the openssl3 merge) I'm now seeing various
> kerberos issues that I wasn't seeing before.

Can you share `ldd /usr/libexec/kadmind' on the machine where it's
crashing?  Wondering whether it's mixing shlib versions in one address
space, like https://gnats.netbsd.org/57603 (though that's not the same
issue because you're only using things in base).

Can you install the debug set on one of the affected systems where you
can reproduce a problem to get more information out of the stack
traces?

kerberos issues with 10.0_BETA post openssl update

[Mark Davies]

Having updated from a  10.0_BETA built in march to one built couple of 
weeks ago (post the openssl3 merge) I'm now seeing various kerberos 
issues that I wasn't seeing before.


1.  pam_ksu not working

On an old system su gets a kerberos specific password prompt:

www-cache% su
mark/r...@ecs.vuw.ac.nz's password:

On a new system drops straight through to the no kerberos prompt

smb2% su
Password:


with pam debug enabled on the pam_ksu module the old system prints:

www-cache su: in openpam_dispatch(): calling pam_sm_authenticate() in 
/usr/lib/security/pam_ksu.so.4

www-cache su: in pam_sm_authenticate(): Got user: root
www-cache su: in pam_sm_authenticate(): Got ruser: mark
www-cache su: in get_su_principal(): Default principal name: 
m...@ecs.vuw.ac.nz
www-cache su: in get_su_principal(): Target principal name: 
mark/r...@ecs.vuw.ac.nz
www-cache su: in pam_sm_authenticate(): kuserok: mark/r...@ecs.vuw.ac.nz 
-> root


but the new system:

smb2 su: in openpam_dispatch(): calling pam_sm_authenticate() in 
/usr/lib/security/pam_ksu.so.4

smb2 su: in pam_sm_authenticate(): Got user: root
smb2 su: in pam_sm_authenticate(): Got ruser: mark
smb2 su: in get_su_principal(): Default principal name: m...@ecs.vuw.ac.nz
smb2 su: in get_su_principal(): Target principal name: 
mark/r...@ecs.vuw.ac.nz

smb2 su: in pam_sm_authenticate(): kuserok: mark/r...@ecs.vuw.ac.nz -> root
smb2 su: in openpam_dispatch(): /usr/lib/security/pam_ksu.so.4: 
pam_sm_authenticate(): Authentication error




2.  ktutil causes kadmind to segfault.

A command such as
  ktutil -k /tmp/k.keytab get -p mark/admin host/xx.ecs.vuw.ac.nz

fails to work. Gets the error
   ktutil: kadm5_create_principal(host/xx.ecs.vuw.ac.nz): End of file

because the kadmind on the kerberos server  segfaults

(No debugging symbols found in /usr/libexec/kadmind)
[New process 3300]
Core was generated by `kadmind'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x in ?? ()
(gdb) where
#0  0x in ?? ()
#1  0x73f5d68423d4 in ?? () from /usr/lib/libkrb5.so.27
#2  0x73f5d6841531 in krb5_string_to_key_data_salt () from 
/usr/lib/libkrb5.so.27
#3  0x73f5d7011d25 in hdb_generate_key_set_password_with_ks_tuple () 
from /usr/lib/libhdb.so.15

#4  0x73f5d740ef90 in _kadm5_set_keys () from /usr/lib/libkadm5srv.so.15
#5  0x73f5d7409eb0 in kadm5_s_create_principal () from 
/usr/lib/libkadm5srv.so.15

#6  0x058078df in kadmind_dispatch.isra ()
#7  0x058084f3 in kadmind_loop ()
#8  0x05809323 in main ()



3.   pam_krb5 will seemingly randomly die while validating perfectly 
valid username/password pairs.


Both dovecot's auth and saslauthd (configured to do pam, and pam to do 
pam_krb5) would get segmentation faults processing some connections 
while others (giving the same credentials) would succeed.


[...]
Sep  3 19:33:05 mail1 dovecot: auth: Error: auth-worker: Aborted PASSV 
request for mark: Worker process died unexpectedly
Sep  3 19:33:25 mail1 dovecot: auth: Error: auth-worker: Aborted PASSV 
request for xxx: Worker process died unexpectedly
Sep  3 19:33:43 mail1 dovecot: auth: Error: auth-worker: Aborted PASSV 
request for yyy: Worker process died unexpectedly

[...]

   [...]
Sep 03 19:33:04 auth: Debug: client passdb out: OK  1 
user=mark
Sep 03 19:33:04 auth: Debug: client passdb out: OK  1 
user=mark
Sep 03 19:33:07 auth: Debug: client passdb out: FAIL1 
user=mark   code=temp_fail
Sep 03 19:33:09 auth: Debug: client passdb out: OK  1 
user=mark
Sep 03 19:33:25 auth: Debug: client passdb out: OK  1   user=zzz 

Sep 03 19:33:27 auth: Debug: client passdb out: FAIL1   user=xxx 
   code=temp_fail
Sep 03 19:33:45 auth: Debug: client passdb out: FAIL1   user=yyy 
   code=temp_fail

   [...]


I didn't get a coredump from dovecot before I had to roll back that 
machine to the older system but I did get one from saslauthd


Core was generated by `saslauthd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x736565442b9f in ?? () from /usr/lib/libkrb5.so.27
(gdb) where
#0  0x736565442b9f in ?? () from /usr/lib/libkrb5.so.27
#1  0x736565442cc0 in ?? () from /usr/lib/libkrb5.so.27
#2  0x7365654429ec in krb5_error_from_rd_error () from 
/usr/lib/libkrb5.so.27
#3  0x73656542cf22 in krb5_init_creds_step () from 
/usr/lib/libkrb5.so.27

#4  0x73656542de98 in krb5_init_creds_get () from /usr/lib/libkrb5.so.27
#5  0x73656542b963 in krb5_get_init_creds_password () from 
/usr/lib/libkrb5.so.27
#6  0x73656020279b in pam_sm_authenticate () from 
/usr/lib/security/pam_krb5.so.4

#7  0x736563804cee in openpam_dispatch () from /usr/lib/libpam.so.4
#8  0x736563803e66 in pam_authenticate () from /usr/lib/libpam.so.4
#9  0x00019e203ca9 in ?? ()
#10 0x00019e2083cc in ?? ()
#11 0x00019e20758d in ?? ()
#12 0x00019e207c8c in ?? ()
#13 0x00019e20a1ab in ?? ()
#14