[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

STINNER Victor  added the comment:

Ned: so Linux was also impacted, but Linux only has an issue with more than 
65536 groups :-D macOS MAX_GROUPS is now only 16 (Python uses MAX_GROUPS+1)! 
Maybe MAX_GROUPS was reduced recently. I'm not sure.

Anyway, os.getgrouplist() does no longer depend on MAX_GROUPS hardcoded limit, 
but grow the group list dynamically. It should now work with any number of 
groups an all platforms (which provide the getgrouplist() function ;-)) on 
Python 3.7, 3.8 and master branches.

Thanks Dong-hee Na for the bug report and your PR.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

miss-islington  added the comment:


New changeset af6fd1faa68f57c11c862624798f8510b7cac68a by Miss Islington (bot) 
in branch '3.8':
bpo-40014: Fix os.getgrouplist() (GH-19126)
https://github.com/python/cpython/commit/af6fd1faa68f57c11c862624798f8510b7cac68a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

miss-islington  added the comment:


New changeset 5753fc69977bd9f70ecb4d466bda650efccf9e0a by Miss Islington (bot) 
in branch '3.7':
bpo-40014: Fix os.getgrouplist() (GH-19126)
https://github.com/python/cpython/commit/5753fc69977bd9f70ecb4d466bda650efccf9e0a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

Change by miss-islington :


--
pull_requests: +18505
pull_request: https://github.com/python/cpython/pull/19144

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

Change by miss-islington :


--
pull_requests: +18504
pull_request: https://github.com/python/cpython/pull/19143

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f5c7cabb2be4e42a5975ba8aac8bb458c8d9d6d7 by Victor Stinner in 
branch 'master':
bpo-40014: Fix os.getgrouplist() (GH-19126)
https://github.com/python/cpython/commit/f5c7cabb2be4e42a5975ba8aac8bb458c8d9d6d7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

STINNER Victor  added the comment:

> Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is 
> too small. In that case, ngroups is updated to the number of groups and so 
> can be used to enlarge the list!

I tested by manually initializing ngroups to a value way lower than MAX_GROUPS 
(65536 on my Fedora 31): os.getgrouplist() raises OSError() with a random 
errno, getgrouplist() doesn't set errno on failure.

I wrote PR 19126 to fix the issue on all platforms. The glibc implementation is 
detected and used: use updated ngroups value (only if it's larger).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18487
pull_request: https://github.com/python/cpython/pull/19126

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

miss-islington  added the comment:


New changeset 1cdc61c7673f71f2cef57715e482c84efda6d9e0 by Miss Islington (bot) 
in branch '3.7':
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
https://github.com/python/cpython/commit/1cdc61c7673f71f2cef57715e482c84efda6d9e0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

miss-islington  added the comment:


New changeset 21bee0bd71e1ad270274499f9f58194ebb52e236 by Miss Islington (bot) 
in branch '3.8':
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
https://github.com/python/cpython/commit/21bee0bd71e1ad270274499f9f58194ebb52e236


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +18484
pull_request: https://github.com/python/cpython/pull/19123

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8ec7370c89aa522602eb9604086ce9f09770953d by Victor Stinner in 
branch 'master':
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
https://github.com/python/cpython/commit/8ec7370c89aa522602eb9604086ce9f09770953d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[miss-islington]

Change by miss-islington :


--
pull_requests: +18485
pull_request: https://github.com/python/cpython/pull/19124

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

STINNER Victor  added the comment:

test.pythoninfo of the macOS job on the PR:

os.getgrouplist: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 
250, 395, 398, 400
os.getgroups: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 
395, 398, 400

So yeah, there are 18 groups ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[STINNER Victor]

STINNER Victor  added the comment:

> Most likely this hasn't been an issue before is that 10.15 creates more 
> groups for system services than the earlier versions did.

Aha, interesting. I updated my PR to remove mentions of the macOS version 
number. I just wrote "on macOS". Yeah, I wouldn't be surprised that the 
function always failed.

--

I'm not sure that the Linux implementation is correct. It rely on errno, 
whereas Linux manual page of getgrouplist() doesn't say anything about errno 
being set on error.

Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is 
too small. In that case, ngroups is updated to the number of groups and so can 
be used to enlarge the list!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS if the user has more than 17 groups

[Ned Deily]

Ned Deily  added the comment:

Thanks for the PR, Victor, it does address the problem.  I went back and 
explicitly tested on macOS 10.14 and 10.13 by creating some more groups and 
adding the user to them and the same failure occurred there so this is not just 
a 10.15 issue.  Most likely this hasn't been an issue before is that 10.15 
creates more groups for system services than the earlier versions did.  So the 
PR should be updated to remove the 10.15 (and 10.5!) references.

--
title: os.getgrouplist() fails on macOS 10.15 (Catalina) if the user has more 
than 17 groups -> os.getgrouplist() can fail on macOS if the user has more than 
17 groups

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS

[STINNER Victor]

STINNER Victor  added the comment:

tl; dr os.getgrouplist() is limited to 17 groups and fails with a random 
OSError if the request user has more groups. PR 19118 fix the issue.

--

On macOS-10.15.1-x86_64-i386-64bit (python -m platform), os.getgrouplist() 
fails with my user name and group identifier:

>>> os.getgrouplist('haypo', 20)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 0] Error

Full example:

macbook:master haypo$ ./python.exe 
Python 3.9.0a4+ (heads/master:da2914d, Mar 20 2020, 09:45:36) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, pwd
>>> entry=pwd.getpwuid(os.getuid())

>>> os.getgrouplist(entry.pw_name, entry.pw_gid)
Traceback (most recent call last):
  File "", line 1, in 
InterruptedError: [Errno 4] Interrupted system call

>>> os.getgrouplist(entry.pw_name, entry.pw_gid)
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 0] Error

>>> entry
pwd.struct_passwd(pw_name='haypo', pw_passwd='', pw_uid=502, pw_gid=20, 
pw_gecos='Victor Stinner', pw_dir='/Users/haypo', pw_shell='/bin/bash')


My user has the following groups:

macbook:master haypo$ id
uid=502(haypo) gid=20(staff) 
groups=20(staff),702(com.apple.sharepoint.group.2),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),703(com.apple.sharepoint.group.3)

My user has 18 groups.

MAX_GROUPS=16, Python uses 1 + MAX_GROUPS.

getgrouplist() manual page says:

RETURN VALUES
 The getgrouplist() function returns 0 on success.  If the size of the 
group list is too small to hold
 all the user's groups, getgrouplist() returns -1 to indicate failure.  In 
this case, the group array
 will be filled with as many groups as will fit.


In short, getgrouplist() doesn't set errno on failure.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18479
pull_request: https://github.com/python/cpython/pull/19118

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18478
pull_request: https://github.com/python/cpython/pull/19117

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40014] os.getgrouplist() can fail on macOS

[Ned Deily]

Change by Ned Deily :


--
title: os.getgrouplist() fails on macOS of GH Actions (Azure) -> 
os.getgrouplist() can fail on macOS

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com