Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Brice Goglin 
If you don't care about the overhead, tell python to use the output of
shell command "hwloc-calc -N pu all".

Brice


Le 31/08/2020 à 18:38, Brock Palen a écrit :
> Thanks,
>
> yeah I was looking for an API that would take into consideration most
> cases, like I find with hwloc-bind --get   where I can find the number
> the process has access to.  Wether is cgroups,  other sorts of
> affinity setting etc.
>
> Brock Palen
> IG: brockpalen1984
> www.umich.edu/~brockp 
> Director Advanced Research Computing - TS
> bro...@umich.edu 
> (734)936-1985
>
>
> On Mon, Aug 31, 2020 at 12:37 PM Guy Streeter  > wrote:
>
> I forgot that the cpuset value is still available in cgroups v2. You
> would want the cpuset.cpus.effective value.
> More information is available here:
> https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
>
> On Mon, Aug 31, 2020 at 11:19 AM Guy Streeter
> mailto:guy.stree...@gmail.com>> wrote:
> >
> > As I said, cgroups doesn't limit the group to a number of cores, it
> > limits processing time, either as an absolute amount or as a
> share of
> > what is available.
> > A docker process can be restricted to a set of cores, but that
> is done
> > with cpu affinity, not cgroups.
> >
> > You could try to figure out an equivalency. For instance if you are
> > using cpu.shares to limit the cgroups, then figure the ratio of a
> > cgroup's share to the shares of all the cgroups at that level, and
> > apply that ratio to the number of available cores to get an
> estimated
> > number of threads you should start.
> >
> > On Mon, Aug 31, 2020 at 10:40 AM Brock Palen  > wrote:
> > >
> > > Sorry if wasn't clear, I'm trying to find out what is
> available to my process before it starts up threads.  If the user
> is jailed in a cgroup (docker, slurm, other)  and the program
> tries to start 36 threads, when it only has access to 4 cores,
> it's probably not a huge deal, but not desirable.
> > >
> > > I do allow the user to specify number of threads, but would
> like to automate it for least astonishment.
> > >
> > > Brock Palen
> > > IG: brockpalen1984
> > > www.umich.edu/~brockp 
> > > Director Advanced Research Computing - TS
> > > bro...@umich.edu 
> > > (734)936-1985
> > >
> > >
> > > On Mon, Aug 31, 2020 at 11:34 AM Guy Streeter
> mailto:guy.stree...@gmail.com>> wrote:
> > >>
> > >> My very basic understanding of cgroups is that it can be used
> to limit
> > >> cpu processing time for a group, and to ensure fair
> distribution of
> > >> processing time within the group, but I don't know of a way
> to use
> > >> cgroups to limit the number of CPUs available to a cgroup.
> > >>
> > >> On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  > wrote:
> > >> >
> > >> > Hello,
> > >> >
> > >> > I have a small utility,  it is currently using 
> multiprocess.cpu_count()
> > >> > Which currently ignores cgroups etc.
> > >> >
> > >> > I see https://gitlab.com/guystreeter/python-hwloc
> > >> > But appears stale,
> > >> >
> > >> > How would you detect number of threads that are safe to
> start in a cgroup from Python3 ?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > Brock Palen
> > >> > IG: brockpalen1984
> > >> > www.umich.edu/~brockp 
> > >> > Director Advanced Research Computing - TS
> > >> > bro...@umich.edu 
> > >> > (734)936-1985
> > >> > ___
> > >> > hwloc-users mailing list
> > >> > hwloc-users@lists.open-mpi.org
> 
> > >> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> > >> ___
> > >> hwloc-users mailing list
> > >> hwloc-users@lists.open-mpi.org
> 
> > >> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> > >
> > > ___
> > > hwloc-users mailing list
> > > hwloc-users@lists.open-mpi.org
> 
> > > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Guy Streeter 
I wrote a set of python bindings for libcgroup v1 about 5 years ago,
and abandoned it because I couldn't find anybody interested in using
cgroups at the time.

On Mon, Aug 31, 2020 at 11:40 AM Brock Palen  wrote:
>
> Thanks,
>
> yeah I was looking for an API that would take into consideration most cases, 
> like I find with hwloc-bind --get   where I can find the number the process 
> has access to.  Wether is cgroups,  other sorts of affinity setting etc.
>
> Brock Palen
> IG: brockpalen1984
> www.umich.edu/~brockp
> Director Advanced Research Computing - TS
> bro...@umich.edu
> (734)936-1985
>
>
> On Mon, Aug 31, 2020 at 12:37 PM Guy Streeter  wrote:
>>
>> I forgot that the cpuset value is still available in cgroups v2. You
>> would want the cpuset.cpus.effective value.
>> More information is available here:
>> https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
>>
>> On Mon, Aug 31, 2020 at 11:19 AM Guy Streeter  wrote:
>> >
>> > As I said, cgroups doesn't limit the group to a number of cores, it
>> > limits processing time, either as an absolute amount or as a share of
>> > what is available.
>> > A docker process can be restricted to a set of cores, but that is done
>> > with cpu affinity, not cgroups.
>> >
>> > You could try to figure out an equivalency. For instance if you are
>> > using cpu.shares to limit the cgroups, then figure the ratio of a
>> > cgroup's share to the shares of all the cgroups at that level, and
>> > apply that ratio to the number of available cores to get an estimated
>> > number of threads you should start.
>> >
>> > On Mon, Aug 31, 2020 at 10:40 AM Brock Palen  wrote:
>> > >
>> > > Sorry if wasn't clear, I'm trying to find out what is available to my 
>> > > process before it starts up threads.  If the user is jailed in a cgroup 
>> > > (docker, slurm, other)  and the program tries to start 36 threads, when 
>> > > it only has access to 4 cores, it's probably not a huge deal, but not 
>> > > desirable.
>> > >
>> > > I do allow the user to specify number of threads, but would like to 
>> > > automate it for least astonishment.
>> > >
>> > > Brock Palen
>> > > IG: brockpalen1984
>> > > www.umich.edu/~brockp
>> > > Director Advanced Research Computing - TS
>> > > bro...@umich.edu
>> > > (734)936-1985
>> > >
>> > >
>> > > On Mon, Aug 31, 2020 at 11:34 AM Guy Streeter  
>> > > wrote:
>> > >>
>> > >> My very basic understanding of cgroups is that it can be used to limit
>> > >> cpu processing time for a group, and to ensure fair distribution of
>> > >> processing time within the group, but I don't know of a way to use
>> > >> cgroups to limit the number of CPUs available to a cgroup.
>> > >>
>> > >> On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  wrote:
>> > >> >
>> > >> > Hello,
>> > >> >
>> > >> > I have a small utility,  it is currently using  
>> > >> > multiprocess.cpu_count()
>> > >> > Which currently ignores cgroups etc.
>> > >> >
>> > >> > I see https://gitlab.com/guystreeter/python-hwloc
>> > >> > But appears stale,
>> > >> >
>> > >> > How would you detect number of threads that are safe to start in a 
>> > >> > cgroup from Python3 ?
>> > >> >
>> > >> > Thanks!
>> > >> >
>> > >> > Brock Palen
>> > >> > IG: brockpalen1984
>> > >> > www.umich.edu/~brockp
>> > >> > Director Advanced Research Computing - TS
>> > >> > bro...@umich.edu
>> > >> > (734)936-1985
>> > >> > ___
>> > >> > hwloc-users mailing list
>> > >> > hwloc-users@lists.open-mpi.org
>> > >> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> > >> ___
>> > >> hwloc-users mailing list
>> > >> hwloc-users@lists.open-mpi.org
>> > >> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> > >
>> > > ___
>> > > hwloc-users mailing list
>> > > hwloc-users@lists.open-mpi.org
>> > > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Brock Palen 
Thanks,

yeah I was looking for an API that would take into consideration most
cases, like I find with hwloc-bind --get   where I can find the number the
process has access to.  Wether is cgroups,  other sorts of affinity setting
etc.

Brock Palen
IG: brockpalen1984
www.umich.edu/~brockp
Director Advanced Research Computing - TS
bro...@umich.edu
(734)936-1985


On Mon, Aug 31, 2020 at 12:37 PM Guy Streeter 
wrote:

> I forgot that the cpuset value is still available in cgroups v2. You
> would want the cpuset.cpus.effective value.
> More information is available here:
> https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
>
> On Mon, Aug 31, 2020 at 11:19 AM Guy Streeter 
> wrote:
> >
> > As I said, cgroups doesn't limit the group to a number of cores, it
> > limits processing time, either as an absolute amount or as a share of
> > what is available.
> > A docker process can be restricted to a set of cores, but that is done
> > with cpu affinity, not cgroups.
> >
> > You could try to figure out an equivalency. For instance if you are
> > using cpu.shares to limit the cgroups, then figure the ratio of a
> > cgroup's share to the shares of all the cgroups at that level, and
> > apply that ratio to the number of available cores to get an estimated
> > number of threads you should start.
> >
> > On Mon, Aug 31, 2020 at 10:40 AM Brock Palen  wrote:
> > >
> > > Sorry if wasn't clear, I'm trying to find out what is available to my
> process before it starts up threads.  If the user is jailed in a cgroup
> (docker, slurm, other)  and the program tries to start 36 threads, when it
> only has access to 4 cores, it's probably not a huge deal, but not
> desirable.
> > >
> > > I do allow the user to specify number of threads, but would like to
> automate it for least astonishment.
> > >
> > > Brock Palen
> > > IG: brockpalen1984
> > > www.umich.edu/~brockp
> > > Director Advanced Research Computing - TS
> > > bro...@umich.edu
> > > (734)936-1985
> > >
> > >
> > > On Mon, Aug 31, 2020 at 11:34 AM Guy Streeter 
> wrote:
> > >>
> > >> My very basic understanding of cgroups is that it can be used to limit
> > >> cpu processing time for a group, and to ensure fair distribution of
> > >> processing time within the group, but I don't know of a way to use
> > >> cgroups to limit the number of CPUs available to a cgroup.
> > >>
> > >> On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  wrote:
> > >> >
> > >> > Hello,
> > >> >
> > >> > I have a small utility,  it is currently using
> multiprocess.cpu_count()
> > >> > Which currently ignores cgroups etc.
> > >> >
> > >> > I see https://gitlab.com/guystreeter/python-hwloc
> > >> > But appears stale,
> > >> >
> > >> > How would you detect number of threads that are safe to start in a
> cgroup from Python3 ?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > Brock Palen
> > >> > IG: brockpalen1984
> > >> > www.umich.edu/~brockp
> > >> > Director Advanced Research Computing - TS
> > >> > bro...@umich.edu
> > >> > (734)936-1985
> > >> > ___
> > >> > hwloc-users mailing list
> > >> > hwloc-users@lists.open-mpi.org
> > >> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> > >> ___
> > >> hwloc-users mailing list
> > >> hwloc-users@lists.open-mpi.org
> > >> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> > >
> > > ___
> > > hwloc-users mailing list
> > > hwloc-users@lists.open-mpi.org
> > > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Guy Streeter 
I forgot that the cpuset value is still available in cgroups v2. You
would want the cpuset.cpus.effective value.
More information is available here:
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html

On Mon, Aug 31, 2020 at 11:19 AM Guy Streeter  wrote:
>
> As I said, cgroups doesn't limit the group to a number of cores, it
> limits processing time, either as an absolute amount or as a share of
> what is available.
> A docker process can be restricted to a set of cores, but that is done
> with cpu affinity, not cgroups.
>
> You could try to figure out an equivalency. For instance if you are
> using cpu.shares to limit the cgroups, then figure the ratio of a
> cgroup's share to the shares of all the cgroups at that level, and
> apply that ratio to the number of available cores to get an estimated
> number of threads you should start.
>
> On Mon, Aug 31, 2020 at 10:40 AM Brock Palen  wrote:
> >
> > Sorry if wasn't clear, I'm trying to find out what is available to my 
> > process before it starts up threads.  If the user is jailed in a cgroup 
> > (docker, slurm, other)  and the program tries to start 36 threads, when it 
> > only has access to 4 cores, it's probably not a huge deal, but not 
> > desirable.
> >
> > I do allow the user to specify number of threads, but would like to 
> > automate it for least astonishment.
> >
> > Brock Palen
> > IG: brockpalen1984
> > www.umich.edu/~brockp
> > Director Advanced Research Computing - TS
> > bro...@umich.edu
> > (734)936-1985
> >
> >
> > On Mon, Aug 31, 2020 at 11:34 AM Guy Streeter  
> > wrote:
> >>
> >> My very basic understanding of cgroups is that it can be used to limit
> >> cpu processing time for a group, and to ensure fair distribution of
> >> processing time within the group, but I don't know of a way to use
> >> cgroups to limit the number of CPUs available to a cgroup.
> >>
> >> On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  wrote:
> >> >
> >> > Hello,
> >> >
> >> > I have a small utility,  it is currently using  multiprocess.cpu_count()
> >> > Which currently ignores cgroups etc.
> >> >
> >> > I see https://gitlab.com/guystreeter/python-hwloc
> >> > But appears stale,
> >> >
> >> > How would you detect number of threads that are safe to start in a 
> >> > cgroup from Python3 ?
> >> >
> >> > Thanks!
> >> >
> >> > Brock Palen
> >> > IG: brockpalen1984
> >> > www.umich.edu/~brockp
> >> > Director Advanced Research Computing - TS
> >> > bro...@umich.edu
> >> > (734)936-1985
> >> > ___
> >> > hwloc-users mailing list
> >> > hwloc-users@lists.open-mpi.org
> >> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> >> ___
> >> hwloc-users mailing list
> >> hwloc-users@lists.open-mpi.org
> >> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> >
> > ___
> > hwloc-users mailing list
> > hwloc-users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Brice Goglin 

Le 31/08/2020 à 18:19, Guy Streeter a écrit :
> As I said, cgroups doesn't limit the group to a number of cores, it
> limits processing time, either as an absolute amount or as a share of
> what is available.
> A docker process can be restricted to a set of cores, but that is done
> with cpu affinity, not cgroups.


cgroup can actually do lots of things: limit the available cores
(example below with only PU #0), the available NUMA nodes, the amount of
RAM in those nodes, processing time, etc.

$ cat /sys/fs/cgroup/cpuset/foobar/cpuset.cpus
0

Brice


___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Guy Streeter 
As I said, cgroups doesn't limit the group to a number of cores, it
limits processing time, either as an absolute amount or as a share of
what is available.
A docker process can be restricted to a set of cores, but that is done
with cpu affinity, not cgroups.

You could try to figure out an equivalency. For instance if you are
using cpu.shares to limit the cgroups, then figure the ratio of a
cgroup's share to the shares of all the cgroups at that level, and
apply that ratio to the number of available cores to get an estimated
number of threads you should start.

On Mon, Aug 31, 2020 at 10:40 AM Brock Palen  wrote:
>
> Sorry if wasn't clear, I'm trying to find out what is available to my process 
> before it starts up threads.  If the user is jailed in a cgroup (docker, 
> slurm, other)  and the program tries to start 36 threads, when it only has 
> access to 4 cores, it's probably not a huge deal, but not desirable.
>
> I do allow the user to specify number of threads, but would like to automate 
> it for least astonishment.
>
> Brock Palen
> IG: brockpalen1984
> www.umich.edu/~brockp
> Director Advanced Research Computing - TS
> bro...@umich.edu
> (734)936-1985
>
>
> On Mon, Aug 31, 2020 at 11:34 AM Guy Streeter  wrote:
>>
>> My very basic understanding of cgroups is that it can be used to limit
>> cpu processing time for a group, and to ensure fair distribution of
>> processing time within the group, but I don't know of a way to use
>> cgroups to limit the number of CPUs available to a cgroup.
>>
>> On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  wrote:
>> >
>> > Hello,
>> >
>> > I have a small utility,  it is currently using  multiprocess.cpu_count()
>> > Which currently ignores cgroups etc.
>> >
>> > I see https://gitlab.com/guystreeter/python-hwloc
>> > But appears stale,
>> >
>> > How would you detect number of threads that are safe to start in a cgroup 
>> > from Python3 ?
>> >
>> > Thanks!
>> >
>> > Brock Palen
>> > IG: brockpalen1984
>> > www.umich.edu/~brockp
>> > Director Advanced Research Computing - TS
>> > bro...@umich.edu
>> > (734)936-1985
>> > ___
>> > hwloc-users mailing list
>> > hwloc-users@lists.open-mpi.org
>> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Brock Palen 
Sorry if wasn't clear, I'm trying to find out what is available to my
process before it starts up threads.  If the user is jailed in a cgroup
(docker, slurm, other)  and the program tries to start 36 threads, when it
only has access to 4 cores, it's probably not a huge deal, but not
desirable.

I do allow the user to specify number of threads, but would like to
automate it for least astonishment.

Brock Palen
IG: brockpalen1984
www.umich.edu/~brockp
Director Advanced Research Computing - TS
bro...@umich.edu
(734)936-1985


On Mon, Aug 31, 2020 at 11:34 AM Guy Streeter 
wrote:

> My very basic understanding of cgroups is that it can be used to limit
> cpu processing time for a group, and to ensure fair distribution of
> processing time within the group, but I don't know of a way to use
> cgroups to limit the number of CPUs available to a cgroup.
>
> On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  wrote:
> >
> > Hello,
> >
> > I have a small utility,  it is currently using  multiprocess.cpu_count()
> > Which currently ignores cgroups etc.
> >
> > I see https://gitlab.com/guystreeter/python-hwloc
> > But appears stale,
> >
> > How would you detect number of threads that are safe to start in a
> cgroup from Python3 ?
> >
> > Thanks!
> >
> > Brock Palen
> > IG: brockpalen1984
> > www.umich.edu/~brockp
> > Director Advanced Research Computing - TS
> > bro...@umich.edu
> > (734)936-1985
> > ___
> > hwloc-users mailing list
> > hwloc-users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc Python3 Bindings - Correctly Grab number cores available

2020-08-31 Thread Guy Streeter 
My very basic understanding of cgroups is that it can be used to limit
cpu processing time for a group, and to ensure fair distribution of
processing time within the group, but I don't know of a way to use
cgroups to limit the number of CPUs available to a cgroup.

On Mon, Aug 31, 2020 at 8:56 AM Brock Palen  wrote:
>
> Hello,
>
> I have a small utility,  it is currently using  multiprocess.cpu_count()
> Which currently ignores cgroups etc.
>
> I see https://gitlab.com/guystreeter/python-hwloc
> But appears stale,
>
> How would you detect number of threads that are safe to start in a cgroup 
> from Python3 ?
>
> Thanks!
>
> Brock Palen
> IG: brockpalen1984
> www.umich.edu/~brockp
> Director Advanced Research Computing - TS
> bro...@umich.edu
> (734)936-1985
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users