[slurm-dev] Re: FAIR_TREE in SLURM 14.11

Ryan Cox Thu, 04 Jun 2015 14:59:40 -0700

Trey,

The reason why the Fairshare values are not proportionally different isbecause they are essentially that user's ranking compared to otherusers, divided by the total number of user associations. The fairsharevalue for a user is just (rank-- / user_assoc_count) where rank isinitialized to user_assoc_count. That's why it's not proportional forthe fairshare value. There may be some other ways to do this but theranking method guarantees proper ordering and eliminates any worriesover floating point issues but, as you observed, it is not proportional.

http://slurm.schedmd.com/fair_tree.html#algorithm and page 36 onward inthe presentation you linked to describe it a little more, especially the"animation" that starts on page 38. The numbers that pop up in thebottom left when visiting a user show the final fairshare calculation.


Ryan

On 06/04/2015 11:38 AM, Trey Dockendorf wrote:

Re: [slurm-dev] Re: FAIR_TREE in SLURM 14.11
Ryan,
Thanks, I think that clarification plus the slide I found here,http://slurm.schedmd.com/SUG14/fair_tree.pdf, offered the insight Ineeded. With the deep hierarchy the goal is that a deeply nestedaccount be considered a sibling with an account elsewhere in thetree. The key was setting FairShare=parent for the accounts thatcontain no users and that are only used for organizational purposes.Once I made that change the normalized shares of accounts that containusers looked correct. The basic idea is such that all accountscontaining users end up being considered siblings under the rootaccount. Using FairShare=parent achieves this perfectly in 14.11.
The only issue I'm seeing may be a non-issue but could use someclarification for better understanding. I ran 100 5-minute jobs as"test-idhmc" (stakeholder) and "test-hepx" (stakeholder) and 505-minute jobs as "test-general" (non-stakeholder). I looked at thesshare -l output for these 3 users and noticed while the FairSharevalues are correctly ordered such that test-hepx > test-idhmc >test-general, the ratio is not proportional for test-hepx vstest-idhmc. The hepx account has 700 shares while the idhmc accounthas 128 shares. The ratio for the account's normalized shares iscorrect, and so is the Level FS of the account. In this case thecalculated FairShare for test-hepx is not much higher than test-idhmcbut it's still greater so in the end I guess that's what matters.
Thanks,
- Trey

# sshare -u test-hepx,test-idhmc,test-general -l
Account User Raw Shares Norm Shares Raw Usage NormUsage Effectv Usage FairShare Level FS GrpCPUMins CPURunMins-------------------- ---------- ---------- ----------- ---------------------- ------------- ---------- ---------- --------------------------
root                        0.000000       11217    1.000000   0
 grid             parent    0.000000           0    0.000000  0.000000 0
cms 10 0.003315 0 0.000000 0.000000inf 0suragrid 10 0.003315 0 0.0000000.000000 inf 0
 tamu             parent    0.000000       11217    1.000000  1.000000 0
agriculture parent 0.000000 0 0.0000000.000000 0aglife 10 0.003315 0 0.0000000.000000 inf 0genomics 10 0.003315 0 0.0000000.000000 inf 0engineering parent 0.000000 0 0.0000000.000000 0pete 10 0.003315 0 0.0000000.000000 inf 0general 10 0.003315 2262 0.2017280.201728 0.016431 0general test-general 1 0.007874 2262 0.2017281.000000 0.001572 0.007874 0
  geo             parent    0.000000           0    0.000000  0.000000 0
atmo 10 0.003315 0 0.0000000.000000 inf 0liberalarts parent 0.000000 4507 0.4017940.401794 0idhmc 128 0.042426 4507 0.4017940.401794 0.105592 0idhmc test-idhmc 1 0.062500 4507 0.4017941.000000 0.201258 0.062500 0mgmt 2058 0.682135 0 0.0000000.000000 inf 0science parent 0.000000 4447 0.3964780.396478 0acad 10 0.003315 0 0.0000000.000000 inf 0chem 10 0.003315 0 0.0000000.000000 inf 0iamcs 10 0.003315 0 0.0000000.000000 inf 0math-dept parent 0.000000 0 0.0000000.000000 0math 10 0.003315 0 0.0000000.000000 inf 0secant 10 0.003315 0 0.0000000.000000 inf 0physics parent 0.000000 4447 0.3964780.396478 0hepx 700 0.232019 4447 0.3964780.396478 0.585199 0hepx test-hepx 1 0.012821 4447 0.3964781.000000 0.226415 0.012821 0
   stat             parent    0.000000           0    0.000000  0.000000 0
carroll 10 0.003315 0 0.0000000.000000 inf 0
=============================

Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: treyd...@tamu.edu <mailto:treyd...@tamu.edu>
Jabber: treyd...@tamu.edu <mailto:treyd...@tamu.edu>
On Thu, Jun 4, 2015 at 11:51 AM, Ryan Cox <ryan_...@byu.edu<mailto:ryan_...@byu.edu>> wrote:
    Trey,

    In http://slurm.schedmd.com/fair_tree.html#fairshare, take a look
    at the definition for "S".  Basically, the normalized shares only
    matters between sibling associations and will equal 1.0 when
    summed.  If an association has no siblings, the value is 1.0.  If
    each of the four siblings in an account has the same Raw Shares
    (as defined in sacctmgr) value, the normalized shares value for
    each is 0.25.  The reason why is because the Level Fairshare
    calculations are only done within in account, comparing siblings
    to each other.  Note that Norm Usage is still presented in sshare
    but not used in the calculations.

    The sshare manpage has a section about the Fair Tree modifications
    to existing columns:
    http://slurm.schedmd.com/sshare.html#SECTION_FAIR_TREE%20MODIFICATIONS

    Ryan


    On 06/03/2015 02:47 PM, Trey Dockendorf wrote:
    My site is currently on 14.03.10 and we are evaluating and
    testing 14.11.7 as well as moving from
    PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME to using
    PriorityFlags=FAIR_TREE,SMALL_RELATIVE_TO_TIME.

    Our account hierarchy is very deep and is intended to represent
    the org structure of departments and research organizations that
    are using our cluster [1].  We were able to make the normalized
    share ratio match up so all non-stakeholders were equal
    (0.000323) and all stakeholders had the correct ratio based on
    their contributions to the cluster.  The Shares value assigned
    represents CPUs funded.  All the CPUs no longer belonging to
    stakeholders were given to the "mgmt" group so that the Shares
    given to the top level (tamu) had a meaningful value when divided
    up amongst all the accounts.

    While testing FAIR_TREE I noticed the normalized shares were
    drastically different [2].  In particular the current
    stakeholders (idhcm and hepx) both ended up with 1.0.  I'm
    guessing this is due to having no sibling accounts.

    The docs for FAIR_TREE only describe the formula used to
    calculate the Level FairShare. Does the method for calculating
    normalized shares change for FAIR_TREE? Is the hierarchy we are
    using not a good fit for FAIR_TREE?  The description and benefits
    of FAIR_TREE appeal to our use case, so modifying our hierarchy
    is within the realm of things I'm willing to change.
Any advice on migrating into FAIR_TREE is more than welcome.Right now I've been running "sleep" jobs under different UIDs to
    simulate usage to try and work out how we may need to adjust
    things for a migration to FAIR_TREE.

    I used the attached spreadsheet to work out the share values we
    are using with 14.03.10.

    Thanks,
    - Trey

    [1]:
       Account       User Raw Shares Norm Shares   Raw Usage Effectv
    Usage  FairShare
    -------------------- ---------- ---------- -----------
    ----------- ------------- ----------
    root  1.000000   114089982      1.000000 0.870551
root root 1 0.000323 00.000000 1.000000grid 1 0.000323 36880.000032 0.986174cms 10 0.000269 36880.000027 0.986155suragrid 1 0.000027 00.000000 1.000000tamu 3096 0.999354 1140862940.999968 0.870477agriculture 20 0.006671 26970.000024 0.999507aglife 1 0.0033362697 0.000012 0.999507genomics 1 0.003336 00.000000 1.000000engineering 10 0.003336 00.000000 1.000000
       pete                                1    0.003336           0
         0.000000   1.000000
general 10 0.0033365542 0.000049 0.997977geo 10 0.003336 20.000000 0.999999
       atmo                                1    0.003336           2
         0.000000   0.999999
liberalarts 128 0.042696 00.000000 1.000000
       idhmc                                 1  0.042696           0
         0.000000 1.000000
mgmt 2058 0.686472 160.000000 1.000000science 760 0.253508114078034 0.999895 0.578806
       acad                               10    0.003336           0
         0.000000   1.000000
       chem                               10    0.003336           0
         0.000000   1.000000
       iamcs                                10  0.003336     3506549
         0.030649 0.279777
math-dept 20 0.006671 117354110.102422 0.119035
        math                                10  0.003336    11735411
         0.102422 0.014169
secant 10 0.003336 00.000000 1.000000physics 700 0.233494 988360730.919795 0.579205
        hepx                               700  0.233494    98836073
         0.919795 0.579205
       stat                               10    0.003336           0
         0.000000   1.000000
carroll 10 0.003336 00.000000 1.000000
    [2]:
       Account       User Raw Shares Norm Shares   Raw Usage Effectv
    Usage  FairShare
    -------------------- ---------- ---------- -----------
    ----------- ------------- ----------
    root  0.000000       53229      1.000000
root root 1 0.000323 00.000000 1.000000grid 1 0.000323 00.000000cms 10 0.909091 00.000000suragrid 1 0.090909 00.000000tamu 3096 0.999354 532291.000000agriculture 20 0.006676 00.000000aglife 1 0.5000000 0.000000genomics 1 0.500000 00.000000engineering 10 0.003338 00.000000
       pete                                1    1.000000           0
         0.000000
general 10 0.0033386326 0.118860geo 10 0.003338 00.000000
       atmo                                1    1.000000           0
         0.000000
liberalarts 128 0.042724 131220.246522
       idhmc                                 1  1.000000       13122
         1.000000
mgmt 2058 0.686916 209840.394237science 760 0.25367212795 0.240382
       acad                               10    0.013158           0
         0.000000
       chem                               10    0.013158           0
         0.000000
       iamcs                                10  0.013158           0
         0.000000
math-dept 20 0.026316 00.000000
        math                                10  0.500000           0
         0.000000
secant 10 0.500000 00.000000physics 700 0.921053 127951.000000
        hepx                                 1  1.000000       12795
         1.000000
       stat                               10    0.013158           0
         0.000000
carroll 1 1.000000 00.000000
    =============================

    Trey Dockendorf
    Systems Analyst I
    Texas A&M University
    Academy for Advanced Telecommunications and Learning Technologies
    Phone: (979)458-2396
    Email: treyd...@tamu.edu <mailto:treyd...@tamu.edu>
    Jabber: treyd...@tamu.edu <mailto:treyd...@tamu.edu>
--Ryan Cox
    Operations Director
    Fulton Supercomputing Lab
    Brigham Young University

[slurm-dev] Re: FAIR_TREE in SLURM 14.11

Reply via email to