Lahiru Jayathilake created AIRAVATA-3977:
--------------------------------------------
Summary: Allocation Governance POC Inspired by Cloud Management
Platforms
Key: AIRAVATA-3977
URL: https://issues.apache.org/jira/browse/AIRAVATA-3977
Project: Airavata
Issue Type: Task
Components: Airavata Allocation Manager, Custos Airavata Integration
Reporter: Lahiru Jayathilake
h3. Summary
Research cloud governance and account management platforms (such as Kion, AWS
Control Tower, and similar tools) and build a standalone proof-of-concept that
demonstrates how their governance patterns can be applied to HPC compute
allocation management. The goal is to explore concepts like hierarchical
organizational structures, budget threshold enforcement, funding source
prioritization, and self-service allocation with guardrails, and produce a
working POC that the Custos team can learn from and adapt for future allocation
management work.
h3. Problem
HPC sites manage compute allocations from multiple funding sources (ACCESS-CI,
NAIRR, internal discretionary pools). Each source has different credit units,
lifecycles, and rules. As allocation management grows more complex, the system
needs governance capabilities that go beyond simple balance tracking:
* How do you organize projects into departments or groups and apply default
policies to all of them at once?
* When an allocation is running low, how do you automatically notify the PI at
80%, warn at 95%, and block new job submissions at 100%?
* If a researcher has credits from both ACCESS-CI and an internal pool for the
same resource, which source gets drawn down first?
* How does a PI self-allocate their awarded credits across CPU, GPU, and
storage without needing admin approval for every change, while still staying
within the total award?
Cloud governance platforms like [Kion|https://kion.io/] have solved analogous
problems for cloud account and budget management. Kion provides hierarchical
organizational units with policy inheritance, automated budget enforcement with
configurable thresholds, multi-source funding management, and self-service
account provisioning with built-in guardrails. While the domain is different
(cloud spending vs. HPC compute credits), the governance patterns are directly
transferable.
h3. Description
This project is an {*}exploratory research and prototyping task{*}. The student
will study cloud governance platforms, identify which patterns apply to HPC
allocation management, and build a standalone POC demonstrating those patterns.
# *Study cloud governance platforms*
Research how cloud management platforms handle governance at scale. The primary
reference is [Kion|https://kion.io/], but also look at AWS Control Tower, Azure
Management Groups, and GCP Organization Policies to get a broad perspective.
Read their documentation, watch available product demos, and understand the
core concepts:
* {*}Hierarchical organizational model{*}: How organizations, departments, and
projects are structured in a tree. How policies and budgets defined at a higher
level automatically apply to everything underneath.
* {*}Budget and funding management{*}: How funding sources (grants, contracts,
departmental budgets) are attached to organizational units and disseminated to
projects. How budgets can be set at different levels.
* {*}Threshold-based enforcement{*}: How the system takes automated actions
when spending hits configurable thresholds (notifications, warnings, freezes).
Not just tracking, but actually enforcing limits.
* {*}Policy inheritance and exemptions{*}: How governance rules cascade down
the organizational tree, and how specific projects can be exempted from
inherited rules when needed.
* {*}Self-service with guardrails{*}: How end users (PIs, project leads) can
provision and manage their own resources within limits set by administrators,
without needing manual approval for every action.
* {*}Funding source prioritization{*}: When multiple funding sources apply to
the same consumption, how the system determines which source to draw from first.
# *Map these patterns to HPC allocation management*
Produce a mapping document that translates cloud governance concepts to the HPC
context:
||*Cloud Concept*||*HPC Equivalent*||
|Organization / OU hierarchy|Site > Department > Research Group > Project|
|Cloud account|Slurm account (the thing that actually consumes resources)|
|Funding source|Allocation source (ACCESS-CI, NAIRR, internal pool)|
|Budget with threshold actions|Award balance with notification/enforcement at
configurable levels|
|Policy inheritance|Default allocation rules per department (e.g., all ACCESS
allocations get Slurm enforcement)|
|Account vending (self-service)|PI self-allocates credits across resource types
within their award|
Identify where the analogy holds, where it breaks down, and what is unique to
HPC (e.g., heterogeneous resource types like CPU vs. GPU vs. storage, external
awards arriving pre-approved).
# *Build a standalone POC*
Build a working prototype that demonstrates the key governance patterns in an
HPC allocation context. This should be a self-contained application (not
plugged into the existing Custos services) that showcases:
* {*}Hierarchical organization{*}: Create sites, departments, and projects in
a tree structure. Show how a policy or budget set at the department level
propagates to all projects underneath.
* {*}Threshold enforcement{*}: Configure thresholds on an allocation (e.g.,
80%, 95%, 100%) with different actions at each level (notify, warn, block).
Simulate consumption and show the enforcement triggering.
* {*}Multi-source funding{*}: Attach multiple allocation sources to a project.
Demonstrate draw-down ordering (e.g., use ACCESS credits first because they
expire, then fall back to internal credits).
* {*}Self-service allocation{*}: A PI receives an award of N credits. They can
split those credits across CPU, GPU, and storage allocations without admin
approval, as long as the total doesn't exceed the award.
* {*}Policy management{*}: Define a governance rule at the site level (e.g.,
"all allocations must have enforcement enabled"). Show it applying
automatically to all descendant projects. Demonstrate an exemption for a
specific project.
The POC should have a simple UI to demonstrate the interactions and a REST API
backing it.
# *Deliver a findings report*
* Summary of governance patterns studied and their sources
* Mapping of cloud governance concepts to HPC allocation management
* Recommendations for which patterns Custos should adopt and how
* What worked well in the POC, what was tricky, and open questions
* Architecture notes for how these patterns could eventually be integrated
into the Custos allocation management layer
h3. Expected Deliverables
* Research summary of cloud governance platforms (Kion, AWS Control Tower,
Azure Management Groups, GCP Org Policies) and their relevant patterns
* Concept mapping document: cloud governance to HPC allocation governance
* Standalone POC (Go + React/TypeScript) demonstrating hierarchical
organizations, threshold enforcement, multi-source funding, self-service
allocation, and policy inheritance
* Findings report with recommendations for Custos adoption
h3. Required Skills
* Go (backend)
* React/TypeScript (POC UI)
* REST API design
* Database modeling (relational)
* Interest in cloud governance and infrastructure management concepts
h3. Resources
* {*}Custos repository{*}:
[github.com/apache/airavata-custos|https://github.com/apache/airavata-custos]
* {*}Allocation management module{*}: allocations/ directory for context on
how Custos currently handles allocations from ACCESS-CI
* {*}Kion{*}: [https://kion.io/] (primary reference, explore their docs, blog
posts, and product videos)
* {*}Kion higher education use case{*}:
[https://kion.io/industries/higher-education/]
* {*}AWS Control Tower{*}: [https://docs.aws.amazon.com/controltower/]
(organizational governance for AWS accounts)
* {*}Azure Management Groups{*}:
[https://learn.microsoft.com/en-us/azure/governance/management-groups/]
(hierarchical governance for Azure subscriptions)
{*}GCP Organization Policies{*}:
[https://cloud.google.com/resource-manager/docs/organization-policy/overview]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
