Yasith Jayawardana created AIRAVATA-3982:
--------------------------------------------
Summary: Airavata Interactive Session Management via Linkspan
Integration
Key: AIRAVATA-3982
URL: https://issues.apache.org/jira/browse/AIRAVATA-3982
Project: Airavata
Issue Type: Task
Reporter: Yasith Jayawardana
h2. Summary
Extend Apache Airavata to orchestrate interactive development sessions on HPC
clusters by leveraging linkspan as the on-node agent. Airavata gains the
ability to deploy linkspan to compute resources using its existing credential
store and SSO-mapped user credentials, track interactive sessions as
first-class experiments, and use linkspan's FUSE overlay filesystem as a new
data movement provider. CS-Bridge (the VS Code extension) becomes an Airavata
client for this workflow, with a fallback standalone mode for environments
without Airavata.
h2. Problem
Airavata currently manages batch computational workflows (submit a job, stage
data in, execute, stage data out). Interactive development sessions (remote VS
Code, Jupyter, tunneled access) are handled entirely outside Airavata by
CS-Bridge through direct SSH and SLURM. This means:
* Interactive sessions are invisible to Airavata's experiment tracking
* Users must manually configure SSH keys and {{~/.ssh/config}} even when
Airavata already has their credentials
* Data staged through interactive sessions (via linkspan's VFS) is not tracked
in Airavata's replica catalog
* There is no unified view of a user's batch and interactive work
This feature brings interactive sessions under Airavata's umbrella, using its
existing infrastructure for auth, resource management, and experiment tracking.
h2. Description
CS-Bridge will support two operating modes, toggled by a VS Code setting
({{cybershuttle.airavataMode}}):
*Standalone mode (unchanged):*
{noformat}
CS-Bridge → SSH (~/.ssh/config) → SLURM/bash → linkspan (on compute node)
{noformat}
*Airavata mode (new):*
{noformat}
CS-Bridge → Keycloak SSO → Airavata REST API → SSH (CredentialStore) →
SLURM/bash → linkspan
│
Experiment tracking
Data staging via linkspan VFS
{noformat}
In Airavata mode, Airavata authenticates users via Keycloak SSO, resolves
compute resources from its registry and SSH credentials from CredentialStore,
and submits linkspan to HPC nodes as a managed job. Each linkspan session is
tracked as an Airavata experiment with full lifecycle state. Linkspan's VFS
overlay is registered as a data movement interface, enabling Airavata to stage
data through it.
h3. 1. Linkspan as a Managed Application in Airavata
* Register "linkspan" as an Airavata application module with deployment
descriptors per compute resource
* Deployment includes: binary path ({{~/.cybershuttle/bin/linkspan}}), pre-job
commands (download binary if missing), workflow YAML template
* Airavata generates the linkspan workflow YAML server-side (tunnel provider
config, auth tokens, callback URLs), keeping credentials and configuration off
the client
* New Orchestrator process template: {{ENV_SETUP}} (ensure linkspan binary) →
{{JOB_SUBMISSION}} (sbatch/bash with linkspan workflow) → {{JOB_MONITORING}}
(poll linkspan status)
* Airavata uses CredentialStore SSH credentials to connect and submit, so
CS-Bridge never handles SSH keys
h3. 2. Interactive Session Experiment Tracking
Linkspan sessions become first-class Airavata experiments
({{SINGLE_APPLICATION}} type):
||Linkspan session event||Airavata experiment state||
|Job submitted|SCHEDULED → LAUNCHED|
|Linkspan starting up|EXECUTING|
|Tunnel established|EXECUTING (metadata: tunnel_url, ssh_port)|
|User terminates / job ends|COMPLETED|
|Workflow failure|FAILED|
* Experiment metadata stores linkspan outputs: {{tunnel_id}}, {{tunnel_url}},
{{tunnel_token}}, {{ssh_port}}, {{mount_path}}
* Airavata polls linkspan's {{/api/v1/status}} endpoint to drive state
transitions
* Airavata is the single source of truth for session status
* Sessions are recoverable from any CS-Bridge instance via the user's Airavata
experiment history
h3. 3. REST API for Interactive Sessions (airavata-http-server)
* New endpoints for CS-Bridge: compute resource listing with credential-scoped
filtering, experiment CRUD for linkspan sessions, session status polling
* Callback endpoint to receive linkspan status updates and map them to
experiment state transitions
* These endpoints build on the existing REST proxy, evolving into
{{airavata-http-server}}
h3. 4. Linkspan VFS as a Data Movement Provider
* New {{LINKSPAN_VFS}} data movement protocol type in Airavata's model
* When a linkspan session is active, its FUSE overlay mount path is registered
as a {{DataReplicaLocation}} on the compute resource
* Airavata's {{DATA_STAGING}} tasks can read/write directly through the overlay
path instead of spawning separate SCP/SFTP transfers
* Avoids redundant data transfers: workspace files synced via mutagen are
already available; outputs are immediately visible locally
* Airavata's {{DataProductModel}} and replica catalog track data at the overlay
location
* Fallback: standard SCP/SFTP when no active linkspan session exists
h3. 5. Linkspan Changes (Minimal)
Linkspan remains a generic on-node agent. The only additions support Airavata's
need to receive status callbacks:
* New workflow action: {{airavata.report_status}} — POSTs session metadata to
an Airavata callback URL
* New workflow action: {{airavata.register_vfs}} — reports the active overlay
mount path to Airavata
* Workflow YAML gains an optional {{airavata_callback_url}} variable, injected
by Airavata server-side
* No changes to existing tunnel, VFS, or SSH subsystems
h3. 6. CS-Bridge Changes (Client-Side)
* New {{AiravataManager}} class: Keycloak OAuth2 PKCE auth, REST API calls for
resource discovery, experiment CRUD, status polling
* Mode-aware {{SessionManager}}: standalone path unchanged, Airavata mode
delegates to {{AiravataManager}}
* {{Runtime}} model gains optional fields: {{experimentId}}, {{processId}},
{{airavataResourceId}}
* Host selector populated from Airavata compute resources (replaces
{{~/.ssh/config}} parsing)
* Resource form limits from Airavata's {{BatchQueue}} metadata
* New settings: {{cybershuttle.airavataMode}},
{{cybershuttle.airavataServerUrl}}, {{cybershuttle.keycloakUrl}},
{{cybershuttle.keycloakRealm}}
h2. Expected Deliverables
* Linkspan registered as an Airavata application module with per-resource
deployment descriptors
* Orchestrator process template for linkspan deployment (ENV_SETUP →
JOB_SUBMISSION → JOB_MONITORING)
* Server-side linkspan workflow YAML generation in Airavata
* Interactive session experiment tracking with linkspan state mapping
* airavata-http-server REST endpoints for interactive session management
* {{LINKSPAN_VFS}} data movement protocol with overlay mount integration
* Two new linkspan workflow actions ({{airavata.report_status}},
{{airavata.register_vfs}})
* CS-Bridge Airavata mode with Keycloak SSO, resource discovery, and
experiment-backed sessions
* End-to-end demo: CS-Bridge in Airavata mode → Keycloak login → resource
selection → Airavata submits linkspan → tunnel established → session tracked as
experiment
* Integration tests and configuration documentation
h2. Delivery Phases
*Phase 1: Auth + Resource Discovery*
* airavata-http-server endpoints for compute resource listing with
credential-scoped filtering
* Keycloak SSO integration in CS-Bridge
* {{AiravataManager}} scaffolding and settings
*Phase 2: Job Submission via Airavata*
* Linkspan registered as Airavata application
* Orchestrator process template for linkspan deployment
* Server-side workflow YAML generation
* {{airavata.report_status}} workflow action in linkspan
*Phase 3: Experiment Lifecycle Tracking*
* Linkspan sessions tracked as Airavata experiments
* State mapping via linkspan status polling
* Session recovery from experiment history
* CS-Bridge UI shows experiment state
*Phase 4: VFS Data Movement*
* {{LINKSPAN_VFS}} data movement protocol
* {{airavata.register_vfs}} workflow action in linkspan
* Data staging tasks through overlay mount
* Replica catalog integration
h2. Required Skills
* Java (Airavata backend)
* TypeScript (CS-Bridge VS Code extension)
* Go (linkspan agent)
* REST API design and integration
* SSH and HPC job submission concepts (SLURM), or willingness to learn
* OAuth2/OIDC/Keycloak authentication flows, or willingness to learn
h2. Resources
* Apache Airavata repository: [https://github.com/apache/airavata]
* CyberShuttle linkspan: [https://github.com/cyber-shuttle/linkspan]
* CyberShuttle CS-Bridge: [https://github.com/cyber-shuttle/CS-Bridge]
* Airavata documentation: [https://airavata.apache.org/]
* Keycloak documentation: [https://www.keycloak.org/documentation]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
