This is an automated email from the ASF dual-hosted git repository.
kunwp1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new 134d7d2785 feat: Add Python Virtual Environment Support: Installing
User Defined Packages (#4902)
134d7d2785 is described below
commit 134d7d2785d17ad55b99906d769c3465bf88b6c3
Author: Sarah Asad <[email protected]>
AuthorDate: Tue May 12 09:10:55 2026 -0700
feat: Add Python Virtual Environment Support: Installing User Defined
Packages (#4902)
<!--
Thanks for sending a pull request (PR)! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
[Contributing to
Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md)
2. Ensure you have added or run the appropriate tests for your PR
3. If the PR is work in progress, mark it a draft on GitHub.
4. Please write your PR title to summarize what this PR proposes, we
are following Conventional Commits style for PR titles as well.
5. Be sure to keep the PR description updated to reflect all changes.
-->
### What changes were proposed in this PR?
<!--
Please clarify what changes you are proposing. The purpose of this
section
is to outline the changes. Here are some tips for you:
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
3. If it is a refactoring, clarify what has been changed.
3. It would be helpful to include a before-and-after comparison using
screenshots or GIFs.
4. Please consider writing useful notes for better and faster reviews.
-->
This PR is an extension of
PR [#4484](https://github.com/apache/texera/pull/4484). Previously, we
introduced support for creating Python Virtual Environments (PVEs) with
system-level dependencies preinstalled. This PR builds on that
foundation by enabling users to install custom Python packages within a
PVE. In some cases, installing a user package may overwrite system-level
packages. For this reason, System package versions are locked to prevent
these conflicts.
Some duplicate lines of code (previously introduced in
PR [#4484](https://github.com/apache/texera/pull/4484)) were removed
from PveManager to improve clarity and reduce redundancy.
### Any related issues, documentation, discussions?
<!--
Please use this section to link other resources if not mentioned
already.
1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves
#1234`
or `Closes #1234`. If it is only related, simply mention the issue
number.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
This change is part of ongoing efforts to support environment isolation
and reproducibility within Texera. Related issue
includes [#4296](https://github.com/apache/texera/issues/4296). This PR
closes sub-issue [#4465](https://github.com/apache/texera/issues/4465).
A GitHub CI sync check between `requirements.txt` and
`system-requirements-lock.txt` will be added in a follow-up PR. Tracking
issue: #5034.
### How was this PR tested?
<!--
If tests were added, say they were added here. Or simply mention that if
the PR
is tested with existing test cases. Make sure to include/update test
cases that
check the changes thoroughly including negative and positive cases if
possible.
If it was tested in a way different from regular unit tests, please
clarify how
you tested step by step, ideally copy and paste-able, so that other
reviewers can
test and check, and descendants can verify in the future. If tests were
not added,
please describe why they were not added and/or why it was difficult to
add.
-->
Tested Manually and PveResourceSpec test file updated.
To test:
1. On CU click "+" Python Environments.
2. Input environment name.
3. Input package name and version.
4. Click "OK" and wait for pip logs.
To test system dependency conflit management, install the following
packages:
- Scanpy 1.11.1 -> Successful
- Tensorflow 2.16.1 -> Unsuccessful because of system level dependency
conflict
### Was this PR authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
PR,
please include the phrase: 'Generated-by: ' followed by the name of the
tool
and its version. If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for
details.
-->
Co-authored using: ChatGPT (OpenAI)
---
.../pythonvirtualenvironment/PveManager.scala | 217 ++++++++++++++++-----
.../pythonvirtualenvironment/PveResource.scala | 21 +-
.../PveWebsocketResource.scala | 30 ++-
.../pythonvirtualenvironment/PveResourceSpec.scala | 33 +++-
amber/system-requirements-lock.txt | 102 ++++++++++
.../computing-unit-selection.component.html | 91 ++++++++-
.../computing-unit-selection.component.ts | 186 +++++++++++++-----
.../virtual-environment.service.ts | 11 +-
frontend/src/styles.scss | 30 ++-
9 files changed, 613 insertions(+), 108 deletions(-)
diff --git
a/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveManager.scala
b/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveManager.scala
index 0399e386ba..27a3b7be7c 100644
---
a/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveManager.scala
+++
b/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveManager.scala
@@ -32,7 +32,8 @@ import org.apache.texera.amber.config.PythonUtils
* for each Computing Unit
*
* It supports:
- * - Creating and initializing isolated Python environments
+ * - Creating and initializing isolated Python environments (with system
packages)
+ * - Installing user defined packages
* - Streaming pip output logs back to the caller
*
* Each PVE is stored under:
@@ -41,6 +42,11 @@ import org.apache.texera.amber.config.PythonUtils
object PveManager {
+ case class PvePackageResponse(
+ pveName: String,
+ userPackages: Seq[String]
+ )
+
private val VenvRoot: Path = Paths.get("/tmp/texera-pve/venvs")
private def cuidDir(cuid: Int, pveName: String): Path = {
@@ -61,9 +67,63 @@ object PveManager {
"PIP_NO_INPUT" -> "1"
)
- def getSystemPackages(): Seq[String] = {
- val python = PythonUtils.getPythonExecutable
- Process(Seq(python, "-m", "pip",
"freeze")).!!.split("\n").map(_.trim).filter(_.nonEmpty).toSeq
+ private def readPackageFile(path: Path): Seq[String] = {
+ if (Files.exists(path)) {
+ Files
+ .readAllLines(path)
+ .asScala
+ .map(_.trim)
+ .filter(_.nonEmpty)
+ .toSeq
+ } else {
+ Seq()
+ }
+ }
+
+ private def getSystemPath(isLocal: Boolean): Path = {
+ Paths.get(
+ if (isLocal) "amber/system-requirements-lock.txt"
+ else "/tmp/system-requirements-lock.txt"
+ )
+ }
+
+ def getSystemPackages(isLocal: Boolean): Seq[String] = {
+ if (!Files.exists(getSystemPath(isLocal))) {
+ Seq()
+ } else {
+ Files
+ .readAllLines(getSystemPath(isLocal))
+ .asScala
+ .map(_.trim)
+ .filter(line => line.nonEmpty && !line.startsWith("#"))
+ .toSeq
+ }
+ }
+
+ private def runPipInstall(
+ python: String,
+ args: Seq[String],
+ queue: BlockingQueue[String]
+ ): Int = {
+ Process(
+ Seq(
+ python,
+ "-u",
+ "-m",
+ "pip",
+ "install",
+ "--progress-bar",
+ "off",
+ "--no-input"
+ ) ++ args,
+ None,
+ pipEnv.toSeq: _*
+ ).!(
+ ProcessLogger(
+ out => queue.put(s"[pip] $out"),
+ err => queue.put(s"[pip][ERR] $err")
+ )
+ )
}
/**
@@ -85,23 +145,18 @@ object PveManager {
queue.put(s"[PVE] Creating new PVE for cuid: $cuid with name: $pveName")
// NOTE: These paths are derived from computing-unit-master.dockerfile.
- // If requirements.txt or operator-requirements.txt locations change,
update these paths.
+ // If requirements.txt location changes, update these paths.
val requirementsPath =
if (isLocal) Paths.get("amber", "requirements.txt")
else Paths.get("/tmp", "requirements.txt")
- val operatorRequirementsPath =
- if (isLocal) Paths.get("amber", "operator-requirements.txt")
- else Paths.get("/tmp", "operator-requirements.txt")
-
- if (!Files.exists(requirementsPath) ||
!Files.exists(operatorRequirementsPath)) {
+ if (!Files.exists(requirementsPath)) {
queue.put(s"[PVE][ERR] System requirements not found")
return
}
val venvDirPath = pveDir(cuid, pveName).toAbsolutePath
val python = pythonBinPath(cuid, pveName).toAbsolutePath.toString
- val envVars = pipEnv
val createVenvPython = PythonUtils.getPythonExecutable
@@ -121,43 +176,17 @@ object PveManager {
return
}
- if (!Files.exists(requirementsPath)) {
- queue.put(s"[PVE][ERR] requirements.txt not found at
${requirementsPath.toAbsolutePath}")
- return
- }
-
- if (!Files.exists(operatorRequirementsPath)) {
- queue.put(
- s"[PVE][ERR] operator-requirements.txt not found at
${operatorRequirementsPath.toAbsolutePath}"
- )
- return
- }
-
queue.put(
- s"[PVE] Installing requirements from ${requirementsPath.toAbsolutePath}
and ${operatorRequirementsPath.toAbsolutePath}"
+ s"[PVE] Installing requirements from ${requirementsPath.toAbsolutePath}"
)
- val installReqCode = Process(
+ val installReqCode = runPipInstall(
+ python,
Seq(
- python,
- "-u",
- "-m",
- "pip",
- "install",
- "--progress-bar",
- "off",
"-r",
- requirementsPath.toString,
- "-r",
- operatorRequirementsPath.toString
+ requirementsPath.toString
),
- None,
- envVars.toSeq: _*
- ).!(
- ProcessLogger(
- out => queue.put(s"[pip] $out"),
- err => queue.put(s"[pip][ERR] $err")
- )
+ queue
)
queue.put(s"[PVE] requirements install finished with exit code
$installReqCode")
@@ -170,7 +199,8 @@ object PveManager {
queue.put(s"[PVE] Created new environment for cuid = $cuid")
}
- def getEnvironments(cuid: Int): List[String] = {
+ // returns list of PVE names and corresponding user packages for a given CU
+ def getEnvironments(cuid: Int): List[PvePackageResponse] = {
val cuPath = VenvRoot.resolve(cuid.toString)
@@ -185,7 +215,17 @@ object PveManager {
.iterator()
.asScala
.filter(path => Files.isDirectory(path))
- .map(path => path.getFileName.toString)
+ .map { path =>
+ val pveName = path.getFileName.toString
+ val metadataPath = path.resolve("user-packages.txt")
+
+ val userPackages = readPackageFile(metadataPath)
+
+ PvePackageResponse(
+ pveName = pveName,
+ userPackages = userPackages
+ )
+ }
.toList
} finally {
stream.close()
@@ -212,4 +252,93 @@ object PveManager {
stream.close()
}
}
+
+ /**
+ * Installs user requested Python packages into the PVE.
+ *
+ * 1. Executes pip install for each package
+ * 2. Prevents conflicts with system dependencies.
+ * 3. Updates user metadata file
+ * 4. Streams logs back via queue
+ */
+ def installUserPackages(
+ packages: List[String],
+ cuid: Int,
+ queue: BlockingQueue[String],
+ pveName: String,
+ isLocal: Boolean
+ ): Unit = {
+
+ val python = pythonBinPath(cuid, pveName).toAbsolutePath.toString
+
+ if (!Files.exists(Paths.get(python))) {
+ queue.put(s"[PVE][ERR] Python executable not found for PVE: $python")
+ return
+ }
+
+ val metadataPath = cuidDir(cuid, pveName).resolve("user-packages.txt")
+
+ var installedPackages = readPackageFile(metadataPath).toSet
+
+ val systemPackages =
+ if (Files.exists(getSystemPath(isLocal))) {
+ Files
+ .readAllLines(getSystemPath(isLocal))
+ .asScala
+ .map(_.trim)
+ .filter(line => line.nonEmpty && !line.startsWith("#"))
+ .map(line => line.split("==")(0).trim.toLowerCase)
+ .toSet
+ } else {
+ Set[String]()
+ }
+
+ packages.foreach { pkg =>
+ val trimmedPkg = pkg.trim
+
+ if (trimmedPkg.nonEmpty) {
+
+ val userPackageName = trimmedPkg.split("==")(0).trim.toLowerCase
+
+ if (systemPackages.contains(userPackageName)) {
+ queue.put(
+ s"[PVE][ERR] $trimmedPkg is a system package and cannot be
installed or modified by the user."
+ )
+ return
+ }
+
+ queue.put(s"[PVE] Installing package: $trimmedPkg")
+
+ val code = runPipInstall(
+ python,
+ Seq(
+ "--constraint", // check against system-requirements-lock
+ getSystemPath(isLocal).toString,
+ trimmedPkg
+ ),
+ queue
+ )
+
+ queue.put(s"[pip] install($trimmedPkg) finished with exit code $code")
+
+ if (code != 0) {
+ queue.put(s"[PVE][ERR] Failed to install package: $trimmedPkg")
+ return
+ }
+
+ installedPackages = installedPackages + trimmedPkg
+
+ Files.write(
+ metadataPath,
+ installedPackages.toSeq.sorted.asJava
+ )
+ }
+ }
+
+ queue.put("[PVE] Final user package list:")
+
+ installedPackages.toSeq.sorted.foreach { pkg =>
+ queue.put(s"[user-package] $pkg")
+ }
+ }
}
diff --git
a/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResource.scala
b/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResource.scala
index 1040fd64ea..8a6f487529 100644
---
a/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResource.scala
+++
b/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResource.scala
@@ -28,24 +28,32 @@ import java.util
@Consumes(Array(MediaType.APPLICATION_JSON))
class PveResource {
// --------------------------------------------------
- // Get installed packages
+ // Get system packages
// --------------------------------------------------
@GET
@Path("/system")
@Produces(Array(MediaType.APPLICATION_JSON))
def getSystemPackages: util.Map[String, util.List[String]] = {
try {
- val systemPkgs = PveManager.getSystemPackages().toList.asJava
+
+ // TODO: Support Kubernetes environment handling
+ val isLocal = true
+
+ val systemPkgs =
+ PveManager.getSystemPackages(isLocal).toList.asJava
+
Map("system" -> systemPkgs).asJava
} catch {
case e: Exception =>
e.printStackTrace()
- throw new InternalServerErrorException("Failed to get system
packages.")
+ throw new InternalServerErrorException(
+ "Failed to get system packages."
+ )
}
}
// --------------------------------------------------
- // Fetch PVEs
+ // Fetch PVEs and Installed User Packages
// --------------------------------------------------
@GET
@Path("/pves")
@@ -54,9 +62,10 @@ class PveResource {
try {
PveManager
.getEnvironments(cuid)
- .map { pveName =>
+ .map { pve =>
Map(
- "pveName" -> pveName.asInstanceOf[Object]
+ "pveName" -> pve.pveName.asInstanceOf[Object],
+ "userPackages" -> pve.userPackages.asJava.asInstanceOf[Object]
).asJava
}
.asJava
diff --git
a/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveWebsocketResource.scala
b/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveWebsocketResource.scala
index b93d1bfde0..e21f91fada 100644
---
a/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveWebsocketResource.scala
+++
b/amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveWebsocketResource.scala
@@ -26,9 +26,9 @@ import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
/**
- * WebSocket endpoint for PVE creation that streams pip installation logs
- * to the frontend in real time. The environment setup runs asynchronously,
- * and output is pushed to the client until completion.
+ * WebSocket endpoint for PVE creation and user package installation that
streams
+ * pip installation logs to the frontend in real time. The environment
setup runs
+ * asynchronously, and output is pushed to the client until completion.
*/
@ServerEndpoint("/wsapi/pve")
@@ -42,12 +42,33 @@ class PveWebsocketResource {
val cuid = params.get("cuid").get(0).toInt
val pveName = params.get("pveName").get(0)
val isLocal = params.get("isLocal").get(0).toBoolean
+ val action = params.getOrDefault("action",
java.util.List.of("create")).get(0)
val queue = new LinkedBlockingQueue[String]()
Future {
try {
- PveManager.createNewPve(cuid, queue, pveName, isLocal)
+ action match {
+ case "create" =>
+ PveManager.createNewPve(cuid, queue, pveName, isLocal)
+
+ case "install" =>
+ val packages =
+ params
+ .getOrDefault("packages", java.util.List.of("[]"))
+ .get(0)
+ .stripPrefix("[")
+ .stripSuffix("]")
+ .split(",")
+ .toList
+ .map(_.replace("\"", "").trim)
+ .filter(_.nonEmpty)
+
+ PveManager.installUserPackages(packages, cuid, queue, pveName,
isLocal)
+
+ case _ =>
+ queue.put(s"[ERR] Unknown action: $action")
+ }
} catch {
case e: Exception =>
queue.put(s"[ERR] ${e.getMessage}")
@@ -61,7 +82,6 @@ class PveWebsocketResource {
while (!done && session.isOpen) {
val line = queue.take()
-
session.getBasicRemote.sendText(line)
if (line == "__DONE__") {
diff --git
a/amber/src/test/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResourceSpec.scala
b/amber/src/test/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResourceSpec.scala
index a093cf1ad2..10e952c8bd 100644
---
a/amber/src/test/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResourceSpec.scala
+++
b/amber/src/test/scala/org/apache/texera/web/resource/pythonvirtualenvironment/PveResourceSpec.scala
@@ -64,7 +64,38 @@ class PveResourceSpec extends AnyFlatSpec with Matchers with
BeforeAndAfterEach
Files.exists(pythonPath) shouldBe true
Files.exists(pipPath) shouldBe true
- PveManager.getEnvironments(testCuid) should contain(testPveName)
+ PveManager.getEnvironments(testCuid).map(_.pveName) should
contain(testPveName)
+ }
+
+ "PveManager" should "install a user package and list it for the PVE" in {
+ PveManager.createNewPve(testCuid, queue, testPveName, isLocal = true)
+
+ val packageName = "colorama"
+ val packageVersion = "0.4.6"
+ val packageSpec = s"$packageName==$packageVersion"
+
+ queue.clear()
+
+ PveManager.installUserPackages(
+ List(packageSpec),
+ testCuid,
+ queue,
+ testPveName,
+ isLocal = true
+ )
+
+ val logs = queueText()
+
+ logs should not include "[PVE][ERR]"
+ logs should include(s"[PVE] Installing package: $packageSpec")
+ logs should include(s"[user-package] $packageSpec")
+
+ val pve = PveManager
+ .getEnvironments(testCuid)
+ .find(_.pveName == testPveName)
+
+ pve should not be empty
+ pve.get.userPackages should contain(packageSpec)
}
"PveManager" should "delete all PVEs for a computing unit" in {
diff --git a/amber/system-requirements-lock.txt
b/amber/system-requirements-lock.txt
new file mode 100644
index 0000000000..67ea725c88
--- /dev/null
+++ b/amber/system-requirements-lock.txt
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is manually generated to track system packages used in PVEs.
+# NOTE: This file must be updated whenever requirements.txt or
+# operator-requirements.txt changes.
+
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.5
+aioitertools==0.13.0
+aiobotocore==2.25.1
+aiosignal==1.4.0
+annotated-types==0.7.0
+appdirs==1.4.4
+asn1crypto==1.5.1
+attrs==26.1.0
+betterproto==2.0.0b7
+bidict==0.22.0
+boto3==1.40.53
+botocore==1.40.53
+cached_property==1.5.2
+cachetools==6.2.6
+certifi==2026.4.22
+charset_normalizer==3.4.7
+click==8.3.3
+Deprecated==1.2.14
+frozenlist==1.8.0
+fs==2.4.16
+fsspec==2025.9.0
+grpclib==0.4.9
+h2==4.3.0
+hpack==4.1.0
+hyperframe==6.1.0
+idna==3.14
+iniconfig==1.1.1
+jmespath==1.1.0
+loguru==0.7.0
+markdown-it-py==4.2.0
+mdurl==0.1.2
+mmh3==5.2.1
+multidict==6.7.1
+numpy==2.1.0
+overrides==7.4.0
+packaging==26.2
+pampy==0.3.0
+pandas==2.2.3
+pg8000==1.31.5
+pluggy==1.6.0
+praw==7.6.1
+prawcore==2.4.0
+propcache==0.5.2
+protobuf==7.34.1
+psutil==5.9.0
+pyarrow==21.0.0
+pydantic==2.13.4
+pydantic-core==2.46.4
+pygments==2.20.0
+pyiceberg==0.11.1
+pympler==1.1
+pyparsing==3.3.2
+pyroaring==1.1.0
+pytest==7.4.0
+pytest-reraise==2.1.2
+pytest-timeout==2.2.0
+python-dateutil==2.8.2
+pytz==2026.2
+readerwriterlock==1.0.9
+requests==2.34.0
+rich==14.3.4
+ruff==0.14.7
+s3fs==2025.9.0
+s3transfer==0.14.0
+scramp==1.4.8
+setuptools==80.10.2
+six==1.17.0
+SQLAlchemy==2.0.37
+strictyaml==1.7.3
+tenacity==8.5.0
+typing-inspection==0.4.2
+typing_extensions==4.14.1
+tzdata==2026.2
+tzlocal==2.1
+update-checker==0.18.0
+urllib3==2.7.0
+websocket-client==1.9.0
+wrapt==1.17.3
+yarl==1.23.0
+zstandard==0.25.0
\ No newline at end of file
diff --git
a/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.html
b/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.html
index b742c71581..6f16073073 100644
---
a/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.html
+++
b/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.html
@@ -480,7 +480,7 @@
<div
*ngFor="let pkg of systemPackages"
class="package-row system-row">
- <div class="package-inputs">
+ <div class="system-package-inputs">
<input
nz-input
class="system-input"
@@ -528,6 +528,95 @@
[(ngModel)]="pve.name"
[disabled]="pve.isLocked || pve.isInstalling" />
</div>
+
+ <!-- USER PACKAGES -->
+ <div
+ *ngFor="let pkg of pve.userPackages; let i = index"
+ class="package-row">
+ <div class="user-package-inputs">
+ <div class="field">
+ <input
+ nz-input
+ [ngModel]="pkg.name"
+ [disabled]="true" />
+ </div>
+
+ <div class="field">
+ <input
+ nz-input
+ [ngModel]="pkg.versionOp"
+ [disabled]="true" />
+ </div>
+
+ <div class="field">
+ <input
+ nz-input
+ [ngModel]="pkg.version"
+ [disabled]="true" />
+ </div>
+ </div>
+ </div>
+
+ <!-- NEW PACKAGES -->
+ <div class="new-packages-section">
+ <div
+ class="user-package-header-row"
+ *ngIf="pve.newPackages.length > 0">
+ <div class="package-column-label">Package</div>
+ <label style="visibility: hidden">Op</label>
+ <div class="package-column-label">Version</div>
+ </div>
+
+ <div
+ *ngFor="let pkg of pve.newPackages; let i = index"
+ class="package-row">
+ <div class="user-package-inputs">
+ <div class="field">
+ <input
+ nz-input
+ placeholder="Package Name"
+ [(ngModel)]="pve.newPackages[i].name" />
+ </div>
+
+ <div class="field operator operator-select">
+ <nz-select
+ nzPlaceHolder="Select"
+ nzCentered
+ [(ngModel)]="pve.newPackages[i].versionOp">
+ <nz-option
+ nzValue="=="
+ nzLabel="=="></nz-option>
+ <nz-option
+ nzValue=">="
+ nzLabel=">="></nz-option>
+ <nz-option
+ nzValue="<="
+ nzLabel="<="></nz-option>
+ </nz-select>
+ </div>
+
+ <div class="field">
+ <input
+ nz-input
+ placeholder="Package Version"
+ [(ngModel)]="pve.newPackages[i].version" />
+ </div>
+ </div>
+ </div>
+ </div>
+
+ <div class="add-btn">
+ <button
+ nz-button
+ nzType="primary"
+ nzShape="circle"
+ (click)="addPackage(envIndex)">
+ <i
+ nz-icon
+ nzType="plus"></i>
+ </button>
+ </div>
+
<!-- Per-environment OK/Install button -->
<div class="env-footer">
<button
diff --git
a/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.ts
b/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.ts
index a8723e04b0..1b3dfb2322 100644
---
a/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.ts
+++
b/frontend/src/app/workspace/component/power-button/computing-unit-selection.component.ts
@@ -77,8 +77,16 @@ import { NzSliderComponent } from "ng-zorro-antd/slider";
import { NzAlertComponent } from "ng-zorro-antd/alert";
import { NzCollapseComponent, NzCollapsePanelComponent } from
"ng-zorro-antd/collapse";
+type PveUserPackageRow = {
+ name: string;
+ versionOp?: "==" | ">=" | "<=";
+ version?: string;
+};
+
type PveDraft = {
name: string;
+ userPackages: PveUserPackageRow[];
+ newPackages: PveUserPackageRow[];
pipOutput: string;
prettyPipOutput: string;
expanded: boolean;
@@ -722,9 +730,16 @@ export class ComputingUnitSelectionComponent implements
OnInit {
return index;
}
+ addPackage(index: number): void {
+ const env = this.pves[index];
+ env.newPackages.push({ name: "", version: "", versionOp: undefined });
+ }
+
addEnvironment(): void {
this.pves.push({
name: "",
+ userPackages: [],
+ newPackages: [],
pipOutput: "",
prettyPipOutput: "",
expanded: true,
@@ -750,6 +765,7 @@ export class ComputingUnitSelectionComponent implements
OnInit {
getPVEs(): void {
const cuId = this.selectedComputingUnit!.computingUnit.cuid;
+ const isLocal = this.selectedComputingUnit?.computingUnit.type === "local";
this.workflowPveService
.fetchPVEs(cuId)
@@ -758,6 +774,8 @@ export class ComputingUnitSelectionComponent implements
OnInit {
next: (resp: PvePackageResponse[]) => {
this.pves = resp.map(pve => ({
name: pve.pveName,
+ userPackages: this.parsePackageRows(pve.userPackages),
+ newPackages: [],
expanded: false,
isInstalling: false,
pipOutput: "",
@@ -766,7 +784,7 @@ export class ComputingUnitSelectionComponent implements
OnInit {
}));
this.workflowPveService
- .getSystemPackages()
+ .getSystemPackages(isLocal)
.pipe(untilDestroyed(this))
.subscribe({
next: installedResp => {
@@ -829,43 +847,34 @@ export class ComputingUnitSelectionComponent implements
OnInit {
.replace(/^(\[(?:PVE|pip|pve)\]\[ERR\].*)$/gm, '<span class="pip-exit
err"><strong>$1</strong></span>')
+ .replace(/^(\[PVE\] Skipped.*)$/gm, '<span class="pip-exit
err"><strong>$1</strong></span>')
+
.replace(/\n/g, "<br/>");
}
- createVirtualEnvironment(index: number): void {
+ private runPveWebSocket(
+ index: number,
+ action: "create" | "install",
+ initialMessage: string,
+ packages: string[] = [],
+ onDone?: () => void
+ ): void {
const cuId = this.selectedComputingUnit!.computingUnit.cuid;
-
const env = this.pves[index];
-
const trimmedName = env.name.trim();
-
- if (!/^[a-zA-Z0-9]+$/.test(trimmedName)) {
- this.notificationService.error("Environment name must contain only
letters and numbers.");
- return;
- }
-
- const duplicateExists = this.pves.some((pve, i) => i !== index &&
(pve.name ?? "").trim() === trimmedName);
-
- if (duplicateExists) {
- this.notificationService.error("An environment with this name already
exists.");
- return;
- }
-
- const packageArray: string[] = [];
+ const isLocal = this.selectedComputingUnit?.computingUnit.type === "local";
env.socket?.close();
- const isLocal = this.selectedComputingUnit?.computingUnit.type === "local";
+ const websocketUrl = this.workflowPveService.getPveWebSocketUrl(cuId,
trimmedName, isLocal, action, packages);
- const websocketUrl = this.workflowPveService.createPveWebSocketUrl(cuId,
trimmedName, isLocal, packageArray);
- console.log("PVE websocketUrl", websocketUrl);
const socket = new WebSocket(websocketUrl);
this.pves[index] = {
...env,
name: trimmedName,
socket,
- pipOutput: "Starting ...\n",
+ pipOutput: initialMessage,
isInstalling: true,
isLocked: true,
};
@@ -874,8 +883,6 @@ export class ComputingUnitSelectionComponent implements
OnInit {
this.scrollToBottomOfPipModal(index);
socket.onmessage = event => {
- console.log("PVE WS received:", event.data);
-
this.ngZone.run(() => {
const currentEnv = this.pves[index];
@@ -888,19 +895,7 @@ export class ComputingUnitSelectionComponent implements
OnInit {
};
socket.close();
- this.workflowPveService
- .getSystemPackages()
- .pipe(untilDestroyed(this))
- .subscribe({
- next: resp => {
- this.systemPackages = resp.system.map(pkg => {
- const [name, version] = pkg.split("==");
- return { name: name.trim(), version: (version ?? "").trim()
};
- });
- this.cdr.detectChanges();
- },
- error: (e: unknown) => console.error("Failed to refresh
packages", e),
- });
+ onDone?.();
this.cdr.detectChanges();
return;
@@ -917,9 +912,7 @@ export class ComputingUnitSelectionComponent implements
OnInit {
});
};
- socket.onerror = err => {
- console.log("PVE WS error", err);
-
+ socket.onerror = () => {
this.ngZone.run(() => {
const currentEnv = this.pves[index];
@@ -936,13 +929,114 @@ export class ComputingUnitSelectionComponent implements
OnInit {
this.cdr.detectChanges();
});
};
+ }
- socket.onclose = event => {
- console.log("PVE WS closed", {
- code: event.code,
- reason: event.reason,
- wasClean: event.wasClean,
+ private refreshUserPackages(index: number): void {
+ const env = this.pves[index];
+
+ this.workflowPveService
+ .getUserPackages(this.selectedComputingUnit!.computingUnit.cuid,
env.name)
+ .pipe(untilDestroyed(this))
+ .subscribe({
+ next: pkgs => {
+ env.userPackages = env.userPackages = this.parsePackageRows(pkgs);
+ this.cdr.detectChanges();
+ },
+ error: (e: unknown) => console.error("Failed to refresh user
packages", e),
});
- };
+ }
+
+ createVirtualEnvironment(index: number): void {
+ const env = this.pves[index];
+ const trimmedName = env.name.trim();
+
+ if (!/^[a-zA-Z0-9]+$/.test(trimmedName)) {
+ this.notificationService.error("Environment name must contain only
letters and numbers.");
+ return;
+ }
+
+ if (env.isLocked) {
+ this.installUserPackages(index);
+ return;
+ }
+
+ const duplicateExists = this.pves.some((pve, i) => i !== index &&
(pve.name ?? "").trim() === trimmedName);
+
+ if (duplicateExists) {
+ this.notificationService.error("An environment with this name already
exists.");
+ return;
+ }
+
+ this.runPveWebSocket(index, "create", "Creating virtual environment...\n",
[], () => {
+ this.installUserPackages(index);
+ });
+ }
+
+ private installUserPackages(index: number): void {
+ const env = this.pves[index];
+
+ const missingVersionPackage = env.newPackages?.find(
+ pkg => pkg.name?.trim() && (!pkg.versionOp?.trim() ||
!pkg.version?.trim())
+ );
+
+ if (missingVersionPackage) {
+ this.notificationService.error("Please specify an operator and version
for each package.");
+ return;
+ }
+
+ const systemPackageNames = new Set(this.systemPackages.map(pkg =>
pkg.name.trim().toLowerCase()));
+
+ const userPackageNames = new Set(env.userPackages.map(pkg =>
pkg.name.trim().toLowerCase()));
+
+ const skippedMessages: string[] = [];
+
+ const packageArray =
+ env.newPackages
+ ?.filter(pkg => pkg.name?.trim())
+ .filter(pkg => {
+ const packageName = pkg.name.trim().toLowerCase();
+
+ if (systemPackageNames.has(packageName)) {
+ this.notificationService.error(`Skipped ${pkg.name}: already
installed as a system package.`);
+ return false;
+ }
+
+ if (userPackageNames.has(packageName)) {
+ this.notificationService.error(`Skipped ${pkg.name}: already
installed in this environment.`);
+ return false;
+ }
+
+ return true;
+ })
+ .map(pkg => `${pkg.name.trim()}${pkg.versionOp}${(pkg.version ??
"").trim()}`) ?? [];
+
+ if (skippedMessages.length > 0) {
+ this.pves[index].pipOutput = `${this.pves[index].pipOutput ?? ""}` +
skippedMessages.join("\n") + "\n";
+
+ this.updatePrettyPipOutput(index);
+ this.scrollToBottomOfPipModal(index);
+ }
+
+ if (packageArray.length === 0) {
+ this.pves[index].newPackages = [];
+ this.refreshUserPackages(index);
+ return;
+ }
+
+ this.runPveWebSocket(index, "install", "Installing user packages...\n",
packageArray, () => {
+ this.pves[index].newPackages = [];
+ this.refreshUserPackages(index);
+ });
+ }
+
+ private parsePackageRows(packages: string[]): PveUserPackageRow[] {
+ return packages.map(pkgStr => {
+ const [name, version] = pkgStr.split("==");
+ return {
+ name: name.trim(),
+ versionOp: "==" as const,
+ version: (version ?? "").trim(),
+ };
+ });
}
}
diff --git
a/frontend/src/app/workspace/service/virtual-environment/virtual-environment.service.ts
b/frontend/src/app/workspace/service/virtual-environment/virtual-environment.service.ts
index a581305759..9c7c123df1 100644
---
a/frontend/src/app/workspace/service/virtual-environment/virtual-environment.service.ts
+++
b/frontend/src/app/workspace/service/virtual-environment/virtual-environment.service.ts
@@ -18,12 +18,12 @@
import { Injectable } from "@angular/core";
import { Observable } from "rxjs";
+import { map } from "rxjs/operators";
import { HttpClient, HttpParams } from "@angular/common/http";
import { AuthService } from "../../../common/service/user/auth.service";
export interface PackageResponse {
system: string[];
- user: string[];
}
export interface PvePackageResponse {
@@ -49,7 +49,7 @@ export class WorkflowPveService {
return params;
}
- getSystemPackages(): Observable<PackageResponse> {
+ getSystemPackages(isLocal: boolean): Observable<PackageResponse> {
const params = this.buildBaseParams();
return this.http.get<PackageResponse>("/pve/system", { params });
}
@@ -59,11 +59,15 @@ export class WorkflowPveService {
return this.http.get<PvePackageResponse[]>("/pve/pves", { params });
}
+ getUserPackages(cuid: number, pveName: string): Observable<string[]> {
+ return this.fetchPVEs(cuid).pipe(map(pves => pves.find(pve => pve.pveName
=== pveName)?.userPackages ?? []));
+ }
+
deleteEnvironments(cuid: number) {
return this.http.delete(`/pve/pves/${cuid}`);
}
- createPveWebSocketUrl(cuid: number, pveName: string, isLocal: boolean,
packages: string[] = []): string {
+ getPveWebSocketUrl(cuid: number, pveName: string, isLocal: boolean, action:
string, packages: string[] = []): string {
const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
const query = encodeURIComponent(JSON.stringify(packages));
@@ -76,6 +80,7 @@ export class WorkflowPveService {
`&cuid=${cuid}` +
`&pveName=${encodeURIComponent(pveName)}` +
`&isLocal=${isLocal}` +
+ `&action=${action}` +
tokenParam
);
}
diff --git a/frontend/src/styles.scss b/frontend/src/styles.scss
index 6915b7f795..27b2a5362a 100644
--- a/frontend/src/styles.scss
+++ b/frontend/src/styles.scss
@@ -239,7 +239,7 @@ hr {
.ve-form {
display: flex;
flex-direction: column;
- gap: 14px;
+ gap: 5px;
}
.fieldRow {
@@ -270,7 +270,7 @@ hr {
background: #ffffff;
}
- .package-inputs {
+ .system-package-inputs {
flex: 1;
display: grid;
grid-template-columns: 1fr 1fr;
@@ -278,6 +278,14 @@ hr {
width: 100%;
}
+ .user-package-inputs {
+ flex: 1;
+ display: grid;
+ grid-template-columns: 1fr 160px 1fr;
+ gap: 14px;
+ width: 100%;
+ }
+
.field {
display: flex;
flex-direction: column;
@@ -361,6 +369,24 @@ hr {
background: transparent;
}
+ .user-package-header-row .package-column-label {
+ font-weight: 600;
+ }
+
+ .new-packages-section {
+ display: flex;
+ flex-direction: column;
+ gap: 2px;
+ }
+
+ .user-package-header-row {
+ display: grid;
+ grid-template-columns: 1fr 160px 1fr;
+ gap: 14px;
+ margin-bottom: 0;
+ padding: 0;
+ }
+
.system-header {
display: flex;
flex-direction: column;
