(yunikorn-release) branch gh-pages updated: Keep index.md updated with latest version
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/gh-pages by this push: new a0c4f93 Keep index.md updated with latest version a0c4f93 is described below commit a0c4f93771f36b1725519f41b82443dd2d9dffc9 Author: Wilfred Spiegelenburg AuthorDate: Mon Sep 23 11:03:35 2024 +1000 Keep index.md updated with latest version --- index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/index.md b/index.md index 0767ff9..2c10f43 100644 --- a/index.md +++ b/index.md @@ -74,6 +74,8 @@ YuniKorn has builtin web UIs for queue hierarchies and apps. See below: | 1.27.x | 1.4.0 | - | | 1.28.x | 1.4.0 | - | | 1.29.x | 1.5.0 | - | +| 1.30.x | 1.6.0 | - | +| 1.31.x | 1.6.0 | - | ## Useful links - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch gh-pages updated: fix typo in version 1.6.0
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/gh-pages by this push: new d3d5042 fix typo in version 1.6.0 d3d5042 is described below commit d3d504288fb98722f3f6b54c20c5b933043d7490 Author: Wilfred Spiegelenburg AuthorDate: Mon Sep 23 10:42:27 2024 +1000 fix typo in version 1.6.0 artifact hub logs error on parsing 1.6 version --- index.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.yaml b/index.yaml index ea505bf..34129c0 100644 --- a/index.yaml +++ b/index.yaml @@ -303,5 +303,5 @@ entries: name: yunikorn urls: - https://github.com/apache/yunikorn-release/releases/download/v1.6.0/yunikorn-1.6.0.tgz - version: 1.6.2 0 + version: 1.6.0 generated: "2024-09-20T12:00:00.00+01:00" - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-2882] Update supported versions (#472)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 93757ad905 [YUNIKORN-2882] Update supported versions (#472) 93757ad905 is described below commit 93757ad90526ee7dd10d862fd47c32e375bc8e60 Author: Peter Bacsko AuthorDate: Mon Sep 23 10:35:46 2024 +1000 [YUNIKORN-2882] Update supported versions (#472) Closes: #472 Signed-off-by: Wilfred Spiegelenburg --- docs/get_started/version.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/get_started/version.md b/docs/get_started/version.md index 9ec67bbe62..34d44606a4 100644 --- a/docs/get_started/version.md +++ b/docs/get_started/version.md @@ -55,4 +55,6 @@ Older build dependencies in earlier releases could prevent you from using some n | 1.27.x |1.4.0| - | | 1.28.x |1.4.0| - | | 1.29.x |1.5.0| - | +| 1.30.x |1.6.0| - | +| 1.31.x |1.6.0| - | - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch YUNIKORN-2870 deleted (was f8f0778a30)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch YUNIKORN-2870 in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git was f8f0778a30 Review The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) branch branch-1.6 created (now 602ca5e)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch branch-1.6 in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git at 602ca5e [YUNIKORN-2799] Fix Dockerfile warnings (#201) No new revisions were added by this update. - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2835] Refactor switch statement in UpdateApplication (#901)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 2ce84319 [YUNIKORN-2835] Refactor switch statement in
UpdateApplication (#901)
2ce84319 is described below
commit 2ce84319027bea6a246c61da068dec866e9e6d70
Author: Tzu-Hua Lan
AuthorDate: Tue Aug 27 13:41:45 2024 +1000
[YUNIKORN-2835] Refactor switch statement in UpdateApplication (#901)
Closes: #901
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/scheduler_callback.go | 8 +++-
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/pkg/cache/scheduler_callback.go b/pkg/cache/scheduler_callback.go
index 0c43b34d..d058be39 100644
--- a/pkg/cache/scheduler_callback.go
+++ b/pkg/cache/scheduler_callback.go
@@ -146,11 +146,9 @@ func (callback *AsyncRMCallback)
UpdateApplication(response *si.ApplicationRespo
ev :=
NewResumingApplicationEvent(updated.ApplicationID)
dispatcher.Dispatch(ev)
}
- default:
- if updated.State == ApplicationStates().Failing ||
updated.State == ApplicationStates().Failed {
- ev :=
NewFailApplicationEvent(updated.ApplicationID, updated.Message)
- dispatcher.Dispatch(ev)
- }
+ case ApplicationStates().Failing, ApplicationStates().Failed:
+ ev := NewFailApplicationEvent(updated.ApplicationID,
updated.Message)
+ dispatcher.Dispatch(ev)
}
}
return nil
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-scheduler-interface) branch master updated: [YUNIKORN-2809] Fix layout of node transition diagram (#141)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository
https://gitbox.apache.org/repos/asf/yunikorn-scheduler-interface.git
The following commit(s) were added to refs/heads/master by this push:
new 68e8c6c [YUNIKORN-2809] Fix layout of node transition diagram (#141)
68e8c6c is described below
commit 68e8c6cca28a743d797e7908b1225392a3a2
Author: Tzu-Hua Lan
AuthorDate: Tue Aug 27 11:56:55 2024 +1000
[YUNIKORN-2809] Fix layout of node transition diagram (#141)
Fix formatting of text to align the diagram
Closes: #141
Signed-off-by: Wilfred Spiegelenburg
---
scheduler-interface-spec.md | 20 +---
1 file changed, 9 insertions(+), 11 deletions(-)
diff --git a/scheduler-interface-spec.md b/scheduler-interface-spec.md
index b51f58a..2a77f20 100644
--- a/scheduler-interface-spec.md
+++ b/scheduler-interface-spec.md
@@ -548,17 +548,15 @@ message AllocationRelease {
State transition of node:
```
-
-+---+ ++ +---+
-|SCHEDULABLE|+>|DRAINING|+-->|REMOVED|
-+---+ ++ +---+
-^ Asked by + Aasked by
-| RM to DRAIN | RM to REMOVE
-| |
-+-+
-Asked by RM to
-SCHEDULE again
-
+ +---+ +++---+
+ |SCHEDULABLE|+>|DRAINING|+-->|REMOVED|
+ +---+ +++---+
+ ^ Asked by + Asked by
+ | RM to DRAIN|RM to REMOVE
+ | |
+ +-+
+ Asked by RM to
+ SCHEDULE again
```
See protocol below:
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-scheduler-interface) branch master updated: [YUNIKORN-2838] SI: Update protobuf dependencies (#142)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-scheduler-interface.git The following commit(s) were added to refs/heads/master by this push: new 2e961af [YUNIKORN-2838] SI: Update protobuf dependencies (#142) 2e961af is described below commit 2e961afe55e4ed44da712b32ff805bf413e97c66 Author: Craig Condit AuthorDate: Tue Aug 27 11:54:31 2024 +1000 [YUNIKORN-2838] SI: Update protobuf dependencies (#142) Update grpc to v1.65.0 and protobuf to v1.34.2 to match upstream Kubernetes usage. Closes: #142 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 13 +-- go.sum | 23 +++- lib/go/si/si.pb.go | 64 +++--- 3 files changed, 46 insertions(+), 54 deletions(-) diff --git a/go.mod b/go.mod index d3b1878..fb47a4e 100644 --- a/go.mod +++ b/go.mod @@ -22,16 +22,15 @@ module github.com/apache/yunikorn-scheduler-interface go 1.21 require ( - google.golang.org/grpc v1.58.3 - google.golang.org/protobuf v1.33.0 + google.golang.org/grpc v1.65.0 + google.golang.org/protobuf v1.34.2 ) require ( - github.com/golang/protobuf v1.5.3 // indirect - golang.org/x/net v0.20.0 // indirect - golang.org/x/sys v0.18.0 // indirect - golang.org/x/text v0.14.0 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 // indirect + golang.org/x/net v0.25.0 // indirect + golang.org/x/sys v0.20.0 // indirect + golang.org/x/text v0.15.0 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20240528184218-531527333157 // indirect ) replace ( diff --git a/go.sum b/go.sum index d36f03f..21696de 100644 --- a/go.sum +++ b/go.sum @@ -1,21 +1,14 @@ -github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= -github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= -github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= -github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs= golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4= golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 h1:bVf09lpb+OJbByTj913DRJioFFAjf/ZGxEz7MajTp2U= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98/go.mod h1:TUfxEVdsvPg18p6AslUXFoLdpED4oBnGwyqk3dV1XzM= -google.golang.org/grpc v1.58.3 h1:BjnpXut1btbtgN/6sp+brB2Kbm2LjNXnidYujAVbSoQ= -google.golang.org/grpc v1.58.3/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0= -google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= -google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI= -google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= +google.golang.org/genproto/googleapis/rpc v0.0.0-20240528184218-531527333157 h1:Zy9XzmMEflZ/MAaA7vNcoebnRAld7FsPW1EeBB7V0m8= +google.golang.org/genproto/googleapis/rpc v0.0.0-20240528184218-531527333157/go.mod h1:EfXuqaE1J41VCDicxHzUDm+8rk+7ZdXzHV0IhO/I6s0= +google.golang.org/grpc v1.65.0 h1:bs/cUb4lp1G5iImFFd3u5ixQzweKizoZJAwBNLR42lc= +google.golang.org/grpc v1.65.0/go.mod h1:WgYC2ypjlB0EiQi6wdKixMqukr6lBc0Vo+oOgjrM5ZQ= +google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg= +google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw= diff --git a/lib/go/si/si.pb.go b/lib/go/si/si.pb.go index d8b941f..f708540 100644 --- a/lib/go/si/si.pb.go +++ b/lib/go/si/si.pb.go @@ -16,7 +16,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: -// protoc-gen-go v1.33.0 +// protoc-gen-go v1.34.2 // protocv4.23.2 // source: yunikorn-scheduler-interface/si.proto @@ -3082,7 +3082,7 @@ func
(yunikorn-core) branch master updated: [YUNIKORN-2796] Prune zero resource type from root and partition (#943)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new f439d805 [YUNIKORN-2796] Prune zero resource type from root and
partition (#943)
f439d805 is described below
commit f439d8053607a40d0048c039796f6dc893d01741
Author: Wilfred Spiegelenburg
AuthorDate: Wed Aug 14 16:32:36 2024 +1000
[YUNIKORN-2796] Prune zero resource type from root and partition (#943)
After the last node is removed that registers a specific resource type
the root queue and partition show it as a resource type with 0 maximum.
A resource type that does not exist should not be shown at all.
Pruning the 0 valued resource types for root and partition will fix that
Clean up of the partition code to use the same function to update
partition and root settings for create, update and delete of a node.
Fix root queue check for max resource.
Fix tests that fake allocation or node registration to correctly set the
root queue limits.
Closes: #943
Signed-off-by: Wilfred Spiegelenburg
---
pkg/common/resources/resources.go| 15 +++-
pkg/common/resources/resources_test.go | 37 -
pkg/scheduler/objects/node.go| 11 ++-
pkg/scheduler/objects/preemption_test.go | 34 -
pkg/scheduler/objects/queue.go | 6 ++
pkg/scheduler/objects/queue_test.go | 126 ++-
pkg/scheduler/partition.go | 56 +-
pkg/scheduler/partition_test.go | 36 ++---
8 files changed, 217 insertions(+), 104 deletions(-)
diff --git a/pkg/common/resources/resources.go
b/pkg/common/resources/resources.go
index 1b2a036b..c5bc7c54 100644
--- a/pkg/common/resources/resources.go
+++ b/pkg/common/resources/resources.go
@@ -144,7 +144,20 @@ func (r *Resource) Clone() *Resource {
return ret
}
-// Add additional resource to the base updating the base resource
+// Prune removes any resource type that has a zero value set.
+// Note that a zero value set and a type not being set are interpreted
differently for quotas.
+func (r *Resource) Prune() {
+ if r == nil {
+ return
+ }
+ for k, v := range r.Resources {
+ if v == 0 {
+ delete(r.Resources, k)
+ }
+ }
+}
+
+// AddTo adds the resource to the base updating the base resource
// Should be used by temporary computation only
// A nil base resource does not change
// A nil passed in resource is treated as a zero valued resource and leaves
base unchanged
diff --git a/pkg/common/resources/resources_test.go
b/pkg/common/resources/resources_test.go
index 49df426a..88c2c421 100644
--- a/pkg/common/resources/resources_test.go
+++ b/pkg/common/resources/resources_test.go
@@ -24,9 +24,10 @@ import (
"reflect"
"testing"
- "github.com/apache/yunikorn-scheduler-interface/lib/go/common"
-
+ "golang.org/x/exp/maps"
"gotest.tools/v3/assert"
+
+ "github.com/apache/yunikorn-scheduler-interface/lib/go/common"
)
func CheckLenOfResource(res *Resource, expected int) (bool, string) {
@@ -2137,3 +2138,35 @@ func TestResource_DominantResource(t *testing.T) {
})
}
}
+
+func TestResource_PruneNil(t *testing.T) {
+ // make sure we're nil safe IDE will complain about the receiver being
nil
+ defer func() {
+ if r := recover(); r != nil {
+ t.Fatal("panic on nil resource in prune test")
+ }
+ }()
+ var empty *Resource
+ empty.Prune()
+}
+
+func TestResource_Prune(t *testing.T) {
+ var tests = []struct {
+ caseName string
+ inputmap[string]Quantity
+ output map[string]Quantity
+ }{
+ {"no types", map[string]Quantity{}, map[string]Quantity{}},
+ {"all types with value", map[string]Quantity{"first": 1,
"second": -2, "third": 3}, map[string]Quantity{"first": 1, "second": -2,
"third": 3}},
+ {"zero type", map[string]Quantity{"first": 1, "zero": 0,
"third": 3}, map[string]Quantity{"first": 1, "third": 3}},
+ {"no types with value", map[string]Quantity{"first": 0,
"second": 0}, map[string]Quantity{}},
+ }
+ for _, tt := range tests {
+ t.Run(tt.caseName, func(t *testing.T) {
+ original := NewResourceFromMap(tt.input)
+ original.Prune()
+ assert.Equal(t, len(tt.output),
len(original.Re
(yunikorn-site) branch master updated: [YUNIKORN-2524] add documentation for recovery queue (root.@recovery@) (#459)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new ae8fe6a8df [YUNIKORN-2524] add documentation for recovery queue
(root.@recovery@) (#459)
ae8fe6a8df is described below
commit ae8fe6a8df0efec5820821d8eddfd268621d69fc
Author: Hsien-Cheng(Ryan) Huang
AuthorDate: Tue Jul 30 11:53:21 2024 +1000
[YUNIKORN-2524] add documentation for recovery queue (root.@recovery@)
(#459)
Closes: #459
Signed-off-by: Wilfred Spiegelenburg
---
docs/user_guide/queue_config.md | 9 +
1 file changed, 9 insertions(+)
diff --git a/docs/user_guide/queue_config.md b/docs/user_guide/queue_config.md
index 05594e0144..c665fc07a3 100644
--- a/docs/user_guide/queue_config.md
+++ b/docs/user_guide/queue_config.md
@@ -196,6 +196,15 @@ partitions:
{memory: 5G, vcore: 50}
```
+### Recovery queue
+
+The recovery queue, identified by the name `root.@recovery@`, is a dynamic
queue that is not directly queryable. It is used exclusively during the
initialization phase for already running allocations that are part of
applications failing placement. Its primary function is to handle tasks that
need to be reassigned or recovered without user intervention.
+
+- The queue is created dynamically and will disappear when it is no longer in
use.
+- The queue does not have quotas or Access Control Lists (ACLs).
+- It cannot be submitted to directly by users. It is managed internally by
YuniKorn for specific recovery operations.
+- While the queue is unqueryable directly, its existence and activities can be
observed through the application RESTful API at
[/ws/v1/partition/:partition/applications/:state](../api/scheduler#partition-applications).
+
### Placement rules
The placement rules are defined and documented in the [placement
rule](placement_rules.md) document.
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2738] Only check failure reason once when failing app (#877)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new c6f69b7b [YUNIKORN-2738] Only check failure reason once when failing
app (#877)
c6f69b7b is described below
commit c6f69b7bd8b6b6914807b9f27ef54466b2bb1f97
Author: Wilfred Spiegelenburg
AuthorDate: Fri Jul 12 18:20:24 2024 +1000
[YUNIKORN-2738] Only check failure reason once when failing app (#877)
Optimisation in failure type determination by moving it outside the
loop.
Closes: #877
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/application.go | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/pkg/cache/application.go b/pkg/cache/application.go
index 9833a6f2..3bedb132 100644
--- a/pkg/cache/application.go
+++ b/pkg/cache/application.go
@@ -615,12 +615,14 @@ func (app *Application) handleFailApplicationEvent(errMsg
string) {
unalloc = append(unalloc, app.getTasks(TaskStates().Pending)...)
unalloc = append(unalloc, app.getTasks(TaskStates().Scheduling)...)
+ timeout := strings.Contains(errMsg,
constants.ApplicationInsufficientResourcesFailure)
+ rejected := strings.Contains(errMsg,
constants.ApplicationRejectedFailure)
// publish pod level event to unallocated pods
for _, task := range unalloc {
// Only need to fail the non-placeholder pod(s)
- if strings.Contains(errMsg,
constants.ApplicationInsufficientResourcesFailure) {
+ if timeout {
failTaskPodWithReasonAndMsg(task,
constants.ApplicationInsufficientResourcesFailure, "Scheduling has timed out
due to insufficient resources")
- } else if strings.Contains(errMsg,
constants.ApplicationRejectedFailure) {
+ } else if rejected {
errMsgArr := strings.Split(errMsg, ":")
failTaskPodWithReasonAndMsg(task,
constants.ApplicationRejectedFailure, errMsgArr[1])
}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2323] Gang scheduling user experience new events (#876)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 4b19450f [YUNIKORN-2323] Gang scheduling user experience new events
(#876)
4b19450f is described below
commit 4b19450fff7642c428619f48b8dc9568dbfe39f6
Author: Wilfred Spiegelenburg
AuthorDate: Fri Jul 12 18:19:24 2024 +1000
[YUNIKORN-2323] Gang scheduling user experience new events (#876)
Additional events for gang scheduling covering:
* placeholder timeout (resuming state)
* placeholder creation
* placeholder create failure(s)
Closes: #876
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/application.go | 25 +++--
pkg/cache/application_state.go | 4
2 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/pkg/cache/application.go b/pkg/cache/application.go
index ff4db058..9833a6f2 100644
--- a/pkg/cache/application.go
+++ b/pkg/cache/application.go
@@ -495,12 +495,28 @@ func (app *Application) postAppAccepted() {
dispatcher.Dispatch(ev)
}
+// onResuming triggered when entering the resuming state which is triggered by
the time out of the gang placeholders
+// if SOFT gang scheduling is configured.
+func (app *Application) onResuming() {
+ if app.originatingTask != nil {
+
events.GetRecorder().Eventf(app.originatingTask.GetTaskPod().DeepCopy(), nil,
v1.EventTypeWarning, "GangScheduling",
+ "GangSchedulingFailed", "Application %s resuming as
non-gang application (SOFT)", app.applicationID)
+ }
+}
+
+// onReserving triggered when entering the reserving state.
+// During normal operation this creates all the placeholders. During recovery
this call could cause the application
+// in the shim and core to progress to the next state.
func (app *Application) onReserving() {
- // happens after recovery - if placeholders already exist, we need to
send
+ // if any placeholder already exist during recovery we might need to
send
// an event to trigger Application state change in the core
if len(app.getPlaceHolderTasks()) > 0 {
ev := NewUpdateApplicationReservationEvent(app.applicationID)
dispatcher.Dispatch(ev)
+ } else if app.originatingTask != nil {
+ // not recovery or no placeholders created yet add an event to
the pod
+
events.GetRecorder().Eventf(app.originatingTask.GetTaskPod().DeepCopy(), nil,
v1.EventTypeNormal, "GangScheduling",
+ "CreatingPlaceholders", "Application %s creating
placeholders", app.applicationID)
}
go func() {
@@ -511,6 +527,11 @@ func (app *Application) onReserving() {
getPlaceholderManager().cleanUp(app)
ev := NewRunApplicationEvent(app.applicationID)
dispatcher.Dispatch(ev)
+ // failed at least one placeholder creation progress as
a normal application
+ if app.originatingTask != nil {
+
events.GetRecorder().Eventf(app.originatingTask.GetTaskPod().DeepCopy(), nil,
v1.EventTypeWarning, "GangScheduling",
+ "PlaceholderCreateFailed", "Application
%s fall back to normal scheduling", app.applicationID)
+ }
}
}()
}
@@ -520,7 +541,7 @@ func (app *Application) onReserving() {
func (app *Application) onReservationStateChange() {
if app.originatingTask != nil {
events.GetRecorder().Eventf(app.originatingTask.GetTaskPod().DeepCopy(), nil,
v1.EventTypeNormal, "GangScheduling",
- "Placeholder Allocated", "Application %s placeholder
has been allocated.", app.applicationID)
+ "PlaceholderAllocated", "Application %s placeholder has
been allocated.", app.applicationID)
}
desireCounts := make(map[string]int32, len(app.taskGroups))
for _, tg := range app.taskGroups {
diff --git a/pkg/cache/application_state.go b/pkg/cache/application_state.go
index ef251e2f..aee1c3e1 100644
--- a/pkg/cache/application_state.go
+++ b/pkg/cache/application_state.go
@@ -506,6 +506,10 @@ func newAppState() *fsm.FSM { //nolint:funlen
app := event.Args[0].(*Application)
//nolint:errcheck
app.onReserving()
},
+ states.Resuming: func(_ context.Context, event
*fsm.Event) {
+ app := event.Args[0].(*Application)
//nolint:errcheck
+
(yunikorn-site) branch master updated: [YUNIKORN-2705] Fix DOAP syntax (#444)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new d503d246d1 [YUNIKORN-2705] Fix DOAP syntax (#444) d503d246d1 is described below commit d503d246d1e9a446687a27f4669f247bcbcfbfd8 Author: Arnout Engelen AuthorDate: Fri Jun 28 14:14:37 2024 +1000 [YUNIKORN-2705] Fix DOAP syntax (#444) Update the doap file to comply with the RDF syntax. Syntax checker run: https://www.w3.org/RDF/Validator/ Closes: #444 Signed-off-by: Wilfred Spiegelenburg --- doap_YuniKorn.rdf | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doap_YuniKorn.rdf b/doap_YuniKorn.rdf index 0d46056084..39758aa9ce 100644 --- a/doap_YuniKorn.rdf +++ b/doap_YuniKorn.rdf @@ -40,6 +40,8 @@ 2024-05-16 1.5.1 + + YuniKorn 1.5.0 2024-03-14 - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2581] Expose running placement rules in REST (#857)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 81d69749 [YUNIKORN-2581] Expose running placement rules in REST (#857)
81d69749 is described below
commit 81d697494d972d01c01135dd981e1ec2fc89f57d
Author: Wilfred Spiegelenburg
AuthorDate: Fri May 31 11:26:57 2024 +1000
[YUNIKORN-2581] Expose running placement rules in REST (#857)
Add a new REST endpoint under the partition that returns the currently
active placement rules:
/ws/v1/partition/:partition/placementrules
The call retrieves the active set from the placement manager. This will
include the recovery rule. The rule interface is extended with a new
private method ruleDAO() that must be implemented by all rules and
returns the RuleDAO object.
Add the currently active set of placement rules to the state dump. This
handles multiple partitions. It wraps the list of RuleDAO objects in a
RuleDAOInfo per partition.
Closes: #857
Signed-off-by: Wilfred Spiegelenburg
---
go.mod| 1 +
go.sum| 2 +
pkg/scheduler/partition.go| 9 ++-
pkg/scheduler/placement/filter.go | 47 -
pkg/scheduler/placement/filter_test.go| 65 +-
pkg/scheduler/placement/fixed_rule.go | 19 +
pkg/scheduler/placement/fixed_rule_test.go| 38 ++
pkg/scheduler/placement/placement.go | 23 +--
pkg/scheduler/placement/placement_test.go | 51 --
pkg/scheduler/placement/provided_rule.go | 17 +
pkg/scheduler/placement/provided_rule_test.go | 33 +
pkg/scheduler/placement/recovery_rule.go | 10 +++
pkg/scheduler/placement/recovery_rule_test.go | 9 +++
pkg/scheduler/placement/rule.go | 44
pkg/scheduler/placement/rule_test.go | 9 +++
pkg/scheduler/placement/tag_rule.go | 18 +
pkg/scheduler/placement/tag_rule_test.go | 38 ++
pkg/scheduler/placement/testrule.go | 17 +
pkg/scheduler/placement/user_rule.go | 17 +
pkg/scheduler/placement/user_rule_test.go | 33 +
pkg/webservice/dao/rule_info.go | 39 +++
pkg/webservice/handlers.go| 32 +
pkg/webservice/handlers_test.go | 99 ---
pkg/webservice/routes.go | 6 ++
pkg/webservice/state_dump.go | 2 +
25 files changed, 623 insertions(+), 55 deletions(-)
diff --git a/go.mod b/go.mod
index 65b2c5a8..5f180926 100644
--- a/go.mod
+++ b/go.mod
@@ -33,6 +33,7 @@ require (
github.com/prometheus/common v0.45.0
github.com/sasha-s/go-deadlock v0.3.1
go.uber.org/zap v1.26.0
+ golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8
golang.org/x/net v0.21.0
golang.org/x/time v0.5.0
google.golang.org/grpc v1.58.3
diff --git a/go.sum b/go.sum
index 35ee67e4..f04fef47 100644
--- a/go.sum
+++ b/go.sum
@@ -50,6 +50,8 @@ go.uber.org/multierr v1.10.0
h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
go.uber.org/multierr v1.10.0/go.mod
h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
+golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8
h1:ESSUROHIBHg7USnszlcdmjBEwdMj9VUvU+OPk4yl2mc=
+golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8/go.mod
h1:/lliqkxwWAhPjf5oSOIJup2XcqJaw8RGS6k3TGEc7GI=
golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
diff --git a/pkg/scheduler/partition.go b/pkg/scheduler/partition.go
index 5662bd75..fc2c5404 100644
--- a/pkg/scheduler/partition.go
+++ b/pkg/scheduler/partition.go
@@ -481,14 +481,19 @@ func (pc *PartitionContext) getQueueInternal(name string)
*objects.Queue {
return queue
}
-// Get the queue info for the whole queue structure to pass to the webservice
+// GetPartitionQueues builds the queue info for the whole queue structure to
pass to the webservice
func (pc *PartitionContext) GetPartitionQueues() dao.PartitionQueueDAOInfo {
partitionQueueDAOInfo := pc.root.GetPartitionQueueDAOInfo(true)
partitionQueueDAOInfo.Partition =
common.GetPartitionNameWithoutClusterID(pc.Name)
return partitionQueueDAOInfo
}
-// Create the recovery queue.
+// GetPlacementRules returns the current active rule set as dao to expose to
the webservice
+func (pc *PartitionContext) GetPlacementRules() []*dao.RuleDAO
(yunikorn-site) branch master updated: [YUNIKORN-2628] revert relative path for release announcement (#430)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new a29e6105c3 [YUNIKORN-2628] revert relative path for release announcement (#430) a29e6105c3 is described below commit a29e6105c3e10e8f80d616bf2acf52dbeec81fac Author: Wilfred Spiegelenburg AuthorDate: Fri May 17 11:44:46 2024 +1000 [YUNIKORN-2628] revert relative path for release announcement (#430) Closes: #430 Signed-off-by: Wilfred Spiegelenburg --- src/pages/community/download.md | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/pages/community/download.md b/src/pages/community/download.md index e3ae0259e3..683a866263 100644 --- a/src/pages/community/download.md +++ b/src/pages/community/download.md @@ -33,11 +33,11 @@ We publish prebuilt docker images for everyone's convenience. The latest release of Apache YuniKorn is v1.5.0. -| Version | Release date | Source download | Docker images [...] -|-|--|---|-- [...] -| v1.5.0 | 2024-03-14 | [Download](https://www.apache.org/dyn/closer.lua/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz)[Checksum](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.sha512) & [Signature](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.5.0/images/sha256-9cefd0df164b9c4d39f9e10b010eaf7d8f89b130de1648e94f75b9b95d300a00)[admission- [...] -| v1.4.0 | 2023-11-20 | [Download](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz)[Checksum](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.4.0/images/sha256-d013be8e3ad7eb8e51ce23951e6899a4b74088e52c3767f3fcc7efcdcc0904f5)[admission- [...] -| v1.3.0 | 2023-06-12 | [Download](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz)[Checksum](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.3.0/images/sha256-99a1973728c6684b1da7631dbf015daa1dbf519dbab1ffc8b23fccdfa7ffd0c5)[admission- [...] +| Version | Release date | Source download | Docker images [...] +|-|--|---|-- [...] +| v1.5.0 | 2024-03-14 | [Download](https://www.apache.org/dyn/closer.lua/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz)[Checksum](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.sha512) & [Signature](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/schedule
(yunikorn-k8shim) branch master updated: [YUNIKORN-2627] Add K8s 1.30 to the e2e matrix (#840)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 5f80f49b [YUNIKORN-2627] Add K8s 1.30 to the e2e matrix (#840)
5f80f49b is described below
commit 5f80f49b2ee5acb3432b2d5534dbe7f3d3bcc2fc
Author: Tseng Hsi-Huang <9501...@gmail.com>
AuthorDate: Thu May 16 17:28:41 2024 +1000
[YUNIKORN-2627] Add K8s 1.30 to the e2e matrix (#840)
Closes: #840
Signed-off-by: Wilfred Spiegelenburg
---
.github/workflows/pre-commit.yml | 2 +-
Makefile | 2 +-
scripts/run-e2e-tests.sh | 3 ++-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
index 4131fde9..afed3906 100644
--- a/.github/workflows/pre-commit.yml
+++ b/.github/workflows/pre-commit.yml
@@ -43,7 +43,7 @@ jobs:
strategy:
fail-fast: false
matrix:
-k8s: [v1.29.2, v1.28.7, v1.27.11, v1.26.14, v1.25.16, v1.24.17]
+k8s: [v1.30.0, v1.29.2, v1.28.7, v1.27.11, v1.26.14, v1.25.16,
v1.24.17]
plugin: ['', '--plugin']
steps:
- name: Checkout source code
diff --git a/Makefile b/Makefile
index 50b3a659..76dd81fc 100644
--- a/Makefile
+++ b/Makefile
@@ -155,7 +155,7 @@ KUBECTL_VERSION=v1.27.7
KUBECTL_BIN=$(TOOLS_DIR)/kubectl
# kind
-KIND_VERSION=v0.20.0
+KIND_VERSION=v0.23.0
KIND_BIN=$(TOOLS_DIR)/kind
# helm
diff --git a/scripts/run-e2e-tests.sh b/scripts/run-e2e-tests.sh
index 07073c4e..02c21ec7 100755
--- a/scripts/run-e2e-tests.sh
+++ b/scripts/run-e2e-tests.sh
@@ -164,9 +164,10 @@ Examples:
${NAME} -a test -n yk8s -v kindest/node:v1.27.11
${NAME} -a test -n yk8s -v kindest/node:v1.28.7
${NAME} -a test -n yk8s -v kindest/node:v1.29.2
+ ${NAME} -a test -n yk8s -v kindest/node:v1.30.0
Use a local helm chart path:
-${NAME} -a test -n yk8s -v kindest/node:v1.29.2 -p
../yunikorn-release/helm-charts/yunikorn
+${NAME} -a test -n yk8s -v kindest/node:v1.30.0 -p
../yunikorn-release/helm-charts/yunikorn
EOF
}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2531] Create unit tests for AsyncRMCallback (#832)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 4f8da23f [YUNIKORN-2531] Create unit tests for AsyncRMCallback (#832)
4f8da23f is described below
commit 4f8da23f4f482ccfa1c57a1f943777a72b3c07dd
Author: Peter Bacsko
AuthorDate: Tue May 14 17:01:18 2024 +1000
[YUNIKORN-2531] Create unit tests for AsyncRMCallback (#832)
Closes: #832
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/context.go | 5 -
pkg/cache/scheduler_callback_test.go | 629 +++
pkg/common/events/recorder.go| 6 -
3 files changed, 629 insertions(+), 11 deletions(-)
diff --git a/pkg/cache/context.go b/pkg/cache/context.go
index b97b9289..c58b7914 100644
--- a/pkg/cache/context.go
+++ b/pkg/cache/context.go
@@ -664,11 +664,6 @@ func (ctx *Context) IsPodFitNode(name, node string,
allocate bool) error {
}
func (ctx *Context) IsPodFitNodeViaPreemption(name, node string, allocations
[]string, startIndex int) (int, bool) {
- // assume minimal pods need killing if running in testing mode
- if ctx.apiProvider.IsTestingMode() {
- return startIndex, false
- }
-
ctx.lock.RLock()
defer ctx.lock.RUnlock()
if pod := ctx.schedulerCache.GetPod(name); pod != nil {
diff --git a/pkg/cache/scheduler_callback_test.go
b/pkg/cache/scheduler_callback_test.go
new file mode 100644
index ..ce2633ac
--- /dev/null
+++ b/pkg/cache/scheduler_callback_test.go
@@ -0,0 +1,629 @@
+/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+package cache
+
+import (
+ "encoding/json"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "gotest.tools/v3/assert"
+ v1 "k8s.io/api/core/v1"
+ apis "k8s.io/apimachinery/pkg/apis/meta/v1"
+ k8sEvents "k8s.io/client-go/tools/events"
+ "k8s.io/kubernetes/pkg/scheduler/framework"
+
+ "github.com/apache/yunikorn-k8shim/pkg/client"
+ "github.com/apache/yunikorn-k8shim/pkg/common/constants"
+ "github.com/apache/yunikorn-k8shim/pkg/common/events"
+ "github.com/apache/yunikorn-k8shim/pkg/common/test"
+ "github.com/apache/yunikorn-k8shim/pkg/common/utils"
+ "github.com/apache/yunikorn-k8shim/pkg/dispatcher"
+ "github.com/apache/yunikorn-k8shim/pkg/plugin/predicates"
+ "github.com/apache/yunikorn-scheduler-interface/lib/go/si"
+)
+
+func TestUpdateAllocation_NewTask(t *testing.T) {
+ callback, context := initCallbackTest(t, false, false)
+ defer dispatcher.UnregisterAllEventHandlers()
+ defer dispatcher.Stop()
+
+ err := callback.UpdateAllocation(&si.AllocationResponse{
+ New: []*si.Allocation{
+ {
+ ApplicationID: appID,
+ AllocationKey: taskUID1,
+ NodeID:fakeNodeName,
+ },
+ },
+ })
+ assert.NilError(t, err, "error updating allocation")
+ assert.Assert(t, context.schedulerCache.IsAssumedPod(taskUID1))
+ task := context.getTask(appID, taskUID1)
+ err = utils.WaitForCondition(func() bool {
+ return task.GetTaskState() == TaskStates().Bound
+ }, 10*time.Millisecond, time.Second)
+ assert.NilError(t, err, "task has not transitioned to Bound state")
+}
+
+func TestUpdateAllocation_NewTask_TaskNotFound(t *testing.T) {
+ callback, context := initCallbackTest(t, false, false)
+ defer dispatcher.UnregisterAllEventHandlers()
+ defer dispatcher.Stop()
+ pod := context.getTask(appID, taskUID1).pod
+ context.RemoveTask(appID, taskUID1)
+ context.DeletePod(pod)
+
+ err := callback.UpdateAllocation(&si.AllocationResponse{
+
(yunikorn-k8shim) branch master updated: [YUNIKORN-2615] Remove named returns from predicate_manager.go (#834)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new db84a698 [YUNIKORN-2615] Remove named returns from
predicate_manager.go (#834)
db84a698 is described below
commit db84a698246f574c2672d38ea160295059af0e25
Author: Wilfred Spiegelenburg
AuthorDate: Tue May 14 16:57:21 2024 +1000
[YUNIKORN-2615] Remove named returns from predicate_manager.go (#834)
Remove named returns, decrease calls to get plugin name.
Simplify skip map used in runPreFilterPlugins and runFilterPlugins to
a boolean map
Closes: #834
Signed-off-by: Wilfred Spiegelenburg
---
pkg/plugin/predicates/predicate_manager.go | 42 ++
1 file changed, 19 insertions(+), 23 deletions(-)
diff --git a/pkg/plugin/predicates/predicate_manager.go
b/pkg/plugin/predicates/predicate_manager.go
index ebf9d19d..ec394803 100644
--- a/pkg/plugin/predicates/predicate_manager.go
+++ b/pkg/plugin/predicates/predicate_manager.go
@@ -125,7 +125,7 @@ func (p *predicateManagerImpl) Predicates(pod *v1.Pod, node
*framework.NodeInfo,
return p.predicatesReserve(pod, node)
}
-func (p *predicateManagerImpl) PreemptionPredicates(pod *v1.Pod, node
*framework.NodeInfo, victims []*v1.Pod, startIndex int) (index int, ok bool) {
+func (p *predicateManagerImpl) PreemptionPredicates(pod *v1.Pod, node
*framework.NodeInfo, victims []*v1.Pod, startIndex int) (int, bool) {
ctx := context.Background()
state := framework.NewCycleState()
@@ -178,19 +178,19 @@ func (p *predicateManagerImpl)
removePodFromNodeNoFail(node *framework.NodeInfo,
}
}
-func (p *predicateManagerImpl) predicatesReserve(pod *v1.Pod, node
*framework.NodeInfo) (plugin string, error error) {
+func (p *predicateManagerImpl) predicatesReserve(pod *v1.Pod, node
*framework.NodeInfo) (string, error) {
ctx := context.Background()
state := framework.NewCycleState()
return p.podFitsNode(ctx, state, *p.reservationPreFilters,
*p.reservationFilters, pod, node)
}
-func (p *predicateManagerImpl) predicatesAllocate(pod *v1.Pod, node
*framework.NodeInfo) (plugin string, error error) {
+func (p *predicateManagerImpl) predicatesAllocate(pod *v1.Pod, node
*framework.NodeInfo) (string, error) {
ctx := context.Background()
state := framework.NewCycleState()
return p.podFitsNode(ctx, state, *p.allocationPreFilters,
*p.allocationFilters, pod, node)
}
-func (p *predicateManagerImpl) podFitsNode(ctx context.Context, state
*framework.CycleState, preFilters []framework.PreFilterPlugin, filters
[]framework.FilterPlugin, pod *v1.Pod, node *framework.NodeInfo) (plugin
string, error error) {
+func (p *predicateManagerImpl) podFitsNode(ctx context.Context, state
*framework.CycleState, preFilters []framework.PreFilterPlugin, filters
[]framework.FilterPlugin, pod *v1.Pod, node *framework.NodeInfo) (string,
error) {
// Run "prefilter" plugins.
status, plugin, skip := p.runPreFilterPlugins(ctx, state, preFilters,
pod, node)
if !status.IsSuccess() && !status.IsSkip() {
@@ -205,27 +205,26 @@ func (p *predicateManagerImpl) podFitsNode(ctx
context.Context, state *framework
return "", nil
}
-func (p *predicateManagerImpl) runPreFilterPlugins(ctx context.Context, state
*framework.CycleState, plugins []framework.PreFilterPlugin, pod *v1.Pod, node
*framework.NodeInfo) (status *framework.Status, plugin string, skip
map[string]interface{}) {
- var mergedNodes *framework.PreFilterResult = nil
- skip = nil
+func (p *predicateManagerImpl) runPreFilterPlugins(ctx context.Context, state
*framework.CycleState, plugins []framework.PreFilterPlugin, pod *v1.Pod, node
*framework.NodeInfo) (*framework.Status, string, map[string]bool) {
+ var mergedNodes *framework.PreFilterResult
+ skip := make(map[string]bool)
for _, pl := range plugins {
+ plugin := pl.Name()
nodes, status := p.runPreFilterPlugin(ctx, pl, state, pod)
if status.IsSkip() {
- if skip == nil {
- skip = make(map[string]interface{})
- }
- skip[pl.Name()] = nil
+ skip[plugin] = true
} else if !status.IsSuccess() {
if status.IsRejected() {
- return status, plugin, skip
+ return status, "", skip
}
err := errors.New(status.Message())
log.Log(log.ShimPredicates).Error("failed running
PreFilter plugin",
- zap.String("pluginName", pl.Name(
(yunikorn-k8shim) branch master updated: [YUNIKORN-2601] Update kindest/node (#833)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new b7499e13 [YUNIKORN-2601] Update kindest/node (#833)
b7499e13 is described below
commit b7499e1317f2be751358df476ec7e78e889682cf
Author: ryan
AuthorDate: Thu May 9 12:41:51 2024 +1000
[YUNIKORN-2601] Update kindest/node (#833)
Update K8s version used in testing by updating kind image versions:
v1.29.1 to v1.29.2
v1.28.6 to v1.28.7
v1.27.10 to v1.27.11
v1.26.13 to v1.26.14
Closes: #833
Signed-off-by: Wilfred Spiegelenburg
---
.github/workflows/pre-commit.yml | 2 +-
scripts/run-e2e-tests.sh | 14 +++---
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
index 26a01085..4131fde9 100644
--- a/.github/workflows/pre-commit.yml
+++ b/.github/workflows/pre-commit.yml
@@ -43,7 +43,7 @@ jobs:
strategy:
fail-fast: false
matrix:
-k8s: [v1.29.1, v1.28.6, v1.27.10, v1.26.13, v1.25.16, v1.24.17]
+k8s: [v1.29.2, v1.28.7, v1.27.11, v1.26.14, v1.25.16, v1.24.17]
plugin: ['', '--plugin']
steps:
- name: Checkout source code
diff --git a/scripts/run-e2e-tests.sh b/scripts/run-e2e-tests.sh
index 8f1d493e..07073c4e 100755
--- a/scripts/run-e2e-tests.sh
+++ b/scripts/run-e2e-tests.sh
@@ -158,15 +158,15 @@ Usage: ${NAME} -a -n -v
[
--plugin use scheduler plugin image instead of default
mode image
Examples:
- ${NAME} -a test -n yk8s -v kindest/node:v1.24.15
- ${NAME} -a test -n yk8s -v kindest/node:v1.25.11
- ${NAME} -a test -n yk8s -v kindest/node:v1.26.6
- ${NAME} -a test -n yk8s -v kindest/node:v1.27.3
- ${NAME} -a test -n yk8s -v kindest/node:v1.28.0
- ${NAME} -a test -n yk8s -v kindest/node:v1.29.0
+ ${NAME} -a test -n yk8s -v kindest/node:v1.24.17
+ ${NAME} -a test -n yk8s -v kindest/node:v1.25.16
+ ${NAME} -a test -n yk8s -v kindest/node:v1.26.14
+ ${NAME} -a test -n yk8s -v kindest/node:v1.27.11
+ ${NAME} -a test -n yk8s -v kindest/node:v1.28.7
+ ${NAME} -a test -n yk8s -v kindest/node:v1.29.2
Use a local helm chart path:
-${NAME} -a test -n yk8s -v kindest/node:v1.29.0 -p
../yunikorn-release/helm-charts/yunikorn
+${NAME} -a test -n yk8s -v kindest/node:v1.29.2 -p
../yunikorn-release/helm-charts/yunikorn
EOF
}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-2591] Document placement rules always (#426)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 72099e7465 [YUNIKORN-2591] Document placement rules always (#426) 72099e7465 is described below commit 72099e7465f808b00996d36c512ff3d45b010c73 Author: ryan AuthorDate: Tue May 7 12:22:40 2024 +1000 [YUNIKORN-2591] Document placement rules always (#426) Remove mention of no placement rules and set it to use the provided rule if no rules are defined. This is the same behaviour as it was without using placement rules always. Closes: #426 Signed-off-by: Wilfred Spiegelenburg --- docs/user_guide/queue_config.md | 3 ++- versioned_docs/version-1.4.0/user_guide/queue_config.md | 3 ++- versioned_docs/version-1.5.0/user_guide/queue_config.md | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/user_guide/queue_config.md b/docs/user_guide/queue_config.md index 75bd24133a..05594e0144 100644 --- a/docs/user_guide/queue_config.md +++ b/docs/user_guide/queue_config.md @@ -201,7 +201,8 @@ partitions: The placement rules are defined and documented in the [placement rule](placement_rules.md) document. Each partition can have only one set of placement rules defined. -If no rules are defined the placement manager is not started and each application *must* have a queue set on submit. +If no rules are defined, [provided rule](placement_rules#provided-rule) will be applied. +Each application *must* have a queue set on submit. ### Limits Limits define a set of limit objects for a queue, and can be set on a queue at any level. diff --git a/versioned_docs/version-1.4.0/user_guide/queue_config.md b/versioned_docs/version-1.4.0/user_guide/queue_config.md index ca1712010c..f8584d0b02 100644 --- a/versioned_docs/version-1.4.0/user_guide/queue_config.md +++ b/versioned_docs/version-1.4.0/user_guide/queue_config.md @@ -202,7 +202,8 @@ partitions: The placement rules are defined and documented in the [placement rule](placement_rules.md) document. Each partition can have only one set of placement rules defined. -If no rules are defined the placement manager is not started and each application *must* have a queue set on submit. +If no rules are defined, [provided rule](placement_rules#provided-rule) will be applied. +Each application *must* have a queue set on submit. ### Statedump filepath diff --git a/versioned_docs/version-1.5.0/user_guide/queue_config.md b/versioned_docs/version-1.5.0/user_guide/queue_config.md index 75bd24133a..05594e0144 100644 --- a/versioned_docs/version-1.5.0/user_guide/queue_config.md +++ b/versioned_docs/version-1.5.0/user_guide/queue_config.md @@ -201,7 +201,8 @@ partitions: The placement rules are defined and documented in the [placement rule](placement_rules.md) document. Each partition can have only one set of placement rules defined. -If no rules are defined the placement manager is not started and each application *must* have a queue set on submit. +If no rules are defined, [provided rule](placement_rules#provided-rule) will be applied. +Each application *must* have a queue set on submit. ### Limits Limits define a set of limit objects for a queue, and can be set on a queue at any level. - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-2596] layout update for release announce page (#425)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new c9a51d470a [YUNIKORN-2596] layout update for release announce page
(#425)
c9a51d470a is described below
commit c9a51d470a2fd880f138199d3796c0ecd333cb53
Author: Wilfred Spiegelenburg
AuthorDate: Tue May 7 12:21:09 2024 +1000
[YUNIKORN-2596] layout update for release announce page (#425)
Add blank lines to separate content
Change to a bullet list for all release announcements
Closes: #425
Signed-off-by: Wilfred Spiegelenburg
---
src/pages/release-announce/index.js | 9 ++---
src/pages/styles.module.css | 9 -
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/src/pages/release-announce/index.js
b/src/pages/release-announce/index.js
index 44370c28b5..6bf73e10fa 100644
--- a/src/pages/release-announce/index.js
+++ b/src/pages/release-announce/index.js
@@ -31,21 +31,24 @@ const LinkPage = () => {
return (
+
-Release Announcements
+Release Announcements
List of all release announcements for Apache YuniKorn.
Sorted by version, not by release date.
+
{releaseNames.map((release) => {
-const name = release.replace("./","").replace(".md",
"");
+const name = release.replace("./", "").replace(".md",
"");
const link = `/release-announce/${name}`;
return (
-
+
Release v{name}
)
})}
+
diff --git a/src/pages/styles.module.css b/src/pages/styles.module.css
index c2e8b9a559..b77eba0db8 100644
--- a/src/pages/styles.module.css
+++ b/src/pages/styles.module.css
@@ -60,7 +60,7 @@
}
.postContainer {
- alignment: center;
+ align-content: center;
min-width: 300px;
max-width: 800px;
margin: 0 20%;
@@ -98,3 +98,10 @@
color: cadetblue;
margin-bottom: 30px;
}
+
+.indentReleases {
+ display: list-item;
+ list-style-type: square;
+ list-style-position: inside;
+ margin-left: 10px
+}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-2595] use cdn for source download (#424)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 86fe2349af [YUNIKORN-2595] use cdn for source download (#424) 86fe2349af is described below commit 86fe2349afa098892cbe4b492826b5a0f5c8ce4e Author: Wilfred Spiegelenburg AuthorDate: Tue May 7 12:20:43 2024 +1000 [YUNIKORN-2595] use cdn for source download (#424) Add closer.lua to first src link, remove the 1.2.0 release Fix IDE warnings on download page. Closes: #424 Signed-off-by: Wilfred Spiegelenburg --- src/pages/community/download.md | 13 ++--- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/src/pages/community/download.md b/src/pages/community/download.md index de81b23106..e3ae0259e3 100644 --- a/src/pages/community/download.md +++ b/src/pages/community/download.md @@ -33,12 +33,11 @@ We publish prebuilt docker images for everyone's convenience. The latest release of Apache YuniKorn is v1.5.0. -| Version | Release date | Source download | Docker images | Release notes | -|-|--|---|-|-| -| 1.5.0 | 2024-03-14 | [Download](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz)[Checksum](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.sha512) & [Signature](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.5.0/images/sha256-9cefd0df164b9c4d39f9e10b010eaf7d8f89b130de1648e94f75b9b95d300a00?context=explore) [...] -| v1.4.0 | 2023-11-20 | [Download](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz)[Checksum](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.4.0/images/sha256-d013be8e3ad7eb8e51ce23951e6899a4b74088e52c3767f3fcc7efcdcc0904f5?context=explore) [...] -| v1.3.0 | 2023-06-12 | [Download](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz)[Checksum](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.3.0/images/sha256-99a1973728c6684b1da7631dbf015daa1dbf519dbab1ffc8b23fccdfa7ffd0c5?context=explore) [...] -| v1.2.0 | 2023-02-02 | [Download](https://archive.apache.org/dist/yunikorn/1.2.0/apache-yunikorn-1.2.0-src.tar.gz)[Checksum](https://archive.apache.org/dist/yunikorn/1.2.0/apache-yunikorn-1.2.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.2.0/apache-yunikorn-1.2.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.2.0/images/sha256-c3b1a7b2cfec3f3560415519278cc4d94748f0f60ee80dfaf23fcc22dbb8b8e5)[admission- [...] +| Version | Release date | Source download
(yunikorn-k8shim) branch master updated: [YUNIKORN-2575] Clarify error returned by IsPodFitNode (#824)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 29981633 [YUNIKORN-2575] Clarify error returned by IsPodFitNode (#824)
29981633 is described below
commit 299816338caa1053e76e8f0d627321335d159cd2
Author: Wilfred Spiegelenburg
AuthorDate: Wed Apr 24 10:42:01 2024 +1000
[YUNIKORN-2575] Clarify error returned by IsPodFitNode (#824)
IsPodFitNode should clearly set the error if the pod or node is missing
from the cache when run. The result, if it is an error, contains the
predicate plugin name that failed to improve identification in the
events.
Closes: #824
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/context.go | 35 ---
1 file changed, 24 insertions(+), 11 deletions(-)
diff --git a/pkg/cache/context.go b/pkg/cache/context.go
index c805f0ba..6c9f34e2 100644
--- a/pkg/cache/context.go
+++ b/pkg/cache/context.go
@@ -21,6 +21,7 @@ package cache
import (
"context"
"encoding/json"
+ "errors"
"fmt"
"sort"
"strconv"
@@ -56,6 +57,11 @@ import (
const registerNodeContextHandler = "RegisterNodeContextHandler"
+var (
+ ErrorPodNotFound = errors.New("predicates were not run because pod was
not found in cache")
+ ErrorNodeNotFound = errors.New("predicates were not run because node
was not found in cache")
+)
+
// context maintains scheduling state, like apps and apps' tasks.
type Context struct {
applications map[string]*Application// apps
@@ -633,21 +639,28 @@ func (ctx *Context) EventsToRegister(queueingHintFn
framework.QueueingHintFn) []
return ctx.predManager.EventsToRegister(queueingHintFn)
}
-// evaluate given predicates based on current context
+// IsPodFitNode evaluates given predicates based on current context
func (ctx *Context) IsPodFitNode(name, node string, allocate bool) error {
ctx.lock.RLock()
defer ctx.lock.RUnlock()
- if pod, ok := ctx.schedulerCache.GetPod(name); ok {
- // if pod exists in cache, try to run predicates
- if targetNode := ctx.schedulerCache.GetNode(node); targetNode
!= nil {
- // need to lock cache here as predicates need a stable
view into the cache
- ctx.schedulerCache.LockForReads()
- defer ctx.schedulerCache.UnlockForReads()
- _, err := ctx.predManager.Predicates(pod, targetNode,
allocate)
- return err
- }
+ var pod *v1.Pod
+ var ok bool
+ if pod, ok = ctx.schedulerCache.GetPod(name); !ok {
+ return ErrorPodNotFound
+ }
+ // if pod exists in cache, try to run predicates
+ targetNode := ctx.schedulerCache.GetNode(node)
+ if targetNode == nil {
+ return ErrorNodeNotFound
+ }
+ // need to lock cache here as predicates need a stable view into the
cache
+ ctx.schedulerCache.LockForReads()
+ defer ctx.schedulerCache.UnlockForReads()
+ plugin, err := ctx.predManager.Predicates(pod, targetNode, allocate)
+ if err != nil {
+ err = errors.Join(fmt.Errorf("failed plugin: '%s'", plugin),
err)
}
- return fmt.Errorf("predicates were not running because pod or node was
not found in cache")
+ return err
}
func (ctx *Context) IsPodFitNodeViaPreemption(name, node string, allocations
[]string, startIndex int) (index int, ok bool) {
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2520] PVC errors in AssumePod() are not handled properly (#810)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new b0cfdf83 [YUNIKORN-2520] PVC errors in AssumePod() are not handled
properly (#810)
b0cfdf83 is described below
commit b0cfdf83d414aa66630f241198774abe5d07eb62
Author: Peter Bacsko
AuthorDate: Thu Apr 4 19:46:59 2024 +1100
[YUNIKORN-2520] PVC errors in AssumePod() are not handled properly (#810)
Handle PV/PVC volume errors in Context.AssumePod() properly.
If volume errors occur fail the task as the chance that they will fix
themselves without outside changes is negligible.
Enhance test coverage.
Closes: #810
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/context.go | 72 -
pkg/cache/context_test.go | 147 --
pkg/cache/external/scheduler_cache.go | 8 +-
pkg/cache/scheduler_callback.go | 11 ++-
pkg/cache/task.go | 37 +
pkg/cache/task_state.go | 2 +-
pkg/cache/task_test.go| 41 ++
pkg/client/apifactory_mock.go | 7 +-
pkg/common/si_helper.go | 48 ---
pkg/common/si_helper_test.go | 10 +--
pkg/common/test/volumebinder_mock.go | 108 +
pkg/shim/scheduler_test.go| 46 +++
12 files changed, 435 insertions(+), 102 deletions(-)
diff --git a/pkg/cache/context.go b/pkg/cache/context.go
index 1ad4adf3..8e003336 100644
--- a/pkg/cache/context.go
+++ b/pkg/cache/context.go
@@ -768,7 +768,7 @@ func (ctx *Context) bindPodVolumes(pod *v1.Pod) error {
// be running on it. And we keep this cache in-sync between core and the shim.
// this way, the core can make allocation decisions with consideration of
// other assumed pods before they are actually bound to the node (bound is
slow).
-func (ctx *Context) AssumePod(name string, node string) error {
+func (ctx *Context) AssumePod(name, node string) error {
ctx.lock.Lock()
defer ctx.lock.Unlock()
if pod, ok := ctx.schedulerCache.GetPod(name); ok {
@@ -781,45 +781,43 @@ func (ctx *Context) AssumePod(name string, node string)
error {
// assume pod volumes before assuming the pod
// this will update scheduler cache with essential
PV/PVC binding info
var allBound = true
- // volume builder might be null in UTs
- if ctx.apiProvider.GetAPIs().VolumeBinder != nil {
- var err error
- // retrieve the volume claims
- podVolumeClaims, err :=
ctx.apiProvider.GetAPIs().VolumeBinder.GetPodVolumeClaims(ctx.klogger, pod)
- if err != nil {
- log.Log(log.ShimContext).Error("Failed
to get pod volume claims",
- zap.String("podName",
assumedPod.Name),
- zap.Error(err))
- return err
- }
+ var err error
+ // retrieve the volume claims
+ podVolumeClaims, err :=
ctx.apiProvider.GetAPIs().VolumeBinder.GetPodVolumeClaims(ctx.klogger, pod)
+ if err != nil {
+ log.Log(log.ShimContext).Error("Failed to get
pod volume claims",
+ zap.String("podName", assumedPod.Name),
+ zap.Error(err))
+ return err
+ }
- // retrieve volumes
- volumes, reasons, err :=
ctx.apiProvider.GetAPIs().VolumeBinder.FindPodVolumes(ctx.klogger, pod,
podVolumeClaims, targetNode.Node())
- if err != nil {
- log.Log(log.ShimContext).Error("Failed
to find pod volumes",
- zap.String("podName",
assumedPod.Name),
- zap.String("nodeName",
assumedPod.Spec.NodeName),
- zap.Error(err))
- return err
- }
- if len(reasons) > 0 {
- sReasons := make([]string, 0)
- for _, reason := range reasons {
-
(yunikorn-core) branch master updated: [YUNIKORN-2519] Remove bypass ACL check from placement rules (#829)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 5716f462 [YUNIKORN-2519] Remove bypass ACL check from placement rules
(#829)
5716f462 is described below
commit 5716f4627dc948182a5268c7c12974421cefa571
Author: Wilfred Spiegelenburg
AuthorDate: Wed Apr 3 22:41:59 2024 +1100
[YUNIKORN-2519] Remove bypass ACL check from placement rules (#829)
Instead of returning a flag to not bypass the ACL check by all rules
except for the recovery rule special case the recovery rule to bypass
checks.
Change logger handler to SchedApplication for placements
Closes: #829
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/partition.go| 14 ++-
pkg/scheduler/placement/fixed_rule.go | 37
pkg/scheduler/placement/fixed_rule_test.go| 29 ++
pkg/scheduler/placement/placement.go | 121 ++
pkg/scheduler/placement/placement_test.go | 87 ++
pkg/scheduler/placement/provided_rule.go | 26 +++---
pkg/scheduler/placement/provided_rule_test.go | 32 +++
pkg/scheduler/placement/recovery_rule.go | 16 ++--
pkg/scheduler/placement/recovery_rule_test.go | 7 +-
pkg/scheduler/placement/rule.go | 3 +-
pkg/scheduler/placement/rule_test.go | 13 +--
pkg/scheduler/placement/tag_rule.go | 26 +++---
pkg/scheduler/placement/tag_rule_test.go | 35 +++-
pkg/scheduler/placement/testrule.go | 8 +-
pkg/scheduler/placement/user_rule.go | 24 ++---
pkg/scheduler/placement/user_rule_test.go | 29 ++
16 files changed, 274 insertions(+), 233 deletions(-)
diff --git a/pkg/scheduler/partition.go b/pkg/scheduler/partition.go
index 26ae861c..207a0ad5 100644
--- a/pkg/scheduler/partition.go
+++ b/pkg/scheduler/partition.go
@@ -289,7 +289,9 @@ func (pc *PartitionContext) getPlacementManager()
*placement.AppPlacementManager
return pc.placementManager
}
-// Add a new application to the partition.
+// AddApplication adds a new application to the partition.
+// Runs the placement rules for the queue resolution. Creates a new dynamic
queue if the queue does not yet
+// exists.
// NOTE: this is a lock free call. It must NOT be called holding the
PartitionContext lock.
func (pc *PartitionContext) AddApplication(app *objects.Application) error {
if pc.isDraining() || pc.isStopped() {
@@ -302,16 +304,13 @@ func (pc *PartitionContext) AddApplication(app
*objects.Application) error {
return fmt.Errorf("adding application %s to partition %s, but
application already existed", appID, pc.Name)
}
- // Put app under the queue
- pm := pc.getPlacementManager()
- err := pm.PlaceApplication(app)
+ // Resolve the queue for this app using the placement rules
+ // We either have an error or a queue name is set on the application.
+ err := pc.getPlacementManager().PlaceApplication(app)
if err != nil {
return fmt.Errorf("failed to place application %s: %v", appID,
err)
}
queueName := app.GetQueuePath()
- if queueName == "" {
- return fmt.Errorf("application rejected by placement rules:
%s", appID)
- }
// lock the partition and make the last change: we need to do this
before creating the queues.
// queue cleanup might otherwise remove the queue again before we can
add the application
@@ -322,7 +321,6 @@ func (pc *PartitionContext) AddApplication(app
*objects.Application) error {
// create the queue if necessary
if queue == nil {
- var err error
if common.IsRecoveryQueue(queueName) {
queue, err = pc.createRecoveryQueue()
if err != nil {
diff --git a/pkg/scheduler/placement/fixed_rule.go
b/pkg/scheduler/placement/fixed_rule.go
index 0dce3847..b62abd51 100644
--- a/pkg/scheduler/placement/fixed_rule.go
+++ b/pkg/scheduler/placement/fixed_rule.go
@@ -30,16 +30,16 @@ import (
"github.com/apache/yunikorn-core/pkg/scheduler/placement/types"
)
+// A rule to place an application based on the queue in the configuration.
+// If the queue provided is fully qualified, starts with "root.", the parent
rule is skipped and the queue is created as
+// configured. If the queue is not qualified all "." characters will be
replaced and the parent rule run before making
+// the queue name fully qualified.
type fixedRule struct {
basicRule
queue string
qualified bool
}
-// A rule to place an application based on the queue in the configuration.
-// If the queue provide
(yunikorn-core) branch master updated: [YUNIKORN-2527] Reset state of configured queue on re-add (#831)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 5ddcbbbd [YUNIKORN-2527] Reset state of configured queue on re-add
(#831)
5ddcbbbd is described below
commit 5ddcbbbddd38019f47e22456b1c6a797edd7fe73
Author: Wilfred Spiegelenburg
AuthorDate: Wed Apr 3 22:39:49 2024 +1100
[YUNIKORN-2527] Reset state of configured queue on re-add (#831)
When a queue is removed and re-added the configured queue status is not
reset. If that happens within the cleanup cycle, the cleanup still gets
triggered. The queue is removed causing an inconsistency between the
config and the queues available in the core.
The object state needs to allow for moving from Draining back to Active.
Updated tests to cover the new state transition.
Closes: #831
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/object_state.go | 2 +-
pkg/scheduler/objects/object_state_test.go | 10 ++---
pkg/scheduler/objects/queue.go | 8
pkg/scheduler/objects/queue_test.go| 33 +++-
pkg/scheduler/partition_test.go| 61 ++
5 files changed, 106 insertions(+), 8 deletions(-)
diff --git a/pkg/scheduler/objects/object_state.go
b/pkg/scheduler/objects/object_state.go
index 7c8a678b..3afbe595 100644
--- a/pkg/scheduler/objects/object_state.go
+++ b/pkg/scheduler/objects/object_state.go
@@ -68,7 +68,7 @@ func NewObjectState() *fsm.FSM {
Dst: Draining.String(),
}, {
Name: Start.String(),
- Src: []string{Active.String(),
Stopped.String()},
+ Src: []string{Active.String(),
Stopped.String(), Draining.String()},
Dst: Active.String(),
}, {
Name: Stop.String(),
diff --git a/pkg/scheduler/objects/object_state_test.go
b/pkg/scheduler/objects/object_state_test.go
index 20cab62d..dbe0180c 100644
--- a/pkg/scheduler/objects/object_state_test.go
+++ b/pkg/scheduler/objects/object_state_test.go
@@ -50,15 +50,15 @@ func TestStateTransition(t *testing.T) {
assert.Assert(t, err == nil)
assert.Equal(t, stateMachine.Current(), Draining.String())
- // start on draining not allowed
- err = stateMachine.Event(context.Background(), Start.String(),
"test_object")
- assert.Assert(t, err != nil)
- assert.Equal(t, stateMachine.Current(), Draining.String())
-
// stop on draining not allowed
err = stateMachine.Event(context.Background(), Stop.String(),
"test_object")
assert.Assert(t, err != nil)
assert.Equal(t, stateMachine.Current(), Draining.String())
+
+ // draining to active
+ err = stateMachine.Event(context.Background(), Start.String(),
"test_object")
+ assert.Assert(t, err == nil)
+ assert.Equal(t, stateMachine.Current(), Active.String())
}
func TestTransitionToSelf(t *testing.T) {
diff --git a/pkg/scheduler/objects/queue.go b/pkg/scheduler/objects/queue.go
index 59a3d371..9f272284 100644
--- a/pkg/scheduler/objects/queue.go
+++ b/pkg/scheduler/objects/queue.go
@@ -328,6 +328,14 @@ func (sq *Queue) applyConf(conf configs.QueueConfig) error
{
sq.isManaged = true
}
+ // if the queue is marked for removal reverse that state
+ if !sq.IsRunning() {
+ err = sq.handleQueueEvent(Start)
+ if err != nil {
+ log.Log(log.SchedQueue).Info("managed queue state
change failed",
+ zap.String("queue", sq.QueuePath))
+ }
+ }
prevLeaf := sq.isLeaf
sq.isLeaf = !conf.Parent
// Make sure the parent flag is set correctly: config might expect auto
parent type creation
diff --git a/pkg/scheduler/objects/queue_test.go
b/pkg/scheduler/objects/queue_test.go
index ff48eadd..f9111d65 100644
--- a/pkg/scheduler/objects/queue_test.go
+++ b/pkg/scheduler/objects/queue_test.go
@@ -632,8 +632,8 @@ func TestQueueStates(t *testing.T) {
t.Errorf("leaf queue is not marked draining: %v", err)
}
err = leaf.handleQueueEvent(Start)
- if err == nil || !leaf.IsDraining() {
- t.Errorf("leaf queue changed state which should not happen:
%v", err)
+ if err != nil || !leaf.IsRunning() {
+ t.Errorf("leaf queue is not marked running: %v", err)
}
}
@@ -2210,6 +2210,35 @@ func TestNewConfiguredQueue(t *testing.T) {
assert.Assert(t, childNonLeaf.maxResource == nil)
}
+func TestResetRunningState(t *testin
(yunikorn-k8shim) branch master updated: [YUNIKORN-2498] Implement force create flag for recovery queue (#807)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new aa20c7b0 [YUNIKORN-2498] Implement force create flag for recovery
queue (#807)
aa20c7b0 is described below
commit aa20c7b0cfc4cede2f34bf7f6c77b3e562e6028e
Author: Wilfred Spiegelenburg
AuthorDate: Tue Apr 2 16:06:29 2024 +1100
[YUNIKORN-2498] Implement force create flag for recovery queue (#807)
To place already running allocations on startup the core
supports a force create flag for an application. The flag wil place the
application into the recovery queue if all other placements fail.
The shim sets the flag in the metadata if the pod has a nodename set
when the metadata is created.
Closes: #807
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/metadata.go | 15 ++
pkg/cache/metadata_test.go | 28
test/e2e/framework/configmanager/constants.go | 1 +
.../framework/helpers/yunikorn/rest_api_utils.go | 105 ++--
.../restart_changed_config_suite_test.go | 50 ++
.../restart_changed_config_test.go | 176 +
6 files changed, 330 insertions(+), 45 deletions(-)
diff --git a/pkg/cache/metadata.go b/pkg/cache/metadata.go
index b11a42d6..1cbe4e30 100644
--- a/pkg/cache/metadata.go
+++ b/pkg/cache/metadata.go
@@ -31,6 +31,7 @@ import (
"github.com/apache/yunikorn-k8shim/pkg/common/utils"
"github.com/apache/yunikorn-k8shim/pkg/conf"
"github.com/apache/yunikorn-k8shim/pkg/log"
+ "github.com/apache/yunikorn-scheduler-interface/lib/go/common"
)
func getTaskMetadata(pod *v1.Pod) (TaskMetadata, bool) {
@@ -77,6 +78,20 @@ func getAppMetadata(pod *v1.Pod) (ApplicationMetadata, bool)
{
tags[constants.AppTagNamespace] = pod.Namespace
}
+ // Make sure we set the force create flag to true if the pod has been
scheduled already.
+ // When we create and link the metadata to an application the
application does not exist yet.
+ // The force flag prevents rejections during initialisation of already
allocated pods in a changed
+ // queue configuration.
+ // It will also pick up static (mirror) and DaemonSet pods. In certain
circumstances this could cause
+ // pods to be allowed into the recovery queue while they should not. If
this becomes an issue we can
+ // add a filter here.
+ // NOTE: this could fail to set the flag if the oldest pod for the
application is not scheduled and
+ // later pods are.
+ tags[common.AppTagCreateForce] = constants.False
+ if utils.IsAssignedPod(pod) {
+ tags[common.AppTagCreateForce] = constants.True
+ }
+
// attach imagePullSecrets if present
secrets := pod.Spec.ImagePullSecrets
if len(secrets) > 0 {
diff --git a/pkg/cache/metadata_test.go b/pkg/cache/metadata_test.go
index d26138b5..755ff65d 100644
--- a/pkg/cache/metadata_test.go
+++ b/pkg/cache/metadata_test.go
@@ -29,6 +29,7 @@ import (
"github.com/apache/yunikorn-k8shim/pkg/common/constants"
"github.com/apache/yunikorn-k8shim/pkg/common/utils"
"github.com/apache/yunikorn-k8shim/pkg/conf"
+ "github.com/apache/yunikorn-scheduler-interface/lib/go/common"
)
const taskGroupInfo = `
@@ -141,6 +142,7 @@ func TestGetAppMetadata(t *testing.T) { //nolint:funlen
assert.Equal(t, app.Tags["namespace"], "default")
assert.Equal(t, app.Tags[constants.AnnotationSchedulingPolicyParam],
"gangSchedulingStyle=Soft")
assert.Equal(t, app.Tags[constants.AppTagImagePullSecrets],
"secret1,secret2")
+ assert.Equal(t, app.Tags[common.AppTagCreateForce], "false")
assert.Assert(t, app.Tags[constants.AnnotationTaskGroups] != "")
assert.Equal(t, app.TaskGroups[0].Name, "test-group-1")
assert.Equal(t, app.TaskGroups[0].MinMember, int32(3))
@@ -180,6 +182,7 @@ func TestGetAppMetadata(t *testing.T) { //nolint:funlen
assert.Equal(t, app.QueueName, "root.b")
assert.Equal(t, app.User, constants.DefaultUser)
assert.Equal(t, app.Tags["namespace"], "app-namespace-01")
+ assert.Equal(t, app.Tags[common.AppTagCreateForce], "false")
assert.Equal(t, len(app.TaskGroups), 0)
assert.Equal(t,
app.SchedulingPolicyParameters.GetGangSchedulingStyle(), "Hard")
@@ -209,6 +212,7 @@ func TestGetAppMetadata(t *testing.T) { //nolint:funlen
app, ok = getAppMetadata(&pod)
assert.Equal(t, ok, true)
assert.Equal(t,
app.SchedulingPol
(yunikorn-core) branch master updated: [YUNIKORN-2494] Add new preemption calculations (#822)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new b07f2646 [YUNIKORN-2494] Add new preemption calculations (#822)
b07f2646 is described below
commit b07f264676cc675983a75495c67657f7a1b207fc
Author: Manikandan R
AuthorDate: Thu Mar 28 18:07:14 2024 +1100
[YUNIKORN-2494] Add new preemption calculations (#822)
As part of preemption hardening change the functions to calculate the
amount of resources that can be preempted and are remaining in the
guaranteed resources for a queue.
Only adds the new functions and testing integration will follow in
follow up PRs
Closes: #822
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/preemption.go| 57 +++
pkg/scheduler/objects/preemption_queue_test.go | 219 +
2 files changed, 276 insertions(+)
diff --git a/pkg/scheduler/objects/preemption.go
b/pkg/scheduler/objects/preemption.go
index 14919274..a9ca050a 100644
--- a/pkg/scheduler/objects/preemption.go
+++ b/pkg/scheduler/objects/preemption.go
@@ -749,6 +749,63 @@ func (qps *QueuePreemptionSnapshot)
GetRemainingGuaranteed() *resources.Resource
return resources.ComponentWiseMin(remaining, parentResult)
}
+func (qps *QueuePreemptionSnapshot) GetPreemptableResource()
*resources.Resource {
+ if qps == nil {
+ return nil
+ }
+ parentPreemptableResource := qps.Parent.GetPreemptableResource()
+ actual := qps.AllocatedResource.Clone()
+
+ // No usage, so nothing to preempt
+ if actual.IsEmpty() {
+ return nil
+ }
+ actual.SubOnlyExisting(qps.PreemptingResource)
+
+ // Calculate preemptable resource. +ve means Over utilized, -ve means
Under utilized, 0 means correct utilization
+ guaranteed := qps.GuaranteedResource
+ actual.SubOnlyExisting(guaranteed)
+ preemptableResource := actual
+
+ // Keep only the resource type which needs to be preempted
+ for k, v := range actual.Resources {
+ // Under-utilized or completely used resource types
+ if v <= 0 {
+ delete(preemptableResource.Resources, k)
+ } else { // Over utilized resource types
+ preemptableResource.Resources[k] = v
+ }
+ }
+ // When nothing to preempt or usage equals guaranteed in current queue,
return as is.
+ // Otherwise, doing min calculation with parent level (for a different
res types) would lead to a wrong perception
+ // of choosing this current queue to select the victims when that is
not the fact.
+ // As you move down the hierarchy, results calculated at lower level
has higher precedence.
+ if preemptableResource.IsEmpty() {
+ return preemptableResource
+ }
+ return resources.ComponentWiseMinPermissive(preemptableResource,
parentPreemptableResource)
+}
+
+func (qps *QueuePreemptionSnapshot) GetRemainingGuaranteedResource()
*resources.Resource {
+ if qps == nil {
+ return nil
+ }
+ parent := qps.Parent.GetRemainingGuaranteedResource()
+ remainingGuaranteed := qps.GuaranteedResource.Clone()
+
+ // No Guaranteed set, so nothing remaining
+ // In case of guaranteed not set for queues at specific level, inherits
the same from parent queue.
+ // If the parent too (or ancestors all the way upto root) doesn't have
guaranteed set, then nil is returned.
+ // Otherwise, parent's guaranteed (or ancestors) would be used.
+ if parent.IsEmpty() && remainingGuaranteed.IsEmpty() {
+ return nil
+ }
+ used := qps.AllocatedResource.Clone()
+ used.SubOnlyExisting(qps.PreemptingResource)
+ remainingGuaranteed.SubOnlyExisting(used)
+ return resources.ComponentWiseMinPermissive(remainingGuaranteed, parent)
+}
+
// GetGuaranteedResource computes the current guaranteed resources considering
parent guaranteed
func (qps *QueuePreemptionSnapshot) GetGuaranteedResource()
*resources.Resource {
if qps == nil {
diff --git a/pkg/scheduler/objects/preemption_queue_test.go
b/pkg/scheduler/objects/preemption_queue_test.go
new file mode 100644
index ..6ce96f5e
--- /dev/null
+++ b/pkg/scheduler/objects/preemption_queue_test.go
@@ -0,0 +1,219 @@
+/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a co
(yunikorn-site) branch master updated: [YUNIKORN-2490] update PMC and committers ADDENDUM
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 1f2b9db5da [YUNIKORN-2490] update PMC and committers ADDENDUM 1f2b9db5da is described below commit 1f2b9db5da0e387c4abe621ba063f03db52f0447 Author: Wilfred Spiegelenburg AuthorDate: Fri Mar 15 14:19:42 2024 +1100 [YUNIKORN-2490] update PMC and committers ADDENDUM --- src/pages/community/people.md | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/pages/community/people.md b/src/pages/community/people.md index 287cce17d8..d6ec35883f 100644 --- a/src/pages/community/people.md +++ b/src/pages/community/people.md @@ -67,16 +67,16 @@ members and [committers](https://infra.apache.org/new-committers-guide.html#what > The listing below excludes the PMC members -| Name | ApacheID | Organization | Timezone (UTC) | -|---|---|--|-| -| Hsuan Zong Wu | samhxwu | | +8 | -| Kuan-Po Tseng | brandboat | SUSE | +8 | -| Li Gao| ligao | Databricks | -8 | -| PoAn Yang | payang| | +8 | -| Qi Zhu| zhuqi | Cloudera | +8 | -| Ryan Lo | wenchih | Fazz | +8 | -| Ted Lin | steinsgateted | NCKU | +8 | -| Yu-Lin Chen | chenyulin0719 | NCKU | +8 | +| Name | ApacheID | Organization | Timezone (UTC) | +|---|---||-| +| Hsuan Zong Wu | samhxwu || +8 | +| Kuan-Po Tseng | brandboat | SUSE | +8 | +| Li Gao| ligao | Databricks | -8 | +| PoAn Yang | payang|| +8 | +| Qi Zhu| zhuqi | Cloudera | +8 | +| Ryan Lo | wenchih | Fazz | +8 | +| Ted Lin | steinsgateted | NCKU | +8 | +| Yu-Lin Chen | chenyulin0719 | opensource4you | +8 | ## Become a Committer - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-2490] update PMC and committers (#410)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 05051d6479 [YUNIKORN-2490] update PMC and committers (#410) 05051d6479 is described below commit 05051d64795b720b9b1bf60c6454f578a273e9bb Author: Wilfred Spiegelenburg AuthorDate: Fri Mar 15 14:16:18 2024 +1100 [YUNIKORN-2490] update PMC and committers (#410) Closes: #410 Signed-off-by: Wilfred Spiegelenburg --- src/pages/community/people.md | 19 +++ 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/src/pages/community/people.md b/src/pages/community/people.md index b3dd0e2e29..287cce17d8 100644 --- a/src/pages/community/people.md +++ b/src/pages/community/people.md @@ -38,6 +38,7 @@ members and [committers](https://infra.apache.org/new-committers-guide.html#what | Carlo Curino| curino | Microsoft| -8 | | Chaoran Yu | yuchaoran | Apple| -8 | | Chenya Zhang| chenya | Apple| -8 | +| Chia-Ping Tsai | chia7712| Alpha Link | +8 | | Craig Condit| ccondit | Cloudera | -6 | | DB Tsai | dbtsai | Apple| -8 | | Felix Cheung| felixcheung | SafeGraph| -8 | @@ -66,14 +67,16 @@ members and [committers](https://infra.apache.org/new-committers-guide.html#what > The listing below excludes the PMC members -| Name | ApacheID | Organization | Timezone (UTC) | -||---|--|-| -| Chia-Ping Tsai | chia7712 | Is-Land | +8 | -| Li Gao | ligao | Databricks | -8 | -| PoAn Yang | payang| | +8 | -| Hsuan Zong Wu | samhxwu | | +8 | -| Ted Lin| steinsgateted | NCKU | +8 | -| Qi Zhu | zhuqi | Cloudera | +8 | +| Name | ApacheID | Organization | Timezone (UTC) | +|---|---|--|-| +| Hsuan Zong Wu | samhxwu | | +8 | +| Kuan-Po Tseng | brandboat | SUSE | +8 | +| Li Gao| ligao | Databricks | -8 | +| PoAn Yang | payang| | +8 | +| Qi Zhu| zhuqi | Cloudera | +8 | +| Ryan Lo | wenchih | Fazz | +8 | +| Ted Lin | steinsgateted | NCKU | +8 | +| Yu-Lin Chen | chenyulin0719 | NCKU | +8 | ## Become a Committer - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch gh-pages updated: Update index.yaml: fix time offset for 1.5.0
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/gh-pages by this push: new 998275a Update index.yaml: fix time offset for 1.5.0 998275a is described below commit 998275a81500d5735bb546bf3137c40157f2da48 Author: Wilfred Spiegelenburg AuthorDate: Fri Mar 15 02:21:05 2024 +1100 Update index.yaml: fix time offset for 1.5.0 --- index.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.yaml b/index.yaml index a59ef9b..3ba6972 100644 --- a/index.yaml +++ b/index.yaml @@ -230,7 +230,7 @@ entries: version: 1.4.0 - apiVersion: v1 appVersion: 1.5.0 - created: "2024-03-14T14:00:00.00+8:00" + created: "2024-03-14T14:00:00.00+08:00" description: YuniKorn scheduler for Kubernetes digest: 251c4c7375d20efd66870813ac557cac3a7f775c58e7ec45a4faedfb401adff2 home: https://yunikorn.apache.org @@ -247,4 +247,4 @@ entries: urls: - https://github.com/apache/yunikorn-release/releases/download/v1.5.0/yunikorn-1.5.0.tgz version: 1.5.0 -generated: "2024-03-14T14:00:00.00+8:00" +generated: "2024-03-14T14:00:00.00+08:00" - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-scheduler-interface) branch master updated: [YUNIKORN-2488] SI: Remove stateaware constants (#128)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-scheduler-interface.git The following commit(s) were added to refs/heads/master by this push: new 658196f [YUNIKORN-2488] SI: Remove stateaware constants (#128) 658196f is described below commit 658196f3b2bbdbb4a2e97ee7eae97268b04c735b Author: Craig Condit AuthorDate: Thu Mar 14 13:58:38 2024 +1100 [YUNIKORN-2488] SI: Remove stateaware constants (#128) Stateaware was deprecated in 1.5 and removed in 1.6. Use FIFO with max applications for a similar behaviour. Closes: #128 Signed-off-by: Wilfred Spiegelenburg --- lib/go/common/constants.go | 1 - scheduler-interface-spec.md | 1 - 2 files changed, 2 deletions(-) diff --git a/lib/go/common/constants.go b/lib/go/common/constants.go index a8898ef..c28b8f9 100644 --- a/lib/go/common/constants.go +++ b/lib/go/common/constants.go @@ -63,7 +63,6 @@ const ( AppTagNamespaceResourceQuota = "namespace.resourcequota" AppTagNamespaceResourceGuaranteed = "namespace.resourceguaranteed" AppTagNamespaceResourceMaxApps= "namespace.resourcemaxapps" - AppTagStateAwareDisable = "application.stateaware.disable" AppTagCreateForce = "application.create.force" NodeReadyAttribute= "ready" ) diff --git a/scheduler-interface-spec.md b/scheduler-interface-spec.md index cd899fa..58cd721 100644 --- a/scheduler-interface-spec.md +++ b/scheduler-interface-spec.md @@ -772,7 +772,6 @@ const ( AppTagNamespaceResourceQuota = "namespace.resourcequota" AppTagNamespaceResourceGuaranteed = "namespace.resourceguaranteed" AppTagNamespaceResourceMaxApps= "namespace.resourcemaxapps" - AppTagStateAwareDisable = "application.stateaware.disable" AppTagCreateForce = "application.create.force" NodeReadyAttribute= "ready" ) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch branch-1.5 updated: [YUNIKORN-2456] Remove weak ciphers (#795)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch branch-1.5
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/branch-1.5 by this push:
new 83583cb5 [YUNIKORN-2456] Remove weak ciphers (#795)
83583cb5 is described below
commit 83583cb506d19d78eebb3433f30cd77536e3c70f
Author: Wilfred Spiegelenburg
AuthorDate: Tue Feb 27 21:09:52 2024 +1100
[YUNIKORN-2456] Remove weak ciphers (#795)
Set limited ciphers on TLS connections, removing weak ciphers.
Based on the list maintained in the go standard TLS library.
Closes: #795
Signed-off-by: Wilfred Spiegelenburg
(cherry picked from commit 288661cb3f4628832949ad780353357d17843f49)
---
pkg/cmd/admissioncontroller/main.go | 20 +---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/pkg/cmd/admissioncontroller/main.go
b/pkg/cmd/admissioncontroller/main.go
index 851ad036..e96dd1b0 100644
--- a/pkg/cmd/admissioncontroller/main.go
+++ b/pkg/cmd/admissioncontroller/main.go
@@ -21,6 +21,7 @@ package main
import (
"context"
"crypto/tls"
+ "errors"
"fmt"
"net/http"
"os"
@@ -142,14 +143,17 @@ func (wh *WebHook) Startup(certs *tls.Certificate) {
wh.server = &http.Server{
Addr: fmt.Sprintf(":%v", wh.port),
TLSConfig: &tls.Config{
- MinVersion: tls.VersionTLS12,
- Certificates: []tls.Certificate{*certs}},
+ MinVersion: tls.VersionTLS12, // No SSL,
TLS 1.0 or TLS 1.1 support
+ NextProtos: []string{"h2", "http/1.1"}, // prefer
HTTP/2 over HTTP/1.1
+ CipherSuites: wh.getCipherSuites(), // limit
cipher suite to secure ones
+ Certificates: []tls.Certificate{*certs},
+ },
Handler: mux,
}
go func() {
if err := wh.server.ListenAndServeTLS("", ""); err != nil {
- if err == http.ErrServerClosed {
+ if errors.Is(err, http.ErrServerClosed) {
log.Log(log.Admission).Info("existing server
closed")
} else {
log.Log(log.Admission).Fatal("failed to start
admission controller", zap.Error(err))
@@ -175,3 +179,13 @@ func (wh *WebHook) Shutdown() {
wh.server = nil
}
}
+
+// getCipherSuites returns the IDs of the currently considered secure ciphers.
+// Order of choice is defined in the cipherSuitesPreferenceOrder
+func (wh *WebHook) getCipherSuites() []uint16 {
+ var ids []uint16
+ for _, cs := range tls.CipherSuites() {
+ ids = append(ids, cs.ID)
+ }
+ return ids
+}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2456] Remove weak ciphers (#795)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 288661cb [YUNIKORN-2456] Remove weak ciphers (#795)
288661cb is described below
commit 288661cb3f4628832949ad780353357d17843f49
Author: Wilfred Spiegelenburg
AuthorDate: Tue Feb 27 21:09:52 2024 +1100
[YUNIKORN-2456] Remove weak ciphers (#795)
Set limited ciphers on TLS connections, removing weak ciphers.
Based on the list maintained in the go standard TLS library.
Closes: #795
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cmd/admissioncontroller/main.go | 20 +---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/pkg/cmd/admissioncontroller/main.go
b/pkg/cmd/admissioncontroller/main.go
index 851ad036..e96dd1b0 100644
--- a/pkg/cmd/admissioncontroller/main.go
+++ b/pkg/cmd/admissioncontroller/main.go
@@ -21,6 +21,7 @@ package main
import (
"context"
"crypto/tls"
+ "errors"
"fmt"
"net/http"
"os"
@@ -142,14 +143,17 @@ func (wh *WebHook) Startup(certs *tls.Certificate) {
wh.server = &http.Server{
Addr: fmt.Sprintf(":%v", wh.port),
TLSConfig: &tls.Config{
- MinVersion: tls.VersionTLS12,
- Certificates: []tls.Certificate{*certs}},
+ MinVersion: tls.VersionTLS12, // No SSL,
TLS 1.0 or TLS 1.1 support
+ NextProtos: []string{"h2", "http/1.1"}, // prefer
HTTP/2 over HTTP/1.1
+ CipherSuites: wh.getCipherSuites(), // limit
cipher suite to secure ones
+ Certificates: []tls.Certificate{*certs},
+ },
Handler: mux,
}
go func() {
if err := wh.server.ListenAndServeTLS("", ""); err != nil {
- if err == http.ErrServerClosed {
+ if errors.Is(err, http.ErrServerClosed) {
log.Log(log.Admission).Info("existing server
closed")
} else {
log.Log(log.Admission).Fatal("failed to start
admission controller", zap.Error(err))
@@ -175,3 +179,13 @@ func (wh *WebHook) Shutdown() {
wh.server = nil
}
}
+
+// getCipherSuites returns the IDs of the currently considered secure ciphers.
+// Order of choice is defined in the cipherSuitesPreferenceOrder
+func (wh *WebHook) getCipherSuites() []uint16 {
+ var ids []uint16
+ for _, cs := range tls.CipherSuites() {
+ ids = append(ids, cs.ID)
+ }
+ return ids
+}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch branch-1.5 updated (3148d2b8 -> 381bc9a8)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch branch-1.5 in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git from 3148d2b8 [YUNIKORN-2413] Initialism fixes for allocationId, clusterId (#806) new d24a25a2 [YUNIKORN-2042] REST API for specific queue (#687) new 381bc9a8 [YUNIKORN-2030] headroom check for reserved allocations (#793) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pkg/scheduler/objects/application.go | 23 pkg/scheduler/objects/application_test.go | 33 pkg/scheduler/objects/queue.go| 16 -- pkg/scheduler/objects/queue_test.go | 20 +++ pkg/scheduler/objects/utilities_test.go | 1 + pkg/scheduler/partition.go| 2 +- pkg/webservice/dao/queue_info.go | 1 + pkg/webservice/handlers.go| 30 +++ pkg/webservice/handlers_test.go | 90 ++- pkg/webservice/routes.go | 6 +++ 10 files changed, 184 insertions(+), 38 deletions(-) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) 01/02: [YUNIKORN-2042] REST API for specific queue (#687)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch branch-1.5
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
commit d24a25a2a3331fef70bfb95dbd6a45b675cf2127
Author: steinsgateted
AuthorDate: Mon Feb 26 20:09:07 2024 +1100
[YUNIKORN-2042] REST API for specific queue (#687)
Expose a REST API for specific queue:
/ws/v1/partition/%s/queue/%s
/ws/v1/partition/%s/queue/%s?aubtree
The call takes one query parameter "subtree" if provided the whole tree
of queues rooted at the level requested in the call will be returned. If
"subtree" is not set only the queue requested will be returned
Closes: #687
Signed-off-by: Wilfred Spiegelenburg
(cherry picked from commit e9805c956bb806437dcf103c0350af3756cc8ba1)
---
pkg/scheduler/objects/queue.go | 16 ---
pkg/scheduler/objects/queue_test.go | 20 -
pkg/scheduler/partition.go | 2 +-
pkg/webservice/dao/queue_info.go| 1 +
pkg/webservice/handlers.go | 30 +
pkg/webservice/handlers_test.go | 90 +++--
pkg/webservice/routes.go| 6 +++
7 files changed, 136 insertions(+), 29 deletions(-)
diff --git a/pkg/scheduler/objects/queue.go b/pkg/scheduler/objects/queue.go
index ebf70e29..43c66e8b 100644
--- a/pkg/scheduler/objects/queue.go
+++ b/pkg/scheduler/objects/queue.go
@@ -624,17 +624,23 @@ func (sq *Queue) CheckAdminAccess(user
security.UserGroup) bool {
}
// GetPartitionQueueDAOInfo returns the queue hierarchy as an object for a
REST call.
-func (sq *Queue) GetPartitionQueueDAOInfo() dao.PartitionQueueDAOInfo {
+// Exclude is true, which means that returns the specified queue object, but
does not return the children of the specified queue.
+func (sq *Queue) GetPartitionQueueDAOInfo(exclude bool)
dao.PartitionQueueDAOInfo {
queueInfo := dao.PartitionQueueDAOInfo{}
- childes := sq.GetCopyOfChildren()
- queueInfo.Children = make([]dao.PartitionQueueDAOInfo, 0, len(childes))
- for _, child := range childes {
- queueInfo.Children = append(queueInfo.Children,
child.GetPartitionQueueDAOInfo())
+ children := sq.GetCopyOfChildren()
+ if !exclude {
+ queueInfo.Children = make([]dao.PartitionQueueDAOInfo, 0,
len(children))
+ for _, child := range children {
+ queueInfo.Children = append(queueInfo.Children,
child.GetPartitionQueueDAOInfo(false))
+ }
}
// we have held the read lock so following method should not take lock
again.
sq.RLock()
defer sq.RUnlock()
+ for _, child := range children {
+ queueInfo.ChildrenNames = append(queueInfo.ChildrenNames,
child.QueuePath)
+ }
queueInfo.QueueName = sq.QueuePath
queueInfo.Status = sq.stateMachine.Current()
queueInfo.PendingResource = sq.pending.DAOMap()
diff --git a/pkg/scheduler/objects/queue_test.go
b/pkg/scheduler/objects/queue_test.go
index 4c9e3303..7c46484e 100644
--- a/pkg/scheduler/objects/queue_test.go
+++ b/pkg/scheduler/objects/queue_test.go
@@ -1685,7 +1685,7 @@ func TestGetPartitionQueueDAOInfo(t *testing.T) {
// test properties
root.properties = getProperties()
- assert.DeepEqual(t, root.properties,
root.GetPartitionQueueDAOInfo().Properties)
+ assert.DeepEqual(t, root.properties,
root.GetPartitionQueueDAOInfo(false).Properties)
// test template
root.template, err = template.FromConf(&configs.ChildTemplate{
@@ -1697,23 +1697,23 @@ func TestGetPartitionQueueDAOInfo(t *testing.T) {
},
})
assert.NilError(t, err)
- assert.Equal(t, root.template.GetMaxApplications(),
root.GetPartitionQueueDAOInfo().TemplateInfo.MaxApplications)
- assert.DeepEqual(t, root.template.GetProperties(),
root.GetPartitionQueueDAOInfo().TemplateInfo.Properties)
- assert.DeepEqual(t, root.template.GetMaxResource().DAOMap(),
root.template.GetMaxResource().DAOMap())
- assert.DeepEqual(t, root.template.GetGuaranteedResource().DAOMap(),
root.template.GetGuaranteedResource().DAOMap())
+ assert.Equal(t, root.template.GetMaxApplications(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.MaxApplications)
+ assert.DeepEqual(t, root.template.GetProperties(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.Properties)
+ assert.DeepEqual(t, root.template.GetMaxResource().DAOMap(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.MaxResource)
+ assert.DeepEqual(t, root.template.GetGuaranteedResource().DAOMap(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.GuaranteedResource)
// test resources
root.maxResource = getResource(t)
root.guaranteedResource = getResource(t)
- assert.DeepEqual(t, root.GetMaxResource().DAOMap(),
root.
(yunikorn-core) 02/02: [YUNIKORN-2030] headroom check for reserved allocations (#793)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch branch-1.5
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
commit 381bc9a8a64d4fac99e37d04de6be4e436adf1a9
Author: Yongjun Zhang
AuthorDate: Mon Feb 26 20:12:25 2024 +1100
[YUNIKORN-2030] headroom check for reserved allocations (#793)
During the allocation cycle for reservations we fall back to trying all
reservations on all nodes for an application. During this fallback a
headroom check was missing which caused queue update failures to be
logged.
The allocation did not succeed but it caused scheduling delays and log
spew.
Closes: #793
Signed-off-by: Wilfred Spiegelenburg
(cherry picked from commit 908d1cb07083b989230a9a18bb0b66180122fb73)
---
pkg/scheduler/objects/application.go | 23 -
pkg/scheduler/objects/application_test.go | 33 +++
pkg/scheduler/objects/utilities_test.go | 1 +
3 files changed, 48 insertions(+), 9 deletions(-)
diff --git a/pkg/scheduler/objects/application.go
b/pkg/scheduler/objects/application.go
index cde6fbd9..a064d321 100644
--- a/pkg/scheduler/objects/application.go
+++ b/pkg/scheduler/objects/application.go
@@ -1223,6 +1223,12 @@ func (sa *Application)
tryPlaceholderAllocate(nodeIterator func() NodeIterator,
return allocResult
}
+// check ask against both user headRoom and queue headRoom
+func (sa *Application) checkHeadRooms(ask *AllocationAsk, userHeadroom
*resources.Resource, headRoom *resources.Resource) bool {
+ // check if this fits in the users' headroom first, if that fits check
the queues' headroom
+ return userHeadroom.FitInMaxUndef(ask.GetAllocatedResource()) &&
headRoom.FitInMaxUndef(ask.GetAllocatedResource())
+}
+
// Try a reserved allocation of an outstanding reservation
func (sa *Application) tryReservedAllocate(headRoom *resources.Resource,
nodeIterator func() NodeIterator) *Allocation {
sa.Lock()
@@ -1250,13 +1256,8 @@ func (sa *Application) tryReservedAllocate(headRoom
*resources.Resource, nodeIte
alloc := newUnreservedAllocation(reserve.nodeID,
unreserveAsk)
return alloc
}
- // check if this fits in the users' headroom first, if that
fits check the queues' headroom
- if !userHeadroom.FitInMaxUndef(ask.GetAllocatedResource()) {
- continue
- }
- // check if this fits in the queue's headroom
- if !headRoom.FitInMaxUndef(ask.GetAllocatedResource()) {
+ if !sa.checkHeadRooms(ask, userHeadroom, headRoom) {
continue
}
@@ -1280,12 +1281,16 @@ func (sa *Application) tryReservedAllocate(headRoom
*resources.Resource, nodeIte
// lets try this on all other nodes
for _, reserve := range sa.reservations {
// Other nodes cannot be tried if the ask has a required node
- if reserve.ask.GetRequiredNode() != "" {
+ ask := reserve.ask
+ if ask.GetRequiredNode() != "" {
continue
}
iterator := nodeIterator()
if iterator != nil {
- alloc := sa.tryNodesNoReserve(reserve.ask, iterator,
reserve.nodeID)
+ if !sa.checkHeadRooms(ask, userHeadroom, headRoom) {
+ continue
+ }
+ alloc := sa.tryNodesNoReserve(ask, iterator,
reserve.nodeID)
// have a candidate return it, including the node that
was reserved
if alloc != nil {
return alloc
@@ -1490,7 +1495,7 @@ func (sa *Application) tryNode(node *Node, ask
*AllocationAsk) *Allocation {
alloc := NewAllocation(node.NodeID, ask)
if node.AddAllocation(alloc) {
if err :=
sa.queue.IncAllocatedResource(alloc.GetAllocatedResource(), false); err != nil {
- log.Log(log.SchedApplication).Warn("queue update failed
unexpectedly",
+ log.Log(log.SchedApplication).DPanic("queue update
failed unexpectedly",
zap.Error(err))
// revert the node update
node.RemoveAllocation(alloc.GetAllocationID())
diff --git a/pkg/scheduler/objects/application_test.go
b/pkg/scheduler/objects/application_test.go
index bc055ee6..04439672 100644
--- a/pkg/scheduler/objects/application_test.go
+++ b/pkg/scheduler/objects/application_test.go
@@ -2601,6 +2601,39 @@ func TestGetRateLimitedAppLog(t *testing.T) {
assert.Check(t, l != nil)
}
+func TestTryAllocateWithReservedHeadRoomChecking(t *testin
(yunikorn-core) branch master updated: [YUNIKORN-2030] headroom check for reserved allocations (#793)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 908d1cb0 [YUNIKORN-2030] headroom check for reserved allocations (#793)
908d1cb0 is described below
commit 908d1cb07083b989230a9a18bb0b66180122fb73
Author: Yongjun Zhang
AuthorDate: Mon Feb 26 20:12:25 2024 +1100
[YUNIKORN-2030] headroom check for reserved allocations (#793)
During the allocation cycle for reservations we fall back to trying all
reservations on all nodes for an application. During this fallback a
headroom check was missing which caused queue update failures to be
logged.
The allocation did not succeed but it caused scheduling delays and log
spew.
Closes: #793
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/application.go | 23 -
pkg/scheduler/objects/application_test.go | 33 +++
pkg/scheduler/objects/utilities_test.go | 1 +
3 files changed, 48 insertions(+), 9 deletions(-)
diff --git a/pkg/scheduler/objects/application.go
b/pkg/scheduler/objects/application.go
index cde6fbd9..a064d321 100644
--- a/pkg/scheduler/objects/application.go
+++ b/pkg/scheduler/objects/application.go
@@ -1223,6 +1223,12 @@ func (sa *Application)
tryPlaceholderAllocate(nodeIterator func() NodeIterator,
return allocResult
}
+// check ask against both user headRoom and queue headRoom
+func (sa *Application) checkHeadRooms(ask *AllocationAsk, userHeadroom
*resources.Resource, headRoom *resources.Resource) bool {
+ // check if this fits in the users' headroom first, if that fits check
the queues' headroom
+ return userHeadroom.FitInMaxUndef(ask.GetAllocatedResource()) &&
headRoom.FitInMaxUndef(ask.GetAllocatedResource())
+}
+
// Try a reserved allocation of an outstanding reservation
func (sa *Application) tryReservedAllocate(headRoom *resources.Resource,
nodeIterator func() NodeIterator) *Allocation {
sa.Lock()
@@ -1250,13 +1256,8 @@ func (sa *Application) tryReservedAllocate(headRoom
*resources.Resource, nodeIte
alloc := newUnreservedAllocation(reserve.nodeID,
unreserveAsk)
return alloc
}
- // check if this fits in the users' headroom first, if that
fits check the queues' headroom
- if !userHeadroom.FitInMaxUndef(ask.GetAllocatedResource()) {
- continue
- }
- // check if this fits in the queue's headroom
- if !headRoom.FitInMaxUndef(ask.GetAllocatedResource()) {
+ if !sa.checkHeadRooms(ask, userHeadroom, headRoom) {
continue
}
@@ -1280,12 +1281,16 @@ func (sa *Application) tryReservedAllocate(headRoom
*resources.Resource, nodeIte
// lets try this on all other nodes
for _, reserve := range sa.reservations {
// Other nodes cannot be tried if the ask has a required node
- if reserve.ask.GetRequiredNode() != "" {
+ ask := reserve.ask
+ if ask.GetRequiredNode() != "" {
continue
}
iterator := nodeIterator()
if iterator != nil {
- alloc := sa.tryNodesNoReserve(reserve.ask, iterator,
reserve.nodeID)
+ if !sa.checkHeadRooms(ask, userHeadroom, headRoom) {
+ continue
+ }
+ alloc := sa.tryNodesNoReserve(ask, iterator,
reserve.nodeID)
// have a candidate return it, including the node that
was reserved
if alloc != nil {
return alloc
@@ -1490,7 +1495,7 @@ func (sa *Application) tryNode(node *Node, ask
*AllocationAsk) *Allocation {
alloc := NewAllocation(node.NodeID, ask)
if node.AddAllocation(alloc) {
if err :=
sa.queue.IncAllocatedResource(alloc.GetAllocatedResource(), false); err != nil {
- log.Log(log.SchedApplication).Warn("queue update failed
unexpectedly",
+ log.Log(log.SchedApplication).DPanic("queue update
failed unexpectedly",
zap.Error(err))
// revert the node update
node.RemoveAllocation(alloc.GetAllocationID())
diff --git a/pkg/scheduler/objects/application_test.go
b/pkg/scheduler/objects/application_test.go
index bc055ee6..04439672 100644
--- a/pkg/scheduler/objects/application_test.go
+++ b/pkg/scheduler/objects/application_test.go
@@ -2601,6 +2601,39 @@ func TestGetRateL
(yunikorn-core) branch master updated: [YUNIKORN-2042] REST API for specific queue (#687)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new e9805c95 [YUNIKORN-2042] REST API for specific queue (#687)
e9805c95 is described below
commit e9805c956bb806437dcf103c0350af3756cc8ba1
Author: steinsgateted
AuthorDate: Mon Feb 26 20:09:07 2024 +1100
[YUNIKORN-2042] REST API for specific queue (#687)
Expose a REST API for specific queue:
/ws/v1/partition/%s/queue/%s
/ws/v1/partition/%s/queue/%s?aubtree
The call takes one query parameter "subtree" if provided the whole tree
of queues rooted at the level requested in the call will be returned. If
"subtree" is not set only the queue requested will be returned
Closes: #687
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/queue.go | 16 ---
pkg/scheduler/objects/queue_test.go | 20 -
pkg/scheduler/partition.go | 2 +-
pkg/webservice/dao/queue_info.go| 1 +
pkg/webservice/handlers.go | 30 +
pkg/webservice/handlers_test.go | 90 +++--
pkg/webservice/routes.go| 6 +++
7 files changed, 136 insertions(+), 29 deletions(-)
diff --git a/pkg/scheduler/objects/queue.go b/pkg/scheduler/objects/queue.go
index ebf70e29..43c66e8b 100644
--- a/pkg/scheduler/objects/queue.go
+++ b/pkg/scheduler/objects/queue.go
@@ -624,17 +624,23 @@ func (sq *Queue) CheckAdminAccess(user
security.UserGroup) bool {
}
// GetPartitionQueueDAOInfo returns the queue hierarchy as an object for a
REST call.
-func (sq *Queue) GetPartitionQueueDAOInfo() dao.PartitionQueueDAOInfo {
+// Exclude is true, which means that returns the specified queue object, but
does not return the children of the specified queue.
+func (sq *Queue) GetPartitionQueueDAOInfo(exclude bool)
dao.PartitionQueueDAOInfo {
queueInfo := dao.PartitionQueueDAOInfo{}
- childes := sq.GetCopyOfChildren()
- queueInfo.Children = make([]dao.PartitionQueueDAOInfo, 0, len(childes))
- for _, child := range childes {
- queueInfo.Children = append(queueInfo.Children,
child.GetPartitionQueueDAOInfo())
+ children := sq.GetCopyOfChildren()
+ if !exclude {
+ queueInfo.Children = make([]dao.PartitionQueueDAOInfo, 0,
len(children))
+ for _, child := range children {
+ queueInfo.Children = append(queueInfo.Children,
child.GetPartitionQueueDAOInfo(false))
+ }
}
// we have held the read lock so following method should not take lock
again.
sq.RLock()
defer sq.RUnlock()
+ for _, child := range children {
+ queueInfo.ChildrenNames = append(queueInfo.ChildrenNames,
child.QueuePath)
+ }
queueInfo.QueueName = sq.QueuePath
queueInfo.Status = sq.stateMachine.Current()
queueInfo.PendingResource = sq.pending.DAOMap()
diff --git a/pkg/scheduler/objects/queue_test.go
b/pkg/scheduler/objects/queue_test.go
index 4c9e3303..7c46484e 100644
--- a/pkg/scheduler/objects/queue_test.go
+++ b/pkg/scheduler/objects/queue_test.go
@@ -1685,7 +1685,7 @@ func TestGetPartitionQueueDAOInfo(t *testing.T) {
// test properties
root.properties = getProperties()
- assert.DeepEqual(t, root.properties,
root.GetPartitionQueueDAOInfo().Properties)
+ assert.DeepEqual(t, root.properties,
root.GetPartitionQueueDAOInfo(false).Properties)
// test template
root.template, err = template.FromConf(&configs.ChildTemplate{
@@ -1697,23 +1697,23 @@ func TestGetPartitionQueueDAOInfo(t *testing.T) {
},
})
assert.NilError(t, err)
- assert.Equal(t, root.template.GetMaxApplications(),
root.GetPartitionQueueDAOInfo().TemplateInfo.MaxApplications)
- assert.DeepEqual(t, root.template.GetProperties(),
root.GetPartitionQueueDAOInfo().TemplateInfo.Properties)
- assert.DeepEqual(t, root.template.GetMaxResource().DAOMap(),
root.template.GetMaxResource().DAOMap())
- assert.DeepEqual(t, root.template.GetGuaranteedResource().DAOMap(),
root.template.GetGuaranteedResource().DAOMap())
+ assert.Equal(t, root.template.GetMaxApplications(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.MaxApplications)
+ assert.DeepEqual(t, root.template.GetProperties(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.Properties)
+ assert.DeepEqual(t, root.template.GetMaxResource().DAOMap(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.MaxResource)
+ assert.DeepEqual(t, root.template.GetGuaranteedResource().DAOMap(),
root.GetPartitionQueueDAOInfo(false).TemplateInfo.GuaranteedResource)
// test resources
root.maxResource = getResource(t)
root.guaranteed
(yunikorn-core) branch master updated: [YUNIKORN-2413] Initialism fixes for allocationId, clusterId (#806)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 3148d2b8 [YUNIKORN-2413] Initialism fixes for allocationId, clusterId
(#806)
3148d2b8 is described below
commit 3148d2b8300c5d0b333a4d8f5d22bc05d4e33f4d
Author: Priyansh Choudhary
AuthorDate: Fri Feb 23 13:47:05 2024 +1100
[YUNIKORN-2413] Initialism fixes for allocationId, clusterId (#806)
unifying allocationId to allocationID.
updating clusterId in PartitionNodesUtilDAOInfo
Closes: #806
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/node.go | 4 ++--
pkg/scheduler/partition.go | 42 -
pkg/scheduler/partition_test.go | 28 +--
pkg/webservice/dao/node_util.go | 2 +-
pkg/webservice/handlers.go | 2 +-
pkg/webservice/handlers_test.go | 6 +++---
6 files changed, 42 insertions(+), 42 deletions(-)
diff --git a/pkg/scheduler/objects/node.go b/pkg/scheduler/objects/node.go
index 7c5b8b92..edce1281 100644
--- a/pkg/scheduler/objects/node.go
+++ b/pkg/scheduler/objects/node.go
@@ -360,8 +360,8 @@ func (sn *Node) ReplaceAllocation(allocationID string,
replace *Allocation, delt
sn.availableResource.SubFrom(delta)
if !before.FitIn(sn.allocatedResource) {
log.Log(log.SchedNode).Warn("unexpected increase in node usage
after placeholder replacement",
- zap.String("placeholder allocationId", allocationID),
- zap.String("allocation allocationId",
replace.GetAllocationID()),
+ zap.String("placeholder allocationID", allocationID),
+ zap.String("allocation allocationID",
replace.GetAllocationID()),
zap.Stringer("delta", delta))
}
}
diff --git a/pkg/scheduler/partition.go b/pkg/scheduler/partition.go
index ab05972b..ec84c9f0 100644
--- a/pkg/scheduler/partition.go
+++ b/pkg/scheduler/partition.go
@@ -402,7 +402,7 @@ func (pc *PartitionContext) removeApplication(appID string)
[]*objects.Allocatio
if nodeAlloc :=
node.RemoveAllocation(currentAllocationID); nodeAlloc == nil {
log.Log(log.SchedPartition).Warn("unknown
allocation: not found on the node",
zap.String("appID", appID),
- zap.String("allocationId",
currentAllocationID),
+ zap.String("allocationID",
currentAllocationID),
zap.String("nodeID", alloc.GetNodeID()))
}
}
@@ -748,9 +748,9 @@ func (pc *PartitionContext) removeNodeAllocations(node
*objects.Node) ([]*object
released = append(released, alloc)
log.Log(log.SchedPartition).Info("allocation removed from node and replacement
confirmed",
zap.String("nodeID",
node.NodeID),
- zap.String("allocationId",
allocID),
+ zap.String("allocationID",
allocID),
zap.String("replacement
nodeID", release.GetNodeID()),
- zap.String("replacement
allocationId", release.GetAllocationID()))
+ zap.String("replacement
allocationID", release.GetAllocationID()))
continue
}
askAlloc = release
@@ -765,7 +765,7 @@ func (pc *PartitionContext) removeNodeAllocations(node
*objects.Node) ([]*object
zap.String("appID",
askAlloc.GetApplicationID()),
zap.String("allocationKey",
askAlloc.GetAsk().GetAllocationKey()),
zap.String("nodeID", node.NodeID),
- zap.String("replacement allocationId",
askAlloc.GetAllocationID()))
+ zap.String("replacement allocationID",
askAlloc.GetAllocationID()))
} else {
log.Log(log.SchedPartition).Error("node
removal: repeat update failure for inflight replacement",
zap.String("appID",
askAlloc.GetApplicationID()),
@
(yunikorn-web) branch master updated: [YUNIKORN-2437] Include NOTICE and LICENSE files in Docker image (#174)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git The following commit(s) were added to refs/heads/master by this push: new b061d65 [YUNIKORN-2437] Include NOTICE and LICENSE files in Docker image (#174) b061d65 is described below commit b061d651ca5b05434ee8001bdb71fa50ed6748e7 Author: Craig Condit AuthorDate: Thu Feb 22 22:18:32 2024 +1100 [YUNIKORN-2437] Include NOTICE and LICENSE files in Docker image (#174) Closes: #174 Signed-off-by: Wilfred Spiegelenburg --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 787a5a5..8a41d68 100644 --- a/Dockerfile +++ b/Dockerfile @@ -29,7 +29,7 @@ RUN yarn build:prod # Imagestage: use scratch base image FROM --platform=$TARGETPLATFORM scratch -COPY --chown=0:0 build/prod/yunikorn-web / +COPY --chown=0:0 NOTICE LICENSE build/prod/yunikorn-web / COPY --chown=0:0 --from=buildstage /work/dist/yunikorn-web /html/ EXPOSE 9889 ENV DOCUMENT_ROOT /html - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2116] Track user/group events (#800)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 174c5fd8 [YUNIKORN-2116] Track user/group events (#800)
174c5fd8 is described below
commit 174c5fd8f1199c1f8a57e436dcc2bad8782e9723
Author: Peter Bacsko
AuthorDate: Thu Feb 22 17:19:50 2024 +1100
[YUNIKORN-2116] Track user/group events (#800)
Add events t the event system for user and group quotas. Additional
events are generated for configuration changes and usage changes for
both the user and groups.
The (un)linking of and app to a group is tracked in separate events.
Closes: #800
Signed-off-by: Wilfred Spiegelenburg
---
pkg/events/events.go | 4 +
pkg/events/events_test.go| 3 +
pkg/scheduler/objects/application.go | 42
pkg/scheduler/objects/application_events.go | 40 +++-
pkg/scheduler/objects/application_events_test.go | 92 +
pkg/scheduler/objects/application_test.go| 55 ++
pkg/scheduler/objects/queue.go | 5 +-
pkg/scheduler/tests/application_tracking_test.go | 62 +++---
pkg/scheduler/ugm/group_tracker.go | 28 ++-
pkg/scheduler/ugm/group_tracker_test.go | 137 +
pkg/scheduler/ugm/manager.go | 68 +++
pkg/scheduler/ugm/ugm_events.go | 116 +++
pkg/scheduler/ugm/ugm_events_test.go | 239 +++
pkg/scheduler/ugm/user_tracker.go| 41 +++-
pkg/scheduler/ugm/user_tracker_test.go | 164
15 files changed, 942 insertions(+), 154 deletions(-)
diff --git a/pkg/events/events.go b/pkg/events/events.go
index be6f72b8..819dd661 100644
--- a/pkg/events/events.go
+++ b/pkg/events/events.go
@@ -54,3 +54,7 @@ func CreateNodeEventRecord(objectID, message, referenceID
string, changeType si.
func CreateQueueEventRecord(objectID, message, referenceID string, changeType
si.EventRecord_ChangeType, changeDetail si.EventRecord_ChangeDetail, resource
*resources.Resource) *si.EventRecord {
return createEventRecord(si.EventRecord_QUEUE, objectID, referenceID,
message, changeType, changeDetail, resource)
}
+
+func CreateUserGroupEventRecord(objectID, message, referenceID string,
changeType si.EventRecord_ChangeType, changeDetail si.EventRecord_ChangeDetail,
resource *resources.Resource) *si.EventRecord {
+ return createEventRecord(si.EventRecord_USERGROUP, objectID,
referenceID, message, changeType, changeDetail, resource)
+}
diff --git a/pkg/events/events_test.go b/pkg/events/events_test.go
index dc86fcd1..11bca877 100644
--- a/pkg/events/events_test.go
+++ b/pkg/events/events_test.go
@@ -58,4 +58,7 @@ func TestCreateEventRecordTypes(t *testing.T) {
record = CreateQueueEventRecord("queue", "message", "app",
si.EventRecord_NONE, si.EventRecord_DETAILS_NONE, nil)
assert.Equal(t, record.Type, si.EventRecord_QUEUE)
+
+ record = CreateUserGroupEventRecord("user", "message", "queue",
si.EventRecord_NONE, si.EventRecord_DETAILS_NONE, nil)
+ assert.Equal(t, record.Type, si.EventRecord_USERGROUP)
}
diff --git a/pkg/scheduler/objects/application.go
b/pkg/scheduler/objects/application.go
index b8d8f60a..cde6fbd9 100644
--- a/pkg/scheduler/objects/application.go
+++ b/pkg/scheduler/objects/application.go
@@ -112,6 +112,8 @@ type Application struct {
placeholderData map[string]*PlaceholderData // track placeholder
and gang related info
askMaxPriority int32 // highest priority
value of outstanding asks
hasPlaceholderAlloc bool// Whether there is at
least one allocated placeholder
+ runnableInQueue bool// whether the
application is runnable/schedulable in the queue. Default is true.
+ runnableByUserLimit bool// whether the
application is runnable/schedulable based on user/group quota. Default is true.
rmEventHandlerhandler.EventHandler
rmID string
@@ -171,6 +173,8 @@ func NewApplication(siApp *si.AddApplicationRequest, ugi
security.UserGroup, eve
askMaxPriority:configs.MinPriority,
sortedRequests:sortedRequests{},
sendStateChangeEvents: true,
+ runnableByUserLimit: true,
+ runnableInQueue: true,
}
placeholderTimeout := common.ConvertSITimeoutWithAdjustment(siApp,
defaultPlaceholderTimeout)
gangSchedStyle := siApp.GetGangSchedulingStyle()
@@ -2083,3 +2087,41 @@ func getRateLimitedAppLog() *log.RateLimitedLogger {
})
(yunikorn-core) branch master updated: [YUNIKORN-2441] Wildcard limits not set on root tracker (#813)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new f0e7d6d9 [YUNIKORN-2441] Wildcard limits not set on root tracker (#813)
f0e7d6d9 is described below
commit f0e7d6d9c4a284b47011d299a9fa3519c12daf9b
Author: Peter Bacsko
AuthorDate: Thu Feb 22 17:15:35 2024 +1100
[YUNIKORN-2441] Wildcard limits not set on root tracker (#813)
When the root queue gets created the wildcard settings are not applied
correctly. The path to lookup the wildcard configs is incorrectly
constructed. It does not take into account that the path could be empty.
This change fixes the path constructioni for wildcard lookup..
Closes: #813
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/ugm/manager_test.go | 79 +
pkg/scheduler/ugm/queue_tracker.go | 8 ++--
pkg/scheduler/ugm/queue_tracker_test.go | 33 ++
3 files changed, 78 insertions(+), 42 deletions(-)
diff --git a/pkg/scheduler/ugm/manager_test.go
b/pkg/scheduler/ugm/manager_test.go
index 8749d117..2b7add0d 100644
--- a/pkg/scheduler/ugm/manager_test.go
+++ b/pkg/scheduler/ugm/manager_test.go
@@ -332,7 +332,7 @@ func TestUpdateConfig(t *testing.T) {
func TestUseWildCard(t *testing.T) {
setupUGM()
manager := GetUserManager()
- user := security.UserGroup{User: "user1", Groups: []string{"group1"}}
+ user1 := security.UserGroup{User: "user1", Groups: []string{"group1"}}
expectedResource, err :=
resources.NewResourceFromConf(map[string]string{"memory": "50", "vcores": "50"})
if err != nil {
@@ -348,70 +348,73 @@ func TestUseWildCard(t *testing.T) {
t.Errorf("new resource create returned error or wrong resource:
error %t, res %v", err, expectedHeadroom)
}
- user1 := security.UserGroup{User: "user2", Groups: []string{"group2"}}
- conf := createUpdateConfigWithWildCardUsersAndGroups(user1.User,
user1.Groups[0], "*", "*", "50", "50")
+ user2 := security.UserGroup{User: "user2", Groups: []string{"group2"}}
+ conf := createUpdateConfigWithWildCardUsersAndGroups(user2.User,
user2.Groups[0], "*", "*", "50", "50")
assert.NilError(t, manager.UpdateConfig(conf.Queues[0], "root"))
// user1 fallback on wild card user limit. user1 max resources and max
applications would be overwritten with wild card user limit settings
- headroom := manager.Headroom(queuePath1, TestApp1, user)
- assert.Equal(t, resources.Equals(headroom, expectedHeadroom), true)
+ headroom := manager.Headroom(queuePath1, TestApp1, user1)
+ assert.Assert(t, resources.Equals(headroom, expectedHeadroom))
// user2 has its own settings, so doesn't fallback on wild card user
limit.
- headroom = manager.Headroom(queuePath1, TestApp1, user1)
- assert.Equal(t, resources.Equals(headroom, resources.Multiply(usage,
7)), true)
+ headroom = manager.Headroom(queuePath1, TestApp1, user2)
+ assert.Assert(t, resources.Equals(headroom, resources.Multiply(usage,
7)))
// user1 uses wild card user limit settings.
- assert.Equal(t,
manager.GetUserTracker(user.User).queueTracker.maxRunningApps, uint64(0))
- assert.Equal(t,
manager.GetUserTracker(user.User).queueTracker.childQueueTrackers["parent"].maxRunningApps,
uint64(10))
- assert.Equal(t,
manager.GetUserTracker(user.User).queueTracker.childQueueTrackers["parent"].childQueueTrackers["child1"].maxRunningApps,
uint64(0))
- assert.Equal(t,
resources.Equals(manager.GetUserTracker(user.User).queueTracker.maxResources,
nil), true)
- assert.Equal(t,
resources.Equals(manager.GetUserTracker(user.User).queueTracker.childQueueTrackers["parent"].maxResources,
expectedHeadroom), true)
- assert.Equal(t,
resources.Equals(manager.GetUserTracker(user.User).queueTracker.childQueueTrackers["parent"].childQueueTrackers["child1"].maxResources,
nil), true)
- assert.Equal(t,
manager.GetUserTracker(user.User).queueTracker.useWildCard, false)
- assert.Equal(t,
manager.GetUserTracker(user.User).queueTracker.childQueueTrackers["parent"].useWildCard,
true)
- assert.Equal(t,
manager.GetUserTracker(user.User).queueTracker.childQueueTrackers["parent"].childQueueTrackers["child1"].useWildCard,
false)
-
- // user2 uses its own settings.
assert.Equal(t,
manager.GetUserTracker(user1.User).queueTracker.maxRunningApps, uint64(20
(yunikorn-site) branch master updated: [YUNIKORN-2337] Update documentation about event streaming (#401)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 2801670fca [YUNIKORN-2337] Update documentation about event streaming
(#401)
2801670fca is described below
commit 2801670fca9d257631c406f1ee673a3fbc3578fe
Author: Peter Bacsko
AuthorDate: Thu Feb 22 10:37:48 2024 +1100
[YUNIKORN-2337] Update documentation about event streaming (#401)
Closes: #401
Signed-off-by: Wilfred Spiegelenburg
---
docs/api/scheduler.md | 36 +++-
1 file changed, 35 insertions(+), 1 deletion(-)
diff --git a/docs/api/scheduler.md b/docs/api/scheduler.md
index 6ab615bb54..4517603ac3 100644
--- a/docs/api/scheduler.md
+++ b/docs/api/scheduler.md
@@ -2093,4 +2093,38 @@ Endpoint is used to retrieve a batch of event records.
### Error response
-**Code** : `500 Internal Server Error`
\ No newline at end of file
+**Code** : `500 Internal Server Error`
+
+### Event stream
+
+Creates a persistent HTTP connection for event streaming. New events are sent
to the clients immediately, so unlike the batch interface, there is no need for
polling.
+The number of active connections is limited. The default setting is 100
connections total and 15 connections per host. The respective configmap
properties are `event.maxStreams` and `event.maxStreamsPerHost`.
+
+**URL**: `/ws/v1/events/stream`
+
+**METHOD** : `GET`
+
+**Auth required** : NO
+
+**URL query parameters**:
+- `count` (optional) : Specifies the number of past events (those which have
been generated before the connection establishment) to include in the response.
Default value is 0.
+
+### Success response
+
+**Code**: `200 OK`
+
+**Content examples**
+
+```json
+{"type":2,"objectID":"app-1","timestampNano":1708465452903045265,"eventChangeType":1,"eventChangeDetail":204,"resource":{}}
+{"type":2,"objectID":"app-1","timestampNano":1708465452903192898,"eventChangeType":2,"eventChangeDetail":201,"referenceID":"alloc-1","resource":{"resources":{"memory":{"value":1000},"vcore":{"value":1000
+{"type":3,"objectID":"node-1:1234","timestampNano":1708465452903312146,"eventChangeType":2,"eventChangeDetail":303,"referenceID":"alloc-1","resource":{"resources":{"memory":{"value":1000},"vcore":{"value":1000
+{"type":2,"objectID":"app-1","timestampNano":1708465452903474210,"eventChangeType":1,"eventChangeDetail":205,"resource":{}}
+{"type":5,"objectID":"testuser","timestampNano":1708465452903506166,"eventChangeType":2,"eventChangeDetail":603,"referenceID":"root.singleleaf","resource":{"resources":{"memory":{"value":1000},"vcore":{"value":1000
+```
+
+### Error responses
+
+**Code** : `400 Bad Request` (URL query is invalid)
+**Code** : `503 Service Unavailable` (Too many active streaming connections)
+**Code** : `500 Internal Server Error`
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch master updated: [YUNIKORN-2425] Use 'go mod edit' to replace module path (#170)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git
The following commit(s) were added to refs/heads/master by this push:
new d707f57 [YUNIKORN-2425] Use 'go mod edit' to replace module path
(#170)
d707f57 is described below
commit d707f573d87eb863388de39f02fdb57f0daaf15e
Author: Craig Condit
AuthorDate: Tue Feb 20 14:50:22 2024 +1100
[YUNIKORN-2425] Use 'go mod edit' to replace module path (#170)
Instead of manual file editing, use the go standard tooling to replace
references to yunikorn-core and yunikorn-scheduler-interface in go.mod
files.
Closes: #170
Signed-off-by: Wilfred Spiegelenburg
---
tools/build-release.py | 27 +--
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/tools/build-release.py b/tools/build-release.py
index fba52aa..8331855 100755
--- a/tools/build-release.py
+++ b/tools/build-release.py
@@ -210,12 +210,15 @@ def update_dep_ref_k8shim(local_repo_path):
mod_file = os.path.join(local_repo_path, "go.mod")
if not os.path.isfile(mod_file):
fail("k8shim go.mod does not exist")
-with open(mod_file, "a") as file_object:
-file_object.write("\n")
-file_object.write("replace github.com/apache/yunikorn-core => ../core
\n")
-file_object.write(
-"replace github.com/apache/yunikorn-scheduler-interface =>
../scheduler-interface \n")
-
+path = os.getcwd()
+os.chdir(local_repo_path)
+command = ['go', 'mod', 'edit']
+command.extend(['-replace', 'github.com/apache/yunikorn-core=../core'])
+command.extend(['-replace',
'github.com/apache/yunikorn-scheduler-interface=../scheduler-interface'])
+retcode = subprocess.call(command)
+if retcode:
+fail("failed to update k8shim go.mod references")
+os.chdir(path)
# core depends on scheduler-interface
def update_dep_ref_core(local_repo_path):
@@ -223,10 +226,14 @@ def update_dep_ref_core(local_repo_path):
mod_file = os.path.join(local_repo_path, "go.mod")
if not os.path.isfile(mod_file):
fail("core go.mod does not exist")
-with open(mod_file, "a") as file_object:
-file_object.write("\n")
-file_object.write(
-"replace github.com/apache/yunikorn-scheduler-interface =>
../scheduler-interface \n")
+path = os.getcwd()
+os.chdir(local_repo_path)
+command = ['go', 'mod', 'edit']
+command.extend(['-replace',
'github.com/apache/yunikorn-scheduler-interface=../scheduler-interface'])
+retcode = subprocess.call(command)
+if retcode:
+fail("failed to update core go.mod references")
+os.chdir(path)
# update go mod in the repos
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2392] Ensure scheduling tasks are accessed under lock (#784)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new e9012794 [YUNIKORN-2392] Ensure scheduling tasks are accessed under
lock (#784)
e9012794 is described below
commit e9012794fba6e1cd5e0f5f761ca57ddc6b01865a
Author: Craig Condit
AuthorDate: Fri Feb 9 13:57:39 2024 +1100
[YUNIKORN-2392] Ensure scheduling tasks are accessed under lock (#784)
Concurrent map write detected during performance testing only. The code
has been through all unit and e2e tested and none have triggered the
concurrent write.
No test cases added.
Closes: #784
Signed-off-by: Wilfred Spiegelenburg
---
pkg/cache/external/scheduler_cache.go | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/pkg/cache/external/scheduler_cache.go
b/pkg/cache/external/scheduler_cache.go
index a81977da..5aa6b2fc 100644
--- a/pkg/cache/external/scheduler_cache.go
+++ b/pkg/cache/external/scheduler_cache.go
@@ -375,8 +375,10 @@ func (cache *SchedulerCache)
removePriorityClass(priorityClass *schedulingv1.Pri
// NotifyTaskSchedulerAction registers the fact that a task has been evaluated
for scheduling, and consequently the
// scheduler plugin should move it to the activeQ if requested to do so.
func (cache *SchedulerCache) NotifyTaskSchedulerAction(taskID string) {
+ cache.lock.Lock()
+ defer cache.lock.Unlock()
// verify that the pod exists in the cache, otherwise ignore
- if _, ok := cache.GetPod(taskID); !ok {
+ if _, ok := cache.GetPodNoLock(taskID); !ok {
return
}
cache.addSchedulingTask(taskID)
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2393] Upgrade codecov/codecov-action to v4 (#783)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new 5f8d20b6 [YUNIKORN-2393] Upgrade codecov/codecov-action to v4 (#783)
5f8d20b6 is described below
commit 5f8d20b68ee4a3bbb9d78acf976fc1143a29ec29
Author: Yu-Lin Chen
AuthorDate: Fri Feb 9 13:44:06 2024 +1100
[YUNIKORN-2393] Upgrade codecov/codecov-action to v4 (#783)
Closes: #783
Signed-off-by: Wilfred Spiegelenburg
---
.github/workflows/pre-commit.yml | 6 +-
.github/workflows/push-master.yml | 6 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
index 6d3edb67..2b6a1fe9 100644
--- a/.github/workflows/pre-commit.yml
+++ b/.github/workflows/pre-commit.yml
@@ -29,9 +29,13 @@ jobs:
- name: Unit tests
run: make test
- name: Code coverage
-uses: codecov/codecov-action@v3
+uses: codecov/codecov-action@v4
with:
files: build/coverage.txt
+ # After codecov/codecov-action@v4, tokenless uploading of coverage
files to non-public repo is unsupported.
+ # To enable codecov analysis in your forked repo. Please configure
CODECOV_TOKEN in your repository secrets.
+ # Ref: https://docs.codecov.com/docs/adding-the-codecov-token
+ token: ${{ secrets.CODECOV_TOKEN }}
e2e-tests:
needs: build
diff --git a/.github/workflows/push-master.yml
b/.github/workflows/push-master.yml
index 9757a602..03bd406b 100644
--- a/.github/workflows/push-master.yml
+++ b/.github/workflows/push-master.yml
@@ -20,6 +20,10 @@ jobs:
- name: Unit tests
run: make test
- name: Code coverage
-uses: codecov/codecov-action@v3
+uses: codecov/codecov-action@v4
with:
files: build/coverage.txt
+ # After codecov/codecov-action@v4, tokenless uploading of coverage
files to non-public repo is unsupported.
+ # To enable codecov analysis in your forked repo. Please configure
CODECOV_TOKEN in your repository secrets.
+ # Ref: https://docs.codecov.com/docs/adding-the-codecov-token
+ token: ${{ secrets.CODECOV_TOKEN }}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch master updated: [YUNIKORN-2382] Update helm chart readme (#167)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/master by this push: new 6dcf8b1 [YUNIKORN-2382] Update helm chart readme (#167) 6dcf8b1 is described below commit 6dcf8b1ac8f3f7620b48fcafa232148292d10cc3 Author: Wilfred Spiegelenburg AuthorDate: Thu Feb 8 12:59:19 2024 +1100 [YUNIKORN-2382] Update helm chart readme (#167) New k8s supported versions Additional functionality, clean up text Closes: #167 Signed-off-by: Wilfred Spiegelenburg --- helm-charts/yunikorn/README.md | 19 ++- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/helm-charts/yunikorn/README.md b/helm-charts/yunikorn/README.md index 9a697e6..82b76c9 100644 --- a/helm-charts/yunikorn/README.md +++ b/helm-charts/yunikorn/README.md @@ -18,26 +18,25 @@ # Apache YuniKorn - A Universal Scheduler Apache YuniKorn is a light-weight, universal resource scheduler for container orchestrator systems. -It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, -and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist -of stateless batch workloads and stateful services. +It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of AI, Machine Learning, stateless batch workloads and stateful services. -YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different -shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems. +YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems. ## Feature highlights -- Features to support both batch jobs and long-running/stateful services. -- Hierarchy queues with min/max resource quotas. +- Features to support both AI, Machine Learning or batch jobs and long-running/stateful services. +- Hierarchical queues with guaranteed/maximum resource quotas and applications. +- User and group quotas configurable for each queue with maximum applicatins and resources. - Resource fairness between queues, users and apps. +- Scheduling policies configurable per queue: FIFO, priority and state based. - Cross-queue preemption based on fairness. - Automatically map incoming container requests to queues by policies. - Node partition: partition cluster to sub-clusters with dedicated quota/ACL management. - Fully compatible with K8s predicates, events, PV/PVC and admin commands. -- Supports to work with [Cluster AutoScaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) to drive cluster scales up and down. +- Support for [Cluster AutoScaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) and [Karpenter](https://karpenter.sh/) to drive cluster scale up and down. ## Deployment model -YuniKorn can be deployed with [helm-charts](https://hub.helm.sh/charts/yunikorn/yunikorn) on an existing K8s cluster. It can be deployed with or without the admission controller. When the admission controller is enabled, YuniKorn will be the primary scheduler that takes over the resource scheduling (the admission controller runs a mutation webhook that automatically mutates pod spec's schedulerName to yunikorn); when it is disabled, user needs to manually change the schedulerName to `yun [...] +YuniKorn can be deployed with [helm-charts](https://artifacthub.io/packages/helm/yunikorn/yunikorn) on an existing K8s cluster. It can be deployed with or without the admission controller. When the admission controller is enabled, YuniKorn will be the primary scheduler that takes over the resource scheduling (the admission controller runs a mutation webhook that automatically mutates pod spec's schedulerName to yunikorn); when it is disabled, user needs to manually change the schedulerNa [...] ## Supported K8s versions @@ -48,6 +47,8 @@ YuniKorn can be deployed with [helm-charts](https://hub.helm.sh/charts/yunikorn/ | 1.25.x |√ | | 1.26.x |√ | | 1.27.x |√ | +| 1.28.x |√ | +| 1.29.x |√ | ## Installing the chart ``` - To unsubscribe, e-mail: issues-unsubscr...@yun
(yunikorn-release) branch master updated: [YUNIKORN-2389] Update golang dependencies (#166)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/master by this push: new 00fff32 [YUNIKORN-2389] Update golang dependencies (#166) 00fff32 is described below commit 00fff320e752246a76f7cffc97730484a34a758f Author: Yu-Lin Chen AuthorDate: Wed Feb 7 22:14:29 2024 +1100 [YUNIKORN-2389] Update golang dependencies (#166) Closes: #166 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 16 go.sum | 40 +++- 2 files changed, 35 insertions(+), 21 deletions(-) diff --git a/go.mod b/go.mod index ccafe69..379b67c 100644 --- a/go.mod +++ b/go.mod @@ -68,10 +68,10 @@ require ( go.uber.org/atomic v1.7.0 // indirect go.uber.org/multierr v1.6.0 // indirect golang.org/x/image v0.14.0 // indirect - golang.org/x/net v0.15.0 // indirect + golang.org/x/net v0.20.0 // indirect golang.org/x/oauth2 v0.10.0 // indirect - golang.org/x/sys v0.13.0 // indirect - golang.org/x/term v0.13.0 // indirect + golang.org/x/sys v0.16.0 // indirect + golang.org/x/term v0.16.0 // indirect golang.org/x/text v0.14.0 // indirect golang.org/x/time v0.4.0 // indirect google.golang.org/appengine v1.6.7 // indirect @@ -89,10 +89,10 @@ require ( ) replace ( - golang.org/x/crypto => golang.org/x/crypto v0.14.0 + golang.org/x/crypto => golang.org/x/crypto v0.18.0 golang.org/x/lint => golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 - golang.org/x/net => golang.org/x/net v0.17.0 - golang.org/x/sys => golang.org/x/sys v0.13.0 - golang.org/x/text => golang.org/x/text v0.13.0 - golang.org/x/tools => golang.org/x/tools v0.13.0 + golang.org/x/net => golang.org/x/net v0.20.0 + golang.org/x/sys => golang.org/x/sys v0.16.0 + golang.org/x/text => golang.org/x/text v0.14.0 + golang.org/x/tools => golang.org/x/tools v0.17.0 ) diff --git a/go.sum b/go.sum index 8d87013..52a0007 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,6 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= git.sr.ht/~sbinet/cmpimg v0.1.0 h1:E0zPRk2muWuCqSKSVZIWsgtU9pjsw3eKHi8VmQeScxo= +git.sr.ht/~sbinet/cmpimg v0.1.0/go.mod h1:FU12psLbF4TfNXkKH2ZZQ29crIqoiqTZmeQ7dkp/pxE= git.sr.ht/~sbinet/gg v0.4.1 h1:YccqPPS57/TpqX2fFnSRlisrqQ43gEdqVm3JtabPrp0= git.sr.ht/~sbinet/gg v0.4.1/go.mod h1:xKrQ22W53kn8Hlq+gzYeyyohGMwR8yGgSMlVpY/mHGc= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= @@ -14,6 +15,7 @@ github.com/apache/yunikorn-core v0.0.0-20230609084156-d2d7ce89457c/go.mod h1:wWl github.com/apache/yunikorn-scheduler-interface v0.0.0-20230517151935-1714a5751931 h1:3MJuXAq/BxWhGdwxDMwmLKX3a2YDDYQz+upupHAtQsc= github.com/apache/yunikorn-scheduler-interface v0.0.0-20230517151935-1714a5751931/go.mod h1:r4amgZ4uYT1K0KpfYoNV4Ao2GVIYOUU0Rxru89g0WsQ= github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= +github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= @@ -26,7 +28,9 @@ github.com/emicklei/go-restful/v3 v3.9.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/go-fonts/dejavu v0.1.0 h1:JSajPXURYqpr+Cu8U9bt8K+XcACIHWqWrvWCKyeFmVQ= +github.com/go-fonts/dejavu v0.1.0/go.mod h1:4Wt4I4OU2Nq9asgDCteaAaWZOV24E+0/Pwo0gppep4g= github.com/go-fonts/latin-modern v0.3.1 h1:/cT8A7uavYKvglYXvrdDw4oS5ZLkcOU22fa2HJ1/JVM= +github.com/go-fonts/latin-modern v0.3.1/go.mod h1:ysEQXnuT/sCDOAONxC7ImeEDVINbltClhasMAqEtRK0= github.com/go-fonts/liberation v0.3.1 h1:9RPT2NhUpxQ7ukUvz3jeUckmN42T9D9TpjtQcqK/ceM= github.com/go-fonts/liberation v0.3.1/go.mod h1:jdJ+cqF+F4SUL2V+qxBth8fvBpBDS7yloUL5Fi8GTGY= github.com/go-latex/latex v0.0.0-20230307184459-12ec69307ad9 h1:NxXI5pTAtpEaU49bpLpQoDsu1zrteW/vxzTz8Cd2UAs= @@ -43,6 +47,7 @@ github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+ github.com/go-pdf/fpdf v0.8.0 h1:IJKpdaagnWUeSkUFUjTcSzTppFxmv8ucGQyNPQWxYOQ= github.com/go-pdf/fpdf v0.8.0/go.mod h1:gfqhcNwXrsd3XYKte9a7vM3smvU/jB4ZRDrmWSxpfdc= github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 h1:p104kn46Q8WdvHunIJ9dAyjPVtrBPhSr3KT2yUst43
(yunikorn-web) branch master updated: [YUNIKORN-2388] Update gotest.tools/v3 (#168)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git The following commit(s) were added to refs/heads/master by this push: new a6e780f [YUNIKORN-2388] Update gotest.tools/v3 (#168) a6e780f is described below commit a6e780f54371935dd2b234b150e97ea3c5d3bc96 Author: Ryan Lo AuthorDate: Wed Feb 7 19:10:34 2024 +1100 [YUNIKORN-2388] Update gotest.tools/v3 (#168) Closes: #168 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 4 ++-- go.sum | 32 2 files changed, 6 insertions(+), 30 deletions(-) diff --git a/go.mod b/go.mod index 8ea7640..b319fcd 100644 --- a/go.mod +++ b/go.mod @@ -21,6 +21,6 @@ module github.com/apache/yunikorn-web go 1.21 -require gotest.tools/v3 v3.4.0 +require gotest.tools/v3 v3.5.1 -require github.com/google/go-cmp v0.5.5 // indirect +require github.com/google/go-cmp v0.5.9 // indirect diff --git a/go.sum b/go.sum index 6f2155e..7dd4ab5 100644 --- a/go.sum +++ b/go.sum @@ -1,28 +1,4 @@ -github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU= -github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -gotest.tools/v3 v3.4.0 h1:ZazjZUfuVeZGLAmlKKuyv3IKP5orXcwtOwDQH6YVr6o= -gotest.tools/v3 v3.4.0/go.mod h1:CtbdzLSsqVhDgMtKsx03ird5YTGB3ar27v0u/yKBW5g= +github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= +github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +gotest.tools/v3 v3.5.1 h1:EENdUnS3pdur5nybKYIh2Vfgc8IUNBjxDPSjtiJcOzU= +gotest.tools/v3 v3.5.1/go.mod h1:isy3WKz7GK6uNw/sbHzfKBLvlvXwUyV06n6brMxxopU= - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-scheduler-interface) branch master updated: [YUNIKORN-1697] Set max apps via namespace annotations (#117)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-scheduler-interface.git The following commit(s) were added to refs/heads/master by this push: new fa0365f [YUNIKORN-1697] Set max apps via namespace annotations (#117) fa0365f is described below commit fa0365fc70e2855436f7d8c9c9de305327f29052 Author: qzhu AuthorDate: Wed Feb 7 14:14:04 2024 +1100 [YUNIKORN-1697] Set max apps via namespace annotations (#117) Additional to max and guaranteed resources allow setting max applications on a queue via the annotations. Closes: #117 Signed-off-by: Wilfred Spiegelenburg --- lib/go/common/constants.go | 1 + scheduler-interface-spec.md | 1 + 2 files changed, 2 insertions(+) diff --git a/lib/go/common/constants.go b/lib/go/common/constants.go index afa666f..a8898ef 100644 --- a/lib/go/common/constants.go +++ b/lib/go/common/constants.go @@ -62,6 +62,7 @@ const ( CPU = "vcore" AppTagNamespaceResourceQuota = "namespace.resourcequota" AppTagNamespaceResourceGuaranteed = "namespace.resourceguaranteed" + AppTagNamespaceResourceMaxApps= "namespace.resourcemaxapps" AppTagStateAwareDisable = "application.stateaware.disable" AppTagCreateForce = "application.create.force" NodeReadyAttribute= "ready" diff --git a/scheduler-interface-spec.md b/scheduler-interface-spec.md index bb97603..bc78e25 100644 --- a/scheduler-interface-spec.md +++ b/scheduler-interface-spec.md @@ -771,6 +771,7 @@ const ( CPU = "vcore" AppTagNamespaceResourceQuota = "namespace.resourcequota" AppTagNamespaceResourceGuaranteed = "namespace.resourceguaranteed" + AppTagNamespaceResourceMaxApps= "namespace.resourcemaxapps" AppTagStateAwareDisable = "application.stateaware.disable" AppTagCreateForce = "application.create.force" NodeReadyAttribute= "ready" - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2388] Update gotest.tools/v3 (#780)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git The following commit(s) were added to refs/heads/master by this push: new 38fae598 [YUNIKORN-2388] Update gotest.tools/v3 (#780) 38fae598 is described below commit 38fae5980967383261090512e06ab36bdd5641b3 Author: Ryan Lo AuthorDate: Wed Feb 7 13:54:52 2024 +1100 [YUNIKORN-2388] Update gotest.tools/v3 (#780) Closes: #780 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 2 +- go.sum | 6 ++ 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index e9c94a54..a505863b 100644 --- a/go.mod +++ b/go.mod @@ -31,7 +31,7 @@ require ( github.com/prometheus/client_golang v1.16.0 go.uber.org/zap v1.26.0 gopkg.in/yaml.v3 v3.0.1 - gotest.tools/v3 v3.0.3 + gotest.tools/v3 v3.5.1 k8s.io/api v0.29.1 k8s.io/apimachinery v0.29.1 k8s.io/cli-runtime v0.29.1 diff --git a/go.sum b/go.sum index dd418cfc..1463d67a 100644 --- a/go.sum +++ b/go.sum @@ -202,7 +202,6 @@ github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+ github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc= github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= -github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= @@ -227,7 +226,6 @@ github.com/soheilhy/cmux v0.1.5 h1:jjzc5WVemNEDTLwv9tlmemhC73tI08BNOIGwBOo10Js= github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0= github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I= github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0= -github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stoewer/go-strcase v1.2.0 h1:Z2iHWqGXH00XYgqDmNgQbIBxf3wrNq0F3feEy0ainaU= @@ -374,8 +372,8 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gotest.tools/v3 v3.0.3 h1:4AuOwCGf4lLR9u3YOe2awrHygurzhO/HeQ6laiA6Sx0= -gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8= +gotest.tools/v3 v3.5.1 h1:EENdUnS3pdur5nybKYIh2Vfgc8IUNBjxDPSjtiJcOzU= +gotest.tools/v3 v3.5.1/go.mod h1:isy3WKz7GK6uNw/sbHzfKBLvlvXwUyV06n6brMxxopU= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= k8s.io/api v0.29.1 h1:DAjwWX/9YT7NQD4INu49ROJuZAAAP/Ijki48GUPzxqw= - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2386] Upgrade zap logger to 1.26 (#779)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git The following commit(s) were added to refs/heads/master by this push: new 0954d2b5 [YUNIKORN-2386] Upgrade zap logger to 1.26 (#779) 0954d2b5 is described below commit 0954d2b5b9f5223dd08d99353197757d9a1bb100 Author: Ryan Lo AuthorDate: Wed Feb 7 13:52:47 2024 +1100 [YUNIKORN-2386] Upgrade zap logger to 1.26 (#779) Closes: #779 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 2 +- go.sum | 6 ++ 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index f1f3674d..e9c94a54 100644 --- a/go.mod +++ b/go.mod @@ -29,7 +29,7 @@ require ( github.com/onsi/ginkgo/v2 v2.15.0 github.com/onsi/gomega v1.30.0 github.com/prometheus/client_golang v1.16.0 - go.uber.org/zap v1.24.0 + go.uber.org/zap v1.26.0 gopkg.in/yaml.v3 v3.0.1 gotest.tools/v3 v3.0.3 k8s.io/api v0.29.1 diff --git a/go.sum b/go.sum index cc3068d3..dd418cfc 100644 --- a/go.sum +++ b/go.sum @@ -21,8 +21,6 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPd github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= -github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= -github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= @@ -300,8 +298,8 @@ go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= -go.uber.org/zap v1.24.0 h1:FiJd5l1UOLj0wCgbSE0rwwXHzEdAZS6hiiSnxJN/D60= -go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg= +go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= +go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2387]: Upgrade ginkgo (#781)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git The following commit(s) were added to refs/heads/master by this push: new c1fc6fd9 [YUNIKORN-2387]: Upgrade ginkgo (#781) c1fc6fd9 is described below commit c1fc6fd9d1437a91ee22daf2278fb0c19593c36c Author: Ryan Lo AuthorDate: Wed Feb 7 13:44:04 2024 +1100 [YUNIKORN-2387]: Upgrade ginkgo (#781) Closes: #781 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 4 ++-- go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index f9684568..f1f3674d 100644 --- a/go.mod +++ b/go.mod @@ -26,8 +26,8 @@ require ( github.com/google/go-cmp v0.6.0 github.com/google/uuid v1.6.0 github.com/looplab/fsm v1.0.1 - github.com/onsi/ginkgo/v2 v2.13.0 - github.com/onsi/gomega v1.29.0 + github.com/onsi/ginkgo/v2 v2.15.0 + github.com/onsi/gomega v1.30.0 github.com/prometheus/client_golang v1.16.0 go.uber.org/zap v1.24.0 gopkg.in/yaml.v3 v3.0.1 diff --git a/go.sum b/go.sum index f0325af7..cc3068d3 100644 --- a/go.sum +++ b/go.sum @@ -192,10 +192,10 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= -github.com/onsi/ginkgo/v2 v2.13.0 h1:0jY9lJquiL8fcf3M4LAXN5aMlS/b2BV86HFFPCPMgE4= -github.com/onsi/ginkgo/v2 v2.13.0/go.mod h1:TE309ZR8s5FsKKpuB1YAQYBzCaAfUgatB/xlT/ETL/o= -github.com/onsi/gomega v1.29.0 h1:KIA/t2t5UBzoirT4H9tsML45GEbo3ouUnBHsCfD2tVg= -github.com/onsi/gomega v1.29.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= +github.com/onsi/ginkgo/v2 v2.15.0 h1:79HwNRBAZHOEwrczrgSOPy+eFTTlIGELKy5as+ClttY= +github.com/onsi/ginkgo/v2 v2.15.0/go.mod h1:HlxMHtYF57y6Dpf+mc5529KKmSq9h2FpCF+/ZkwUxKM= +github.com/onsi/gomega v1.30.0 h1:hvMK7xYz4D3HapigLTeGdId/NcfQx1VHMJc60ew99+8= +github.com/onsi/gomega v1.30.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/selinux v1.11.0 h1:+5Zbo97w3Lbmb3PeqQtpmTkMwsW5nRI3YaLpt7tQ7oU= - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch master updated: [YUNIKORN-2396] dependency sync between k8shim and core (#778)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git The following commit(s) were added to refs/heads/master by this push: new 6ac96e4a [YUNIKORN-2396] dependency sync between k8shim and core (#778) 6ac96e4a is described below commit 6ac96e4a8d55b3e7d7241350efb5a658fd5e7187 Author: Kuan-Po Tseng AuthorDate: Wed Feb 7 13:42:36 2024 +1100 [YUNIKORN-2396] dependency sync between k8shim and core (#778) Closes: #778 Signed-off-by: Wilfred Spiegelenburg --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 023f48b4..f9684568 100644 --- a/go.mod +++ b/go.mod @@ -24,7 +24,7 @@ require ( github.com/apache/yunikorn-core v0.0.0-20240119185010-bce75ba32684 github.com/apache/yunikorn-scheduler-interface v0.0.0-20240102192148-d4b43d6910c9 github.com/google/go-cmp v0.6.0 - github.com/google/uuid v1.3.1 + github.com/google/uuid v1.6.0 github.com/looplab/fsm v1.0.1 github.com/onsi/ginkgo/v2 v2.13.0 github.com/onsi/gomega v1.29.0 diff --git a/go.sum b/go.sum index e5c55777..f0325af7 100644 --- a/go.sum +++ b/go.sum @@ -130,8 +130,8 @@ github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJY github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= -github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4= -github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc= github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2132] Show active event streaming in the state dump (#707)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new b62164a2 [YUNIKORN-2132] Show active event streaming in the state dump
(#707)
b62164a2 is described below
commit b62164a213bca1f488328bf67b9a08b4e8cbe508
Author: Peter Bacsko
AuthorDate: Thu Feb 1 13:17:36 2024 +1100
[YUNIKORN-2132] Show active event streaming in the state dump (#707)
A list of active streams for events is added to the state dump.
Creation time and originating host name are shown. No usage data is
included.
Closes: #707
Signed-off-by: Wilfred Spiegelenburg
---
pkg/events/event_streaming.go| 23 ++-
pkg/events/event_streaming_test.go | 20
pkg/events/event_system.go | 8
pkg/events/event_system_test.go | 13 +
pkg/scheduler/objects/common_test.go | 4
pkg/webservice/state_dump.go | 3 +++
6 files changed, 70 insertions(+), 1 deletion(-)
diff --git a/pkg/events/event_streaming.go b/pkg/events/event_streaming.go
index 4f7b9d26..3896e24d 100644
--- a/pkg/events/event_streaming.go
+++ b/pkg/events/event_streaming.go
@@ -36,7 +36,7 @@ type EventStreaming struct {
buffer *eventRingBuffer
stopCh chan struct{}
eventStreams map[*EventStream]eventConsumerDetails
- sync.Mutex
+ sync.RWMutex
}
type eventConsumerDetails struct {
@@ -47,6 +47,12 @@ type eventConsumerDetails struct {
createdAt time.Time
}
+// EventStreamData contains data about an event stream.
+type EventStreamData struct {
+ Name string
+ CreatedAt time.Time
+}
+
// EventStream handle type returned to the client that wants to capture the
stream of events.
type EventStream struct {
Events <-chan *si.EventRecord
@@ -169,6 +175,21 @@ func (e *EventStreaming) Close() {
close(e.stopCh)
}
+// GetEventStreams returns the current active event streams.
+func (e *EventStreaming) GetEventStreams() []EventStreamData {
+ e.RLock()
+ defer e.RUnlock()
+ var streams []EventStreamData
+ for _, s := range e.eventStreams {
+ streams = append(streams, EventStreamData{
+ Name: s.name,
+ CreatedAt: s.createdAt,
+ })
+ }
+
+ return streams
+}
+
// NewEventStreaming creates a new event streaming infrastructure.
func NewEventStreaming(eventBuffer *eventRingBuffer) *EventStreaming {
return &EventStreaming{
diff --git a/pkg/events/event_streaming_test.go
b/pkg/events/event_streaming_test.go
index 8afc770a..c559c424 100644
--- a/pkg/events/event_streaming_test.go
+++ b/pkg/events/event_streaming_test.go
@@ -134,6 +134,26 @@ func TestEventStreaming_SlowConsumer(t *testing.T) {
assert.Equal(t, 0, len(streaming.eventStreams))
}
+func TestGetEventStreams(t *testing.T) {
+ buffer := newEventRingBuffer(10)
+ streaming := NewEventStreaming(buffer)
+ defer streaming.Close()
+
+ streaming.CreateEventStream("test-1", 0)
+ streams := streaming.GetEventStreams()
+ assert.Equal(t, 1, len(streams))
+ assert.Equal(t, "test-1", streams[0].Name)
+
+ streaming.CreateEventStream("test-2", 0)
+ streams = streaming.GetEventStreams()
+ assert.Equal(t, 2, len(streams))
+ names := make(map[string]bool)
+ names[streams[0].Name] = true
+ names[streams[1].Name] = true
+ assert.Assert(t, names["test-2"])
+ assert.Assert(t, names["test-1"])
+}
+
func receive(t *testing.T, input <-chan *si.EventRecord) *si.EventRecord {
select {
case event := <-input:
diff --git a/pkg/events/event_system.go b/pkg/events/event_system.go
index e89f6289..70ce2f26 100644
--- a/pkg/events/event_system.go
+++ b/pkg/events/event_system.go
@@ -73,6 +73,9 @@ type EventSystem interface {
// Consumers that no longer wish to be updated (e.g., a remote client
// disconnected) *must* call this method to gracefully stop the
streaming.
RemoveStream(*EventStream)
+
+ // GetEventStreams returns the current active event streams.
+ GetEventStreams() []EventStreamData
}
// EventSystemImpl main implementation of the event system which is used for
history tracking.
@@ -254,6 +257,11 @@ func (ec *EventSystemImpl) Restart() {
ec.StartServiceWithPublisher(true)
}
+// GetEventStreams returns the current active event streams.
+func (ec *EventSystemImpl) GetEventStreams() []EventStreamData {
+ return ec.streaming.GetEventStreams()
+}
+
// VisibleForTesting
func (ec *EventSystemImpl) CloseAllStreams() {
ec.streaming.Lock()
diff --git a/pkg/events/event_system_test.go b/
(yunikorn-core) branch master updated: [YUNIKORN-2212] Outstanding request collection optimisation (#745)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 8c6f4c53 [YUNIKORN-2212] Outstanding request collection optimisation
(#745)
8c6f4c53 is described below
commit 8c6f4c533a5d18f10baea495b6b668f8a7040e4f
Author: Peter Bacsko
AuthorDate: Thu Jan 25 16:59:04 2024 +1100
[YUNIKORN-2212] Outstanding request collection optimisation (#745)
Auto scaling n K8s relies on the pod to be marked unschedulable. That is
done via a regular callback from the core to the shim. The logic for
building the pod list in this callback is simple. This could cause over
scaling nodes or excessive K8s API server calls.
Improvements added:
* Don't collect requests that have not been scheduled yet.
* If already triggered scale up do not sent again
* Check for gang placeholders for any real ask
* Don't scale if pod has required node set (daemon set pod)
Track the state of scheduling and auto scale triggering.
Closes: #745
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/allocation_ask.go | 26
pkg/scheduler/objects/application.go | 11 +++-
pkg/scheduler/objects/application_test.go | 103 ++
pkg/scheduler/objects/queue_test.go | 25 +---
pkg/scheduler/partition_test.go | 67 +++
pkg/scheduler/scheduler.go| 17 -
pkg/scheduler/scheduler_test.go | 92 ++
pkg/webservice/dao/allocation_ask_info.go | 2 +
pkg/webservice/handlers.go| 2 +
9 files changed, 329 insertions(+), 16 deletions(-)
diff --git a/pkg/scheduler/objects/allocation_ask.go
b/pkg/scheduler/objects/allocation_ask.go
index 38370aec..7a42d04c 100644
--- a/pkg/scheduler/objects/allocation_ask.go
+++ b/pkg/scheduler/objects/allocation_ask.go
@@ -55,6 +55,8 @@ type AllocationAsk struct {
allocLogmap[string]*AllocationLogEntry
preemptionTriggered bool
preemptCheckTimetime.Time
+ schedulingAttempted bool // whether scheduler core has
tried to schedule this ask
+ scaleUpTriggeredbool // whether this ask has triggered
autoscaling or not
resKeyPerNode map[string]string // reservation key for a given
node
sync.RWMutex
@@ -299,6 +301,30 @@ func (aa *AllocationAsk) LessThan(other *AllocationAsk)
bool {
return aa.priority < other.priority
}
+func (aa *AllocationAsk) SetSchedulingAttempted(attempted bool) {
+ aa.Lock()
+ defer aa.Unlock()
+ aa.schedulingAttempted = attempted
+}
+
+func (aa *AllocationAsk) IsSchedulingAttempted() bool {
+ aa.RLock()
+ defer aa.RUnlock()
+ return aa.schedulingAttempted
+}
+
+func (aa *AllocationAsk) SetScaleUpTriggered(triggered bool) {
+ aa.Lock()
+ defer aa.Unlock()
+ aa.scaleUpTriggered = triggered
+}
+
+func (aa *AllocationAsk) HasTriggeredScaleUp() bool {
+ aa.RLock()
+ defer aa.RUnlock()
+ return aa.scaleUpTriggered
+}
+
// completedPendingAsk How many pending asks has been completed or processed
so far?
func (aa *AllocationAsk) completedPendingAsk() int {
aa.RLock()
diff --git a/pkg/scheduler/objects/application.go
b/pkg/scheduler/objects/application.go
index bc7793d9..1899ce62 100644
--- a/pkg/scheduler/objects/application.go
+++ b/pkg/scheduler/objects/application.go
@@ -899,13 +899,16 @@ func (sa *Application) getOutstandingRequests(headRoom
*resources.Resource, user
return
}
for _, request := range sa.sortedRequests {
- if request.GetPendingAskRepeat() == 0 {
+ if request.GetPendingAskRepeat() == 0 ||
!request.IsSchedulingAttempted() {
continue
}
+
// ignore nil checks resource function calls are nil safe
if headRoom.FitInMaxUndef(request.GetAllocatedResource()) &&
userHeadRoom.FitInMaxUndef(request.GetAllocatedResource()) {
- // if headroom is still enough for the resources
- *total = append(*total, request)
+ if !request.HasTriggeredScaleUp() &&
request.requiredNode == common.Empty && !sa.canReplace(request) {
+ // if headroom is still enough for the resources
+ *total = append(*total, request)
+ }
headRoom.SubOnlyExisting(request.GetAllocatedResource())
userHeadRoom.SubOnlyExisting(request.GetAllocatedResource())
}
@@ -951,6 +954,8 @@ func (sa *Application) tryAllocate(headRoom
(yunikorn-site) branch master updated: [YUNIKORN-2266] Correct Spark related application ID label (#382)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new dfd23dbab1 [YUNIKORN-2266] Correct Spark related application ID label (#382) dfd23dbab1 is described below commit dfd23dbab11508f2d0806ccb1970c09b2758167b Author: Yu-Lin Chen AuthorDate: Thu Jan 25 16:27:06 2024 +1100 [YUNIKORN-2266] Correct Spark related application ID label (#382) The label SparkLabelAppID is the name of the constant in the code to reference the label Spark sets for its application ID. The documentation should show the real name of the label: spark-app-selector Closes: #382 Signed-off-by: Wilfred Spiegelenburg --- .../labels_and_annotations_in_yunikorn.md | 42 ++--- .../labels_and_annotations_in_yunikorn.md | 44 +++--- 2 files changed, 43 insertions(+), 43 deletions(-) diff --git a/docs/user_guide/labels_and_annotations_in_yunikorn.md b/docs/user_guide/labels_and_annotations_in_yunikorn.md index a09051e107..efd9ee8df6 100644 --- a/docs/user_guide/labels_and_annotations_in_yunikorn.md +++ b/docs/user_guide/labels_and_annotations_in_yunikorn.md @@ -25,31 +25,31 @@ under the License. YuniKorn utilizes several Kubernetes labels and annotations to support various features: ### Labels in YuniKorn -| Name| Description | -| --- | --- | -| `applicationId` | Associates this pod with an application. | -| `queue` | Selects the YuniKorn queue this application should be scheduled in. This may be ignored if a placement policy is in effect. | -| `SparkLabelAppID ` | Alternative method of specifying `applicationId` used by Spark Operator if the label `applicationId` and annotation `yunikorn.apache.org/app-id` unset. | -| `disableStateAware` | If present, disables the YuniKorn state-aware scheduling policy for this pod. Set internally by the YuniKorn admission controller. | -| `placeholder` | Set if this pod represents a placeholder for gang scheduling. Set internally by YuniKorn. | +| Name | Description | +|--|-| +| `applicationId` | Associates this pod with an application. | +| `queue` | Selects the YuniKorn queue this application should be scheduled in. This may be ignored if a placement policy is in effect. | +| `spark-app-selector` | Alternative method of specifying `applicationId` used by Spark Operator if the label `applicationId` and annotation `yunikorn.apache.org/app-id` unset. | +| `disableStateAware` | If present, disables the YuniKorn state-aware scheduling policy for this pod. Set internally by the YuniKorn admission controller. | +| `placeholder`| Set if this pod represents a placeholder for gang scheduling. Set internally by YuniKorn. | ### Annotations in YuniKorn All annotations are under the namespace `yunikorn.apache.org`. For example `yunikorn.apache.org/app-id`. -| Name| Description | -| --- | --- | -| `app-id`| Assoiates this pod with an application.The priority of applicationID is determined by: annotation
(yunikorn-site) branch master updated: [YUNIKORN-2224] add preemption to core feature (#375)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new c4fbb6934c [YUNIKORN-2224] add preemption to core feature (#375) c4fbb6934c is described below commit c4fbb6934cf36636006c5abcb42ac52eb65df617 Author: DouPache AuthorDate: Thu Jan 25 16:12:29 2024 +1100 [YUNIKORN-2224] add preemption to core feature (#375) Closes: #375 Signed-off-by: Wilfred Spiegelenburg --- docs/get_started/core_features.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/get_started/core_features.md b/docs/get_started/core_features.md index 8c9076fe50..ddb3039219 100644 --- a/docs/get_started/core_features.md +++ b/docs/get_started/core_features.md @@ -74,6 +74,9 @@ This mechanism can avoid this pod gets starved by later submitted smaller, less- This feature is important in the batch workloads scenario because when a large amount of heterogeneous pods is submitted to the cluster, it's very likely some pods can be starved even they are submitted much earlier. +## Preemption +YuniKorn's preemption feature allows higher-priority tasks to dynamically reallocate resources by preempting lower-priority ones, ensuring critical workloads get necessary resources in a multi-tenant Kubernetes environment. This proactive mechanism maintains system stability and fairness, integrating with Kubernetes' priority classes and YuniKorn's hierarchical queue system. + ## Throughput Throughput is a key criterion to measure scheduler performance. It is critical for a large scale distributed system. If throughput is bad, applications may waste time on waiting for scheduling, and further impact service SLAs. - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-2111] Update core application states (#364)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 899aa4af13 [YUNIKORN-2111] Update core application states (#364) 899aa4af13 is described below commit 899aa4af13488103d0adcefa22c972858c020c38 Author: Yu-Lin Chen AuthorDate: Thu Jan 25 16:05:39 2024 +1100 [YUNIKORN-2111] Update core application states (#364) Moved scheduler_object_states.md from design folder to developer_guider folder. Updated links to the document and a new graph based on the latest states for the application. Closes: #364 Signed-off-by: Wilfred Spiegelenburg --- docs/assets/application-state.png | Bin 117964 -> 136449 bytes docs/design/gang_scheduling.md | 2 +- .../scheduler_object_states.md | 5 ++--- docs/user_guide/sorting_policies.md| 2 +- .../current/user_guide/sorting_policies.md | 2 +- sidebars.js| 2 +- 6 files changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/assets/application-state.png b/docs/assets/application-state.png index 6c7f27e092..2af46e96e1 100644 Binary files a/docs/assets/application-state.png and b/docs/assets/application-state.png differ diff --git a/docs/design/gang_scheduling.md b/docs/design/gang_scheduling.md index 1a09f729bc..44560ef11b 100644 --- a/docs/design/gang_scheduling.md +++ b/docs/design/gang_scheduling.md @@ -181,7 +181,7 @@ The scheduler processes the AllocationAsk as follows: ## Application completion Application completion has been a long standing issue. Currently, applications do not transition to a _completed_ state when done. -The current states for the application are [documented here](./scheduler_object_states.md). +The current states for the application are [documented here](developer_guide/scheduler_object_states.md). However, at this point in time an application will not reach the _completed_ state and will be stuck in _waiting_. This provides a number of issues specifically around memory usage and cleanup of queues in long running deployments. diff --git a/docs/design/scheduler_object_states.md b/docs/developer_guide/scheduler_object_states.md similarity index 97% rename from docs/design/scheduler_object_states.md rename to docs/developer_guide/scheduler_object_states.md index 306e2eb031..86cc135aea 100644 --- a/docs/design/scheduler_object_states.md +++ b/docs/developer_guide/scheduler_object_states.md @@ -60,11 +60,10 @@ This is a final state and after this state the application cannot be tracked any * Failing: An application marked for failing, what still has some allocations or asks what needs to be cleaned up before entering into the Failed state. The application can be Failing when the partition it belongs to is removed or during gang scheduling, if the placeholder processing times out, and the application has no real allocations yet. * Failed: An application is considered failed when it was marked for failure and all the pending requests and allocations were already removed. -This is a final state. The application cannot change state after entering. +From this state the application can only move to the Expired state. * Rejected: The application was rejected when it was added to the scheduler. This only happens when a resource manager tries to add a new application, when it gets created in a New state, and the scheduler rejects the creation. -Applications can be rejected due ACLs denying access to a queue the application has specified, or a placement via placement rules has failed. -This is a final state. The application cannot change state after entering. +Applications can be rejected due ACLs denying access to a queue the application has specified, or a placement via placement rules has failed. From this state the application can only move to the Expired state. The events that can trigger a state change: * Reject: rejecting the application by the scheduler (source: core scheduler) diff --git a/docs/user_guide/sorting_policies.md b/docs/user_guide/sorting_policies.md index f806f9ef1b..982877a138 100644 --- a/docs/user_guide/sorting_policies.md +++ b/docs/user_guide/sorting_policies.md @@ -92,7 +92,7 @@ Short description: limit of one (1) application in Starting or Accepted state Config value: `stateaware` This sorting policy requires an understanding of the application states. -Applications states are described in the [application states](design/scheduler_object_states.md#application-state) documentation. +Applications states are described in the [application states](developer_guide/scheduler_object_states.md#application-state) documentation. Before sorting applications the following filters are applied to all applications in
(yunikorn-web) branch master updated: [YUNIKORN-2008] Hide empty columns in node view (#148)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git
The following commit(s) were added to refs/heads/master by this push:
new 9bbe0dd [YUNIKORN-2008] Hide empty columns in node view (#148)
9bbe0dd is described below
commit 9bbe0dd1f7fb69f2ef019d0d5e7fd71b3b8c5688
Author: wusamzong
AuthorDate: Thu Jan 25 15:47:00 2024 +1100
[YUNIKORN-2008] Hide empty columns in node view (#148)
Columns that only contain empty ivalues or have all 'n/a' values are no
longer rendered in the table for nodes.
Closes: #148
Signed-off-by: Wilfred Spiegelenburg
---
.../components/nodes-view/nodes-view.component.ts | 31 +-
1 file changed, 18 insertions(+), 13 deletions(-)
diff --git a/src/app/components/nodes-view/nodes-view.component.ts
b/src/app/components/nodes-view/nodes-view.component.ts
index a9c77d7..9bcf58a 100644
--- a/src/app/components/nodes-view/nodes-view.component.ts
+++ b/src/app/components/nodes-view/nodes-view.component.ts
@@ -193,21 +193,26 @@ export class NodesViewComponent implements OnInit {
}
formatColumn(){
-if(this.nodeDataSource.data.length==0){
- return
+if(this.nodeDataSource.data.length===0){
+ return;
}
this.nodeColumnIds.forEach((colId)=>{
- let emptyCell=this.nodeDataSource.data.filter((node: NodeInfo)=>{
-if (colId === 'indicatorIcon'){
- return false;
-}
-if (!(colId in node)) {
- console.error(`Property '${colId}' does not exist on Node.`);
- return false;
-}
-return (node as any)[colId]==="" || (node as any)[colId]==="n/a";
- })
- if (emptyCell.length==this.nodeDataSource.data.length){
+ if (colId==='indicatorIcon'){
+return;
+ }
+
+ // Verify whether all cells in the column are empty.
+ let isEmpty:boolean = true;
+ Object.values(this.nodeDataSource.data).forEach((node) => {
+Object.entries(node).forEach(entry => {
+ const [key, value] = entry;
+ if (key===colId && !(value==='' || value==='n/a')){
+isEmpty=false;
+ }
+});
+ });
+
+ if (isEmpty){
this.nodeColumnIds = this.nodeColumnIds.filter(el => el!==colId);
this.nodeColumnIds = this.nodeColumnIds.filter(colId =>
colId!=="attributes");
}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) branch master updated: [YUNIKORN-1362] Filtering node list on UI (#153)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git
The following commit(s) were added to refs/heads/master by this push:
new 1a51966 [YUNIKORN-1362] Filtering node list on UI (#153)
1a51966 is described below
commit 1a51966f637852b8e1ef53e85744cdf3cb4b
Author: wusamzong
AuthorDate: Thu Jan 25 14:16:55 2024 +1100
[YUNIKORN-1362] Filtering node list on UI (#153)
Filter nodes based on the node attributes:
* Attributes are exposed in the node table (detail mode)
* A filter input has been introduced
* Nodes not matching filter are hidden
* The filter results are highlighted
Two nodes have been explicitly added to db.json for testing and validating
the filter functionality.
Closes: #153
Signed-off-by: Wilfred Spiegelenburg
---
json-db.json | 130 +
src/app/app.module.ts | 2 +
.../nodes-view/highlighttable-search.pipe.ts} | 31 +++--
.../nodes-view/nodes-view.component.html | 48 ++--
.../nodes-view/nodes-view.component.scss | 56 ++---
.../nodes-view/nodes-view.component.spec.ts| 8 ++
.../components/nodes-view/nodes-view.component.ts | 71 ++-
src/app/models/node-info.model.ts | 7 +-
src/app/services/scheduler/scheduler.service.ts| 3 +-
src/styles.scss| 5 -
10 files changed, 306 insertions(+), 55 deletions(-)
diff --git a/json-db.json b/json-db.json
index d495481..4dede38 100644
--- a/json-db.json
+++ b/json-db.json
@@ -589,6 +589,20 @@
"nodes": [
{
"nodeID": "lima-rancher-desktop",
+ "attributes":{
+"beta.kubernetes.io/arch":"amd64",
+"beta.kubernetes.io/os":"linux",
+"kubernetes.io/arch":"amd64",
+"kubernetes.io/hostname":"lima-rancher-desktop",
+"kubernetes.io/os":"linux",
+"node-role.kubernetes.io/control-plane":"",
+"node.kubernetes.io/exclude-from-external-load-balancers":"",
+"ready":"true",
+"si.io/hostname":"lima-rancher-desktop",
+"si.io/rackname":"/rack-default",
+"si/instance-type":"",
+"si/node-partition":"[mycluster]default"
+ },
"hostName": "",
"rackName": "",
"capacity": {
@@ -749,6 +763,122 @@
"schedulable": true,
"isReserved": false,
"reservations": []
+},
+{
+ "nodeID": "lima-rancher-desktop2",
+ "attributes":{
+"beta.kubernetes.io/arch":"arm64",
+"beta.kubernetes.io/os":"linux",
+"kubernetes.io/arch":"arm64",
+"kubernetes.io/hostname":"lima-rancher-desktop2",
+"kubernetes.io/os":"linux",
+"node-role.kubernetes.io/control-plane":"",
+"node.kubernetes.io/exclude-from-external-load-balancers":"",
+"ready":"true",
+"si.io/hostname":"lima-rancher-desktop2",
+"si.io/rackname":"/rack-default",
+"si/instance-type":"",
+"si/node-partition":"[mycluster]default"
+ },
+ "hostName": "",
+ "rackName": "",
+ "capacity": {
+"ephemeral-storage": 99833802265,
+"hugepages-1Gi": 0,
+"hugepages-2Mi": 0,
+"hugepages-32Mi": 0,
+"hugepages-64Ki": 0,
+"memory": 4110970880,
+"pods": 110,
+"vcore": 2000
+ },
+ "allocated": {
+"memory": 0,
+"pods": 0,
+"vcore": 0
+ },
+ "occupied": {
+"memory": 0,
+"pods": 0,
+"vcore": 0
+ },
+ "available": {
+"ephemeral-storage": 99833802265,
+"hugepages-1Gi": 0,
+"hugepages-2Mi": 0,
+"hugepages-32Mi": 0,
+"hugepages-64Ki": 0,
+"memory": 4110970880,
+"pods": 110,
+"vcore": 2000
+ },
+ "utilized": {
+"memory": 0,
+
(yunikorn-web) branch master updated: [YUNIKORN-2198] Format 'utilized' column correctly (#152)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git The following commit(s) were added to refs/heads/master by this push: new a3b8f5d [YUNIKORN-2198] Format 'utilized' column correctly (#152) a3b8f5d is described below commit a3b8f5dbb959327f647b44f91672bee781997842 Author: wusamzong AuthorDate: Thu Jan 25 14:04:49 2024 +1100 [YUNIKORN-2198] Format 'utilized' column correctly (#152) The 'utilized' column in the 'nodes' table should be formatted as a resource list nit as a set of comma seperated values of strings. Closes: #152 Signed-off-by: Wilfred Spiegelenburg --- src/app/components/nodes-view/nodes-view.component.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/app/components/nodes-view/nodes-view.component.html b/src/app/components/nodes-view/nodes-view.component.html index c7ead42..2b9ea57 100644 --- a/src/app/components/nodes-view/nodes-view.component.html +++ b/src/app/components/nodes-view/nodes-view.component.html @@ -44,7 +44,7 @@ - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) branch master updated (69e0eb8 -> 221662f)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git from 69e0eb8 [YUNIKORN-2231] Show node list when hovering mouse over the node utitutilization bar chart (#157) add 221662f [YUNIKORN-2081] Update karma in web UI test (#151) No new revisions were added by this update. Summary of changes: package.json | 3 +-- src/app/app.component.spec.ts | 5 ++-- .../app-history/app-history.component.spec.ts | 5 ++-- .../app-status/app-status.component.spec.ts| 5 ++-- .../area-chart/area-chart.component.spec.ts| 5 ++-- .../container-history.component.spec.ts| 5 ++-- .../container-status.component.spec.ts | 5 ++-- .../dashboard/dashboard.component.spec.ts | 5 ++-- .../donut-chart/donut-chart.component.spec.ts | 5 ++-- .../error-view/error-view.component.spec.ts| 5 ++-- .../nodes-view/nodes-view.component.spec.ts| 5 ++-- .../queue-rack/queue-rack.component.spec.ts| 5 ++-- .../queues-view/queues-view.component.spec.ts | 5 ++-- .../api-error/api-error.interceptor.spec.ts| 5 ++-- .../services/envconfig/envconfig.service.spec.ts | 5 ++-- .../services/event-bus/event-bus.service.spec.ts | 5 ++-- yarn.lock | 30 +++--- 17 files changed, 40 insertions(+), 68 deletions(-) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) branch master updated: [YUNIKORN-2140] Improve presentation of resources (#146)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git
The following commit(s) were added to refs/heads/master by this push:
new 9ec0016 [YUNIKORN-2140] Improve presentation of resources (#146)
9ec0016 is described below
commit 9ec0016b2a3ad3570422c82f6ed12f55dfb9d7ef
Author: Yu-Lin Chen
AuthorDate: Tue Dec 19 18:13:33 2023 +1100
[YUNIKORN-2140] Improve presentation of resources (#146)
Format resources with units instead of simple integers.
CPU: m, (bare), k, M, G, T, P, E (powers of 10, no units)
Memory: B, KiB, MiB, GiB, TiB, PiB, EiB (powers of 2, byte units)
Ephemeral storage: B, KB, MB, GB, TB, PB, EB (powers of 10, byte units)
All others: (bare), k, M, G, T, P, E (powers of 10, no units)
Replace n/a with 0 for values that are not given. Use n/a if the
resource is not set at all.
Closes: #146
Signed-off-by: Wilfred Spiegelenburg
---
.../services/scheduler/scheduler.service.spec.ts | 114 -
src/app/services/scheduler/scheduler.service.ts| 95 +
src/app/utils/common.util.spec.ts | 37 +--
src/app/utils/common.util.ts | 49 ++---
4 files changed, 227 insertions(+), 68 deletions(-)
diff --git a/src/app/services/scheduler/scheduler.service.spec.ts
b/src/app/services/scheduler/scheduler.service.spec.ts
index 3e10742..2a73c3a 100644
--- a/src/app/services/scheduler/scheduler.service.spec.ts
+++ b/src/app/services/scheduler/scheduler.service.spec.ts
@@ -20,25 +20,129 @@ import {HttpClientTestingModule} from
'@angular/common/http/testing';
import {TestBed} from '@angular/core/testing';
import {EnvconfigService} from '@app/services/envconfig/envconfig.service';
import {MockEnvconfigService} from '@app/testing/mocks';
-import {configureTestSuite} from 'ng-bullet';
import {SchedulerService} from './scheduler.service';
+import {SchedulerResourceInfo} from '@app/models/resource-info.model';
+import {NOT_AVAILABLE} from '@app/utils/constants';
describe('SchedulerService', () => {
let service: SchedulerService;
- configureTestSuite(() => {
+ beforeEach(() => {
TestBed.configureTestingModule({
imports: [HttpClientTestingModule],
providers: [SchedulerService, { provide: EnvconfigService, useValue:
MockEnvconfigService }],
});
- });
-
- beforeEach(() => {
service = TestBed.inject(SchedulerService);
});
it('should create the service', () => {
expect(service).toBeTruthy();
+
+ });
+
+ it('should format SchedulerResourceInfo correctly', () => {
+type TestCase = {
+ description: string;
+ schedulerResourceInfo: SchedulerResourceInfo;
+ expected: string;
+};
+
+const testCases: TestCase[] = [
+ {
+description: 'test simple resourceInfo',
+schedulerResourceInfo: {
+ 'memory': 1024,
+ 'vcore': 2,
+},
+expected: 'Memory: 1 KiB, CPU: 2m'
+ },
+ {
+description: 'test undefined resourceInfo',
+schedulerResourceInfo : undefined as any,
+expected: `${NOT_AVAILABLE}`
+ },
+ {
+description: 'test empty resourceInfo',
+schedulerResourceInfo : {} as any,
+expected: `${NOT_AVAILABLE}`
+ },
+ {
+description: 'Test zero values',
+schedulerResourceInfo: {
+ 'memory': 0,
+ 'vcore': 0,
+ 'ephemeral-storage': 0,
+ 'hugepages-2Mi': 0,
+ 'hugepages-1Gi': 0,
+ 'pods': 0
+},
+expected: 'Memory: 0 B, CPU: 0, pods: 0, ephemeral-storage: 0 B,
hugepages-1Gi: 0 B, hugepages-2Mi: 0 B'
+ },
+ {
+description: 'Test resource ordering',
+schedulerResourceInfo: {
+ 'ephemeral-storage': 2048,
+ 'memory': 1024,
+ 'vcore': 2,
+ 'TPU': 3,
+ 'GPU': 4,
+ 'hugepages-2Mi':2097152,
+ 'hugepages-1Gi':1073741824,
+ 'pods': 1
+},
+expected: 'Memory: 1 KiB, CPU: 2m, pods: 10k, ephemeral-storage: 2.05
kB, GPU: 40k, hugepages-1Gi: 1 GiB, hugepages-2Mi: 2 MiB, TPU: 30k'
+ }
+];
+
+testCases.forEach((testCase: TestCase) => {
+ const result = (service as
any).formatResource(testCase.schedulerResourceInfo); // ignore type typecheck
to access private method
+ expect(result).toEqual(testCase.expected);
+});
+ });
+
+
+ it('should fo
(yunikorn-core) branch master updated: [YUNIKORN-2246] remove deprecated FitIn function (#751)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 4816aa25 [YUNIKORN-2246] remove deprecated FitIn function (#751)
4816aa25 is described below
commit 4816aa2511e9ebd0a685e8a1c3912e4cbd2e9b46
Author: Kuan-Po Tseng
AuthorDate: Tue Dec 12 11:42:21 2023 +1100
[YUNIKORN-2246] remove deprecated FitIn function (#751)
Closes: #751
Signed-off-by: Wilfred Spiegelenburg
---
pkg/common/resources/resources.go | 8
pkg/common/resources/resources_test.go | 2 +-
pkg/scheduler/objects/preemption.go| 6 +++---
pkg/scheduler/objects/queue.go | 2 +-
4 files changed, 5 insertions(+), 13 deletions(-)
diff --git a/pkg/common/resources/resources.go
b/pkg/common/resources/resources.go
index 7b64fe1a..26156100 100644
--- a/pkg/common/resources/resources.go
+++ b/pkg/common/resources/resources.go
@@ -417,14 +417,6 @@ func subNonNegative(left, right *Resource) (*Resource,
string) {
return out, message
}
-// FitIn Checks if smaller fits in larger
-// Types not defined in the larger resource are considered 0 values for
Quantity
-// A nil resource is treated as an empty resource (all types are 0)
-// Deprecated: use receiver version Resource.FitIn
-func FitIn(larger, smaller *Resource) bool {
- return larger.fitIn(smaller, false)
-}
-
// FitIn checks if smaller fits in the defined resource
// Types not defined in resource this is called against are considered 0 for
Quantity
// A nil resource is treated as an empty resource (no types defined)
diff --git a/pkg/common/resources/resources_test.go
b/pkg/common/resources/resources_test.go
index 9be92d79..3917dd6f 100644
--- a/pkg/common/resources/resources_test.go
+++ b/pkg/common/resources/resources_test.go
@@ -1051,7 +1051,7 @@ func TestFitIn(t *testing.T) {
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
- assert.Equal(t, FitIn(tt.larger, tt.smaller), tt.want,
"unexpected FitIn result (parameter method)")
+ assert.Equal(t, tt.larger.FitIn(tt.smaller), tt.want,
"unexpected FitIn result (parameter method)")
assert.Equal(t, tt.larger.FitIn(tt.smaller), tt.want,
"unexpected FitIn result (receiver method)")
})
}
diff --git a/pkg/scheduler/objects/preemption.go
b/pkg/scheduler/objects/preemption.go
index 1dc43d89..ebf82da8 100644
--- a/pkg/scheduler/objects/preemption.go
+++ b/pkg/scheduler/objects/preemption.go
@@ -213,7 +213,7 @@ func (p *Preemptor) calculateVictimsByNode(nodeAvailable
*resources.Resource, po
// Initial check: Will allocation fit on node without preemption? This
is possible if preemption was triggered due
// to queue limits and not node resource limits.
- if resources.FitIn(nodeCurrentAvailable, p.ask.GetAllocatedResource()) {
+ if nodeCurrentAvailable.FitIn(p.ask.GetAllocatedResource()) {
// return empty list so this node is considered for preemption
return -1, make([]*Allocation, 0)
}
@@ -713,7 +713,7 @@ func (qps *QueuePreemptionSnapshot)
IsAtOrAboveGuaranteedResource() bool {
used := resources.Sub(qps.AllocatedResource, qps.PreemptingResource)
// if we don't fit, we're clearly above
- if !resources.FitIn(absGuaranteed, used) {
+ if !absGuaranteed.FitIn(used) {
return true
}
@@ -739,7 +739,7 @@ func (qps *QueuePreemptionSnapshot)
IsWithinGuaranteedResource() bool {
max := qps.GetMaxResource()
absGuaranteed := resources.ComponentWiseMinPermissive(guaranteed, max)
used := resources.Sub(qps.AllocatedResource, qps.PreemptingResource)
- return resources.FitIn(absGuaranteed, used)
+ return absGuaranteed.FitIn(used)
}
func (qps *QueuePreemptionSnapshot) GetRemainingGuaranteed()
*resources.Resource {
diff --git a/pkg/scheduler/objects/queue.go b/pkg/scheduler/objects/queue.go
index de02b35f..9ae27f9d 100644
--- a/pkg/scheduler/objects/queue.go
+++ b/pkg/scheduler/objects/queue.go
@@ -1059,7 +1059,7 @@ func (sq *Queue) DecAllocatedResource(alloc
*resources.Resource) error {
defer sq.Unlock()
// check this queue: failure stops checks
- if alloc != nil && !resources.FitIn(sq.allocatedResource, alloc) {
+ if alloc != nil && !sq.allocatedResource.FitIn(alloc) {
return fmt.Errorf("released allocation (%v) is larger than '%s'
queue allocation (%v)",
alloc, sq.QueuePath, sq.allocatedResource)
}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2265] Populate Originator and PreemptionPolicy fields in SI Allocation (#756)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 8c066c42 [YUNIKORN-2265] Populate Originator and PreemptionPolicy
fields in SI Allocation (#756)
8c066c42 is described below
commit 8c066c42b83a912150ca66ee01f3476d3eeca29f
Author: Craig Condit
AuthorDate: Tue Dec 12 11:41:12 2023 +1100
[YUNIKORN-2265] Populate Originator and PreemptionPolicy fields in SI
Allocation (#756)
Closes: #756
Signed-off-by: Wilfred Spiegelenburg
---
go.mod | 2 +-
go.sum | 4 +--
pkg/scheduler/objects/allocation.go | 8 ++
pkg/scheduler/objects/allocation_test.go | 45 ++--
4 files changed, 54 insertions(+), 5 deletions(-)
diff --git a/go.mod b/go.mod
index 824ce23e..6ba7d67a 100644
--- a/go.mod
+++ b/go.mod
@@ -22,7 +22,7 @@ module github.com/apache/yunikorn-core
go 1.20
require (
- github.com/apache/yunikorn-scheduler-interface
v0.0.0-20231201001639-c81397b31653
+ github.com/apache/yunikorn-scheduler-interface
v0.0.0-20231211235204-ec7bfad7d00e
github.com/google/btree v1.1.2
github.com/google/go-cmp v0.5.9
github.com/google/uuid v1.3.0
diff --git a/go.sum b/go.sum
index d9de3a73..ecb2ed5f 100644
--- a/go.sum
+++ b/go.sum
@@ -40,8 +40,8 @@ github.com/alecthomas/template
v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod
h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod
h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod
h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
-github.com/apache/yunikorn-scheduler-interface
v0.0.0-20231201001639-c81397b31653
h1:pUbVmmR+LWuy0L8dGCZNue9UNpWKsY7yFYcCtPtWAic=
-github.com/apache/yunikorn-scheduler-interface
v0.0.0-20231201001639-c81397b31653/go.mod
h1:zDWV5y9Zh9DM1C65RCVXT1nhNNO8kykVW7bzPFamNYw=
+github.com/apache/yunikorn-scheduler-interface
v0.0.0-20231211235204-ec7bfad7d00e
h1:WiDns+JSNrp1jUfTkwtTwVyfxAhe3vPMtxJxs2CRseE=
+github.com/apache/yunikorn-scheduler-interface
v0.0.0-20231211235204-ec7bfad7d00e/go.mod
h1:zDWV5y9Zh9DM1C65RCVXT1nhNNO8kykVW7bzPFamNYw=
github.com/benbjohnson/clock v1.1.0
h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod
h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod
h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
diff --git a/pkg/scheduler/objects/allocation.go
b/pkg/scheduler/objects/allocation.go
index f165ad56..9fefcb83 100644
--- a/pkg/scheduler/objects/allocation.go
+++ b/pkg/scheduler/objects/allocation.go
@@ -153,6 +153,9 @@ func NewAllocationFromSI(alloc *si.Allocation) *Allocation {
placeholder: alloc.Placeholder,
createTime:time.Unix(creationTime, 0),
allocLog: make(map[string]*AllocationLogEntry),
+ originator:alloc.Originator,
+ allowPreemptSelf: alloc.PreemptionPolicy.GetAllowPreemptSelf(),
+ allowPreemptOther:
alloc.PreemptionPolicy.GetAllowPreemptOther(),
}
newAlloc := NewAllocation(alloc.NodeID, ask)
newAlloc.allocationID = alloc.AllocationID
@@ -175,6 +178,11 @@ func (a *Allocation) NewSIFromAllocation() *si.Allocation {
ResourcePerAlloc: a.GetAllocatedResource().ToProto(), // needed
in tests for restore
TaskGroupName:a.GetTaskGroup(),
Placeholder: a.IsPlaceholder(),
+ Originator: a.GetAsk().IsOriginator(),
+ PreemptionPolicy: &si.PreemptionPolicy{
+ AllowPreemptSelf: a.GetAsk().IsAllowPreemptSelf(),
+ AllowPreemptOther: a.GetAsk().IsAllowPreemptOther(),
+ },
}
}
diff --git a/pkg/scheduler/objects/allocation_test.go
b/pkg/scheduler/objects/allocation_test.go
index be218109..4d92ebca 100644
--- a/pkg/scheduler/objects/allocation_test.go
+++ b/pkg/scheduler/objects/allocation_test.go
@@ -24,7 +24,6 @@ import (
"testing"
"time"
- "github.com/google/go-cmp/cmp/cmpopts"
"gotest.tools/v3/assert"
"github.com/apache/yunikorn-core/pkg/common/resources"
@@ -136,14 +135,37 @@ func TestSIFromAlloc(t *testing.T) {
NodeID: "node-1",
ApplicationID:"app-1",
ResourcePerAlloc: res.ToProto(),
+ Originator: true,
+ PreemptionPol
(yunikorn-k8shim) branch master updated: [YUNIKORN-2037] Testing the throughput of YuniKorn (#724)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git
The following commit(s) were added to refs/heads/master by this push:
new c01047fb [YUNIKORN-2037] Testing the throughput of YuniKorn (#724)
c01047fb is described below
commit c01047fb1456266f911bf3c010544ea7b2a121a6
Author: wusamzong
AuthorDate: Thu Dec 7 11:46:14 2023 +1100
[YUNIKORN-2037] Testing the throughput of YuniKorn (#724)
Add scripts for setting up KWOK based throughput tests.
* kwok-setup.sh: Initiates a Kwok instance within a cluster and generates
required nodes based on user specifications.
* deploy-tool.sh: Creates or deletes a specified number of applications,
along with the desired number of replicas for each application.
Closes: #724
Signed-off-by: Wilfred Spiegelenburg
---
deployments/kwok-perf-test/deploy-tool.sh | 121 ++
deployments/kwok-perf-test/kwok-setup.sh | 76 +++
2 files changed, 197 insertions(+)
diff --git a/deployments/kwok-perf-test/deploy-tool.sh
b/deployments/kwok-perf-test/deploy-tool.sh
new file mode 100755
index ..64bb64fd
--- /dev/null
+++ b/deployments/kwok-perf-test/deploy-tool.sh
@@ -0,0 +1,121 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+show_help() {
+ cat << EOF
+Invalid option: -$OPTARG
+Usage: $0 [-d] [-i ]
+
+Options:
+ -d, --delete Delete the specified number of deployments.
+ -i, --interval Set the interval between deployments in seconds.
+
+Arguments:
+ Number of deployments to create or delete
(required).
+ Number of replicas for each deployment (required).
+EOF
+}
+
+deploy_deployments() {
+ for (( i=0; ihttp://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+if [ $# -eq 0 ]; then
+ echo "Error: Please provide the number of nodes to create."
+ echo "Usage: $0 "
+ exit 1
+fi
+
+KWOK_REPO=kubernetes-sigs/kwok
+KWOK_LATEST_RELEASE=$(curl
"https://api.github.com/repos/${KWOK_REPO}/releases/latest"; | jq -r '.tag_name')
+kubectl apply -f
"https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwok.yaml";
+kubectl apply -f
"https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/stage-fast.yaml";
+
+for (( i=0;i<$1; i++))
+do
+ kubectl apply -f - <
(yunikorn-scheduler-interface) branch master updated: [YUNIKORN-2211] Replace Allocation uuid with allocationID (#113)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository
https://gitbox.apache.org/repos/asf/yunikorn-scheduler-interface.git
The following commit(s) were added to refs/heads/master by this push:
new c81397b [YUNIKORN-2211] Replace Allocation uuid with allocationID
(#113)
c81397b is described below
commit c81397b316531102078e5412d02315204ba50c3c
Author: Manikandan R
AuthorDate: Fri Dec 1 11:16:39 2023 +1100
[YUNIKORN-2211] Replace Allocation uuid with allocationID (#113)
Closes: #113
Signed-off-by: Wilfred Spiegelenburg
---
lib/go/si/si.pb.go | 610 ++--
scheduler-interface-spec.md | 17 +-
si.proto| 17 +-
3 files changed, 329 insertions(+), 315 deletions(-)
diff --git a/lib/go/si/si.pb.go b/lib/go/si/si.pb.go
index 1bb07d0..204efc4 100644
--- a/lib/go/si/si.pb.go
+++ b/lib/go/si/si.pb.go
@@ -1856,8 +1856,6 @@ type Allocation struct {
AllocationKey string `protobuf:"bytes,1,opt,name=allocationKey,proto3"
json:"allocationKey,omitempty"`
// Allocation tags from AllocationAsk
AllocationTags map[string]string
`protobuf:"bytes,2,rep,name=allocationTags,proto3"
json:"allocationTags,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3"
protobuf_val:"bytes,2,opt,name=value,proto3"`
- // UUID of the allocation
- UUID string `protobuf:"bytes,3,opt,name=UUID,proto3"
json:"UUID,omitempty"`
// Resource for each allocation
ResourcePerAlloc *Resource
`protobuf:"bytes,5,opt,name=resourcePerAlloc,proto3"
json:"resourcePerAlloc,omitempty"`
// Priority of ask
@@ -1873,6 +1871,8 @@ type Allocation struct {
// Is this a placeholder allocation (true) or a real allocation
(false), defaults to false
// ignored if the taskGroupName is not set
Placeholder bool `protobuf:"varint,12,opt,name=placeholder,proto3"
json:"placeholder,omitempty"`
+ // AllocationID of the allocation
+ AllocationID string `protobuf:"bytes,13,opt,name=allocationID,proto3"
json:"allocationID,omitempty"`
}
func (x *Allocation) Reset() {
@@ -1921,13 +1921,6 @@ func (x *Allocation) GetAllocationTags()
map[string]string {
return nil
}
-func (x *Allocation) GetUUID() string {
- if x != nil {
- return x.UUID
- }
- return ""
-}
-
func (x *Allocation) GetResourcePerAlloc() *Resource {
if x != nil {
return x.ResourcePerAlloc
@@ -1977,6 +1970,13 @@ func (x *Allocation) GetPlaceholder() bool {
return false
}
+func (x *Allocation) GetAllocationID() string {
+ if x != nil {
+ return x.AllocationID
+ }
+ return ""
+}
+
type AllocationReleasesRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
@@ -2046,15 +2046,15 @@ type AllocationRelease struct {
PartitionName string `protobuf:"bytes,1,opt,name=partitionName,proto3"
json:"partitionName,omitempty"`
// The application the allocation belongs to
ApplicationID string `protobuf:"bytes,2,opt,name=applicationID,proto3"
json:"applicationID,omitempty"`
- // The UUID of the allocation to release, if not set all allocations
are released for
- // the applicationID
- UUID string `protobuf:"bytes,3,opt,name=UUID,proto3"
json:"UUID,omitempty"`
// Termination type of the released allocation
TerminationType TerminationType
`protobuf:"varint,4,opt,name=terminationType,proto3,enum=si.v1.TerminationType"
json:"terminationType,omitempty"`
// human-readable message
Message string `protobuf:"bytes,5,opt,name=message,proto3"
json:"message,omitempty"`
// AllocationKey from AllocationAsk
AllocationKey string `protobuf:"bytes,6,opt,name=allocationKey,proto3"
json:"allocationKey,omitempty"`
+ // AllocationID of the allocation to release, if not set all
allocations are released for
+ // the applicationID
+ AllocationID string `protobuf:"bytes,7,opt,name=allocationID,proto3"
json:"allocationID,omitempty"`
}
func (x *AllocationRelease) Reset() {
@@ -2103,13 +2103,6 @@ func (x *AllocationRelease) GetApplicationID() string {
return ""
}
-func (x *AllocationRelease) GetUUID() string {
- if x != nil {
- return x.UUID
- }
- return ""
-}
-
func (x *AllocationRelease) GetTerminationType() TerminationType {
if x != nil {
return x.TerminationType
@@ -2131,6 +2124,13 @@ func (x *AllocationRelease) GetAllocationKe
(yunikorn-core) branch master updated: [YUNIKORN-2204] Use ask unique id for allocation (#740)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new b516ca13 [YUNIKORN-2204] Use ask unique id for allocation (#740)
b516ca13 is described below
commit b516ca13bf10df1e87a6350abe7ef39919ef6f07
Author: Manikandan R
AuthorDate: Thu Nov 30 19:45:04 2023 +1100
[YUNIKORN-2204] Use ask unique id for allocation (#740)
Do not generate a new UUID for an allocation when it gets added. Re-use
the ask key which is the same as the K8s pod UID to improve on the
troubleshooting experience.
The new allocationID will be the ask key followed by the current repeat
counter, seperated by a dash '-'
The UUID is renamed to allocationID on the allocation. Scheduler
interface changes will follow. Replaced GetUUID with GetAllocationID in
already modified lines
Closes: #740
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/health_checker_test.go | 4 +-
pkg/scheduler/objects/allocation.go| 37 --
pkg/scheduler/objects/allocation_ask.go| 7 ++
pkg/scheduler/objects/allocation_test.go | 11 +-
pkg/scheduler/objects/application.go | 7 +-
pkg/scheduler/objects/application_events_test.go | 12 +-
pkg/scheduler/objects/application_test.go | 140 ++---
pkg/scheduler/objects/node_collection_test.go | 9 +-
pkg/scheduler/objects/node_test.go | 57 +
pkg/scheduler/objects/nodesorting_test.go | 7 +-
pkg/scheduler/objects/preemption_test.go | 14 +--
pkg/scheduler/objects/queue_test.go| 10 +-
.../objects/required_node_preemptor_test.go| 20 +--
pkg/scheduler/objects/utilities_test.go| 8 +-
pkg/scheduler/partition_test.go| 69 +-
pkg/scheduler/tests/smoke_test.go | 4 +-
pkg/webservice/handlers_test.go| 27 ++--
17 files changed, 235 insertions(+), 208 deletions(-)
diff --git a/pkg/scheduler/health_checker_test.go
b/pkg/scheduler/health_checker_test.go
index a8043ed3..a3f97f2a 100644
--- a/pkg/scheduler/health_checker_test.go
+++ b/pkg/scheduler/health_checker_test.go
@@ -200,7 +200,7 @@ func TestGetSchedulerHealthStatusContext(t *testing.T) {
// add orphan allocation to a node
node := schedulerContext.partitions[partName].nodes.GetNode("node")
- alloc := objects.NewAllocation(allocID, "node", newAllocationAsk("key",
"appID", resources.NewResource()))
+ alloc := objects.NewAllocation("node", newAllocationAsk("key", "appID",
resources.NewResource()))
node.AddAllocation(alloc)
healthInfo = GetSchedulerHealthStatus(schedulerMetrics,
schedulerContext)
assert.Assert(t, !healthInfo.Healthy, "Scheduler should not be healthy")
@@ -216,7 +216,7 @@ func TestGetSchedulerHealthStatusContext(t *testing.T) {
assert.Assert(t, healthInfo.HealthChecks[9].Succeeded, "The orphan
allocation check on the node should be successful")
// remove the allocation from the node, so we will have an orphan
allocation assigned to the app
- node.RemoveAllocation(allocID)
+ node.RemoveAllocation("key-0")
healthInfo = GetSchedulerHealthStatus(schedulerMetrics,
schedulerContext)
assert.Assert(t, healthInfo.HealthChecks[9].Succeeded, "The orphan
allocation check on the node should be successful")
assert.Assert(t, !healthInfo.HealthChecks[10].Succeeded, "The orphan
allocation check on the app should not be successful")
diff --git a/pkg/scheduler/objects/allocation.go
b/pkg/scheduler/objects/allocation.go
index b81991a0..6d9eeb04 100644
--- a/pkg/scheduler/objects/allocation.go
+++ b/pkg/scheduler/objects/allocation.go
@@ -57,7 +57,7 @@ type Allocation struct {
taskGroupName string // task group this allocation belongs to
placeholder bool // is this a placeholder allocation
nodeIDstring
- uuid string
+ allocationID string
priority int32
tags map[string]string
allocatedResource *resources.Resource
@@ -77,7 +77,7 @@ type Allocation struct {
sync.RWMutex
}
-func NewAllocation(uuid, nodeID string, ask *AllocationAsk) *Allocation {
+func NewAllocation(nodeID string, ask *AllocationAsk) *Allocation {
var createTime time.Time
if ask.GetTag(siCommon.CreationTime) == "" {
createTime = time.Now()
@@ -92,7 +92,7 @@ func NewAllocation(uuid, nodeID string, ask *AllocationAsk)
*Allocation {
bindTime:
(yunikorn-site) branch master updated: [YUNIKORN-2166] Translate 1.4 release notes and roadmap to zh-cn (#370)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new d88efc52a2 [YUNIKORN-2166] Translate 1.4 release notes and roadmap to zh-cn (#370) d88efc52a2 is described below commit d88efc52a21a25516aa437cbf2322596209901ef Author: haser AuthorDate: Wed Nov 29 15:22:09 2023 +1100 [YUNIKORN-2166] Translate 1.4 release notes and roadmap to zh-cn (#370) Closes: #370 Signed-off-by: Wilfred Spiegelenburg --- .../community/roadmap.md | 39 +- .../release-announce/1.4.0.md | 86 ++ 2 files changed, 56 insertions(+), 69 deletions(-) diff --git a/i18n/zh-cn/docusaurus-plugin-content-pages/community/roadmap.md b/i18n/zh-cn/docusaurus-plugin-content-pages/community/roadmap.md index a1294dc7eb..14c9086ab2 100644 --- a/i18n/zh-cn/docusaurus-plugin-content-pages/community/roadmap.md +++ b/i18n/zh-cn/docusaurus-plugin-content-pages/community/roadmap.md @@ -24,32 +24,31 @@ under the License. # Kubernetes 版本支持 支持的 Kubernetes 版本和 Kubernetes 依赖项将在开发周期中进行选择和最终确定。 -随着 Kubernetes 依赖项更改计划作为该版本的主要功能之一，一些信息已经为人所知。 -有关初步详细信息，请参阅 YuniKorn 1.3 [发布公告](/release-announce/1.3.0)。 +目前的构建依赖版本: 1.27 # 下一个发布研发内容 发布详情： -- 版本：1.4.0 -- 目标发布日期：2023 年 10 月/11 月 +- 版本：1.5.0 +- 目标发布日期：2024 年 2 月/3 月 - 发布负责人：尚未决定 - 开发状态：[问题跟踪器](https://issues.apache.org/jira/issues/?filter=12348416) 计划的主要功能： -- [YUNIKORN-1699](https://issues.apache.org/jira/browse/YUNIKORN-1699) K8s 1.27 支持，包括预先入队hook -- [YUNIKORN-1573](https://issues.apache.org/jira/browse/YUNIKORN-1573) [Umbrella] 基于用户和群组的配额执行 -- [YUNIKORN-1628](https://issues.apache.org/jira/browse/YUNIKORN-1628) [Umbrella] 应用跟踪历史 -- [YUNIKORN-1777](https://issues.apache.org/jira/browse/YUNIKORN-1777) [Umbrella] 恢复和启动 -- [YUNIKORN-22](https://issues.apache.org/jira/browse/YUNIKORN-22) 节点的分区支持 -- [YUNIKORN-1728](https://issues.apache.org/jira/browse/YUNIKORN-1728) 队列最大应用量和资源的百分比支持 - -- Web UI更改: - - [还没有jira] 显示抢占信息 - - [还没有jira] 资源显示返工 - - [YUNIKORN-1727](https://issues.apache.org/jira/browse/YUNIKORN-1727) 构建信息扩展 - - [YUNIKORN-1362](https://issues.apache.org/jira/browse/YUNIKORN-1362) 在 UI 中过滤节点 +- [YUNIKORN-970](https://issues.apache.org/jira/browse/YUNIKORN-970) 将队列指标改为可标记的 +- [YUNIKORN-1544](https://issues.apache.org/jira/browse/YUNIKORN-1544) 使用者与群组配额实施 - 阶段 2 +- [YUNIKORN-2099](https://issues.apache.org/jira/browse/YUNIKORN-2099) [Umbrella] 简化 K8shim +- [YUNIKORN-2115](https://issues.apache.org/jira/browse/YUNIKORN-2125) [Umbrella] 应用程式追踪历史 - 阶段 2 +- [YUNIKORN-1362](https://issues.apache.org/jira/browse/YUNIKORN-1362) UI 中的节点过滤 +- [YUNIKORN-1727](https://issues.apache.org/jira/browse/YUNIKORN-1727) 建置资讯扩展 +- [YUNIKORN-1922](https://issues.apache.org/jira/browse/YUNIKORN-1922) 在网页UI中显示待处理的资源 +- [YUNIKORN-2140](https://issues.apache.org/jira/browse/YUNIKORN-2140) Web UI: 资源显示重制 +- [no jira yet] 在 web UI 中显示抢占讯息 # 未来发展：长期的目标 -- REST 和 Web UI：authn、authz和加密 -- 支持隊列型態雙向改變(从叶队列到父队列和从父队列到叶队列) -- 将应用从一个队列移动到另一个队列 -- 压缩队列配置（configmap 1MB 限制） \ No newline at end of file +- [YUNIKORN-22](https://issues.apache.org/jira/browse/YUNIKORN-22) 支援节点分区 +- [YUNIKORN-1728](https://issues.apache.org/jira/browse/YUNIKORN-1728) 支持队列最大应用程式及资源百分比 +- 队列类型从叶节点变为父节点，反之亦然 +- 应用程式的队列间移动 +- 压缩队列配置 (configmap 1MB 限制) +- 可重复的构建 +- FIPS 构建 \ No newline at end of file diff --git a/i18n/zh-cn/docusaurus-plugin-content-pages/release-announce/1.4.0.md b/i18n/zh-cn/docusaurus-plugin-content-pages/release-announce/1.4.0.md index 87aadf64a9..09a4922fc3 100644 --- a/i18n/zh-cn/docusaurus-plugin-content-pages/release-announce/1.4.0.md +++ b/i18n/zh-cn/docusaurus-plugin-content-pages/release-announce/1.4.0.md @@ -33,77 +33,65 @@ Apache YuniKorn社区在此版本中修复了270个[JIRAs](https://issues.apache 发布日期: 2023-11-20 -## Incompatible changes +## 不兼容变更 -### Kubernetes version support -Running on Kubernetes 1.23 or earlier is **not** supported, for earlier Kubernetes releases use YuniKorn 1.3.0. +### Kubernetes 版本支援 +**不**支援在 Kubernetes 1.23 或更早期的版本中运行，对于早期的 Kubernetes 版本请使用 YuniKorn 1.3.0。 -**NOTE**: YuniKorn 1.3.0 is the last release that supports Kubernetes 1.23 or earlier. +**备注**: YuniKorn 1.3.0 是支援 Kubernetes 1.23 或更早期版本的最后一个版本。 -## Highlights +## 亮点 -### Kubernetes version support -YuniKorn 1.4.0 supports running on Kubernetes clusters from version 1.24 through to 1.28. +### Kubernetes 版本支援 +YuniKorn 1.4.0 支援在版本 1.24 到 1.28 的 Kubernetes 丛集上运行。 -Please see [YUNIKORN-1841](https://issues.apache.org/jira/browse/YUNIKORN-1841) for details. +详细资讯请参阅 [YUNIKORN-1841](https://issues.apache.org/jira/browse/YUNIKORN-1841) -### PreEnqueue plugin implementation -YuniKorn has added the pre-enqueue scheduling hook for the scheduler plugin. -This allows gating Pods that are not yet ready for scheduling due to queue
(yunikorn-core) branch master updated: [YUNIKORN-809] Remove deprecated policy inheritance (#737)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 84aac672 [YUNIKORN-809] Remove deprecated policy inheritance (#737)
84aac672 is described below
commit 84aac6726c7f4f6d5dd57b4fc96a2d4ddbeba704
Author: Kuan-Po Tseng
AuthorDate: Wed Nov 29 15:10:02 2023 +1100
[YUNIKORN-809] Remove deprecated policy inheritance (#737)
Remove deprecated behaviour that dynamic queues can inherit
application.sort.policy from their parent via properties.
The child template should be used for this.
Closes: #737
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/queue.go | 12 +---
pkg/scheduler/objects/queue_test.go | 9 -
2 files changed, 1 insertion(+), 20 deletions(-)
diff --git a/pkg/scheduler/objects/queue.go b/pkg/scheduler/objects/queue.go
index afe9fad6..de02b35f 100644
--- a/pkg/scheduler/objects/queue.go
+++ b/pkg/scheduler/objects/queue.go
@@ -919,23 +919,13 @@ func (sq *Queue) addChildQueue(child *Queue) error {
if child.isManaged {
return nil
}
- // this is a story about compatibility. the template is a new
feature, and we want to keep old behavior.
- // 1) try to use template if it is not nil
- // 2) otherwise, child leaf copy the
configs.ApplicationSortPolicy from parent (old behavior)
+ // try to use template if it is not nil
if sq.template != nil {
log.Log(log.SchedQueue).Debug("applying child template
to new leaf queue",
zap.String("child queue", child.QueuePath),
zap.String("parent queue", sq.QueuePath),
zap.Any("template", sq.template))
child.applyTemplate(sq.template)
- } else {
- policyValue, ok :=
sq.properties[configs.ApplicationSortPolicy]
- if ok {
- log.Log(log.Deprecation).Warn("inheriting
application sort policy is deprecated, use child templates",
- zap.String("child queue",
child.QueuePath),
- zap.String("parent queue",
sq.QueuePath))
- child.properties[configs.ApplicationSortPolicy]
= policyValue
- }
}
return nil
}
diff --git a/pkg/scheduler/objects/queue_test.go
b/pkg/scheduler/objects/queue_test.go
index 5d390db3..6044005f 100644
--- a/pkg/scheduler/objects/queue_test.go
+++ b/pkg/scheduler/objects/queue_test.go
@@ -1524,15 +1524,6 @@ func TestQueueProps(t *testing.T) {
assert.Assert(t, leaf.isLeaf && leaf.isManaged, "leaf queue is not
marked as managed leaf")
assert.Equal(t, len(leaf.properties), 2, "leaf queue properties size
incorrect")
- props = map[string]string{"first": "not inherited",
configs.ApplicationSortPolicy: "stateaware"}
- parent, err = createManagedQueueWithProps(root, "parent2", true, nil,
props)
- assert.NilError(t, err, "failed to create parent queue")
- assert.Equal(t, len(parent.properties), 2, "parent queue properties
size incorrect")
- leaf, err = createDynamicQueue(parent, "leaf", false)
- assert.NilError(t, err, "failed to create leaf queue")
- assert.Assert(t, leaf.isLeaf && !leaf.isManaged, "leaf queue is not
marked as unmanaged leaf")
- assert.Equal(t, leaf.properties[configs.ApplicationSortPolicy],
"stateaware", "leaf queue property value not as expected")
-
props = map[string]string{}
leaf, err = createManagedQueueWithProps(parent, "leaf", false, nil,
props)
assert.NilError(t, err, "failed to create leaf queue")
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2197] remove FitInMaxUndef from node code (#735)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 5a40e8ee [YUNIKORN-2197] remove FitInMaxUndef from node code (#735)
5a40e8ee is described below
commit 5a40e8ee01b4b43c9454fb4541e180df0c751c57
Author: Wilfred Spiegelenburg
AuthorDate: Wed Nov 29 14:48:29 2023 +1100
[YUNIKORN-2197] remove FitInMaxUndef from node code (#735)
Nodes should perform a strict FitIn for the resources requested not
allowing resource types in allocation that are not defined on the node.
Includes start of the Resources.FitIn refactor
This is a followup of the cleanup from YUNIKORN-1125.
Discovered while testing YUNIKORN-2174.
Closes: #735
Signed-off-by: Wilfred Spiegelenburg
---
pkg/common/resources/resources.go | 12 -
pkg/common/resources/resources_test.go | 53 +++
pkg/scheduler/objects/node.go | 28 +-
pkg/scheduler/objects/node_test.go | 94 ++
4 files changed, 118 insertions(+), 69 deletions(-)
diff --git a/pkg/common/resources/resources.go
b/pkg/common/resources/resources.go
index efa94bc4..7b64fe1a 100644
--- a/pkg/common/resources/resources.go
+++ b/pkg/common/resources/resources.go
@@ -417,14 +417,22 @@ func subNonNegative(left, right *Resource) (*Resource,
string) {
return out, message
}
-// Check if smaller fits in larger
+// FitIn Checks if smaller fits in larger
// Types not defined in the larger resource are considered 0 values for
Quantity
// A nil resource is treated as an empty resource (all types are 0)
+// Deprecated: use receiver version Resource.FitIn
func FitIn(larger, smaller *Resource) bool {
return larger.fitIn(smaller, false)
}
-// Check if smaller fits in the defined resource
+// FitIn checks if smaller fits in the defined resource
+// Types not defined in resource this is called against are considered 0 for
Quantity
+// A nil resource is treated as an empty resource (no types defined)
+func (r *Resource) FitIn(smaller *Resource) bool {
+ return r.fitIn(smaller, false)
+}
+
+// FitInMaxUndef checks if smaller fits in the defined resource
// Types not defined in resource this is called against are considered the
maximum value for Quantity
// A nil resource is treated as an empty resource (no types defined)
func (r *Resource) FitInMaxUndef(smaller *Resource) bool {
diff --git a/pkg/common/resources/resources_test.go
b/pkg/common/resources/resources_test.go
index 5c32095c..b3846e6c 100644
--- a/pkg/common/resources/resources_test.go
+++ b/pkg/common/resources/resources_test.go
@@ -1034,42 +1034,27 @@ func TestEqualsOrEmpty(t *testing.T) {
}
func TestFitIn(t *testing.T) {
- // simple case (nil checks)
- empty := NewResource()
- if !FitIn(nil, empty) {
- t.Error("fitin empty in nil resource failed")
- }
- if !FitIn(empty, nil) {
- t.Error("fitin nil in empty resource failed")
- }
- // zero set resources
- smaller := &Resource{Resources: map[string]Quantity{"a": 1}}
- if FitIn(empty, smaller) {
- t.Errorf("fitin resource with value %v should not fit in
empty", smaller)
+ tests := []struct {
+ namestring
+ larger *Resource
+ smaller *Resource
+ wantbool
+ }{
+ {"nil larger", nil, NewResource(), true},
+ {"nil smaller", NewResource(), nil, true},
+ {"zero set", NewResource(),
NewResourceFromMap(map[string]Quantity{"a": 1}), false},
+ {"same type", NewResourceFromMap(map[string]Quantity{"a": 5}),
NewResourceFromMap(map[string]Quantity{"a": 1}), true},
+ {"not in smaller",
NewResourceFromMap(map[string]Quantity{"not-in-smaller": 1}),
NewResourceFromMap(map[string]Quantity{"a": 1}), false},
+ {"not in larger",
NewResourceFromMap(map[string]Quantity{"not-in-smaller": 1}),
NewResourceFromMap(map[string]Quantity{"not-in-larger": 1}), false},
+ {"negative larger", NewResourceFromMap(map[string]Quantity{"a":
-10}), NewResourceFromMap(map[string]Quantity{"a": 0, "b": -10}), true},
+ {"negative smaller",
NewResourceFromMap(map[string]Quantity{"a": -5}),
NewResourceFromMap(map[string]Quantity{"a": 0, "b": 10}), false},
}
-
- // simple resources, same type
- larger := NewResourceFromMap(map[string]Quantity{"a": 5})
- if !FitIn(larger, smaller) {
- t.Err
(yunikorn-core) branch master updated: [YUNIKORN-2168] Generate better random data (#728)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new b331cb68 [YUNIKORN-2168] Generate better random data (#728)
b331cb68 is described below
commit b331cb68a89f59635bd3215be2baa5696b033f7b
Author: Kuan-Po Tseng
AuthorDate: Tue Nov 28 12:28:06 2023 +1100
[YUNIKORN-2168] Generate better random data (#728)
In queue_test.go and template_test.go replace the seconds based resource
sizing with random values from math random.
Add nolint remarks to all lines that use the random since it is not a
security related random.
Closes: #728
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/objects/queue_test.go | 5 +++--
pkg/scheduler/objects/template/template_test.go | 8
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/pkg/scheduler/objects/queue_test.go
b/pkg/scheduler/objects/queue_test.go
index b0d32b65..5d390db3 100644
--- a/pkg/scheduler/objects/queue_test.go
+++ b/pkg/scheduler/objects/queue_test.go
@@ -20,6 +20,7 @@ package objects
import (
"fmt"
+ "math/rand"
"reflect"
"sort"
"strconv"
@@ -1728,7 +1729,7 @@ func getAllocatingAcceptedApps() map[string]bool {
func getResourceConf() map[string]string {
resource := make(map[string]string)
- resource["memory"] = strconv.Itoa(time.Now().Second()%1000 + 100)
+ resource["memory"] = strconv.Itoa(rand.Intn(1) + 100) //nolint:gosec
return resource
}
@@ -1747,7 +1748,7 @@ func getZeroResourceConf() map[string]string {
func getProperties() map[string]string {
properties := make(map[string]string)
- properties[strconv.Itoa(time.Now().Second())] =
strconv.Itoa(time.Now().Second())
+ properties[strconv.Itoa(rand.Intn(1))] =
strconv.Itoa(rand.Intn(1)) //nolint:gosec
return properties
}
diff --git a/pkg/scheduler/objects/template/template_test.go
b/pkg/scheduler/objects/template/template_test.go
index 138ca737..cbe78078 100644
--- a/pkg/scheduler/objects/template/template_test.go
+++ b/pkg/scheduler/objects/template/template_test.go
@@ -19,9 +19,9 @@
package template
import (
+ "math/rand"
"strconv"
"testing"
- "time"
"gotest.tools/v3/assert"
@@ -31,14 +31,14 @@ import (
func getResourceConf() map[string]string {
resource := make(map[string]string)
- resource["memory"] = strconv.Itoa(time.Now().Second()%1000 + 10)
- resource["vcore"] = strconv.Itoa(time.Now().Second()%1000 + 10)
+ resource["memory"] = strconv.Itoa(rand.Intn(1) + 10) //nolint:gosec
+ resource["vcore"] = strconv.Itoa(rand.Intn(1) + 10) //nolint:gosec
return resource
}
func getProperties() map[string]string {
properties := make(map[string]string)
- properties[strconv.Itoa(time.Now().Second()%1000)] =
strconv.Itoa(time.Now().Second() % 1000)
+ properties[strconv.Itoa(rand.Intn(1))] =
strconv.Itoa(rand.Intn(1)) //nolint:gosec
return properties
}
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2171] race between node removal and scheduling (#724)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 1c2d0328 [YUNIKORN-2171] race between node removal and scheduling
(#724)
1c2d0328 is described below
commit 1c2d0328eab687cce71d1df4a2a50e62fee9ba86
Author: Wilfred Spiegelenburg
AuthorDate: Tue Nov 28 12:25:11 2023 +1100
[YUNIKORN-2171] race between node removal and scheduling (#724)
When a node gets removed the partition resources and thus the root max
resources are decreased. Cleanup of the allocations happens after that.
This means that for a short period of time the root queue max resources
are already decreased while the usage is not.
If the queue headroom is limited by the root queue then we could have a
race between the removal of the node allocations and scheduling adding a
new allocation. Scheduling could fail with the root queue usage being
over the max resources.
Closes: #724
Signed-off-by: Wilfred Spiegelenburg
---
pkg/scheduler/partition.go | 48 +++---
1 file changed, 33 insertions(+), 15 deletions(-)
diff --git a/pkg/scheduler/partition.go b/pkg/scheduler/partition.go
index 8c12e1cc..25d7f32b 100644
--- a/pkg/scheduler/partition.go
+++ b/pkg/scheduler/partition.go
@@ -611,9 +611,8 @@ func (pc *PartitionContext) addNodeToList(node
*objects.Node) error {
return nil
}
-// Update the partition details when removing a node.
-// This locks the partition. The partition may not be locked when we process
the allocation
-// removal from the node as that takes further app, queue or node locks
+// removeNodeFromList removes the node from the list of partition nodes.
+// This locks the partition.
func (pc *PartitionContext) removeNodeFromList(nodeID string) *objects.Node {
pc.Lock()
defer pc.Unlock()
@@ -628,26 +627,37 @@ func (pc *PartitionContext) removeNodeFromList(nodeID
string) *objects.Node {
// Remove node from list of tracked nodes
metrics.GetSchedulerMetrics().DecActiveNodes()
- // found the node cleanup the available resources, partition resources
cannot be nil at this point
+ log.Log(log.SchedPartition).Info("Removed node from available partition
nodes",
+ zap.String("partitionName", pc.Name),
+ zap.String("nodeID", node.NodeID))
+ return node
+}
+
+// removeNodeResources updates the partition and root queue resources as part
of the node removal process.
+// This locks the partition.
+func (pc *PartitionContext) removeNodeResources(node *objects.Node) {
+ pc.Lock()
+ defer pc.Unlock()
+ // cleanup the available resources, partition resources cannot be nil
at this point
pc.totalPartitionResource.SubFrom(node.GetCapacity())
pc.root.SetMaxResource(pc.totalPartitionResource)
log.Log(log.SchedPartition).Info("Updated available resources from
removed node",
zap.String("partitionName", pc.Name),
zap.String("nodeID", node.NodeID),
zap.Stringer("partitionResource", pc.totalPartitionResource))
- return node
}
-// Remove a node from the partition. It returns all removed and confirmed
allocations.
-// The removed allocations are all linked to the current node.
+// removeNode removes a node from the partition. It returns all released and
confirmed allocations.
+// The released allocations are all linked to the current node.
// The confirmed allocations are real allocations that are linked to
placeholders on the current node and are linked to
// other nodes.
// NOTE: this is a lock free call. It must NOT be called holding the
PartitionContext lock.
func (pc *PartitionContext) removeNode(nodeID string) ([]*objects.Allocation,
[]*objects.Allocation) {
- log.Log(log.SchedPartition).Info("removing node from partition",
+ log.Log(log.SchedPartition).Info("Removing node from partition",
zap.String("partition", pc.Name),
zap.String("nodeID", nodeID))
+ // remove the node: it will no longer be seen by the scheduling cycle
node := pc.removeNodeFromList(nodeID)
if node == nil {
return nil, nil
@@ -659,13 +669,19 @@ func (pc *PartitionContext) removeNode(nodeID string)
([]*objects.Allocation, []
_, app, ask := r.GetObjects()
pc.unReserve(app, node, ask)
}
- // cleanup the allocations linked to the node
- return pc.removeNodeAllocations(node)
+ // cleanup the allocations linked to the node. do this before changing
the root queue max: otherwise if
+ // scheduling and removal of a node race on a
(yunikorn-site) branch master updated: [YUNIKORN-2178] fix markdown format (#371)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new 8e46df7810 [YUNIKORN-2178] fix markdown format (#371) 8e46df7810 is described below commit 8e46df78109b636ba9463bab327f4f98a045e334 Author: targetoee AuthorDate: Mon Nov 27 19:33:17 2023 +1100 [YUNIKORN-2178] fix markdown format (#371) There's 2 markdown format error in Chinese gang scheduling document. One has a missing backtick at the end while another has a misplaced backtick. Closes: #371 Signed-off-by: Wilfred Spiegelenburg --- .../current/user_guide/gang_scheduling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/i18n/zh-cn/docusaurus-plugin-content-docs/current/user_guide/gang_scheduling.md b/i18n/zh-cn/docusaurus-plugin-content-docs/current/user_guide/gang_scheduling.md index 754e29c329..e7e30526d6 100644 --- a/i18n/zh-cn/docusaurus-plugin-content-docs/current/user_guide/gang_scheduling.md +++ b/i18n/zh-cn/docusaurus-plugin-content-docs/current/user_guide/gang_scheduling.md @@ -121,7 +121,7 @@ pods永远卡住，请参考 [故障排除](troubleshooting.md#成组调度) 了 示例 下面的例子是一个工作的yaml文件。这个工作启动了2个 pod，每个 pod 睡眠时间为 30 秒。 -在 pod 规范中值得注意的变化是 *spec.template.metadata.annotations*，在这里我们定义了 `yunikorn.apache.org/task-group-name +在 pod 规范中值得注意的变化是 *spec.template.metadata.annotations*，在这里我们定义了 `yunikorn.apache.org/task-group-name` 和 `yunikorn.apache.org/task-groups` 。 ```yaml @@ -232,7 +232,7 @@ Annotations: - `Hard Style`：当使用这种风格时，我们将有初始行为，更确切地说，如果应用程序不能根据分组调度规则进行调度，并且超时，它将被标记为失败，而不会重新尝试调度。 - `Soft Style`：当应用程序不能被分组调度时，它将退回到正常的调度，并使用非分组调度策略来实现最佳努力的调度。当这种情况发生时，应用程序将过渡到恢复状态，所有剩余的占位符 pod 将被清理掉。 -**使用的默认样式**： ``Soft。 +**使用的默认样式**： `Soft`。 **启用一个特定的风格**：可以通过在应用程序定义中设置'gangSchedulingStyle'参数来改变风格，即 Soft 或 Hard。 - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch master updated: [YUNIKORN-2163] Fix HTTP status codes in REST handlers (#729)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new 3b9c9661 [YUNIKORN-2163] Fix HTTP status codes in REST handlers (#729)
3b9c9661 is described below
commit 3b9c966157962787ec4e03bb3ca3c7093870b48b
Author: Peter Bacsko
AuthorDate: Mon Nov 27 16:47:25 2023 +1100
[YUNIKORN-2163] Fix HTTP status codes in REST handlers (#729)
An object does not exist: 400 Bad Request --> 404 Not Found
Internal metrics is disabled: 501 Not Implemented --> 500 Internal Server
Error
Event tracking disabled: 400 Bad Request --> 500 Internal Server Error
removal of dead code for writing a new config
Closes: #729
Signed-off-by: Wilfred Spiegelenburg
---
pkg/webservice/handlers.go | 47
pkg/webservice/handlers_test.go | 59 ++---
2 files changed, 37 insertions(+), 69 deletions(-)
diff --git a/pkg/webservice/handlers.go b/pkg/webservice/handlers.go
index 382dd230..ee4ecf22 100644
--- a/pkg/webservice/handlers.go
+++ b/pkg/webservice/handlers.go
@@ -358,7 +358,7 @@ func getNodeUtilisation(w http.ResponseWriter, r
*http.Request) {
writeHeaders(w)
partitionContext :=
schedulerContext.GetPartitionWithoutClusterID(configs.DefaultPartition)
if partitionContext == nil {
- buildJSONErrorResponse(w, PartitionDoesNotExists,
http.StatusBadRequest)
+ buildJSONErrorResponse(w, PartitionDoesNotExists,
http.StatusInternalServerError)
return
}
// calculate the dominant resource based on root queue usage and size
@@ -422,7 +422,7 @@ func getApplicationHistory(w http.ResponseWriter, r
*http.Request) {
// There is nothing to return but we did not really encounter a problem
if imHistory == nil {
- buildJSONErrorResponse(w, "Internal metrics collection is not
enabled.", http.StatusNotImplemented)
+ buildJSONErrorResponse(w, "Internal metrics collection is not
enabled.", http.StatusInternalServerError)
return
}
// get a copy of the records: if the array contains nil values they
will always be at the
@@ -439,7 +439,7 @@ func getContainerHistory(w http.ResponseWriter, r
*http.Request) {
// There is nothing to return but we did not really encounter a problem
if imHistory == nil {
- buildJSONErrorResponse(w, "Internal metrics collection is not
enabled.", http.StatusNotImplemented)
+ buildJSONErrorResponse(w, "Internal metrics collection is not
enabled.", http.StatusInternalServerError)
return
}
// get a copy of the records: if the array contains nil values they
will always be at the
@@ -507,20 +507,7 @@ func checkHealthStatus(w http.ResponseWriter, r
*http.Request) {
}
}
-func buildUpdateResponse(err error, w http.ResponseWriter) {
- if err == nil {
- w.WriteHeader(http.StatusOK)
- if _, err = w.Write([]byte("Configuration updated
successfully")); err != nil {
- buildJSONErrorResponse(w, err.Error(),
http.StatusInternalServerError)
- }
- } else {
- log.Log(log.REST).Info("Configuration update failed with
errors",
- zap.Error(err))
- buildJSONErrorResponse(w, err.Error(), http.StatusConflict)
- }
-}
-
-func getPartitions(w http.ResponseWriter, r *http.Request) {
+func getPartitions(w http.ResponseWriter, _ *http.Request) {
writeHeaders(w)
lists := schedulerContext.GetPartitionMapClone()
@@ -543,7 +530,7 @@ func getPartitionQueues(w http.ResponseWriter, r
*http.Request) {
if partition != nil {
partitionQueuesDAOInfo = partition.GetPartitionQueues()
} else {
- buildJSONErrorResponse(w, PartitionDoesNotExists,
http.StatusBadRequest)
+ buildJSONErrorResponse(w, PartitionDoesNotExists,
http.StatusNotFound)
return
}
if err := json.NewEncoder(w).Encode(partitionQueuesDAOInfo); err != nil
{
@@ -566,7 +553,7 @@ func getPartitionNodes(w http.ResponseWriter, r
*http.Request) {
buildJSONErrorResponse(w, err.Error(),
http.StatusInternalServerError)
}
} else {
- buildJSONErrorResponse(w, PartitionDoesNotExists,
http.StatusBadRequest)
+ buildJSONErrorResponse(w, PartitionDoesNotExists,
http.StatusNotFound)
}
}
@@ -583,7 +570,7 @@ func getPartitionNode(w http.ResponseWriter, r
*http.Request) {
nodeID := vars.ByName("node")
(yunikorn-core) branch master updated: [YUNIKORN-2161] Metrics code cleanup (#719)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git
The following commit(s) were added to refs/heads/master by this push:
new ab7ab268 [YUNIKORN-2161] Metrics code cleanup (#719)
ab7ab268 is described below
commit ab7ab268aaa4db9c419e4a18e55de54edb82c050
Author: brandboat
AuthorDate: Tue Nov 21 13:36:03 2023 +1100
[YUNIKORN-2161] Metrics code cleanup (#719)
Remove CoreQueueMetrics interface
Remove CoreSchedulerMetrics interface
Remove unused queue.go#addQueueResource
Rename cqm to qm in queue_test.go
Rename csm to sm in scheduler_test.go
Fix incorrect unregister metrics method
Remove unused testing parameter in unregisterQueueMetrics
Remove unused testing parameter in unregisterMetrics
Remove CoreEventMetrics interface
Rename eventMetrics to EventMetrics
Remove GoRuntimeMetrics interface
Closes: #719
Signed-off-by: Wilfred Spiegelenburg
---
pkg/metrics/event.go| 24 ---
pkg/metrics/init.go | 149 +++-
pkg/metrics/queue.go| 19 +
pkg/metrics/queue_test.go | 89 +++-
pkg/metrics/runtime.go | 2 +
pkg/metrics/scheduler.go| 65 +-
pkg/metrics/scheduler_test.go | 40 +--
pkg/scheduler/health_checker.go | 6 +-
8 files changed, 90 insertions(+), 304 deletions(-)
diff --git a/pkg/metrics/event.go b/pkg/metrics/event.go
index 04124f01..7f0032cd 100644
--- a/pkg/metrics/event.go
+++ b/pkg/metrics/event.go
@@ -20,7 +20,7 @@ package metrics
import "github.com/prometheus/client_golang/prometheus"
-type eventMetrics struct {
+type EventMetrics struct {
totalEventsCreated prometheus.Gauge
totalEventsChanneledprometheus.Gauge
totalEventsNotChanneled prometheus.Gauge
@@ -30,8 +30,8 @@ type eventMetrics struct {
totalEventsCollectedprometheus.Gauge
}
-func initEventMetrics() CoreEventMetrics {
- metrics := &eventMetrics{}
+func initEventMetrics() *EventMetrics {
+ metrics := &EventMetrics{}
metrics.totalEventsCreated = prometheus.NewGauge(
prometheus.GaugeOpts{
@@ -86,7 +86,9 @@ func initEventMetrics() CoreEventMetrics {
return metrics
}
-func (em *eventMetrics) Reset() {
+// Reset all metrics that implement the Set functionality.
+// Should only be used in tests
+func (em *EventMetrics) Reset() {
em.totalEventsCollected.Set(0)
em.totalEventsCreated.Set(0)
em.totalEventsChanneled.Set(0)
@@ -96,30 +98,30 @@ func (em *eventMetrics) Reset() {
em.totalEventsProcessed.Set(0)
}
-func (em *eventMetrics) IncEventsCreated() {
+func (em *EventMetrics) IncEventsCreated() {
em.totalEventsCreated.Inc()
}
-func (em *eventMetrics) IncEventsChanneled() {
+func (em *EventMetrics) IncEventsChanneled() {
em.totalEventsChanneled.Inc()
}
-func (em *eventMetrics) IncEventsNotChanneled() {
+func (em *EventMetrics) IncEventsNotChanneled() {
em.totalEventsNotChanneled.Inc()
}
-func (em *eventMetrics) IncEventsProcessed() {
+func (em *EventMetrics) IncEventsProcessed() {
em.totalEventsProcessed.Inc()
}
-func (em *eventMetrics) IncEventsStored() {
+func (em *EventMetrics) IncEventsStored() {
em.totalEventsStored.Inc()
}
-func (em *eventMetrics) IncEventsNotStored() {
+func (em *EventMetrics) IncEventsNotStored() {
em.totalEventsNotStored.Inc()
}
-func (em *eventMetrics) AddEventsCollected(collectedEvents int) {
+func (em *EventMetrics) AddEventsCollected(collectedEvents int) {
em.totalEventsCollected.Add(float64(collectedEvents))
}
diff --git a/pkg/metrics/init.go b/pkg/metrics/init.go
index 7a4f5b4e..39b2120e 100644
--- a/pkg/metrics/init.go
+++ b/pkg/metrics/init.go
@@ -20,7 +20,6 @@ package metrics
import (
"sync"
- "time"
)
const (
@@ -38,148 +37,18 @@ var once sync.Once
var m *Metrics
type Metrics struct {
- scheduler CoreSchedulerMetrics
- queuesmap[string]CoreQueueMetrics
- event CoreEventMetrics
- runtime GoRuntimeMetrics
+ scheduler *SchedulerMetrics
+ queuesmap[string]*QueueMetrics
+ event *EventMetrics
+ runtime *RuntimeMetrics
lock sync.RWMutex
}
-type CoreQueueMetrics interface {
- IncQueueApplicationsAccepted()
- GetQueueApplicationsAccepted() (int, error)
- IncQueueApplicationsRejected()
- GetQueueApplicationsRejected() (int, error)
- IncQueueApplicationsRunning()
- DecQueueApplicationsRunning()
- GetQueueApplicationsRunning() (int, error)
- IncQueueApplicationsFailed()
- GetQueueApplicationsFailed() (int, error)
- IncQueueApplicationsCompleted()
- GetQu
(yunikorn-site) branch master updated: [YUNIKORN-2154] Fix documentation typo (#368)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git The following commit(s) were added to refs/heads/master by this push: new bf0f2a5793 [YUNIKORN-2154] Fix documentation typo (#368) bf0f2a5793 is described below commit bf0f2a5793eb31732793fd636d22baa6f99bef18 Author: Michael Akinyemi AuthorDate: Tue Nov 21 13:03:43 2023 +1100 [YUNIKORN-2154] Fix documentation typo (#368) Fix misspelled word "wll" to "will". Closes: #368 Signed-off-by: Wilfred Spiegelenburg --- docs/developer_guide/env_setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer_guide/env_setup.md b/docs/developer_guide/env_setup.md index 18c5bfdaff..3ca7e21f8d 100644 --- a/docs/developer_guide/env_setup.md +++ b/docs/developer_guide/env_setup.md @@ -147,7 +147,7 @@ the correct cluster. Without setting the environment minikube might not find the docker images when deploying the scheduler. 1. Make sure minikube is started. -2. In the terminal where you wll run the build, execute: +2. In the terminal where you will run the build, execute: ```shell script eval $(minikube docker-env) ``` - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) 01/01: [YUNIKORN-2102] Publish helm chart for 1.4.0
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git commit 72f7b0ecfd20bed5003a8f0f426582e312183b9a Author: Wilfred Spiegelenburg AuthorDate: Mon Nov 20 15:48:59 2023 +1100 [YUNIKORN-2102] Publish helm chart for 1.4.0 --- index.md | 17 ++--- index.yaml | 21 - 2 files changed, 30 insertions(+), 8 deletions(-) diff --git a/index.md b/index.md index 4a8dee1..cf127f3 100644 --- a/index.md +++ b/index.md @@ -36,12 +36,13 @@ shim layer and adopt to different ResourceManager implementation including Apach ## Feature highlights -- App-aware scheduling +- Gang scheduling +- Application and queue aware preemption - Hierarchy Resource Queues +- Application aware scheduling - Job Ordering and Queuing -- Resource fairness -- Resource Reservation -- Throughput +- High throughput scheduling +- Default Kubernetes workload compatible (daemonsets, jobs, deployments etc) Read the complete list of features from [here](https://yunikorn.apache.org/docs/get_started/core_features). @@ -64,12 +65,14 @@ YuniKorn has builtin web UIs for queue hierarchies and apps. See below: | 1.18.x | 0.10.0 |0.11.0 | | 1.19.x | 0.11.0 | 1.0.0 | | 1.20.x | 0.12.1 | 1.2.0 | -| 1.21.x | 0.12.1 | - | -| 1.22.x | 0.12.2 | - | -| 1.23.x | 0.12.2 | - | +| 1.21.x | 0.12.1 | 1.3.0 | +| 1.22.x | 0.12.2 | 1.3.0 | +| 1.23.x | 0.12.2 | 1.3.0 | | 1.24.x | 1.0.0 | - | | 1.25.x | 1.2.0 | - | | 1.26.x | 1.2.0 | - | +| 1.27.x | 1.4.0 | - | +| 1.28.x | 1.4.0 | - | ## Useful links diff --git a/index.yaml b/index.yaml index c69cb2d..9f88e19 100644 --- a/index.yaml +++ b/index.yaml @@ -209,4 +209,23 @@ entries: urls: - https://github.com/apache/yunikorn-release/releases/download/v1.3.0/yunikorn-1.3.0.tgz version: 1.3.0 -generated: "2023-06-12T15:00:00.00+10:00" +- apiVersion: v1 + appVersion: 1.4.0 + created: "2023-11-20T15:00:00.00+11:00" + description: YuniKorn scheduler for Kubernetes + digest: b27542e1c17e83bbadbbb09bd4a2e6433a84d0a6c674614361f748756eb8b355 + home: https://yunikorn.apache.org + icon: https://raw.githubusercontent.com/apache/yunikorn-site/master/static/img/logo/yunikorn_classic_logo.png + maintainers: +- email: d...@yunikorn.apache.org + name: Developers mailing list + sources: +- https://github.com/apache/yunikorn-core +- https://github.com/apache/yunikorn-k8shim +- https://github.com/apache/yunikorn-scheduler-interface +- https://github.com/apache/yunikorn-web + name: yunikorn + urls: +- https://github.com/apache/yunikorn-release/releases/download/v1.4.0/yunikorn-1.4.0.tgz + version: 1.4.0 +generated: "2023-11-20T15:00:00.00+11:00" - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch gh-pages updated (3ca7642 -> 72f7b0e)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git discard 3ca7642 [YUNIKORN-2102] Publish helm chart for 1.4.0 new 72f7b0e [YUNIKORN-2102] Publish helm chart for 1.4.0 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (3ca7642) \ N -- N -- N refs/heads/gh-pages (72f7b0e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: index.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch gh-pages updated: [YUNIKORN-2102] Publish helm chart for 1.4.0
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/gh-pages by this push: new 3ca7642 [YUNIKORN-2102] Publish helm chart for 1.4.0 3ca7642 is described below commit 3ca7642819940ead5381c5074f50ca3be410568f Author: Wilfred Spiegelenburg AuthorDate: Mon Nov 20 15:48:59 2023 +1100 [YUNIKORN-2102] Publish helm chart for 1.4.0 --- index.md | 17 ++--- index.yaml | 21 - 2 files changed, 30 insertions(+), 8 deletions(-) diff --git a/index.md b/index.md index 4a8dee1..cf127f3 100644 --- a/index.md +++ b/index.md @@ -36,12 +36,13 @@ shim layer and adopt to different ResourceManager implementation including Apach ## Feature highlights -- App-aware scheduling +- Gang scheduling +- Application and queue aware preemption - Hierarchy Resource Queues +- Application aware scheduling - Job Ordering and Queuing -- Resource fairness -- Resource Reservation -- Throughput +- High throughput scheduling +- Default Kubernetes workload compatible (daemonsets, jobs, deployments etc) Read the complete list of features from [here](https://yunikorn.apache.org/docs/get_started/core_features). @@ -64,12 +65,14 @@ YuniKorn has builtin web UIs for queue hierarchies and apps. See below: | 1.18.x | 0.10.0 |0.11.0 | | 1.19.x | 0.11.0 | 1.0.0 | | 1.20.x | 0.12.1 | 1.2.0 | -| 1.21.x | 0.12.1 | - | -| 1.22.x | 0.12.2 | - | -| 1.23.x | 0.12.2 | - | +| 1.21.x | 0.12.1 | 1.3.0 | +| 1.22.x | 0.12.2 | 1.3.0 | +| 1.23.x | 0.12.2 | 1.3.0 | | 1.24.x | 1.0.0 | - | | 1.25.x | 1.2.0 | - | | 1.26.x | 1.2.0 | - | +| 1.27.x | 1.4.0 | - | +| 1.28.x | 1.4.0 | - | ## Useful links diff --git a/index.yaml b/index.yaml index c69cb2d..dfc8358 100644 --- a/index.yaml +++ b/index.yaml @@ -209,4 +209,23 @@ entries: urls: - https://github.com/apache/yunikorn-release/releases/download/v1.3.0/yunikorn-1.3.0.tgz version: 1.3.0 -generated: "2023-06-12T15:00:00.00+10:00" +- apiVersion: v1 + appVersion: 1.4.0 + created: "2023-11-20:00:00.00+11:00" + description: YuniKorn scheduler for Kubernetes + digest: b27542e1c17e83bbadbbb09bd4a2e6433a84d0a6c674614361f748756eb8b355 + home: https://yunikorn.apache.org + icon: https://raw.githubusercontent.com/apache/yunikorn-site/master/static/img/logo/yunikorn_classic_logo.png + maintainers: +- email: d...@yunikorn.apache.org + name: Developers mailing list + sources: +- https://github.com/apache/yunikorn-core +- https://github.com/apache/yunikorn-k8shim +- https://github.com/apache/yunikorn-scheduler-interface +- https://github.com/apache/yunikorn-web + name: yunikorn + urls: +- https://github.com/apache/yunikorn-release/releases/download/v1.4.0/yunikorn-1.4.0.tgz + version: 1.4.0 +generated: "2023-11-20:00:00.00+11:00" - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated: [YUNIKORN-970] Add queue metrics with queue names as labels (#365)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 2a63686b8c [YUNIKORN-970] Add queue metrics with queue names as labels
(#365)
2a63686b8c is described below
commit 2a63686b8c63a8e3aa71c649e681826e1e904e52
Author: William Tom
AuthorDate: Tue Nov 14 13:25:47 2023 -0800
[YUNIKORN-970] Add queue metrics with queue names as labels (#365)
Reverted as part of the YuniKorn 1.4 release steps and re-published
after versioning documentation under a new commit to master.
Co-authored-by: William Tom
---
docs/metrics/queue.md | 61 +--
1 file changed, 54 insertions(+), 7 deletions(-)
diff --git a/docs/metrics/queue.md b/docs/metrics/queue.md
index 0aa1f7cd36..4943fd3092 100644
--- a/docs/metrics/queue.md
+++ b/docs/metrics/queue.md
@@ -22,13 +22,37 @@ specific language governing permissions and limitations
under the License.
-->
-## Queue
-### Application
-Eech queue has a ` queue_app` metric to trace the applications in
the queue.
+## Application
+Each queue has a ` queue_app` metric to trace the applications in
the queue.
` queue_app` metrics records the number of applications in
different states.
These application states include `running`, `accepted`, `rejected`, `failed`
and `completed`.
-` queue_app` metrics record container states including `released`,
`allocated`.
-**Metric Type**: `guage`
+` queue_app` metrics record container states including `released`,
`allocated`.
+
+Prior to `1.5.0` queue metrics were pushed to a separate subsystem for each
queue. In `1.5.0`
+fixed metrics with `` labels were introduced. Metrics using
`` subsystems
+will be **deprecated in `1.6.0` and removed in `1.7.0`** in favor of a `` label based approach.
+
+### Label
+(Introduced in `1.5.0`)
+
+**Metric Type**: `gauge`
+
+**Namespace**: `yunikorn`
+
+**Label**: `queue: `
+
+**TYPE**: `yunikorn_queue_app`
+
+```json
+yunikorn_queue_app{queue="root.default",state="accepted"} 3
+yunikorn_queue_app{queue="root.default",state="running"} 3
+```
+
+### Subsystem
+|:exclamation: To be deprecated in `1.6.0` and removed in `1.7.0`|
+||
+
+**Metric Type**: `gauge`
**Namespace**: `yunikorn`
@@ -41,11 +65,34 @@ yunikorn_root_default_queue_app{state="accepted"} 3
yunikorn_root_default_queue_app{state="running"} 3
```
-### Resource
+## Resource
The ` queue_resource` metric to trace the resource in the queue.
These resource states include `guaranteed`, `max`, `allocated`, `pending`,
`preempting`.
+### Label
+(Introduced in `1.5.0`)
+
+**Metric Type**: `gauge`
+
+**Namespace**: `yunikorn`
+
+**Label**: `queue: `
+
+**TYPE**: `yunikorn_queue_resource`
+
+```json
+yunikorn_queue_resource{queue="root",resource="ephemeral-storage",state="max"}
9.41009558e+10
+yunikorn_queue_resource{queue="root",resource="hugepages-1Gi",state="max"} 0
+yunikorn_queue_resource{queue="root",resource="hugepages-2Mi",state="max"} 0
+yunikorn_queue_resource{queue="root",resource="memory",state="max"}
1.6223076352e+10
+yunikorn_queue_resource{queue="root",resource="pods",state="max"} 110
+yunikorn_queue_resource{queue="root",resource="vcore",state="max"} 8000
+```
+
+### Subsystem
+|:exclamation: To be deprecated in `1.6.0` and removed in `1.7.0`|
+||
-**Metric Type**: `guage`
+**Metric Type**: `gauge`
**Namespace**: `yunikorn`
-
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-site) branch master updated (6d7e69ca2c -> 778ca9e546)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
omit 6d7e69ca2c [YUNIKORN-970] Add queue metrics with queue names as labels
(#365)
new 778ca9e546 [YUNIKORN-2104] website update for release 1.4
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6d7e69ca2c)
\
N -- N -- N refs/heads/master (778ca9e546)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
doap_YuniKorn.rdf | 7 ++
docs/metrics/queue.md | 61 ++--
docusaurus.config.js | 4 +-
.../{current => version-1.4.0}/api/cluster.md | 0
.../{current => version-1.4.0}/api/scheduler.md| 0
.../{current => version-1.4.0}/api/system.md | 0
.../{current => version-1.4.0}/assets | 0
.../design/scheduler_plugin.md | 0
.../version-1.4.0}/design/simple_preemptor.md | 0
.../developer_guide/build.md | 0
.../developer_guide/dependencies.md| 0
.../developer_guide/env_setup.md | 0
.../developer_guide/openshift_development.md | 0
.../developer_guide/translation.md | 0
.../get_started/core_features.md | 0
.../get_started/get_started.md | 0
.../evaluate_perf_function_with_kubemark.md| 0
.../performance/metrics.md | 0
.../performance/performance_tutorial.md| 0
.../performance/profiling.md | 0
.../{current => version-1.4.0}/user_guide/acls.md | 0
.../user_guide/deployment_modes.md | 0
.../user_guide/gang_scheduling.md | 0
.../labels_and_annotations_in_yunikorn.md | 0
.../user_guide/placement_rules.md | 0
.../user_guide/prometheus.md | 0
.../user_guide/queue_config.md | 0
.../user_guide/resource_quota_mgmt.md | 0
.../user_guide/service_config.md | 0
.../user_guide/sorting_policies.md | 0
.../user_guide/troubleshooting.md | 0
.../user_guide/usergroup_resolution.md | 0
.../user_guide/workloads/run_flink.md | 0
.../user_guide/workloads/run_mpi.md| 0
.../user_guide/workloads/run_nvidia.md | 0
.../user_guide/workloads/run_spark.md | 0
.../user_guide/workloads/run_tensorflow.md | 0
.../user_guide/workloads/workload_overview.md | 0
.../community/download.md | 12 +--
.../community/people.md| 62 ++--
.../release-announce/1.4.0.md | 109
src/pages/community/download.md| 12 +--
src/pages/community/people.md | 8 +-
src/pages/community/roadmap.md | 32 +++---
src/pages/release-announce/1.4.0.md| 110 +
.../version-1.4.0}/api/cluster.md | 0
.../version-1.4.0}/api/scheduler.md| 0
.../version-1.4.0}/api/system.md | 0
.../archived_design/cross_queue_preemption.md | 0
.../version-1.4.0}/archived_design/k8shim.md | 0
.../archived_design/namespace_resource_quota.md| 0
.../archived_design/pluggable_app_management.md| 0
.../version-1.4.0}/archived_design/predicates.md | 0
.../archived_design/scheduler_core_design.md | 0
.../version-1.4.0}/assets/RunningSparkOnK8s.png| Bin
.../version-1.4.0}/assets/allocation_4k.png| Bin
.../version-1.4.0}/assets/application-state.png| Bin
.../version-1.4.0}/assets/architecture.png | Bin
.../version-1.4.0}/assets/cpu_profile.jpg | Bin
.../version-1.4.0}/assets/dashboard_secret.png | Bin
.../assets/dashboard_token_select.png | Bin
.../assets/docker-de
(yunikorn-release) annotated tag v1.4.0 updated (05749e0 -> 7c47886)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0 in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git *** WARNING: tag v1.4.0 was modified! *** from 05749e0 (commit) to 7c47886 (tag) tagging 05749e0bae4caa784852ae201d47930bb4c58419 (commit) by Wilfred Spiegelenburg on Mon Nov 20 12:43:59 2023 +1100 - Log - Apache YuniKorn v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-scheduler-interface) annotated tag v1.4.0 updated (45f4a5a -> f92995a)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0 in repository https://gitbox.apache.org/repos/asf/yunikorn-scheduler-interface.git *** WARNING: tag v1.4.0 was modified! *** from 45f4a5a (commit) to f92995a (tag) tagging 45f4a5a0428c7c7f44b6da18f987e177fee720f6 (commit) replaces v1.2.0-1 by Wilfred Spiegelenburg on Mon Nov 20 12:37:25 2023 +1100 - Log - Apache YuniKorn v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) annotated tag v1.4.0 updated (781370e -> 6687266)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0 in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git *** WARNING: tag v1.4.0 was modified! *** from 781370e (commit) to 6687266 (tag) tagging 781370e047edf91d412d50e1b8caa8af7481dc5b (commit) replaces v1.4.0-1 by Wilfred Spiegelenburg on Mon Nov 20 12:37:21 2023 +1100 - Log - Apache YuniKorn v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) annotated tag v1.4.0 updated (c75a39a3 -> 947af5c0)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0 in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git *** WARNING: tag v1.4.0 was modified! *** from c75a39a3 (commit) to 947af5c0 (tag) tagging c75a39a33132aded2f825674b6c670e530449a23 (commit) by Wilfred Spiegelenburg on Mon Nov 20 12:37:13 2023 +1100 - Log - Apache YuniKorn v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) annotated tag v1.4.0 updated (ee4dd0c6 -> 5bae675c)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0 in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git *** WARNING: tag v1.4.0 was modified! *** from ee4dd0c6 (commit) to 5bae675c (tag) tagging ee4dd0c61859be8a2ac6c6cfd517e3405486a064 (commit) replaces v1.4.0-1 by Wilfred Spiegelenburg on Mon Nov 20 12:36:41 2023 +1100 - Log - Apache YuniKorn v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-release) branch branch-1.4 updated: [YUNIKORN-2105] Tag details and changelog for 1.4.0-RC1
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch branch-1.4 in repository https://gitbox.apache.org/repos/asf/yunikorn-release.git The following commit(s) were added to refs/heads/branch-1.4 by this push: new 05749e0 [YUNIKORN-2105] Tag details and changelog for 1.4.0-RC1 05749e0 is described below commit 05749e0bae4caa784852ae201d47930bb4c58419 Author: Wilfred Spiegelenburg AuthorDate: Wed Nov 15 19:59:07 2023 +1100 [YUNIKORN-2105] Tag details and changelog for 1.4.0-RC1 --- release-top-level-artifacts/CHANGELOG | 292 +- tools/release-configs.json| 10 +- 2 files changed, 294 insertions(+), 8 deletions(-) diff --git a/release-top-level-artifacts/CHANGELOG b/release-top-level-artifacts/CHANGELOG index 5c86300..140d7d5 100644 --- a/release-top-level-artifacts/CHANGELOG +++ b/release-top-level-artifacts/CHANGELOG @@ -15,6 +15,292 @@ # limitations under the License. # - -Replace this with the jira release notes - \ No newline at end of file +Release Notes - Apache YuniKorn - Version 1.4.0 + +** Sub-task +* [YUNIKORN-325] - Add a chart to display node resource utilization summary +* [YUNIKORN-1471] - Add e2e tests for preemption +* [YUNIKORN-1575] - Ability to specify guaranteedResources via namespace annotations +* [YUNIKORN-1607] - Configuration processing should support limit hierarchical quota check. +* [YUNIKORN-1643] - Doc need to change for new limit wildcard interpretation syntax rule check. +* [YUNIKORN-1656] - Extend si.EventRecord type +* [YUNIKORN-1657] - Create design document +* [YUNIKORN-1659] - Documentation: add real life preemption examples +* [YUNIKORN-1687] - Add E2E test for User & Group Based Quota Enforcement +* [YUNIKORN-1716] - Improve the performance of baseNodeCollection.getNodeIteratorInternal() +* [YUNIKORN-1744] - Enable recovery for MockScheduler +* [YUNIKORN-1748] - Create test case which measures binding throughput using MockScheduler +* [YUNIKORN-1774] - Event cache: misc cleanup +* [YUNIKORN-1783] - Application: maintain sorted state of requests instead of sorting +* [YUNIKORN-1793] - Handle placement rule and queue changes during initialisation (core) +* [YUNIKORN-1794] - Define new messages in SI to support init per object +* [YUNIKORN-1797] - Implement scoped logging API +* [YUNIKORN-1799] - Create basic ringbuffer implementation +* [YUNIKORN-1800] - Extend Application event wrapper with new events +* [YUNIKORN-1801] - Add allocation events +* [YUNIKORN-1802] - Add node events +* [YUNIKORN-1803] - Add queue events +* [YUNIKORN-1804] - Add REST endpoint for batch event retrieval +* [YUNIKORN-1805] - Create smoke test to validate application tracking via REST interface +* [YUNIKORN-1806] - Add new configuration entries +* [YUNIKORN-1807] - Remove error check from createEventRecord() +* [YUNIKORN-1814] - Remove manager#cleaner method +* [YUNIKORN-1824] - Document configuration changes for scoped logging +* [YUNIKORN-1825] - Core: Convert to scoped logging +* [YUNIKORN-1826] - Shim: Convert to scoped logging +* [YUNIKORN-1827] - Add core.scheduler.application.usage logger +* [YUNIKORN-1830] - Add reservation/unreservation events +* [YUNIKORN-1833] - Rationalize event verification in partition_test.go +* [YUNIKORN-1839] - Core: remove deprecated log.Logger() implementation +* [YUNIKORN-1843] - Shim: Remove deprecated log.Logger() function +* [YUNIKORN-1845] - Handle nil logger handle properly +* [YUNIKORN-1846] - Rewrite event verification in application_test.go +* [YUNIKORN-1859] - Add "omitempty" to json REST object definition +* [YUNIKORN-1860] - Update webapp to handle missing entries +* [YUNIKORN-1861] - Add "omitempty" to k8shim cache dump +* [YUNIKORN-1863] - Remove unused code in processUserConfig and processGroupConfig +* [YUNIKORN-1864] - Apply wild card group limit settings for users without any matching group +* [YUNIKORN-1871] - Improving test coverage for whole user/group enforcement feature +* [YUNIKORN-1881] - Improve the performance of tryPlaceholderAllocate +* [YUNIKORN-1882] - Further performance improvements on HavePodsWithAffinityList() and HavePodsWithRequiredAntiAffinityList() +* [YUNIKORN-1883] - Allow nodes to be created as unschedulable +* [YUNIKORN-1886] - Optimize scheduling cycle if there's no placeholder allocation or reservation +* [YUNIKORN-1887] - SI: Add tag for application recovered flag +* [YUNIKORN-1888] - Allow allocations to be added to existing nodes +* [YUNIKORN-1893] - Group tracker decrease resource usage incorrectly +* [YUNIKORN-1895] - SI: Add RejectedAllocations to AllocationResponse +* [YUNIKORN-1899] - Handle config reload in the event code +
(yunikorn-web) annotated tag v1.4.0-2 updated (781370e -> f3b6e37)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0-2 in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git *** WARNING: tag v1.4.0-2 was modified! *** from 781370e (commit) to f3b6e37 (tag) tagging 781370e047edf91d412d50e1b8caa8af7481dc5b (commit) replaces v1.4.0-1 by Wilfred Spiegelenburg on Wed Nov 15 14:12:44 2023 +1100 - Log - RC1 updated for v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) branch branch-1.4 updated: [YUNIKORN-2150] Use dominant resource utilization in UI (#147)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git
The following commit(s) were added to refs/heads/branch-1.4 by this push:
new 781370e [YUNIKORN-2150] Use dominant resource utilization in UI (#147)
781370e is described below
commit 781370e047edf91d412d50e1b8caa8af7481dc5b
Author: Yu-Lin Chen
AuthorDate: Wed Nov 15 13:38:59 2023 +1100
[YUNIKORN-2150] Use dominant resource utilization in UI (#147)
Node resource utilization chart does not render properly as it expects
multiple resources types. Node utilisation REST was changed to dominant
resource only.i
Multiple resource types in the REST response could lead to showing the
incorrect usage (non dominant one) or for different types after a
reload. The REST content shows types in a random way.
Closes: #147
Signed-off-by: Wilfred Spiegelenburg
---
json-db.json | 473 +++--
json-routes.json | 2 +-
.../components/dashboard/dashboard.component.ts| 16 +-
src/app/services/scheduler/scheduler.service.ts| 4 +-
4 files changed, 67 insertions(+), 428 deletions(-)
diff --git a/json-db.json b/json-db.json
index 9ba1666..09ffc7c 100644
--- a/json-db.json
+++ b/json-db.json
@@ -751,422 +751,63 @@
"reservations": []
}
],
- "utilization": [
-{
- "type": "ephemeral-storage",
- "utilization": [
-{
- "bucketName": "0-10%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "10-20%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "20-30%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "30-40%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "40-50%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "50-60%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "60-70%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "70-80%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "80-90%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "90-100%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-}
- ]
-},
-{
- "type": "hugepages-1Gi",
- "utilization": [
-{
- "bucketName": "0-10%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "10-20%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "20-30%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "30-40%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "40-50%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "50-60%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "60-70%",
- "numOfNodes": -1,
- "nodeNames": [
-
(yunikorn-web) branch master updated: [YUNIKORN-2150] Use dominant resource utilization in UI (#147)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git
The following commit(s) were added to refs/heads/master by this push:
new d1a26e7 [YUNIKORN-2150] Use dominant resource utilization in UI (#147)
d1a26e7 is described below
commit d1a26e7f05d7008435f973974e7601b47b8a89a6
Author: Yu-Lin Chen
AuthorDate: Wed Nov 15 13:38:59 2023 +1100
[YUNIKORN-2150] Use dominant resource utilization in UI (#147)
Node resource utilization chart does not render properly as it expects
multiple resources types. Node utilisation REST was changed to dominant
resource only.i
Multiple resource types in the REST response could lead to showing the
incorrect usage (non dominant one) or for different types after a
reload. The REST content shows types in a random way.
Closes: #147
Signed-off-by: Wilfred Spiegelenburg
---
json-db.json | 473 +++--
json-routes.json | 2 +-
.../components/dashboard/dashboard.component.ts| 16 +-
src/app/services/scheduler/scheduler.service.ts| 4 +-
4 files changed, 67 insertions(+), 428 deletions(-)
diff --git a/json-db.json b/json-db.json
index 9ba1666..09ffc7c 100644
--- a/json-db.json
+++ b/json-db.json
@@ -751,422 +751,63 @@
"reservations": []
}
],
- "utilization": [
-{
- "type": "ephemeral-storage",
- "utilization": [
-{
- "bucketName": "0-10%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "10-20%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "20-30%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "30-40%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "40-50%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "50-60%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "60-70%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "70-80%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "80-90%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "90-100%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-}
- ]
-},
-{
- "type": "hugepages-1Gi",
- "utilization": [
-{
- "bucketName": "0-10%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "10-20%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "20-30%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "30-40%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "40-50%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "50-60%",
- "numOfNodes": -1,
- "nodeNames": [
-"N/A"
- ]
-},
-{
- "bucketName": "60-70%",
- "numOfNodes": -1,
- "nodeNames": [
-
(yunikorn-k8shim) annotated tag v1.4.0-2 updated (ee4dd0c6 -> e7bb640d)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0-2 in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git *** WARNING: tag v1.4.0-2 was modified! *** from ee4dd0c6 (commit) to e7bb640d (tag) tagging ee4dd0c61859be8a2ac6c6cfd517e3405486a064 (commit) replaces v1.4.0-1 by Wilfred Spiegelenburg on Tue Nov 14 18:32:37 2023 +1100 - Log - RC1 updated for v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch dependabot/go_modules/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc-0.46.0 deleted (was 64be4b22)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch dependabot/go_modules/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc-0.46.0 in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git was 64be4b22 Bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) branch branch-1.4 updated (d2b8ebde -> 38a23bd4)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to branch branch-1.4 in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git from d2b8ebde [YUNIKORN-2129] fix go.mod (#722) add 38a23bd4 [YUNIKORN-2105] Update core and scheduler-interface to v1.4.0-1 No new revisions were added by this update. Summary of changes: go.mod | 4 ++-- go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-k8shim) annotated tag v1.4.0-1 updated (38a23bd4 -> 0775700d)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0-1 in repository https://gitbox.apache.org/repos/asf/yunikorn-k8shim.git *** WARNING: tag v1.4.0-1 was modified! *** from 38a23bd4 (commit) to 0775700d (tag) tagging 38a23bd431b470483f5725acebc6bec7cebaa912 (commit) by Wilfred Spiegelenburg on Tue Nov 14 12:47:34 2023 +1100 - Log - RC1 for v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) annotated tag v1.4.0-1 updated (c75a39a3 -> 16453583)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0-1 in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git *** WARNING: tag v1.4.0-1 was modified! *** from c75a39a3 (commit) to 16453583 (tag) tagging c75a39a33132aded2f825674b6c670e530449a23 (commit) by Wilfred Spiegelenburg on Tue Nov 14 12:41:49 2023 +1100 - Log - RC1 for v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-core) branch branch-1.4 updated: [YUNIKORN-2105] Update yunikorn-scheduler-interface to v1.4.0-1
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a commit to branch branch-1.4 in repository https://gitbox.apache.org/repos/asf/yunikorn-core.git The following commit(s) were added to refs/heads/branch-1.4 by this push: new c75a39a3 [YUNIKORN-2105] Update yunikorn-scheduler-interface to v1.4.0-1 c75a39a3 is described below commit c75a39a33132aded2f825674b6c670e530449a23 Author: Wilfred Spiegelenburg AuthorDate: Tue Nov 14 12:34:48 2023 +1100 [YUNIKORN-2105] Update yunikorn-scheduler-interface to v1.4.0-1 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 7da143fa..02595123 100644 --- a/go.mod +++ b/go.mod @@ -22,7 +22,7 @@ module github.com/apache/yunikorn-core go 1.20 require ( - github.com/apache/yunikorn-scheduler-interface v0.0.0-20231020041412-6f80d179257c + github.com/apache/yunikorn-scheduler-interface v1.4.0-1 github.com/google/btree v1.1.2 github.com/google/go-cmp v0.5.9 github.com/google/uuid v1.3.0 diff --git a/go.sum b/go.sum index e2f32a27..e903b68a 100644 --- a/go.sum +++ b/go.sum @@ -40,8 +40,8 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho= -github.com/apache/yunikorn-scheduler-interface v0.0.0-20231020041412-6f80d179257c h1:KTIC3f+3aQdAo42YRxs27VpDWY6y73bxXpWcAii2IlQ= -github.com/apache/yunikorn-scheduler-interface v0.0.0-20231020041412-6f80d179257c/go.mod h1:3NQfrhroMqU++kDTroBrTyCRKAczwwX//Fkj/ag/rsY= +github.com/apache/yunikorn-scheduler-interface v1.4.0-1 h1:ogC/IDvnI2E+eHD6dTiUI/jkDDurzBzDCnfqBQULSK4= +github.com/apache/yunikorn-scheduler-interface v1.4.0-1/go.mod h1:zDWV5y9Zh9DM1C65RCVXT1nhNNO8kykVW7bzPFamNYw= github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
(yunikorn-web) annotated tag v1.4.0-1 updated (271bb72 -> 085912c)
This is an automated email from the ASF dual-hosted git repository. wilfreds pushed a change to annotated tag v1.4.0-1 in repository https://gitbox.apache.org/repos/asf/yunikorn-web.git *** WARNING: tag v1.4.0-1 was modified! *** from 271bb72 (commit) to 085912c (tag) tagging 271bb7210ea4cdccc02d671c088525de65c637b0 (commit) replaces v1.3.0 by Wilfred Spiegelenburg on Tue Nov 14 12:21:08 2023 +1100 - Log - RC1 for v1.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org
