This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new ff1aec4b52 [YUNIKORN-2444] Create user guide for the event system
(#406)
ff1aec4b52 is described below
commit ff1aec4b52e242df8bef987ef16d6f6388946321
Author: Peter Bacsko <pbac...@cloudera.com>
AuthorDate: Mon Mar 25 16:19:35 2024 +0100
[YUNIKORN-2444] Create user guide for the event system (#406)
Closes: #406
Signed-off-by: Peter Bacsko <pbac...@cloudera.com>
---
docs/user_guide/event_system.md | 147 ++++++++++++++++++++++++++++++++++++++
docs/user_guide/service_config.md | 43 +++++++++++
sidebars.js | 1 +
3 files changed, 191 insertions(+)
diff --git a/docs/user_guide/event_system.md b/docs/user_guide/event_system.md
new file mode 100644
index 0000000000..2ea7b57f23
--- /dev/null
+++ b/docs/user_guide/event_system.md
@@ -0,0 +1,147 @@
+---
+id: event_system
+title: Event System
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Overview
+
+The Yunikorn scheduler core generates well-defined events at various points
during the execution. Since an event has a fixed structure, it is an ideal
input for automated tools for further processing. Although Yunikorn has
extensive logging, the output is textual and needs to be parsed. Not only is it
error-prone, but also nothing guarantees that it never changes over time: it
might be completely removed in a future release or the log level can change.
+
+The event object is defined in the scheduler interface repository along with
the constants that describe a certain event.
+
+Certain types of events which are related to pods and nodes are also sent to
Kubernetes itself. This allows the users to identify certain problems that can
arise from misconfiguration, such as insufficient queue or user quota. If a pod
is pending because the quota is exceeded, it will be updated with an event
about why it cannot be scheduled. Previously this had to be inferred from the
logs or state dump.
+
+Various properties of the event subsystem are configurable. Changes are picked
up and applied immediately.
+
+## Event types
+
+Events always describe a specific object type. There are four types of events:
application, queue, node and user/group.
+When it is appropriate, the event will be sent to Kubernetes using the K8s
event API. Since the API requires a "regarding" object (ie. which K8s object
the event is about), we only send node and pod events.
+
+The following table summarizes when they are generated.
+
+| Type | Event | Notes |
+|--|--|--|
+| Application | A new application is created ||
+| Application | Application state transition ||
+| Application | Successful allocation ||
+| Application | An ask is added to the application ||
+| Application | Application removed ||
+| Application | Application rejected ||
+| Application | Ask removed ||
+| Application | Allocation removed ||
+| Application | Placeholder timeout ||
+| Application | Allocation replacement (gang scheduling) ||
+| Application | Application is not runnable due to maxApplications | Queue or
user maxApps limit |
+| Application | Application has become runnable | Only sent if limit was hit
before |
+| Application | Ask is not schedulable due to quota limitations | Sent to
Kubernetes to the respective pod |
+| Application | Ask has become schedulable (there is quota now) | Sent to
Kubernetes to the respective pod<br/> Only sent if quota was not available
before |
+| Application | Ask is not schedulable due to predicate error * | Sent to
Kubernetes to the respective pod<br/> Rate limited: sent once in every 15
seconds |
+| Application | Real allocation is larger than the placeholder | Sent to
Kubernetes to the respective pod |
+| Node | New node added to the scheduler core | Sent to Kubernetes to the
respective node |
+| Node | Node removed from the scheduler core | Sent to Kubernetes to the
respective node |
+| Node | Allocation added to a node ||
+| Node | Allocation removed from a node ||
+| Node | Node "ready" status changed ||
+| Node | Node "schedulable" status changed ||
+| Queue | Queue added to the scheduler core ||
+| Queue | Application added to a queue ||
+| Queue | Queue removed from the scheduler core ||
+| Queue | Max resource changed | From leaf to parent or vice-versa ||
+| Queue | Queue type changed ||
+| User/group | Limit configured for a user ||
+| User/group | Limit configured for a group ||
+| User/group | Limit removed for a user ||
+| User/group | Limit removed for a group ||
+| User/group | Resource usage increased for a user ||
+| User/group | Resource usage increased for a group | Only sent if group limit
is active |
+| User/group | Group tracking got associated with a user| Only sent if group
limit is active |
+| User/group | Group tracking removed from a user| Only sent if group limit is
active |
+
+`*` a predicate is a plugin located in the default scheduler. They're
responsible for functionality like node selectors, affinity, anti-affinity,
etc. Yunikorn runs the predicates for every allocation to see if it fits the
candidate node.
+
+## In-memory storage of historical events
+
+Yunikorn stores a previously generated event objects in a ring buffer. Once it
gets full, oldest elements are overwritten. The default size of the buffer is
100000.
+On a busy cluster this might not be enough. The size can be changed in the
configmap by using the key `event.ringBufferCapacity`.
+Keep in mind that increasing the size results in higher memory usage and
longer GC pause times. Events that are put in the ring buffer are not removed,
only overwritten.
+If the size of the buffer gets smaller after a config change, the oldest
entries that do not fit are discarded.
+
+Every event is assigned an ID starting from 0. This can be used to fetch them
based on ID.
+
+Yunikorn currently does not offer a solution which stores the generated events
in a persistent storage. However, this might change in the upcoming releases.
+
+## Retrieving events
+
+The REST API provides two ways to see the generated events.
+
+### Batch REST endpoint
+
+The batch endpoint is available at `/ws/v1/events/batch`. If not defined, the
start ID is 0 and a maximum of 10000 events are returned. This can be changed
by defining the "start" and "count" URL query parameters.
+
+See [batch interface](../api/scheduler.md#batch-events) for details.
+
+### Streaming REST endpoint
+
+The streaming endpoint is available at `/ws/v1/events/stream`. Unlike the
batch interface, which closes the connection after the query, streaming keeps
the HTTP connection open. There is no idle timeout, so as long as the
connection is stable, it is never closed. New events are sent immediately to
active clients. The URL parameter "count" is also accepted; it tells Yunikorn
how many recent events we want to fetch that were generated before the
connection.
+
+Since this approach uses more resources inside Yunikorn, the number of
streaming connections is limited, but the limits are configurable. See
[streaming](../api/scheduler.md#event-stream) for details. Existing connections
are not closed after configuration change.
+
+The active streaming connections are also shown in the state dump.
+
+## Event structure
+
+A Yunikorn event has the following fields:
+
+| Name | Description | Always set |
+|--|--|--|
+| ObjectID | Identifies the object what the event is about. | Yes |
+| ReferenceID | Identifies a secondary object which is related to the first
one. It can be an allocation ID, queue, etc. | No |
+| Message| Textual details. Relevant if it is sent to Kubernetes. | No |
+| Type | The type of the ObjectID: application, queue, node, user/group. | Yes
|
+| EventChangeType | The nature of the change: adding, setting, removing or
none. | Yes |
+| EventChangeDetail | Further details about the change itself. | Yes |
+| TimestampNano | Unix nanotime inside Yunikorn when the event was generated.
| Yes |
+| Resource | Set if the event involves a resource (eg. allocation occurs).| No|
+
+Example: when an allocation occurs, we generate three events: an application
event, a node event and a user event about the usage (there's a group event if
the limits are configured in a certain way, but it's not relevant in this
example).
+We use the following unique identifiers:
+* Application is "app-1"
+* The request is "req-1"
+* The selected node is "node-1"
+* Application belongs to the user "yunikorn"
+* The application has been placed in the queue "root.test"
+
+| What the event describes | ObjectID | ReferenceID | Type | EventChangeType |
EventChangeDetail |
+|--|--|--|--|--|--|
+| Allocation occurred for a request | app-1 | req-1-0 * | `EventRecord_APP` |
`EventRecord_ADD` | `EventRecord_APP_ALLOC` |
+| Allocation on a node | node-1 | req-1-0 * | `EventRecord_NODE` |
`EventRecord_ADD` | `EventRecord_NODE_ALLOC` |
+| Resource usage changed for a user | yunikorn | root.test |
`EventRecord_USERGROUP` | `EventRecord_ADD` | `EventRecord_UG_USER_RESOURCE` |
+
+`*` the allocation ID is based on the ask ID plus a counter.
+
+All events will contain the allocated resource.
+
+## Disabling the event system
+
+Although it has negligible overhead, it is possible to completely disable
Yunikorn events. This can be achieved by setting `event.trackingEnabled` to
`false`. This does NOT clear the ring buffer contents, so the history recorded
until the change is still available.
+
diff --git a/docs/user_guide/service_config.md
b/docs/user_guide/service_config.md
index 2703ad12b7..91e6ff3414 100644
--- a/docs/user_guide/service_config.md
+++ b/docs/user_guide/service_config.md
@@ -651,6 +651,49 @@ Example:
```yaml
service.instanceTypeNodeLabelKey: "node.kubernetes.io/my-instance-type"
```
+
+### Event system settings
+
+#### event.trackingEnabled
+Enables or disables the event system and event generation.
+
+Default: `true`
+
+Example:
+```yaml
+event.trackingEnabled: "false"
+```
+
+#### event.ringbufferCapacity
+Sets the capacity of the ring buffer which stores Yunikorn generated events.
+
+Default: `10000`
+
+Example:
+```yaml
+event.ringbufferCapacity: "300000"
+```
+
+#### event.maxStreams
+Sets the maximum number of event stream connections.
+
+Default: `100`
+
+Example:
+```yaml
+event.maxStreams: "50"
+```
+
+#### event.maxStreamsPerHost
+Sets the maximum number of event stream connections from a given host.
+
+Default: `15`
+
+Example:
+```yaml
+event.maxStreamsPerHost: "5"
+```
+
### Health settings
#### health.checkInterval
diff --git a/sidebars.js b/sidebars.js
index 31d090cb08..6de952f713 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -36,6 +36,7 @@ module.exports = {
'user_guide/labels_and_annotations_in_yunikorn',
'user_guide/prometheus',
'user_guide/use_cases',
+ 'user_guide/event_system',
{
type: 'category',
label: 'Workloads',
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: issues-h...@yunikorn.apache.org
