This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
The following commit(s) were added to refs/heads/master by this push:
new a37e17ce docs for new workflow transforms, tidy up tips including
foreach example
a37e17ce is described below
commit a37e17ce6071356d1f8546d391cd5bdd50c643cb
Author: Alex Heneveld <a...@cloudsoft.io>
AuthorDate: Tue Oct 10 14:35:22 2023 +0100
docs for new workflow transforms, tidy up tips including foreach example
---
guide/blueprints/workflow/defining.md | 3 ++
guide/blueprints/workflow/settings.md | 29 ++----------
guide/blueprints/workflow/steps/steps.yaml | 43 ++++++++++++------
guide/blueprints/workflow/tips.md | 72 ++++++++++++++++++++++++++++++
guide/ops/configuration/brooklyn_cfg.md | 17 ++++---
5 files changed, 120 insertions(+), 44 deletions(-)
diff --git a/guide/blueprints/workflow/defining.md
b/guide/blueprints/workflow/defining.md
index bba29825..7bee711f 100644
--- a/guide/blueprints/workflow/defining.md
+++ b/guide/blueprints/workflow/defining.md
@@ -92,6 +92,9 @@ The config to define the sensor feed is:
if on the local entity, or a map of `sensor` containing the name and
`entity` containing the entity or entity ID
where the sensor should be listened for, or if just a single local sensor,
that sensor name supplied as a string
* `period`: whether the feed should run periodically
+* `skip_initial_run`: by default sensors and policies will run when created
(if any `condition` is met);
+ this can be set `true` to prevent that,
+ ensuring it is only run after the initial `period` or when one of the
`triggers` fires
In addition to defining the `sensor` name, at least one of `triggers` or
`period` must be supplied.
The `steps` must also be defined, as per `workflow-effector` above,
diff --git a/guide/blueprints/workflow/settings.md
b/guide/blueprints/workflow/settings.md
index 2f76ff31..ab7af87c 100644
--- a/guide/blueprints/workflow/settings.md
+++ b/guide/blueprints/workflow/settings.md
@@ -10,7 +10,7 @@ Some of the common properties permitted on [steps](common.md)
also apply to work
including `condition`, `timeout`, and `on-error`.
This rest of this section describes the remaining properties for more advanced
use cases
-including mutex locking and resilient workflows with replay points, and some
tips on optimizing.
+including mutex locking and resilient workflows with replay points.
## Locks and Mutual Exclusion Behavior
@@ -309,10 +309,11 @@ complicated replay or retention needs.
* **`idempotent`**
* when defining a workflow, as `idempotent: <value>`
- * **`all`**: means that all external steps in this workflow will be
resumable unless explicitly declared otherwise (
- by default, per below, external steps are not resumable); this includes
steps in explicit sub-workflows (where the
+ * **`all`**: means that all external steps in this workflow will be
resumable unless explicitly declared otherwise
+ (by default, per below, external steps are not resumable); this includes
steps in explicit sub-workflows (where the
workflow definition has a `workflow` with `steps`) but not sub-workflows
which are references (effectors or
registered workflow types)
+ * the empty string or null (the default): means normal idempotency
determination applies
* on a step as a key, as `idempotent: <value>`
* **`yes`**: the step is idempotent and the workflow can replay resuming
at this step if interrupted there
* **`no`**: the step is not idempotent and should not resumed at this
step; if interrupted there, replay resuming
@@ -476,25 +477,3 @@ on-error:
# finally restore default retention per parent or system, as details are now
stored on the entity
- workflow retention parent
```
-
-## Optimizing for Workflows
-
-Workflows can generate a huge amount of data which can impact memory usage,
persistence, and the UI.
-The REST API and UI do some filtering (e.g. in the body of the `internal`
sensors used by workflow),
-but when working with large `ssh` `output` and `http` `content` payloads, and
with `update-children`,
-performance can be dramatically improved by following these tips:
-
-* Optimize external calls to return the minimal amount of information needed
- * Use `jq` to filter when using `ssh` or `container` steps
- * Pass filter arguments to `http` endpoints that accept them
- * Loop over small page sizes (e.g. 20 records per cycle from `http`) using
`retry from` steps
-
-* Optimize the data which is stored
- * Override the `output` on `ssh` and `http` steps to remove unnecessary
objects;
- for example `http` returns several `content*` fields, and often just one
is needed.
- Simply settings `output: { content: ${content} }` will achieve this.
- * Set `retention: 1` or `retention: 0` on workflows that use a large amount
of information
- and can simply be replayed from the start; additionally `retention:
disabled` can be used
- to prevent any persistence (even for ongoing workflows), but only for
workflows that do
- not acquire any `lock`
-
diff --git a/guide/blueprints/workflow/steps/steps.yaml
b/guide/blueprints/workflow/steps/steps.yaml
index a62d7dd8..e84fba15 100644
--- a/guide/blueprints/workflow/steps/steps.yaml
+++ b/guide/blueprints/workflow/steps/steps.yaml
@@ -43,21 +43,16 @@
- name: transform
summary: Applies a transformation to a variable or expression
- shorthand: '`transform [TYPE] [ "value" ] VARIABLE_NAME [ = VALUE ] |
TRANSFORM`'
- input: |
- * `variable`: either a string, being the workflow variable name, or a
map, containing the `name` and optionally the `type`;
- if `value` is supplied, this variable will be set to the result of
all transforms and then coercion to any indicated type,
- e.g. to force conversion to an integer or to a bean registered type;
- if `value` is not supplied, the name is treated as a variable name
to resolve and coerce to any indicated type
- (i.e. wrapping the name in `${...}`; this is suppressed if the type
is the special keyword `value`, and `name` is resolved as a normal expression)
- and then used for the transform with the result being returned;
- `name` here can be of the form `x.key` where `x` is an existing map
variable, to user or set a specific `key` within it
- * `value_is_initial`: a boolean, implied in shorthand by the word
"value", means to treat the variable name as a value rather than a variable
name,
- evaluating it as a normal expression rather than wraping it in
`${...}`; this cannot be used it a `value` is supplied
- * `value`: the value to set, with some limited evaluation prior to
transforms as described [here](../variables.html);
- if not supplied, the `variable` is evaluated per the above and used
as the input for the transform
+ shorthand: '`transform [TYPE] [ VALUE | "value" VALUE | "variable"
VARIABLE_NAME | VARIABLE_NAME "=" VALUE ] "|" TRANSFORM`'
+ input: |
+ * `variable`: either a string, being the workflow variable name, or a
map, containing a `name` and/or a `type`;
+ if a variable name is supplied, it is set as the result of the
transform,
+ and if `value` is not supplied, its initial value is used as the
input to the transform (update in place)
+ * `value`: the value to start with; if omitted a `variable` is
required and its value used as the starting value;
+ where the value might be confused with a variable name it must be
explicitly offset with the keyword `"value"`;
+ more information is [here](../variables.html)
* `transform`: a string indicating a transform, or multiple transforms
separated by `|`, where a transform can be
- * `trim` to remove leading and trailing whitespace on strings, null
values from lists or sets, and any entry with a null key or null value in a map
+ * `trim` to remove leading and trailing whitespace on strings, null
and empty string values from lists or sets, and any entry with a null key or
null value in a map
* `replace MODE PATTERN REPLACEMENT` to replace a pattern in a
string with a replacement, supporting mode `regex`, `glob`, or `literal`
* `merge [list|set|map] [deep] [wait] [lax]` to treat the words in
the value as expressions to be combined;
indicates the value will be a space-separated list of values, with
quotes significant, usually including `$(vars)`;
@@ -66,6 +61,8 @@
if `wait`, unavailable sensor values are blocked on before merging;
if `lax` then nulls, and if not waiting unavailable sensors, are
silently removed
* `prepend <item>` and `append <item>` to add `<item>` at the start
or end, respectively, when the input to the transform is a list or a string
+ * `join [<separator>]` to concatenate stringable items from a list
+ * `split [limit <limit>] [keep_delimiters] [literal|regex]
<delimiter>` to split a string around the given `<delimiter>`, optionally
including the delimiters or limiting the count
* `slice <start-index> <optional-end-index>` to take a list and
return the sublist; a negative index is taken from the end
* `remove <item-or-index>` to remove an entry given a key (for a
map, ignoring if not found) or an index (for a list, negative index taken from
the end, failing if out of bounds)
* `wait` to wait on values (such as
`${entity.attributeWhenReady...}`); must precede any expression that has to
resolve values, such as `trim`)
@@ -83,6 +80,8 @@
* `to_string`, `to_upper_case`, and `to_lower_case` are accepted for
strings
* `resolve_expression` to trigger a re-resolution of the value being
transformed, eg if it has nested variable references that need resolution or
has been explicitly prepended and appended with `${` and `}`;
this is not done immediately but on the next step (or at the end),
so the value can be piped to `wait` if desired
+ * `get [<modifier>]` to get the unwrapped value of a supplier (if no
arguments) or get a specific indexed value or modifier sequence,
+ such as `get 0` or `get [0]` to get the first element of a list,
or `get .key.subkey[0]` to take an element `key` in a map, an element `subkey`
in the resulting map, and the first element of that list
* `set VAR` to set the workflow variable `VAR` to the the value at
that point in the transformation
* `return` to return the result of the transformation (not
compatible with supplying a `value` to set in a variable)
* any other word is looked up as a registered type of type
`org.apache.brooklyn.core.workflow.steps.transform.WorkflowTransform` (or
`WorkflowTransformWithContext` to get the context or arguments)
@@ -142,6 +141,7 @@
input: |
* `message`: a string to report in the exception
* `rethrow`: whether to include any error in scope (if being used
within an `on-error` step)
+ * `value`: on optional value that can be included in the failure, and
for callers to inspect additional context
output: _none_ (throws exception)
- name: retry
@@ -292,6 +292,21 @@
* `content_json`: the content, parsed as JSON if possible to return a
map, list, string, or primitive (null if not valid json)
* `duration`: how long the request took
+ - name: shell
+ summary: Runs a command using the local system shell
+ shorthand: '`shell COMMAND`'
+ input: |
+ * `command`: the command to run
+ * `env`: a map of string keys with values whose JSON is taken and
passed to the command be executed
+ * `exit_code`: the word `any`, a number, or a predicate DSL expression
(e.g. `less-than: 32`)
+ * `output_max_size`: by default output from commands is limited to
100k, truncating to keep the end; this can be increased or decreased,
+ or disabled with `-1` (assuming Apache Brooklyn has sufficient
memory to hold the data)
+ * `interpolation_mode` and `interpolation_errors`: as per `let` and
`load`, with defaults `full` and `ignore` so that applicable values are
resolved and others deferred
+ output: |
+ * `stdout`
+ * `stderr`
+ * `exit_code`
+
- name: ssh
summary: Runs a command over ssh
shorthand: '`ssh COMMAND`'
diff --git a/guide/blueprints/workflow/tips.md
b/guide/blueprints/workflow/tips.md
new file mode 100644
index 00000000..611956ce
--- /dev/null
+++ b/guide/blueprints/workflow/tips.md
@@ -0,0 +1,72 @@
+---
+title: Tips and Tricks
+layout: website-normal
+---
+
+## Filtering Lists
+
+There are many `transform` filters available, but none will filter lists.
+Removing elements from a list is easily be done by using `foreach`` with a
`condition`,
+returning elements that meet the condition.
+
+For example, to filter out non-empty strings from a list:
+
+```
+- let list_to_filter = [ "word1", "", "word2" ]
+- step: foreach item in ${list_to_filter}
+ condition:
+ size: { greater-than: 0 }
+ steps:
+ - return ${item}
+```
+
+The output from the `foreach` step is the list of results,
+so this results in the filtered list `["word1", "word2"]`.
+
+(This condition uses `size` to find the length of the string.
+The `condition: { not: { equals: "" } }` could just as well be used,
+or using the fact that `equals` can be made implicit, simply
+`condition: { not: "" }`.)
+
+Here is a more complicated example which filters comma-separated words
+to return known fruits and proper names:
+
+```
+- let requested_foods = "apple, banana, Nutella, invalid"
+- transform value ${requested_foods} | split regex \S*,\S* | trim
+- let list known_fruits = [ "apple", "orange", "banana" ]
+- step: foreach item in ${output}
+ condition:
+ any:
+ - regex: [A-Z].* # match proper name
+ - target: ${known_fruits} # match known fruits
+ contains: ${item}
+ steps:
+ - return ${item}
+```
+
+The above will return the list of `apple`, `banana`, and `Nutella`, dropping
`invalid`.
+
+
+
+## Optimizing for Workflows
+
+Workflows can generate a huge amount of data which can impact memory usage,
persistence, and the UI.
+The REST API and UI do some filtering (e.g. in the body of the `internal`
sensors used by workflow),
+but when working with large `ssh` `output` and `http` `content` payloads, and
with `update-children`,
+performance can be dramatically improved by following these tips:
+
+* Optimize external calls to return the minimal amount of information needed
+ * Use `jq` to filter when using `ssh` or `container` steps
+ * Pass filter arguments to `http` endpoints that accept them
+ * Loop over small page sizes (e.g. 20 records per cycle from `http`) using
`retry from` steps
+
+* Optimize the data which is stored
+ * Override the `output` on `ssh` and `http` steps to remove unnecessary
objects;
+ for example `http` returns several `content*` fields, and often just one
is needed.
+ Simply settings `output: { content: ${content} }` will achieve this.
+ * Set `retention: 1` or `retention: 0` on workflows that use a large amount
of information
+ and can simply be replayed from the start; additionally `retention:
disabled` can be used
+ to prevent any persistence (even for ongoing workflows), but only for
workflows that do
+ not acquire any `lock`
+
diff --git a/guide/ops/configuration/brooklyn_cfg.md
b/guide/ops/configuration/brooklyn_cfg.md
index 7b22351e..0000af1f 100644
--- a/guide/ops/configuration/brooklyn_cfg.md
+++ b/guide/ops/configuration/brooklyn_cfg.md
@@ -187,10 +187,10 @@ brooklyn.entitlements.global=<class>
The default entitlement manager is one which responds to per-user entitlement
rules,
and understands:
-* `root`: full access, including to the Groovy console
-* `poweruser`: full access, excluding to the Groovy console
-* `user`: access to everything but actions that affect the server itself.
Such actions include the
- Groovy console, stopping the server and retrieving management context
configuration
+* `root`: full access, including local scripts and the Groovy console
+* `poweruser`: full access apart from local shell scripts and Groovy scripts
+* `user`: access to everything but actions that affect the server itself.
Such actions include scripts,
+ stopping the server and retrieving management context configuration
* `blueprintAuthor`: same as user but cannot install bundles containing jar
or class files
* `readonly`: read-only access to almost all information
* `minimal`: access only to server stats, for use by monitoring systems
@@ -268,9 +268,16 @@ brooklyn.webconsole.security.login.form=brooklyn-ui-login
## SSH and Script Defaults
-Default values for SSH and script execution behaviour can be set in this file
+Default values for SSH connection and script execution behaviour can be set in
this file
using the prefix `brooklyn.ssh.config.`, as described in
[Locations](/guide/locations#os-setup).
+Local shell access via the `shell` workflow step, available only to users with
`root`-level entitlement,
+can be disabled altogether using the following setting:
+
+```
+brooklyn.security.shell.workflow_step.disabled=true
+```
+
## Certificate Validation
