This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 3c589aa723 Reorganize the hop-conf page, fixes #4335 (#9006)
3c589aa723 is described below
commit 3c589aa723d64535cf3659e3562e9b7e31ce73cf
Author: Adalennis <[email protected]>
AuthorDate: Sun Oct 18 15:30:03 2026 +0200
Reorganize the hop-conf page, fixes #4335 (#9006)
* Reorganize the hop-conf page: projects, environments and configuration
files, and the current options. #4335
* Create the project home folder first, and list -fj and -scv in the
hop-conf options. #4335
---
.../ROOT/pages/hop-tools/hop-conf/hop-conf.adoc | 373 +++++----------------
.../modules/ROOT/pages/projects/advanced.adoc | 2 +-
.../hop-conf-cloud-azure-blob-storage.adoc | 7 +-
.../hop-conf-cloud-google-cloud-storage.adoc | 3 +
.../hop-tools/hop-conf/hop-conf-projects.adoc | 275 ++++++---------
5 files changed, 206 insertions(+), 454 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-conf/hop-conf.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-conf/hop-conf.adoc
index abfa0acdb0..0399e3b452 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-conf/hop-conf.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-conf/hop-conf.adoc
@@ -14,295 +14,22 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-:description: Hop Conf is a command line tool to manage your Hop client
installation
+:description: Hop Conf is a command line tool to manage your Hop
configuration: projects, environments, configuration files and the settings in
hop-config.json.
= Hop Conf - The Hop command line configuration tool
-== Usage
+Hop Conf changes the Hop configuration from the command line, so you can
script what you otherwise do in Hop Gui: create projects and environments, set
variables and change settings.
-Hop Conf is a command line tool to manage environments.
-The *hop-conf.sh* script can be run with the *-h* flag (`./hop-conf.sh -h`) to
display available options.
+It writes to `hop-config.json`, in the folder of the `HOP_CONFIG_FOLDER`
variable or by default in the `config` folder of your Hop installation, and to
the configuration files of projects and environments.
+Run it as `hop-conf.sh`, as `hop-conf.bat` on Windows, or as `hop conf` with
the xref:hop-tools/hop.adoc[hop] command.
- +
+`hop-conf.sh -h` lists all the options of your installation.
+Plugins add their own, for example for search, the File Explorer, linting, the
AI Assistant or cloud storage.
+The options described on this page are listed in <<_options,Options>>.
-.Usage
-[%collapsible]
-=====
-[source,bash]
-----
-Usage: <main class> [-h] [-ec] [-ed] [-el] [-em] [-ey] [-pc] [-pd] [-pl] [-pm]
- [-pn] [-py] [-aza=<account>] [-azi=<blockIncrement>]
- [-azk=<key>] [-cfg=<configFile>]
- [-dc=<defaultProjectConfigFile>] [-de=<defaultEnvironment>]
- [-dp=<defaultProject>] [-dv=<describeVariable>]
- [-e=<environmentName>] [-ep=<environmentProject>]
- [-eu=<environmentPurpose>] [-fj=<fatJarFilename>]
- [-gck=<serviceAccountKeyFile>] [-gdc=<credentialsFile>]
- [-gdt=<tokensFolder>] [-p=<projectName>]
- [-pa=<projectMetadataBaseFolder>]
- [-pb=<projectDataSetsCsvFolder>] [-pf=<projectConfigFile>]
- [-ph=<projectHome>] [-pp=<projectCompany>]
- [-pr=<projectParent>] [-ps=<projectDescription>]
- [-pt=<projectDepartment>] [-pu=<projectUnitTestsBasePath>]
- [-px=<projectEnforceExecutionInHome>]
- [-sj=<standardProjectsFolder>]
- [-sp=<standardParentProject>] [-sv=<setVariable>]
- [-xm=<metadataJsonFilename>]
[-cfd=<configDescribeVariables>
- [,<configDescribeVariables>...]]...
- [-cfv=<configSetVariables>[,<configSetVariables>...]]...
- [-eg=<environmentConfigFiles>[,
- <environmentConfigFiles>...]]... [-pv=<projectVariables>[,
- <projectVariables>...]]...
- -aza, --azure-account=<account>
- The account to use for the Azure VFS
- -azi, --azure-block-increment=<blockIncrement>
- The block increment size for new files on Azure,
- multiples of 512 only.
- -azk, --azure-key=<key>
- The key to use for the Azure VFS
- -cfd, --config-file-describe-variables=<configDescribeVariables>[,
- <configDescribeVariables>...]
- A list of variable=description combinations
separated by
- a comma
- -cfg, --config-file=<configFile>
- Specify the configuration JSON file to manage
- -cfv, --config-file-set-variables=<configSetVariables>[,
- <configSetVariables>...]
- A list of variable=value combinations separated by
a
- comma
- -dc, --default-projects-folder=<defaultProjectConfigFile>
- The standard project configuration filename
proposed
- when creating projects
- -de, --default-environment=<defaultEnvironment>
- The name of the default environment to use when
none is
- specified
- -dp, --default-project=<defaultProject>
- The name of the default project to use when none is
- specified
- -dv, --describe-variable=<describeVariable>
- Describe a variable, use format
VARIABLE=Description
- -e, --environment=<environmentName>
- The name of the lifecycle environment to manage
- -ec, --environment-create
- Create a new project lifecycle environment. Also
specify
- its name, purpose, the project name and the
- configuration files.
- -ed, --environment-delete
- Delete a lifecycle environment
- -eg, --environment-config-files=<environmentConfigFiles>[,
- <environmentConfigFiles>...]
- A list of configuration files for this lifecycle
- environment, comma separated
- --environment-config-file-directory=<environmentConfigFileDirectory>
- Folder of configuration files for this lifecycle
- environment. Matching files replace the
configuration
- file list, together with
--environment-config-files
- when that option is also set.
-
--environment-config-file-exclude-wildcard=<environmentConfigFileExcludeWildcard>
- Regular expression for file names to ignore in
- --environment-config-file-directory. Example:
- .*secret.*
- --environment-config-file-wildcard=<environmentConfigFileWildcard>
- Regular expression matched against file names in
- --environment-config-file-directory. Example:
- .*\.json
- --environment-config-include-subfolders
- Also include matching files from subfolders of
- --environment-config-file-directory
- -el, --environments-list
- List the defined lifecycle environments
- -em, --environment-modify
- Modify a lifecycle environment
- -ep, --environment-project=<environmentProject>
- The project for the environment
- -eu, --environment-purpose=<environmentPurpose>
- The purpose of the environment: Development,
Testing,
- Production, CI, ...
- -ey, --environment-mandatory
- Make it mandatory to reference an environment
- -fj, --generate-fat-jar=<fatJarFilename>
- Specify the filename of the fat jar to generate
from
- your current software installation
- -gck, --google-cloud-service-account-key-file=<serviceAccountKeyFile>
- Configure the path to a Google Cloud service
account
- JSON key file
- -gdc, --google-drive-credentials-file=<credentialsFile>
- Configure the path to a Google Drive credentials
JSON
- file
- -gdt, --google-drive-tokens-folder=<tokensFolder>
- Configure the path to a Google Drive tokens folder
- -h, --help Displays this help message and quits.
- -p, --project=<projectName>
- The name of the project to manage
- -pa, --project-metadata-base=<projectMetadataBaseFolder>
- The metadata base folder (relative to home)
- -pb, --project-datasets-base=<projectDataSetsCsvFolder>
- The data sets CSV folder (relative to home)
- -pc, --project-create Create a new project. Also specify the name and
its home
- -pd, --project-delete Delete a project
- -pf, --project-config-file=<projectConfigFile>
- The configuration file relative to the home
folder. The
- default value is project-config.json
- -ph, --project-home=<projectHome>
- The home directory of the project
- -pl, --projects-list List the defined projects
- -pla, --list-action-types-in-project
- List action types used in this project
- -plm, --list-metadata-types-in-project
- List metadata types used in this project
- -plt, --list-transform-types-in-project
- List transform types used in this project
- -pm, --project-modify Modify a project
- -pn, --projects-enabled
- Enable or disable the projects plugin
- -pp, --project-company=<projectCompany>
- The company
- -pr, --project-parent=<projectParent>
- The name of the parent project to inherit metadata
and
- variables from
- -ps, --project-description=<projectDescription>
- The description of the project
- -pt, --project-department=<projectDepartment>
- The department
- -pu, --project-unit-tests-base=<projectUnitTestsBasePath>
- The unit tests base folder (relative to home)
- -pv, --project-variables=<projectVariables>[,<projectVariables>...]
- A list of variable=value combinations separated by
a
- comma
- -px, --project-enforce-execution=<projectEnforceExecutionInHome>
- Validate before execution that a workflow or
pipeline is
- located in the project home folder or a
sub-folder
- (true/false).
- -py, --project-mandatory
- Make it mandatory to reference a project
- -sj, --standard-projects-folder=<standardProjectsFolder>
- GUI: The standard projects folder proposed when
creating
- projects
- -sp, --standard-parent-project=<standardParentProject>
- The name of the standard project to use as a
parent when
- creating new projects
- -sv, --set-variable=<setVariable>
- Set a variable, use format VAR=Value
- -xm, --export-metadata=<metadataJsonFilename>
- Export project metadata to a single JSON file
which you
- can specify with this option. Also specify the -p
- option.
- -v, --version Print version information and exit
-----
-
-The available options are listed below:
-
-.Hop-conf Options
-[id="hop-conf",cols="10%, 40%,50%",options="header"]
-|===
-|Short Option|Extended Option|Description
-|-h|--help|Displays this help message and quits
-|-v|--version|Print version information and exit
-|-ec|--environment-create|Create an environment
-Also specify the name and its home
-|-ed|--environment-delete|Delete an environment
-|-el|--environment-list|List the defined environments
-|-em|--environment-modify|Modify an environment
-|-pc|--project-create|Create a new project.
-Also specify the name and its home
-|-pd|--prject-delete|Delete a project
-|-pl|--projects-list|List the defined projects
-|-pla|--list-action-types-in-project|List action types used in this project
-|-plm|--list-metadata-types-in-project|List metadata types used in this project
-|-plt|--list-transform-types-in-project|List transform types used in this
project
-|-pm|--project-modify|Modify a project
-|-dv|--describe-variable=<describeVariable>|Describe a variable
-|-e|-environment=<environmentName>|The name of the environment to manage
-|-ep|--environment-project=<environmentProject>|The project for the environment
-|-eu|--environment-purpose=<environmentPurpose>|The purpose of the
environment: Development, Testing, Production, CI, ...
-|-fj|--generate-fat-jar=<fatJarFilename>|Specify the filename of the fat jar
to generate from your current software installation
-|-xm|--export-metadata=<metadataJsonFilename>|Export project metadata to a
single JSON file which you can specify with this option.
-Also specify the -p option to know which metadata to export.
-|-p|--project=<projectName>|The project name
-|-pa|--project-metadata-base=<projectMetadataBaseFolder>|The metadata base
folder (relative to home)
-|-pb|--project-datasets-base-base=<projectDataSetsCsvFolder>|The data sets CSV
folder (relative to home)
-|-pf|--project-config-file=<projectConfigFile>|The configuration file relative
to the home folder.
-The default value is project-config.json
-|-ph|--project-home=<projectHome>|The home directory of the project
-|-pp|--project-company=<projectCompany>|The company
-|-ps|--project-description=<projectDescription>|The description of the project
-|-pt|--project-department=<projectDepartment>|The department
-|-pu|--project-unit-tests-base=<projectUnitTestsBasePath>|The unit tests base
folder (relative to home)
-|-px|--project-enforce-execution=<projectEnforceExecutionInHome>|Validate
before execution that a workflow or pipeline islocated in the project home
folder or a sub-folder (true/false)
-|-sv
-| --set-variable=<setVariable>
-a| Set a variable, use format VAR=Value
-
-TIP: `-sv` can be used to unset a variable by specifying a variable without a
value, e.g. `-sv=myvar=`
-
-|-cfg|--config-file=<configFile>|Specify the configuration JSON file to manage
-|-cfd|--config-file-describe-variables=<configDescribeVariables>[,<configDescribeVariables>...]|A
list of variable=description combinations separated by a comma
-|-cfv|--config-file-set-variables=<configSetVariables>
,<configSetVariables>...]|A list of variable=value combinations separated by a
comma
-|-eg|--environment-config-files=<environmentConfigFiles>[,
<environmentConfigFiles>...]|A list of configuration files for this lifecycle
environment, comma separated
-||--environment-config-file-directory=<environmentConfigFileDirectory>|Folder
of configuration files. Matching files replace the configuration file list,
together with --environment-config-files when that option is also set. The
wildcard is a regular expression, for example `.*\.json`, not a shell glob
-||--environment-config-file-wildcard=<environmentConfigFileWildcard>|Regular
expression matched against file names in --environment-config-file-directory
-||--environment-config-file-exclude-wildcard=<environmentConfigFileExcludeWildcard>|Regular
expression for file names to ignore in that folder
-||--environment-config-include-subfolders|Also include matching files from
subfolders of --environment-config-file-directory
-|-pv|--project-variables=<projectVariables>[,<projectVariables>...]| A list of
variable=value combinations separated by a comma
-||--graph-connection-test=<name>|Test the graph database connection or Neo4j
connection with the given name. See <<_graph_database_connections>>
-||--graph-connection-info=<name>|Print the graph database type and
capabilities of a graph database connection or Neo4j connection as JSON
-||--graph-database-types|Print the installed graph database types and their
capabilities as JSON
-|===
-
-=====
+== Projects and environments
- +
-
-=== project variables Examples
-This is a list of examples on how the parameters on this command are parsed
-
-==== Normal Usage
-----
---project-variables=key1=value1,key2=value2
-----
-Result:
-[options="header"]
-|===
-|Key|value
-|key1|value1
-|key2|value2
-|===
-==== Spaces in value Usage
-----
---project-variables=key1="This value contains spaces",key2=value2
-----
-Result:
-[options="header"]
-|===
-|Key|value
-|key1|This value contains spaces
-|key2|value2
-|===
-
-==== Commas in value Usage
-----
---project-variables=key1=\"value1,value2\"
-----
-Result:
-[options="header"]
-|===
-|Key|value
-|key1|value1,value2
-|===
-==== Forcing quotes in value
-----
---project-variables=key1="\"\"String with spaces\"\""
-----
-Result:
-[options="header"]
-|===
-|Key|value
-|key1|"String with spaces"
-|===
-
-== Project Usage and Configuration
-
-include::../../snippets/hop-tools/hop-conf/hop-conf-projects.adoc[]
+include::../../snippets/hop-tools/hop-conf/hop-conf-projects.adoc[leveloffset=+1]
== Graph database connections
@@ -380,7 +107,7 @@ N/A
=== Azure
-Set the account, block increment size for new files and your Azure key
+Set the account and key of your Azure storage account, or the URL of an Azure
emulator (Azurite)
include::../../snippets/hop-tools/hop-conf/hop-conf-cloud-azure-blob-storage.adoc[]
@@ -388,7 +115,7 @@
include::../../snippets/hop-tools/hop-conf/hop-conf-cloud-azure-blob-storage.ado
==== Google Cloud Storage
-Set the path to your Google Cloud service account JSON key file
+Set the path to your Google Cloud service account JSON key file, and the block
size to read files with
include::../../snippets/hop-tools/hop-conf/hop-conf-cloud-google-cloud-storage.adoc[]
@@ -398,3 +125,81 @@ Set the path to your Google Drive credentials JSON file or
Google Drive tokens f
include::../../snippets/hop-tools/hop-conf/hop-conf-cloud-google-drive.adoc[]
+== Options
+
+The options described on this page.
+Run `hop-conf.sh -h` for the complete list of your installation.
+
+[%header, cols="1,3,4"]
+|===
+|Short option|Long option|Description
+
+3+|*General*
+|`-h`|`--help`|Displays the help message and quits.
+|`-v`|`--version`|Prints version information and exits.
+|`-l`|`--level=<level>`|The log level: NOTHING, ERROR, MINIMAL, BASIC,
DETAILED, DEBUG or ROWLEVEL.
+
+3+|*Projects*
+|`-p`|`--project=<projectName>`|The name of the project to manage.
+|`-pc`|`--project-create`|Create a project. Also specify its name and home
folder.
+|`-pm`|`--project-modify`|Modify a project.
+|`-pd`|`--project-delete`|Remove a project from `hop-config.json`. Its files
are not deleted.
+|`-pl`|`--projects-list`|List the projects.
+|`-ph`|`--project-home=<projectHome>`|The home folder of the project.
+|`-pf`|`--project-config-file=<projectConfigFile>`|The configuration file,
relative to the home folder. The default is `project-config.json`.
+|`-pkf`|`--project-keep-config-file`|Keep the configuration file of the
project if it already exists.
+|`-pr`|`--project-parent=<projectParent>`|The parent project to inherit
metadata and variables from.
+|`-ps`|`--project-description=<projectDescription>`|The description of the
project.
+|`-pp`|`--project-company=<projectCompany>`|The company.
+|`-pt`|`--project-department=<projectDepartment>`|The department.
+|`-pv`|`--project-variables=<projectVariables>[,...]`|Variables of the
project, as `variable=value` separated by commas.
+|`-pa`|`--project-metadata-base=<projectMetadataBaseFolder>`|The metadata
folder, relative to the home folder.
+|`-pb`|`--project-datasets-base=<projectDataSetsCsvFolder>`|The data sets CSV
folder, relative to the home folder.
+|`-pu`|`--project-unit-tests-base=<projectUnitTestsBasePath>`|The unit tests
folder, relative to the home folder.
+|`-px`|`--project-enforce-execution=<projectEnforceExecutionInHome>`|`true` or
`false`: check before execution that a workflow or pipeline is in the home
folder of the project or a subfolder.
+|`-pla`|`--project-list-action-types`|List the action types used in the
project.
+|`-plm`|`--project-list-metadata-types`|List the metadata types used in the
project.
+|`-plt`|`--project-list-transform-types`|List the transform types used in the
project.
+|`-plmi`|`--project-list-metadata-item-usages=<item>`|List the files of the
project that use a given metadata item.
+|`-xm`|`--export-metadata=<metadataJsonFilename>`|Export the metadata of the
project given with `-p` to a single JSON file.
+
+3+|*Environments*
+|`-e`|`--environment=<environmentName>`|The name of the environment to manage.
+|`-ec`|`--environment-create`|Create an environment. Also specify its name,
purpose, project and configuration files.
+|`-em`|`--environment-modify`|Modify an environment.
+|`-ed`|`--environment-delete`|Remove an environment from `hop-config.json`.
Its configuration files are not deleted.
+|`-el`|`--environments-list`|List the environments.
+|`-ep`|`--environment-project=<environmentProject>`|The project of the
environment.
+|`-eu`|`--environment-purpose=<environmentPurpose>`|The purpose of the
environment: Development, Testing, Production, CI, ...
+|`-eg`|`--environment-config-files=<file>[,...]`|The configuration files of
the environment, separated by commas.
+||`--environment-config-file-directory=<folder>`|A folder of configuration
files. Matching files replace the configuration file list, together with
`--environment-config-files` when that option is also set.
+||`--environment-config-file-wildcard=<regex>`|Regular expression matched
against the file names in that folder, for example `.*\.json`.
+||`--environment-config-file-exclude-wildcard=<regex>`|Regular expression for
file names to ignore in that folder, for example `.*secret.*`.
+||`--environment-config-include-subfolders`|Also include matching files from
subfolders of that folder.
+
+3+|*Configuration files and variables*
+|`-cfg`|`--config-file=<configFile>`|The configuration file to manage.
+|`-cfv`|`--config-file-set-variables=<variable=value>[,...]`|Set variables in
that file, separated by commas.
+|`-cfd`|`--config-file-describe-variables=<variable=description>[,...]`|Describe
variables in that file, separated by commas.
+|`-sv`|`--set-variable=<variable=value>`|Set a variable in `hop-config.json`.
Without a value, the variable gets an empty value.
+|`-dv`|`--describe-variable=<variable=description>`|Describe a variable in
`hop-config.json`.
+
+3+|*Projects plugin*
+|`-pn`|`--projects-enabled`|Enable or disable the projects plugin.
+|`-py`|`--project-mandatory`|Make it mandatory to reference a project.
+|`-ey`|`--environment-mandatory`|Make it mandatory to reference an environment.
+|`-dp`|`--default-project=<defaultProject>`|The default project to use when
none is specified.
+|`-de`|`--default-environment=<defaultEnvironment>`|The default environment to
use when none is specified.
+|`-sp`|`--standard-parent-project=<standardParentProject>`|The parent project
proposed when creating a project.
+|`-sj`|`--standard-projects-folder=<standardProjectsFolder>`|The folder
proposed when creating a project.
+|`-dc`|`--default-projects-folder=<defaultProjectConfigFile>`|The
configuration filename for new projects.
+
+3+|*Beam and Spark*, when the Beam plugin is installed
+|`-fj`|`--generate-fat-jar=<fatJarFilename>`|Generate a fat jar with the given
filename from your current Hop installation, for the Beam and Spark engines.
+|`-scv`|`--spark-client-version=<sparkClientVersion>`|The Spark client pack to
embed in the fat jar: a version (folder `lib/spark-clients/<version>`),
`native` or `native-provided`. When omitted, `HOP_SPARK_CLIENT_VERSION` or the
default pack in `lib/spark-client` is used.
+
+3+|*Graph database connections*
+||`--graph-connection-test=<name>`|Test the graph database connection or Neo4j
connection with the given name. See <<_graph_database_connections>>.
+||`--graph-connection-info=<name>`|Print the graph database type and
capabilities of a connection as JSON.
+||`--graph-database-types`|Print the installed graph database types and their
capabilities as JSON.
+|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
b/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
index 429d732530..8c4a555558 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
@@ -224,5 +224,5 @@ See xref:hop-tools/hop.adoc[hop],
xref:hop-run/index.adoc[hop run] and xref:hop-
In addition to the Hop Gui and configuration files, all aspects of and
operations on projects and environments can be managed through the Hop Conf
command line tool.
-include::../snippets/hop-tools/hop-conf/hop-conf-projects.adoc[]
+include::../snippets/hop-tools/hop-conf/hop-conf-projects.adoc[leveloffset=+1]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-azure-blob-storage.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-azure-blob-storage.adoc
index 888ebcd67b..ca9b62c581 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-azure-blob-storage.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-azure-blob-storage.adoc
@@ -19,9 +19,10 @@ under the License.
----
-aza, --azure-account=<account>
The account to use for the Azure VFS
- -azi, --azure-block-increment=<blockIncrement>
- The block increment size for new files on Azure,
- multiples of 512 only.
+ -aze, --azure-emulator=<emulatorUrl>
+ The URL of an Azure emulator (Azurite) to be used
+ for test and development goals instead of
+ official Azure services
-azk, --azure-key=<key>
The key to use for the Azure VFS
----
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-google-cloud-storage.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-google-cloud-storage.adoc
index 3dfc7bf7f0..982a4f2461 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-google-cloud-storage.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-cloud-google-cloud-storage.adoc
@@ -19,4 +19,7 @@ under the License.
----
-gck, --google-cloud-service-account-key-file=<serviceAccountKeyFile>
Configure the path to a Google Cloud service
account JSON key file
+ -gcrbs, --google-cloud-read-block-size=<readBlockSize>
+ Configure the bytes per request when reading a file
+ in blocks, like 8MB
----
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-projects.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-projects.adoc
index 78f082e1d6..99a23ee11c 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-projects.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-tools/hop-conf/hop-conf-projects.adoc
@@ -15,93 +15,112 @@ specific language governing permissions and limitations
under the License.
////
+The examples use `hop-conf.sh` on Linux and macOS.
+On Windows, run `hop-conf.bat` with the same options, and put a command on a
single line or end its lines with `^` instead of `\`.
-=== Configuration on the command line
+== Projects
-The ```hop-conf``` script offers many options to edit environment definitions.
+A project is registered in `hop-config.json` with its name and its home folder.
+Its settings and variables are stored in `project-config.json`, in the home
folder.
+=== Create a project
-==== Creating an environment
+The home folder has to exist: when it doesn't, hop-conf stops with an error
that the home folder does not exist.
-[tabs]
-====
-Windows::
-+
---
[source,bash]
----
-hop-conf.bat --environment-create \
- --environment hop2 \
- --environment-project hop2
- --environment-purpose=Development \
-
--environment-config-files="C:\<YOUR_ENV_FILE_PATH>\env-variables.json"
+mkdir -p /data/projects/sales
+./hop-conf.sh --project-create --project=sales
--project-home=/data/projects/sales
----
-Expected output:
+Output:
[source,shell]
----
-C:\<YOUR_PATH>\hop>echo off
-===[Environment Settings - hop-conf.bat]===================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf
--DHOP_AUTO_CREATE_CONFIG=Y
-Command to start Hop will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\*
--Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit
-DHOP_PLATFORM_OS=Windows
--DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
org.apache.hop.config.HopConfig --environment-create
---environment hop2 --environment-project hop2 --environment-purpose Development
---environment-config-files "C:\<YOUR_ENV_FILE_PATH\env-variables.json"
-===[Starting
HopConfig]=========================================================
-Creating environment 'hop2'
-Environment 'hop2' was created in Hop configuration file
C:\<YOUR_PATH>\hop\config\hop-config.json
-Warning: referenced project 'hop2' doesn\'t exist
-Found existing environment configuration file:
C:\<YOUR_ENV_FILE_PATH>\variables.json
-Purpose: Development
-Project name: hop2
-Config file: C:\<YOUR_ENV_FILE_PATH>\env-variables.json
+Creating project 'sales'
+Project 'sales' was created for home folder : /data/projects/sales
+Configuration file for project 'sales' was saved to :
file:/data/projects/sales/project-config.json
----
---
-Linux, macOS::
-+
---
+You can set more at the same time, for example a parent project to inherit
metadata and variables from (`--project-parent`), a description
(`--project-description`) or variables (`--project-variables`, see
<<hop-conf-project-variables,Project variables>>).
+
+When the home folder already has a `project-config.json`, `--project-create`
updates it with the options you pass and keeps its variables.
+Add `--project-keep-config-file` to leave the file as it is.
+
+=== Modify, list and delete projects
+
+* `--project-modify` changes the options you pass, for example `./hop-conf.sh
--project-modify --project=sales --project-description="Sales data warehouse"`.
+* `--projects-list` lists the projects with their home folder and
configuration file.
+* `--project-delete` removes a project from `hop-config.json`, for example
`./hop-conf.sh --project-delete --project=sales`. The home folder and its files
are not deleted.
+
+[[hop-conf-project-variables]]
+=== Project variables
+
+`--project-variables` adds variables to the project, or changes the value of a
variable that already exists.
+Variables with other names are kept.
+
[source,bash]
----
-$ sh hop-conf.sh \
- --environment-create \
- --environment hop2 \
- --environment-project hop2 \
+./hop-conf.sh --project-modify --project=sales
--project-variables=INPUT_FOLDER=/data/in,OUTPUT_FOLDER=/data/out
+----
+
+Separate variables with commas.
+To use spaces or commas in a value, wrap the value in double quotes and keep
them from the shell:
+
+[options="header", cols="3,2"]
+|===
+|Option, as typed in a shell|Variables
+|`--project-variables=key1=value1,key2=value2`|`key1` = `value1`, `key2` =
`value2`
+|`--project-variables=key1="This value contains spaces",key2=value2`|`key1` =
`This value contains spaces`, `key2` = `value2`
+|`--project-variables='key1="value1,value2"'`|`key1` = `value1,value2`
+|`--project-variables=key1=\"value1,value2\"`|`key1` = `value1,value2`
+|`--project-variables=key1="\"\"String with spaces\"\""`|`key1` = `"String
with spaces"`
+|===
+
+== Environments
+
+An environment belongs to one project.
+It adds configuration files with variables to that project, usually one
environment for every stage of its lifecycle: development, testing, production.
+The project holds the code, and the environment the configuration, such as
server names and folders.
+See xref:projects/projects-environments.adoc[Projects and Environments] for
how they work together.
+
+=== Create an environment
+
+[source,bash]
+----
+./hop-conf.sh --environment-create \
+ --environment=sales-dev \
+ --environment-project=sales \
--environment-purpose=Development \
- --environment-config-files=<YOUR_ENV_FILE_PATH>/env-variables.json
+ --environment-config-files=/data/environments/sales-dev/variables.json
----
-Expected output:
+Output, shortened:
[source,shell]
----
-Creating environment 'hop2'
-Environment 'hop2' was created in Hop configuration file
<YOUR_PATH>/hop/config/hop-config.json
-Warning: referenced project 'hop2' doesn't exist
-Found existing environment configuration file:
<YOUR_ENV_FILE_PATH>/env-variables.json
- hop2
+Creating environment 'sales-dev'
+Environment 'sales-dev' was created in Hop configuration file
/opt/hop/config/hop-config.json
+Created empty environment configuration file :
/data/environments/sales-dev/variables.json
+ sales-dev
Purpose: Development
- Project name: hop2
- Config file: <YOUR_ENV_FILE_PATH>/env-variables.json
+ Project name: sales
+ Config file: /data/environments/sales-dev/variables.json
----
---
-====
-As you can see from the log, an empty file was created to set variables in:
+A configuration file that doesn't exist yet is created, empty.
+Before creating it, hop-conf logs an error that the file doesn't exist to read
variables from.
[source,json]
----
{ }
----
-==== Adding configuration files from a directory
+Add variables to it as described in
<<hop-conf-configuration-files,Configuration files>>.
+
+=== Add configuration files from a directory
-`--environment-config-files` still lists files one by one.
+`--environment-config-files` lists files one by one.
To add every matching file in a folder, pass the folder and a regular
expression.
The expression is matched against the file name, the same way as in the JSON
Input and Get File Names transforms.
`*.json` is not a valid expression; use `.*\.json`.
@@ -113,142 +132,67 @@ When either option is set, that selection replaces the
environment's configurati
[source,bash]
----
-$ sh hop-conf.sh \
- --environment-create \
- --environment test-file \
- --environment-project test-file \
+./hop-conf.sh --environment-create \
+ --environment=sales-test \
+ --environment-project=sales \
--environment-purpose=Testing \
- --environment-config-file-directory="<YOUR_ENV_CONFIG_PATH>/config" \
+ --environment-config-file-directory="/data/environments/sales-test" \
--environment-config-file-wildcard=".*\.json" \
--environment-config-file-exclude-wildcard=".*secret.*" \
--environment-config-include-subfolders
----
-==== Setting variables in an environment
+=== Modify, list and delete environments
-This command adds a variable to the environment configuration file:
+* `--environment-modify` changes the options you pass, for example
`./hop-conf.sh --environment-modify --environment=sales-dev
--environment-purpose=Testing`.
+* `--environments-list` lists the environments with their purpose, project and
configuration files.
+* `--environment-delete` removes an environment from `hop-config.json`, for
example `./hop-conf.sh --environment-delete --environment=sales-dev`. Its
configuration files are not deleted.
- +
-
-[tabs]
-====
-Windows::
-+
---
-[source, shell]
-----
-hop-conf.bat --config-file "C:\<YOUR_ENV_FILE_PATH>\env-variables.json"
--config-file-set-variables "DB_HOSTNAME=localhost,DB_PASSWORD=abcd"
-----
+[[hop-conf-configuration-files]]
+== Configuration files
-Expected output:
+The configuration files of an environment hold its variables.
+`--config-file` selects the file, `--config-file-set-variables` sets the value
of variables, and `--config-file-describe-variables` their description:
-[source,shell]
-----
-C:\<YOUR_PATH\hop>echo off
-===[Environment Settings - hop-conf.bat]===================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
-Command to start Hop will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\*
--Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit
-DHOP_PLATFORM_OS=Windows
--DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
org.apache.hop.config.HopConfig
---config-file "C:\<YOUR_ENV_FILE_PATH>\env-variables.json"
---config-file-set-variables "DB_HOSTNAME=localhost,DB_PASSWORD=abcd"
-===[Starting
HopConfig]=========================================================
-Configuration file 'C:\<YOUR_ENV_FILE_PATH>/env-variables.json' was modified.
-----
---
-
-Linux, macOS::
-+
---
[source,bash]
----
-./hop-conf.sh --config-file <YOUR_ENV_FILE_PATH>/env-variables.json
--config-file-set-variables DB_HOSTNAME=localhost,DB_PASSWORD=abcd
+./hop-conf.sh --config-file=/data/environments/sales-dev/variables.json \
+ --config-file-set-variables=DB_HOSTNAME=localhost,DB_PASSWORD=abcd
+./hop-conf.sh --config-file=/data/environments/sales-dev/variables.json \
+ --config-file-describe-variables="DB_HOSTNAME=The database server"
----
-Expected output:
-
-[source,shell]
-----
-Configuration file '<YOUR_ENV_FILE_PATH>/env-variables.json' was modified.
-----
---
-====
-
-
-If you look at the file `env-variables.json`, you'll see that the variables
were added:
+The file then contains:
[source,json]
----
{
- "variables" : [ {
- "name" : "DB_HOSTNAME",
- "value" : "localhost",
- "description" : ""
- }, {
- "name" : "DB_PASSWORD",
- "value" : "abcd",
- "description" : ""
- } ]
+ "variables" : [
+ {
+ "name" : "DB_HOSTNAME",
+ "value" : "localhost",
+ "description" : "The database server"
+ },
+ {
+ "name" : "DB_PASSWORD",
+ "value" : "abcd",
+ "description" : ""
+ }
+ ]
}
----
-Please note that you can add descriptions for the variables as well with the
`--describe-variable` option.
-Please run hop-conf without options to see all the possibilities.
-
-==== Deleting an environment
-
-The following example deletes an environment from the Hop configuration file:
-
- +
-
-[tabs]
-====
-Windows::
-+
---
-[source,shell]
-----
-hop-conf.bat -ed --environment hop2
-----
-
-Expected output:
-
-[source,shell]
-----
-C:\<YOUR_PATH>\hop>echo off
-===[Environment Settings - hop-conf.bat]===================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
-Command to start Hop will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\* -Djava.library.path=lib\core;lib\beam
-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
org.apache.hop.config.HopConfig -ed --environment hop2
-===[Starting
HopConfig]=========================================================
-Lifecycle environment 'hop2' was deleted from Hop configuration file
C:\<YOUR_PATH>\hop\config\hop-config.json
-----
-
---
+The variables of `hop-config.json` itself, listed as *System variables* in the
configuration perspective of Hop Gui, are set with `--set-variable` and
described with `--describe-variable`:
-Linux, macOS::
-+
---
-[source,shell]
-----
-./hop-conf.sh -ed --environment hop2
-----
-
-Expected output:
-
-[source,shell]
+[source,bash]
----
-Lifecycle environment 'hop2' was deleted from Hop configuration file
<YOUR_PATH>/hop/config/hop-config.json
+./hop-conf.sh --set-variable=COMPANY_NAME=Acme
+./hop-conf.sh --describe-variable="COMPANY_NAME=The company name used in
reports"
----
---
-====
-
+`--set-variable=MY_VARIABLE=` sets a variable to an empty value. The variable
stays in the list.
-=== Projects Plugin configuration
+== Projects plugin configuration
There are various options to configure the behavior of the ```Projects```
plugin itself.
In Hop configuration file ```hop-config.json``` we can find the following
options:
@@ -293,4 +237,3 @@ In Hop configuration file ```hop-config.json``` we can find
the following option
|```--standard-projects-folder```
|===
-
