This is an automated email from the ASF dual-hosted git repository.
mattcasters pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 640d395fae Issue #8509 : Add an optional project id for shared
execution information (#8559)
640d395fae is described below
commit 640d395fae91c5fccc100d1516ca4360703fc6aa
Author: Matt Casters <[email protected]>
AuthorDate: Mon Sep 28 12:14:54 2026 +0200
Issue #8509 : Add an optional project id for shared execution information
(#8559)
Store projectId in project-config.json and copy it onto execution
documents only when it is set. An empty id keeps 2.19.0 behavior:
no filtering, and the property is omitted from JSON so 2.19 can still
read those documents. Caching database, Elastic, and OpenSearch can
filter on the id once the column or keyword mapping is in place.
---
.../execution-information-location.adoc | 19 ++-
.../modules/ROOT/pages/projects/advanced.adoc | 3 +
.../ROOT/pages/projects/projects-environments.adoc | 1 +
.../java/org/apache/hop/execution/Execution.java | 14 ++
.../org/apache/hop/execution/ExecutionBuilder.java | 25 +++-
.../caching/BaseCachingExecutionInfoLocation.java | 45 ++++++
.../apache/hop/execution/caching/CacheEntry.java | 43 ++++++
.../caching/CachingFileExecutionInfoLocation.java | 5 +
.../execution/ExecutionBuilderProjectIdTest.java | 55 +++++++
.../execution/ProjectIdJsonCompatibilityTest.java | 84 +++++++++++
.../CachingFileExecutionInfoLocationTest.java | 47 ++++++
.../CachingDatabaseExecutionInfoLocation.java | 166 +++++++++++++++++----
.../CachingDatabaseExecutionInfoLocationTest.java | 98 +++++++++++-
.../EnvironmentVariablesImportHelper.java | 1 +
.../apache/hop/projects/gui/ProjectsGuiPlugin.java | 1 +
.../project/ManageProjectsOptionPlugin.java | 12 ++
.../org/apache/hop/projects/project/Project.java | 12 ++
.../apache/hop/projects/project/ProjectDialog.java | 53 +++++++
.../org/apache/hop/projects/util/Defaults.java | 8 +
.../hop/projects/util/PathVariableReplacer.java | 1 +
...ojectsVariablesControlSpaceSortOrderPrefix.java | 1 +
.../project/messages/messages_en_US.properties | 2 +
.../apache/hop/projects/project/ProjectTest.java | 79 ++++++++++
.../projects/util/PathVariableReplacerTest.java | 1 +
.../elastic/ElasticExecutionInfoLocation.java | 149 ++++++++++++------
.../ElasticExecutionInfoLocationProjectIdTest.java | 71 +++++++++
.../OpenSearchExecutionInfoLocation.java | 137 +++++++++++------
...enSearchExecutionInfoLocationProjectIdTest.java | 53 +++++++
28 files changed, 1057 insertions(+), 129 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
index 13e1367214..acee2bb32f 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
@@ -119,10 +119,23 @@ Options:
Table layout (logical):
* Primary key `id` (log channel / execution id)
-* Filter columns: `name`, `execution_type`, `parent_id`, `registration_date`,
`execution_start_date`, `execution_end_date`, `failed`, `status_description`,
`duration_ms`
-* `json`: full `CacheEntry` JSON (execution, state, children, samples, logging
text, …)
+* Filter columns: `name`, `execution_type`, `parent_id`, `registration_date`,
`execution_start_date`, `execution_end_date`, `failed`, `status_description`,
`duration_ms`, `project_id`
+* `json`: `CacheEntry` JSON written on insert (execution, children, samples,
logging text, …)
+* `state_json`: later execution state, child states, and samples for a local
run
-Indexes are recommended on the filter columns used by the GUI (start date,
name, type, failed, parent id, status). Use the **Show table and indexes DDL**
button to obtain statements for your database dialect.
+`project_id` is optional and nullable. It is filled from the active project's
id (`HOP_PROJECT_ID`, stored as `projectId` in `project-config.json`). When
that variable is empty, Hop does not filter this table, which is the same
behavior as 2.19. A table created before the column existed keeps working:
writes use the old column list and no project filter is applied until you add
the column. The **Show table and indexes DDL** button prints `CREATE TABLE` for
a new table (`project_id` and ` [...]
+
+`state_json` lets a long run update execution state without rewriting `json`.
A table that does not have that column yet keeps working: every save rewrites
`json`, and Hop logs an `ALTER TABLE` statement at startup. Hop does not change
the table by itself.
+
+When a project id is set and the column exists, the list shows rows for that
id and rows whose `project_id` is null or empty. Those untagged rows are
executions from 2.19 or from a project that still has no id. Rows tagged with a
different id are hidden, including from "last execution" lookups.
+
+Indexes are recommended on the filter columns used by the GUI (start date,
name, type, failed, parent id, status, project id). Use the **Show table and
indexes DDL** button to obtain statements for your database dialect.
+
+=== Project id on Elastic and OpenSearch
+
+Elastic and OpenSearch documents can carry the same value as a root
`projectId` keyword. The create-index action adds that field. If the index
already exists, the same action adds the field to the mapping and does not
recreate the index. Listing uses that keyword when the active project has an
id, and lists every document when it does not. Documents written without a
project id stay visible to every project.
+
+The field is omitted from the JSON document when the project has no id. A 2.19
runtime reads those documents. A document that includes `projectId` is not
readable by 2.19, because that version rejects unknown JSON properties in
execution documents.
=== Cleanup
diff --git a/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
b/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
index ad02208f1d..f63143b149 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/projects/advanced.adoc
@@ -106,6 +106,8 @@ Let's take a look at it:
}
----
+`projectId` is optional. When it is absent, Hop does not filter execution
information by project. A new project suggests the project name; an existing
2.19 configuration file is left unchanged.
+
=====
+
@@ -123,6 +125,7 @@ Hop also sets a few built-in project variables when a
project is activated:
|Variable|Description
|`PROJECT_HOME`|Home folder of the active project
|`HOP_PROJECT_NAME`|Name of the active project
+|`HOP_PROJECT_ID`|Project id from `project-config.json`. Empty when the
project has no id, which leaves execution information unfiltered
|`PARENT_PROJECT_HOME`|Home folder of the immediate parent project (empty if
none)
|`PARENT_PROJECT_NAME`|Name of the immediate parent project (empty if none)
|===
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/projects/projects-environments.adoc
b/docs/hop-user-manual/modules/ROOT/pages/projects/projects-environments.adoc
index 9af778be4a..cbe0e847a5 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/projects/projects-environments.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/projects/projects-environments.adoc
@@ -45,6 +45,7 @@ image:hop-gui/environment/create-project-dialog.png[Project
Properties Basic tab
|===
|Property|Description|Variables Supported|Mandatory|Default
|Name|The project name|Yes|No|
+|Project id|Optional id stored in `project-config.json` and copied onto
execution information. A shared execution database or search index can then be
filtered to this project. Leave empty to keep the 2.19 behavior: no filtering.
The project name is a readable value; a UUID also works. Not inherited from the
parent project.|No|No|empty (the project name is suggested for a new project)
|Home folder|The folder where the project is located. Paths starting with `~`
(e.g. `~/my-project` or `~\my-project` on Windows) automatically resolve to the
user's home directory (`+${user.home}+`).|Yes|No|
|This project is read only|When checked, Hop does not write
`project-config.json` (archives, HTTP, and similar)|No|No|unchecked
|Configuration file (relative path)|The project's configuration json, relative
to the home folder.|Yes|Yes|`project-config.json`
diff --git a/engine/src/main/java/org/apache/hop/execution/Execution.java
b/engine/src/main/java/org/apache/hop/execution/Execution.java
index 4cb53f4400..9fc41b59ef 100644
--- a/engine/src/main/java/org/apache/hop/execution/Execution.java
+++ b/engine/src/main/java/org/apache/hop/execution/Execution.java
@@ -19,6 +19,7 @@
package org.apache.hop.execution;
import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
@@ -34,6 +35,12 @@ import org.apache.hop.core.logging.LogLevel;
@Getter
public class Execution {
+ /**
+ * Set when a project is enabled. Empty means execution information is not
filtered by project.
+ * The projects plugin writes the same name ({@code HOP_PROJECT_ID}).
+ */
+ public static final String VARIABLE_HOP_PROJECT_ID = "HOP_PROJECT_ID";
+
@SuppressWarnings("java:S115")
public enum EnvironmentDetailType {
ContainerId,
@@ -50,6 +57,13 @@ public class Execution {
/** The name of the pipeline or workflow execution */
private String name;
+ /**
+ * Project that registered this execution. Omitted from JSON when empty so a
2.19 reader, which
+ * rejects unknown properties, can still open the document.
+ */
+ @JsonInclude(JsonInclude.Include.NON_EMPTY)
+ private String projectId;
+
/** The filename that is executing */
private String filename;
diff --git
a/engine/src/main/java/org/apache/hop/execution/ExecutionBuilder.java
b/engine/src/main/java/org/apache/hop/execution/ExecutionBuilder.java
index b754264931..0b00ea8422 100644
--- a/engine/src/main/java/org/apache/hop/execution/ExecutionBuilder.java
+++ b/engine/src/main/java/org/apache/hop/execution/ExecutionBuilder.java
@@ -23,6 +23,7 @@ import java.net.UnknownHostException;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.logging.LogLevel;
import org.apache.hop.core.metadata.SerializableMetadataProvider;
@@ -54,6 +55,7 @@ public final class ExecutionBuilder {
public Date registrationDate;
public Date executionStartDate;
public String copyNr;
+ public String projectId;
private ExecutionBuilder() {
this.variableValues = new HashMap<>();
@@ -86,6 +88,7 @@ public final class ExecutionBuilder {
builder.getParameterInformation(pipeline);
builder.updateRuntimeInformation();
+ builder.withProjectIdFromVariables(pipeline);
return builder;
}
@@ -111,6 +114,7 @@ public final class ExecutionBuilder {
builder.getParameterInformation(workflow);
builder.updateRuntimeInformation();
+ builder.withProjectIdFromVariables(workflow);
return builder;
}
@@ -128,7 +132,8 @@ public final class ExecutionBuilder {
.withMetadataJson(null)
.withRunConfigurationName(null)
.withLogLevel(transform.getLogLevel())
- .withExecutionStartDate(pipeline.getExecutionStartDate());
+ .withExecutionStartDate(pipeline.getExecutionStartDate())
+ .withProjectIdFromVariables(pipeline);
}
public static ExecutionBuilder fromAction(
@@ -146,7 +151,8 @@ public final class ExecutionBuilder {
.withMetadataJson(null)
.withRunConfigurationName(null)
.withLogLevel(action.getLogChannel().getLogLevel())
- .withExecutionStartDate(startDate);
+ .withExecutionStartDate(startDate)
+ .withProjectIdFromVariables(workflow);
}
@Deprecated
@@ -266,6 +272,20 @@ public final class ExecutionBuilder {
return this;
}
+ /**
+ * Copy {@link Execution#VARIABLE_HOP_PROJECT_ID} when it is set. An empty
value is left unset.
+ */
+ public ExecutionBuilder withProjectIdFromVariables(IVariables variables) {
+ if (variables == null) {
+ return this;
+ }
+ String value = variables.getVariable(Execution.VARIABLE_HOP_PROJECT_ID);
+ if (StringUtils.isNotEmpty(value)) {
+ this.projectId = value;
+ }
+ return this;
+ }
+
public Execution build() {
Execution executionRegistration = new Execution();
executionRegistration.setName(name);
@@ -283,6 +303,7 @@ public final class ExecutionBuilder {
executionRegistration.setEnvironmentDetails(environmentDetails);
executionRegistration.setRegistrationDate(registrationDate);
executionRegistration.setExecutionStartDate(executionStartDate);
+ executionRegistration.setProjectId(projectId);
return executionRegistration;
}
}
diff --git
a/engine/src/main/java/org/apache/hop/execution/caching/BaseCachingExecutionInfoLocation.java
b/engine/src/main/java/org/apache/hop/execution/caching/BaseCachingExecutionInfoLocation.java
index a4f0de413b..2bb3ddb0b0 100644
---
a/engine/src/main/java/org/apache/hop/execution/caching/BaseCachingExecutionInfoLocation.java
+++
b/engine/src/main/java/org/apache/hop/execution/caching/BaseCachingExecutionInfoLocation.java
@@ -132,6 +132,12 @@ public abstract class BaseCachingExecutionInfoLocation
implements IExecutionInfo
protected int maxAge;
protected int maxSize;
+ /**
+ * {@link Execution#VARIABLE_HOP_PROJECT_ID} captured at {@link
#initialize}. Empty means do not
+ * filter (2.19.0 behavior).
+ */
+ protected String activeProjectId = "";
+
protected BaseCachingExecutionInfoLocation() {
cache = new LinkedHashMap<>(16, 0.75f, true);
this.cacheTimer = null;
@@ -148,6 +154,7 @@ public abstract class BaseCachingExecutionInfoLocation
implements IExecutionInfo
this.delay = location.delay;
this.maxAge = location.maxAge;
this.maxSize = location.maxSize;
+ this.activeProjectId = location.activeProjectId;
}
public abstract BaseCachingExecutionInfoLocation clone();
@@ -167,6 +174,10 @@ public abstract class BaseCachingExecutionInfoLocation
implements IExecutionInfo
throws HopException {
this.variables = variables;
this.metadataProvider = metadataProvider;
+ this.activeProjectId =
+ variables == null
+ ? ""
+ :
Const.NVL(variables.getVariable(Execution.VARIABLE_HOP_PROJECT_ID), "");
// The default persistence delay is 1 minute
//
@@ -428,6 +439,9 @@ public abstract class BaseCachingExecutionInfoLocation
implements IExecutionInfo
}
entry.setExecution(execution);
entry.setName(execution.getName());
+ if (StringUtils.isNotEmpty(execution.getProjectId())) {
+ entry.setProjectId(execution.getProjectId());
+ }
entry.setDirty(true);
entry.setLastWritten(null);
@@ -719,8 +733,36 @@ public abstract class BaseCachingExecutionInfoLocation
implements IExecutionInfo
}
}
+ /**
+ * Active project id empty: every execution matches. Stored id empty: legacy
rows match. Otherwise
+ * the ids must be equal.
+ */
+ protected boolean matchesActiveProject(String storedProjectId) {
+ if (StringUtils.isEmpty(activeProjectId)) {
+ return true;
+ }
+ if (StringUtils.isEmpty(storedProjectId)) {
+ return true;
+ }
+ return activeProjectId.equals(storedProjectId);
+ }
+
+ protected boolean matchesActiveProject(CacheEntry cacheEntry) {
+ if (cacheEntry == null) {
+ return false;
+ }
+ String storedProjectId = cacheEntry.getProjectId();
+ if (StringUtils.isEmpty(storedProjectId) && cacheEntry.getExecution() !=
null) {
+ storedProjectId = cacheEntry.getExecution().getProjectId();
+ }
+ return matchesActiveProject(storedProjectId);
+ }
+
protected synchronized void getExecutionIdsFromCache(Set<DatedId> ids,
boolean includeChildren) {
for (CacheEntry cacheEntry : cache.values()) {
+ if (!matchesActiveProject(cacheEntry)) {
+ continue;
+ }
ids.add(new DatedId(cacheEntry.getId(),
cacheEntry.getExecution().getRegistrationDate()));
if (includeChildren) {
addChildIds(cacheEntry, ids);
@@ -731,6 +773,9 @@ public abstract class BaseCachingExecutionInfoLocation
implements IExecutionInfo
protected synchronized void getExecutionIdsFromCache(
Set<DatedId> ids, IExecutionSelector selector) {
for (CacheEntry cacheEntry : cache.values()) {
+ if (!matchesActiveProject(cacheEntry)) {
+ continue;
+ }
if (selector.isSelected(cacheEntry.getExecution())
&& selector.isSelected(cacheEntry.getExecutionState())) {
ids.add(new DatedId(cacheEntry.getId(),
cacheEntry.getExecution().getRegistrationDate()));
diff --git
a/engine/src/main/java/org/apache/hop/execution/caching/CacheEntry.java
b/engine/src/main/java/org/apache/hop/execution/caching/CacheEntry.java
index c264fa3f56..697e84c925 100644
--- a/engine/src/main/java/org/apache/hop/execution/caching/CacheEntry.java
+++ b/engine/src/main/java/org/apache/hop/execution/caching/CacheEntry.java
@@ -19,6 +19,7 @@
package org.apache.hop.execution.caching;
import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import java.io.OutputStream;
@@ -52,6 +53,13 @@ public class CacheEntry {
// The name of the pipeline of workflow
private String name;
+ /**
+ * Root copy of {@link Execution#getProjectId()}. Elastic and OpenSearch
filter on this field.
+ * Omitted when empty so 2.19 readers can still open the document.
+ */
+ @JsonInclude(JsonInclude.Include.NON_EMPTY)
+ private String projectId;
+
// The creation date of this entry
//
private Date creationDate;
@@ -101,6 +109,41 @@ public class CacheEntry {
dirty = true;
}
+ /**
+ * Copy a non-empty project id between this entry and its execution before
writing JSON. Does not
+ * replace a stored id with an empty one; callers that loaded a previous
document should call
+ * {@link #keepStoredProjectId} first.
+ */
+ public void prepareForPersist() {
+ if (execution == null) {
+ return;
+ }
+ if (StringUtils.isEmpty(projectId)) {
+ projectId = StringUtils.trimToNull(execution.getProjectId());
+ } else if (StringUtils.isEmpty(execution.getProjectId())) {
+ execution.setProjectId(projectId);
+ }
+ }
+
+ /**
+ * Keep a project id already stored on disk or in the database when this
in-memory entry has none.
+ * A later update from a process that has no {@code HOP_PROJECT_ID} must not
wipe it.
+ */
+ public void keepStoredProjectId(CacheEntry stored) {
+ if (stored == null) {
+ return;
+ }
+ if (StringUtils.isEmpty(projectId) &&
StringUtils.isNotEmpty(stored.getProjectId())) {
+ projectId = stored.getProjectId();
+ }
+ if (execution != null
+ && stored.getExecution() != null
+ && StringUtils.isEmpty(execution.getProjectId())
+ && StringUtils.isNotEmpty(stored.getExecution().getProjectId())) {
+ execution.setProjectId(stored.getExecution().getProjectId());
+ }
+ }
+
/**
* Write this cache entry to a file in a folder
*
diff --git
a/engine/src/main/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocation.java
b/engine/src/main/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocation.java
index f02de3d70b..bf26cef081 100644
---
a/engine/src/main/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocation.java
+++
b/engine/src/main/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocation.java
@@ -147,6 +147,7 @@ public class CachingFileExecutionInfoLocation extends
BaseCachingExecutionInfoLo
// Merge child maps from the on-disk file so samples/children written by
other processes
// are not wiped when this process flushes parent metrics/state.
mergeChildrenFromDisk(cacheEntry);
+ cacheEntry.prepareForPersist();
// Before writing to disk, we calculate some summaries for convenience
of other tools.
cacheEntry.calculateSummary();
cacheEntry.writeToDisk(actualRootFolder, variables);
@@ -174,6 +175,7 @@ public class CachingFileExecutionInfoLocation extends
BaseCachingExecutionInfoLo
mergeMap(onDisk.getChildExecutions(), cacheEntry.getChildExecutions());
mergeMap(onDisk.getChildExecutionStates(),
cacheEntry.getChildExecutionStates());
mergeMap(onDisk.getChildExecutionData(),
cacheEntry.getChildExecutionData());
+ cacheEntry.keepStoredProjectId(onDisk);
} catch (Exception e) {
// Best-effort: still write our in-memory view if merge fails
LogChannel.GENERAL.logError(
@@ -252,6 +254,9 @@ public class CachingFileExecutionInfoLocation extends
BaseCachingExecutionInfoLo
// Not much loaded from disk or cache
continue;
}
+ if (!matchesActiveProject(entry)) {
+ continue;
+ }
if (!activeSelector.isSelected(entry.getExecution())) {
continue;
}
diff --git
a/engine/src/test/java/org/apache/hop/execution/ExecutionBuilderProjectIdTest.java
b/engine/src/test/java/org/apache/hop/execution/ExecutionBuilderProjectIdTest.java
new file mode 100644
index 0000000000..bc395c5a87
--- /dev/null
+++
b/engine/src/test/java/org/apache/hop/execution/ExecutionBuilderProjectIdTest.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.execution;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import org.apache.hop.core.variables.Variables;
+import org.junit.jupiter.api.Test;
+
+class ExecutionBuilderProjectIdTest {
+
+ @Test
+ void copiesProjectIdVariable() {
+ Variables variables = new Variables();
+ variables.setVariable(Execution.VARIABLE_HOP_PROJECT_ID, "sales");
+
+ Execution execution =
+ ExecutionBuilder.of()
+ .withName("pipeline")
+ .withId("id")
+ .withProjectIdFromVariables(variables)
+ .build();
+
+ assertEquals("sales", execution.getProjectId());
+ }
+
+ @Test
+ void leavesProjectIdUnsetWhenVariableIsMissing() {
+ Execution execution =
+ ExecutionBuilder.of()
+ .withName("pipeline")
+ .withId("id")
+ .withProjectIdFromVariables(new Variables())
+ .withProjectIdFromVariables(null)
+ .build();
+
+ assertNull(execution.getProjectId());
+ }
+}
diff --git
a/engine/src/test/java/org/apache/hop/execution/ProjectIdJsonCompatibilityTest.java
b/engine/src/test/java/org/apache/hop/execution/ProjectIdJsonCompatibilityTest.java
new file mode 100644
index 0000000000..2f885e567c
--- /dev/null
+++
b/engine/src/test/java/org/apache/hop/execution/ProjectIdJsonCompatibilityTest.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.execution;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hop.execution.caching.CacheEntry;
+import org.junit.jupiter.api.Test;
+
+class ProjectIdJsonCompatibilityTest {
+
+ /** 2.19 execution locations read and write with a strict ObjectMapper. */
+ private final ObjectMapper strictMapper = new ObjectMapper();
+
+ @Test
+ void omitsProjectIdWhenEmpty() throws Exception {
+ Execution execution = new Execution();
+ execution.setName("pipeline");
+ String json = strictMapper.writeValueAsString(execution);
+ assertFalse(json.contains("projectId"));
+ assertNull(strictMapper.readValue(json, Execution.class).getProjectId());
+
+ CacheEntry entry = new CacheEntry();
+ entry.setId("id");
+ entry.setName("pipeline");
+ entry.setExecution(execution);
+ String entryJson = strictMapper.writeValueAsString(entry);
+ assertFalse(entryJson.contains("projectId"));
+ assertNull(strictMapper.readValue(entryJson,
CacheEntry.class).getProjectId());
+ }
+
+ @Test
+ void writesProjectIdWhenSet() throws Exception {
+ Execution execution = new Execution();
+ execution.setName("pipeline");
+ execution.setProjectId("sales");
+ String json = strictMapper.writeValueAsString(execution);
+ assertTrue(json.contains("\"projectId\":\"sales\""));
+
+ CacheEntry entry = new CacheEntry();
+ entry.setId("id");
+ entry.setName("pipeline");
+ entry.setExecution(execution);
+ entry.setProjectId("sales");
+ String entryJson = strictMapper.writeValueAsString(entry);
+ assertTrue(entryJson.contains("\"projectId\":\"sales\""));
+ CacheEntry read = strictMapper.readValue(entryJson, CacheEntry.class);
+ assertEquals("sales", read.getProjectId());
+ assertEquals("sales", read.getExecution().getProjectId());
+ }
+
+ @Test
+ void strictMapperWithoutThePropertyRejectsATaggedDocument() {
+ assertThrows(
+ JsonMappingException.class,
+ () -> strictMapper.readValue("{\"projectId\":\"sales\"}",
UntaggedExecution.class));
+ }
+
+ /** Stand-in for the 2.19 Execution class, which has no projectId property.
*/
+ static class UntaggedExecution {
+ public String name;
+ }
+}
diff --git
a/engine/src/test/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocationTest.java
b/engine/src/test/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocationTest.java
index 568536a7a0..4bd6f68be6 100644
---
a/engine/src/test/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocationTest.java
+++
b/engine/src/test/java/org/apache/hop/execution/caching/CachingFileExecutionInfoLocationTest.java
@@ -117,6 +117,53 @@ class CachingFileExecutionInfoLocationTest {
}
}
+ @Test
+ void getExecutionIdsFiltersByProjectIdAndKeepsLegacyRows() throws Exception {
+ Path root = tempDir.resolve("project-id");
+ Variables variables = new Variables();
+ variables.setVariable(Execution.VARIABLE_HOP_PROJECT_ID, "sales");
+
+ CachingFileExecutionInfoLocation location = new
CachingFileExecutionInfoLocation();
+ location.setRootFolder(root.toAbsolutePath().toString());
+ location.initialize(variables, null);
+ try {
+ location.registerExecution(execution("sales-run", "sales"));
+ location.registerExecution(execution("finance-run", "finance"));
+ location.registerExecution(execution("legacy-run", null));
+ location.clearCaches();
+
+ List<String> filtered = location.getExecutionIds(false, 20);
+ assertTrue(filtered.contains("sales-run"));
+ assertTrue(filtered.contains("legacy-run"));
+ assertFalse(filtered.contains("finance-run"));
+ } finally {
+ location.close();
+ }
+
+ CachingFileExecutionInfoLocation unfiltered = new
CachingFileExecutionInfoLocation();
+ unfiltered.setRootFolder(root.toAbsolutePath().toString());
+ unfiltered.initialize(new Variables(), null);
+ try {
+ List<String> all = unfiltered.getExecutionIds(false, 20);
+ assertTrue(all.contains("sales-run"));
+ assertTrue(all.contains("legacy-run"));
+ assertTrue(all.contains("finance-run"));
+ } finally {
+ unfiltered.close();
+ }
+ }
+
+ private static Execution execution(String id, String projectId) {
+ Execution execution = new Execution();
+ execution.setId(id);
+ execution.setName(id);
+ execution.setExecutionType(ExecutionType.Pipeline);
+ execution.setExecutionStartDate(new Date());
+ execution.setRegistrationDate(new Date());
+ execution.setProjectId(projectId);
+ return execution;
+ }
+
@Test
void testInitializeCreateFolderFalse() throws Exception {
Path targetDir = tempDir.resolve(UUID.randomUUID().toString());
diff --git
a/plugins/misc/execution-database/src/main/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocation.java
b/plugins/misc/execution-database/src/main/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocation.java
index d48c0c0e0b..5135e42601 100644
---
a/plugins/misc/execution-database/src/main/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocation.java
+++
b/plugins/misc/execution-database/src/main/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocation.java
@@ -100,6 +100,7 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
public static final String COL_FAILED = "failed";
public static final String COL_STATUS_DESCRIPTION = "status_description";
public static final String COL_DURATION_MS = "duration_ms";
+ public static final String COL_PROJECT_ID = "project_id";
public static final String COL_JSON = "json";
/**
@@ -108,6 +109,13 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
*/
public static final String COL_STATE_JSON = "state_json";
+ /**
+ * Separates CREATE TABLE (new installs, includes {@code project_id} and
{@code state_json}) from
+ * the ALTER statements that add {@code project_id} to an older table. Run
one block, not both.
+ */
+ public static final String DDL_EXISTING_TABLE_MARKER =
+ "-- Existing table: run the statements below instead of CREATE TABLE.";
+
@GuiWidgetElement(
id = "connectionName",
order = "010",
@@ -158,6 +166,9 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
protected String actualSchemaName;
protected String actualTableName;
+ /** False when the table is missing or still has the 2.19.0 column list. */
+ protected boolean projectIdColumnPresent;
+
/** False only when an existing table could not grow the state column. New
DDL includes it. */
private boolean stateJsonColumnAvailable = true;
@@ -240,6 +251,9 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
throw new HopException(
"Error starting the caching database execution information
location", e);
}
+ synchronized (dbLock) {
+ detectProjectIdColumn();
+ }
LogChannel.GENERAL.logBasic(
"Caching database execution info location ready: connection="
+ Const.NVL(actualConnectionName, databaseMeta.getName())
@@ -491,6 +505,7 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
mergeMap(existing.getChildExecutions(), cacheEntry.getChildExecutions());
mergeMap(existing.getChildExecutionStates(),
cacheEntry.getChildExecutionStates());
mergeMap(existing.getChildExecutionData(),
cacheEntry.getChildExecutionData());
+ cacheEntry.keepStoredProjectId(existing);
} catch (Exception e) {
LogChannel.GENERAL.logError(
"Unable to merge on-database cache entry before persist (non-fatal):
" + e.getMessage());
@@ -520,6 +535,7 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
}
private void writeFullDocument(CacheEntry cacheEntry) throws HopException {
+ cacheEntry.prepareForPersist();
String json = serializeCacheEntry(cacheEntry);
IRowMeta rowMeta = createDataRowMeta();
// A full write is the source of truth, so drop any older state overlay.
@@ -595,6 +611,25 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
return fields;
}
+ /**
+ * An empty project id must not wipe a value written earlier. Light updates
leave the column
+ * alone; only a full write sets it, and only when this entry has an id.
+ */
+ private String[] fullUpdateFields(IRowMeta rowMeta, Object[] data) {
+ String[] fields = fullUpdateFields();
+ int projectIndex = rowMeta.indexOfValue(COL_PROJECT_ID);
+ if (projectIndex < 0
+ || projectIndex >= data.length
+ || data[projectIndex] == null
+ || StringUtils.isEmpty(data[projectIndex].toString())) {
+ return fields;
+ }
+ String[] withProject = new String[fields.length + 1];
+ System.arraycopy(fields, 0, withProject, 0, fields.length);
+ withProject[fields.length] = COL_PROJECT_ID;
+ return withProject;
+ }
+
/** prepareUpdate binds SET fields first, then the WHERE value. */
private void executeUpdate(String[] setFields, Object[] data) throws
HopException {
String[] codes = new String[] {COL_ID};
@@ -637,7 +672,7 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
private void upsertCacheEntry(IRowMeta rowMeta, Object[] data) throws
HopException {
String id = (String) data[0];
if (rowExists(id)) {
- executeUpdate(fullUpdateFields(), data);
+ executeUpdate(fullUpdateFields(rowMeta, data), data);
} else {
database.insertRow(actualSchemaName, actualTableName, rowMeta, data);
}
@@ -786,6 +821,7 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
private void appendSelectorFilters(
IExecutionSelector selector, List<String> where, IRowMeta paramMeta,
List<Object> params) {
+ appendProjectIdFilter(where, paramMeta, params);
if (selector == IExecutionSelector.ALL) {
return;
}
@@ -856,7 +892,36 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
}
}
+ /**
+ * When the column exists and this Hop process has a project id, keep rows
for that id and rows
+ * that have none (2.19.0 and projects that have not set an id).
+ */
+ private void appendProjectIdFilter(List<String> where, IRowMeta paramMeta,
List<Object> params) {
+ if (!projectIdColumnPresent || StringUtils.isEmpty(getActiveProjectId())) {
+ return;
+ }
+ where.add(
+ "("
+ + databaseMeta.quoteField(COL_PROJECT_ID)
+ + " = ? OR "
+ + databaseMeta.quoteField(COL_PROJECT_ID)
+ + " IS NULL OR "
+ + databaseMeta.quoteField(COL_PROJECT_ID)
+ + " = '')");
+ paramMeta.addValueMeta(new ValueMetaString(COL_PROJECT_ID, 256, -1));
+ params.add(getActiveProjectId());
+ }
+
protected IRowMeta createDataRowMeta() {
+ return createDataRowMeta(projectIdColumnPresent, stateJsonColumnAvailable);
+ }
+
+ /**
+ * New-table DDL passes true for both flags. Runtime writes follow the
columns that exist. Order
+ * is the status columns, optional {@code project_id}, {@code json}, then
optional {@code
+ * state_json}.
+ */
+ private IRowMeta createDataRowMeta(boolean includeProjectId, boolean
includeStateJson) {
IRowMeta rowMeta = new RowMeta();
rowMeta.addValueMeta(new ValueMetaString(COL_ID, 100, -1));
rowMeta.addValueMeta(new ValueMetaString(COL_NAME, 1024, -1));
@@ -869,9 +934,12 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
rowMeta.addValueMeta(new ValueMetaString(COL_STATUS_DESCRIPTION, 128, -1));
// length 15 → BIGINT on most dialects (default Integer length maps to
tinyint on H2)
rowMeta.addValueMeta(new ValueMetaInteger(COL_DURATION_MS, 15, 0));
+ if (includeProjectId) {
+ rowMeta.addValueMeta(new ValueMetaString(COL_PROJECT_ID, 256, -1));
+ }
// CLOB for the CacheEntry JSON written on insert
rowMeta.addValueMeta(new ValueMetaString(COL_JSON,
DatabaseMeta.CLOB_LENGTH, -1));
- if (stateJsonColumnAvailable) {
+ if (includeStateJson) {
rowMeta.addValueMeta(new ValueMetaString(COL_STATE_JSON,
DatabaseMeta.CLOB_LENGTH, -1));
}
return rowMeta;
@@ -897,36 +965,30 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
String status = state != null ? state.getStatusDescription() : null;
Long durationMs =
cacheEntry.getSummary() != null ?
cacheEntry.getSummary().getDurationMs() : null;
+ String projectId = cacheEntry.getProjectId();
+ if (StringUtils.isEmpty(projectId) && execution != null) {
+ projectId = execution.getProjectId();
+ }
- if (!stateJsonColumnAvailable) {
- return new Object[] {
- cacheEntry.getId(),
- name,
- executionType,
- parentId,
- registrationDate,
- startDate,
- endDate,
- failed,
- status,
- durationMs,
- json
- };
- }
- return new Object[] {
- cacheEntry.getId(),
- name,
- executionType,
- parentId,
- registrationDate,
- startDate,
- endDate,
- failed,
- status,
- durationMs,
- json,
- stateJson
- };
+ List<Object> values = new ArrayList<>();
+ values.add(cacheEntry.getId());
+ values.add(name);
+ values.add(executionType);
+ values.add(parentId);
+ values.add(registrationDate);
+ values.add(startDate);
+ values.add(endDate);
+ values.add(failed);
+ values.add(status);
+ values.add(durationMs);
+ if (projectIdColumnPresent) {
+ values.add(StringUtils.trimToNull(projectId));
+ }
+ values.add(json);
+ if (stateJsonColumnAvailable) {
+ values.add(stateJson);
+ }
+ return values.toArray();
}
/**
@@ -952,6 +1014,37 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
target.setDirty(false);
}
+ /**
+ * The 2.19.0 table has no {@code project_id} column. Writes and filters
keep the old column list
+ * until the user runs the ALTER from {@link #buildDdl}.
+ */
+ private void detectProjectIdColumn() {
+ projectIdColumnPresent = false;
+ if (database == null) {
+ return;
+ }
+ try {
+ if (!database.checkTableExists(actualSchemaName, actualTableName)) {
+ return;
+ }
+ projectIdColumnPresent =
+ database.checkColumnExists(actualSchemaName, actualTableName,
COL_PROJECT_ID);
+ if (!projectIdColumnPresent &&
StringUtils.isNotEmpty(getActiveProjectId())) {
+ LogChannel.GENERAL.logBasic(
+ "Execution table "
+ + getQuotedSchemaTable()
+ + " has no "
+ + COL_PROJECT_ID
+ + " column. Project filtering is inactive until that column is
added.");
+ }
+ } catch (Exception e) {
+ projectIdColumnPresent = false;
+ LogChannel.GENERAL.logError(
+ "Unable to check for column " + COL_PROJECT_ID + " on table " +
getQuotedSchemaTable(),
+ e);
+ }
+ }
+
/**
* The state column is part of the DDL from {@link #buildDdl}. An existing
table that predates it
* keeps working through {@link #stateJsonColumnAvailable}. This location
does not alter the
@@ -1041,7 +1134,7 @@ public class CachingDatabaseExecutionInfoLocation extends
BaseCachingExecutionIn
// No connect required for DDL generation
String schemaTable = meta.getQuotedSchemaTableCombination(vars, schema,
table);
- IRowMeta fields = createDataRowMeta();
+ IRowMeta fields = createDataRowMeta(true, true);
StringBuilder ddl = new StringBuilder();
ddl.append(db.getCreateTableStatement(schemaTable, fields, null, false,
COL_ID, true));
ddl.append(Const.CR);
@@ -1052,6 +1145,15 @@ public class CachingDatabaseExecutionInfoLocation
extends BaseCachingExecutionIn
addIndexDdl(ddl, db, schemaTable, "idx_hop_exec_failed", COL_FAILED);
addIndexDdl(ddl, db, schemaTable, "idx_hop_exec_parent", COL_PARENT_ID);
addIndexDdl(ddl, db, schemaTable, "idx_hop_exec_status",
COL_STATUS_DESCRIPTION);
+ addIndexDdl(ddl, db, schemaTable, "idx_hop_exec_project", COL_PROJECT_ID);
+
+ ddl.append(Const.CR);
+ ddl.append(DDL_EXISTING_TABLE_MARKER);
+ ddl.append(Const.CR);
+ ddl.append(
+ meta.getAddColumnStatement(
+ schemaTable, new ValueMetaString(COL_PROJECT_ID, 256, -1), null,
false, null, true));
+ addIndexDdl(ddl, db, schemaTable, "idx_hop_exec_project", COL_PROJECT_ID);
return ddl.toString();
}
diff --git
a/plugins/misc/execution-database/src/test/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocationTest.java
b/plugins/misc/execution-database/src/test/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocationTest.java
index 682e24258b..bcbfd0732c 100644
---
a/plugins/misc/execution-database/src/test/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocationTest.java
+++
b/plugins/misc/execution-database/src/test/java/org/apache/hop/execution/database/CachingDatabaseExecutionInfoLocationTest.java
@@ -111,7 +111,7 @@ class CachingDatabaseExecutionInfoLocationTest {
new LoggingObject("CachingDatabaseExecutionInfoLocationTest"),
variables, databaseMeta);
db.connect();
try {
- db.execStatements(ddl);
+ db.execStatements(createTableDdl(ddl));
} finally {
db.disconnect();
}
@@ -377,6 +377,10 @@ class CachingDatabaseExecutionInfoLocationTest {
String ddl = location.buildDdl(variables);
assertTrue(ddl.toLowerCase().contains("create"));
assertTrue(ddl.contains("idx_hop_exec_start") ||
ddl.toLowerCase().contains("index"));
+
assertTrue(ddl.contains(CachingDatabaseExecutionInfoLocation.COL_PROJECT_ID));
+ assertTrue(ddl.contains("idx_hop_exec_project"));
+
assertTrue(ddl.contains(CachingDatabaseExecutionInfoLocation.DDL_EXISTING_TABLE_MARKER));
+ assertTrue(ddl.toLowerCase().contains("alter table"));
assertTrue(ddl.toLowerCase().contains(CachingDatabaseExecutionInfoLocation.COL_STATE_JSON));
assertTrue(
ddl.contains(CachingDatabaseExecutionInfoLocation.COL_JSON)
@@ -387,6 +391,98 @@ class CachingDatabaseExecutionInfoLocationTest {
|| ddl.toLowerCase().contains("character"));
}
+ @Test
+ void legacyTableWithoutProjectIdColumnStillWorks() throws Exception {
+ location.close();
+ String table =
+ databaseMeta.getQuotedSchemaTableCombination(
+ variables, "",
CachingDatabaseExecutionInfoLocation.DEFAULT_TABLE_NAME);
+ Database db =
+ new Database(
+ new LoggingObject("CachingDatabaseExecutionInfoLocationTest"),
variables, databaseMeta);
+ db.connect();
+ try {
+ db.execStatement(
+ "ALTER TABLE "
+ + table
+ + " DROP COLUMN "
+ +
databaseMeta.quoteField(CachingDatabaseExecutionInfoLocation.COL_PROJECT_ID));
+ } finally {
+ db.disconnect();
+ }
+
+ variables.setVariable(Execution.VARIABLE_HOP_PROJECT_ID, "sales");
+ location.initialize(variables, metadataProvider);
+ assertFalse(location.isProjectIdColumnPresent());
+
+ String id = UUID.randomUUID().toString();
+ CacheEntry entry = sampleEntry(id, "Legacy", ExecutionType.Pipeline,
false, "Finished");
+ entry.setProjectId("sales");
+ entry.getExecution().setProjectId("sales");
+ location.persistCacheEntry(entry);
+
+ CacheEntry loaded = location.loadCacheEntry(id);
+ assertNotNull(loaded);
+ assertEquals("sales", loaded.getProjectId());
+
+ Set<DatedId> ids = new HashSet<>();
+ location.retrieveIds(false, ids, 100, IExecutionSelector.ALL);
+ assertEquals(1, ids.size());
+ }
+
+ @Test
+ void retrieveIdsFiltersByProjectIdAndKeepsLegacyRows() throws Exception {
+ String salesId = UUID.randomUUID().toString();
+ String otherId = UUID.randomUUID().toString();
+ String legacyId = UUID.randomUUID().toString();
+
+ CacheEntry sales = sampleEntry(salesId, "SalesPipe",
ExecutionType.Pipeline, false, "Finished");
+ sales.setProjectId("sales");
+ sales.getExecution().setProjectId("sales");
+ CacheEntry other =
+ sampleEntry(otherId, "FinancePipe", ExecutionType.Pipeline, false,
"Finished");
+ other.setProjectId("finance");
+ other.getExecution().setProjectId("finance");
+ CacheEntry legacy = sampleEntry(legacyId, "OldPipe",
ExecutionType.Pipeline, false, "Finished");
+
+ location.persistCacheEntry(sales);
+ location.persistCacheEntry(other);
+ location.persistCacheEntry(legacy);
+
+ location.clearCaches();
+ Set<DatedId> all = new HashSet<>();
+ location.retrieveIds(false, all, 100, IExecutionSelector.ALL);
+ assertEquals(3, all.size());
+
+ variables.setVariable(Execution.VARIABLE_HOP_PROJECT_ID, "sales");
+ location.close();
+ location.initialize(variables, metadataProvider);
+ assertTrue(location.isProjectIdColumnPresent());
+
+ location.clearCaches();
+ Set<DatedId> filtered = new HashSet<>();
+ location.retrieveIds(false, filtered, 100, IExecutionSelector.ALL);
+ assertEquals(2, filtered.size());
+ assertTrue(filtered.stream().anyMatch(dated ->
salesId.equals(dated.getId())));
+ assertTrue(filtered.stream().anyMatch(dated ->
legacyId.equals(dated.getId())));
+ assertTrue(filtered.stream().noneMatch(dated ->
otherId.equals(dated.getId())));
+
+ variables.setVariable(Execution.VARIABLE_HOP_PROJECT_ID, "");
+ location.close();
+ location.initialize(variables, metadataProvider);
+ CacheEntry again = sampleEntry(salesId, "SalesPipe",
ExecutionType.Pipeline, true, "Finished");
+ location.persistCacheEntry(again);
+ CacheEntry loaded = location.loadCacheEntry(salesId);
+ assertNotNull(loaded);
+ assertEquals("sales", loaded.getProjectId());
+ assertEquals("sales", loaded.getExecution().getProjectId());
+ }
+
+ private static String createTableDdl(String ddl) {
+ int marker =
ddl.indexOf(CachingDatabaseExecutionInfoLocation.DDL_EXISTING_TABLE_MARKER);
+ return marker < 0 ? ddl : ddl.substring(0, marker);
+ }
+
@Test
void lruCacheEvictionEnforcesMaxSize() throws Exception {
location.setMaxCacheSize("2");
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/environment/EnvironmentVariablesImportHelper.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/environment/EnvironmentVariablesImportHelper.java
index 5254ae366b..e7eed87f57 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/environment/EnvironmentVariablesImportHelper.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/environment/EnvironmentVariablesImportHelper.java
@@ -83,6 +83,7 @@ public final class EnvironmentVariablesImportHelper {
managed.add(ProjectsUtil.VARIABLE_HOP_DATASETS_FOLDER);
managed.add(ProjectsUtil.VARIABLE_HOP_UNIT_TESTS_FOLDER);
managed.add(Defaults.VARIABLE_HOP_PROJECT_NAME);
+ managed.add(Defaults.VARIABLE_HOP_PROJECT_ID);
managed.add(Defaults.VARIABLE_HOP_ENVIRONMENT_NAME);
managed.add(Const.HOP_METADATA_FOLDER);
managed.add(JAVA_IO_TMPDIR);
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/gui/ProjectsGuiPlugin.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/gui/ProjectsGuiPlugin.java
index 350a41848c..74fb446242 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/gui/ProjectsGuiPlugin.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/gui/ProjectsGuiPlugin.java
@@ -1950,6 +1950,7 @@ public class ProjectsGuiPlugin {
&& !name.contains("HOP_PROJECTS")
&& !name.contains("HOP_PLATFORM_OS")
&& !name.contains("HOP_PROJECT_NAME")
+ && !name.contains("HOP_PROJECT_ID")
&& !name.contains("HOP_SERVER_URL")) {
String value = variables.getVariable(name);
variablesMap.put(name, value);
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ManageProjectsOptionPlugin.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ManageProjectsOptionPlugin.java
index 161a61bb0b..3c35a67d54 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ManageProjectsOptionPlugin.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ManageProjectsOptionPlugin.java
@@ -24,6 +24,7 @@ import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.vfs2.FileObject;
import org.apache.hop.core.Const;
import org.apache.hop.core.config.HopConfig;
import org.apache.hop.core.config.plugin.ConfigPlugin;
@@ -35,6 +36,7 @@ import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.DescribedVariable;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.core.variables.Variables;
+import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.metadata.api.IHasHopMetadataProvider;
import org.apache.hop.projects.config.ProjectsConfig;
import org.apache.hop.projects.config.ProjectsConfigSingleton;
@@ -390,6 +392,11 @@ public class ManageProjectsOptionPlugin implements
IConfigOptions {
log.logBasic(CONST_PROJECT + projectName + "' was created for home folder
: " + projectHome);
+ String configFilename =
projectConfig.getActualProjectConfigFilename(variables);
+ boolean configFileExisted = false;
+ try (FileObject configFile = HopVfs.getFileObject(configFilename)) {
+ configFileExisted = configFile.exists();
+ }
Project project = projectConfig.loadProject(variables);
// Keep an existing parent from a pre-existing project-config.json (Docker
// --project-keep-config-file). Only fall back to the standard parent when
none is set.
@@ -398,6 +405,11 @@ public class ManageProjectsOptionPlugin implements
IConfigOptions {
if (StringUtils.isEmpty(project.getParentProjectName())) {
project.setParentProjectName(config.findRegisteredStandardParentProject());
}
+ // A brand-new config file starts from the project name. An existing file
keeps its id,
+ // including none, so a 2.19.0 project is not switched on to execution
filtering.
+ if (!configFileExisted && StringUtils.isEmpty(project.getProjectId())) {
+ project.setProjectId(projectName);
+ }
modifyProjectSettings(project);
// Check to see if there's not a loop in the project parent hierarchy
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/Project.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/Project.java
index c04ab290cf..87a3bbbaeb 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/Project.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/Project.java
@@ -75,6 +75,13 @@ public class Project extends ConfigFile implements
IConfigFile {
@JsonIgnore private String configFilename;
private String description;
+
+ /**
+ * Optional id copied onto execution information. Empty means a shared
execution store is not
+ * filtered (2.19.0 behavior). Not inherited from the parent project.
+ */
+ private String projectId;
+
private String company;
private String department;
private String version;
@@ -169,6 +176,7 @@ public class Project extends ConfigFile implements
IConfigFile {
Project project = objectMapper.readValue(inputStream, Project.class);
this.description = project.description;
+ this.projectId = project.projectId;
this.company = project.company;
this.department = project.department;
this.version = project.version;
@@ -247,6 +255,10 @@ public class Project extends ConfigFile implements
IConfigFile {
//
variables.setVariable(
Defaults.VARIABLE_HOP_PROJECT_NAME,
Const.NVL(projectConfig.getProjectName(), ""));
+ // Always set, including to empty, so a project without an id does not
keep a parent's value.
+ variables.setVariable(
+ Defaults.VARIABLE_HOP_PROJECT_ID,
+ StringUtils.defaultString(StringUtils.trimToNull(projectId)));
variables.setVariable(Defaults.VARIABLE_HOP_ENVIRONMENT_NAME,
Const.NVL(environmentName, ""));
// To allow circular logic where an environment file is relative to the
project home
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ProjectDialog.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ProjectDialog.java
index c8d0795079..febd4820cf 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ProjectDialog.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/project/ProjectDialog.java
@@ -84,6 +84,7 @@ public class ProjectDialog extends Dialog {
private final PropsUi props;
private TextVar wName;
+ private Text wProjectId;
private TextVar wHome;
private Button wReadOnly;
private ComboVar wParentProject;
@@ -109,6 +110,11 @@ public class ProjectDialog extends Dialog {
private final boolean editMode;
+ /** Create mode only: project id tracks the name until the user edits the
id. */
+ private boolean projectIdFollowsName;
+
+ private boolean updatingProjectId;
+
public ProjectDialog(
Shell parent,
Project project,
@@ -249,6 +255,40 @@ public class ProjectDialog extends Dialog {
wName.setLayoutData(fdName);
Control lastControl = wName;
+ Label wlProjectId = new Label(comp, SWT.RIGHT);
+ PropsUi.setLook(wlProjectId);
+ wlProjectId.setText(BaseMessages.getString(PKG,
"ProjectDialog.Label.ProjectId"));
+ wlProjectId.setToolTipText(
+ BaseMessages.getString(PKG, "ProjectDialog.Label.ProjectId.Tooltip"));
+ FormData fdlProjectId = new FormData();
+ fdlProjectId.left = new FormAttachment(0, 0);
+ fdlProjectId.right = new FormAttachment(middle, 0);
+ fdlProjectId.top = new FormAttachment(lastControl, margin);
+ wlProjectId.setLayoutData(fdlProjectId);
+ wProjectId = new Text(comp, SWT.SINGLE | SWT.BORDER | SWT.LEFT);
+ PropsUi.setLook(wProjectId);
+ wProjectId.setToolTipText(BaseMessages.getString(PKG,
"ProjectDialog.Label.ProjectId.Tooltip"));
+ FormData fdProjectId = new FormData();
+ fdProjectId.left = new FormAttachment(middle, margin);
+ fdProjectId.right = new FormAttachment(100, 0);
+ fdProjectId.top = new FormAttachment(wlProjectId, 0, SWT.CENTER);
+ wProjectId.setLayoutData(fdProjectId);
+ wName.addModifyListener(
+ e -> {
+ if (projectIdFollowsName) {
+ updatingProjectId = true;
+ wProjectId.setText(wName.getText());
+ updatingProjectId = false;
+ }
+ });
+ wProjectId.addModifyListener(
+ e -> {
+ if (!updatingProjectId) {
+ projectIdFollowsName = false;
+ }
+ });
+ lastControl = wProjectId;
+
Label wlHome = new Label(comp, SWT.RIGHT);
PropsUi.setLook(wlHome);
wlHome.setText(BaseMessages.getString(PKG,
"ProjectDialog.Label.HomeFolder"));
@@ -649,6 +689,7 @@ public class ProjectDialog extends Dialog {
boolean editable = !wReadOnly.getSelection();
wbConfigFile.setEnabled(editable);
+ wProjectId.setEnabled(editable);
wParentProject.setEnabled(editable);
wDescription.setEnabled(editable);
wCompany.setEnabled(editable);
@@ -964,6 +1005,17 @@ public class ProjectDialog extends Dialog {
||
ProjectConfig.isArchiveUri(variables.resolve(projectConfig.getProjectHome())));
wDescription.setText(Const.NVL(project.getDescription(), ""));
+ String storedProjectId = StringUtils.trimToNull(project.getProjectId());
+ // New projects suggest the project name. Editing an existing project must
not fill it in:
+ // saving the dialog would otherwise start filtering execution information.
+ projectIdFollowsName = !editMode && storedProjectId == null;
+ String shownProjectId = storedProjectId;
+ if (projectIdFollowsName) {
+ shownProjectId =
StringUtils.defaultString(projectConfig.getProjectName());
+ }
+ updatingProjectId = true;
+ wProjectId.setText(Const.NVL(shownProjectId, ""));
+ updatingProjectId = false;
wCompany.setText(Const.NVL(project.getCompany(), ""));
wDepartment.setText(Const.NVL(project.getDepartment(), ""));
wVersion.setText(Const.NVL(project.getVersion(), ""));
@@ -1025,6 +1077,7 @@ public class ProjectDialog extends Dialog {
projectConfig.setReadOnly(wReadOnly.getSelection());
project.setParentProjectName(wParentProject.getText());
+ project.setProjectId(StringUtils.trimToNull(wProjectId.getText()));
project.setDescription(wDescription.getText());
project.setCompany(wCompany.getText());
project.setDepartment(wDepartment.getText());
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/Defaults.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/Defaults.java
index 6726bcc326..05fd9f6f12 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/Defaults.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/Defaults.java
@@ -19,6 +19,14 @@ package org.apache.hop.projects.util;
public class Defaults {
public static final String VARIABLE_HOP_PROJECT_NAME = "HOP_PROJECT_NAME";
+
+ /**
+ * Same variable execution information locations read ({@code
+ * org.apache.hop.execution.Execution#VARIABLE_HOP_PROJECT_ID}). Empty means
those locations do
+ * not filter by project.
+ */
+ public static final String VARIABLE_HOP_PROJECT_ID = "HOP_PROJECT_ID";
+
public static final String VARIABLE_HOP_ENVIRONMENT_NAME =
"HOP_ENVIRONMENT_NAME";
public static final String EXTENSION_POINT_PROJECT_ACTIVATED =
"ProjectActivated";
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/PathVariableReplacer.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/PathVariableReplacer.java
index e25fb6513b..7f8d394afd 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/PathVariableReplacer.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/util/PathVariableReplacer.java
@@ -42,6 +42,7 @@ public final class PathVariableReplacer {
private static final Set<String> NON_PATH_VARIABLE_NAMES =
Set.of(
Defaults.VARIABLE_HOP_PROJECT_NAME,
+ Defaults.VARIABLE_HOP_PROJECT_ID,
Defaults.VARIABLE_HOP_ENVIRONMENT_NAME,
ProjectsUtil.VARIABLE_PARENT_PROJECT_NAME);
diff --git
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/xp/ProjectsVariablesControlSpaceSortOrderPrefix.java
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/xp/ProjectsVariablesControlSpaceSortOrderPrefix.java
index 7ca0f82a69..b43957212c 100644
---
a/plugins/misc/projects/src/main/java/org/apache/hop/projects/xp/ProjectsVariablesControlSpaceSortOrderPrefix.java
+++
b/plugins/misc/projects/src/main/java/org/apache/hop/projects/xp/ProjectsVariablesControlSpaceSortOrderPrefix.java
@@ -40,6 +40,7 @@ public class ProjectsVariablesControlSpaceSortOrderPrefix
prefixMap.put(ProjectsUtil.VARIABLE_PARENT_PROJECT_HOME, "311_");
prefixMap.put(ProjectsUtil.VARIABLE_PARENT_PROJECT_NAME, "312_");
prefixMap.put(Defaults.VARIABLE_HOP_PROJECT_NAME, "450_");
+ prefixMap.put(Defaults.VARIABLE_HOP_PROJECT_ID, "451_");
prefixMap.put(Defaults.VARIABLE_HOP_ENVIRONMENT_NAME, "450_");
prefixMap.put(ProjectsUtil.VARIABLE_HOP_DATASETS_FOLDER, "450_");
prefixMap.put(ProjectsUtil.VARIABLE_HOP_UNIT_TESTS_FOLDER, "460_");
diff --git
a/plugins/misc/projects/src/main/resources/org/apache/hop/projects/project/messages/messages_en_US.properties
b/plugins/misc/projects/src/main/resources/org/apache/hop/projects/project/messages/messages_en_US.properties
index 5addd1abf1..5f0b5bd38f 100644
---
a/plugins/misc/projects/src/main/resources/org/apache/hop/projects/project/messages/messages_en_US.properties
+++
b/plugins/misc/projects/src/main/resources/org/apache/hop/projects/project/messages/messages_en_US.properties
@@ -60,6 +60,8 @@ ProjectDialog.Label.AutoExportMetadata.Enable=Write a single
metadata JSON file
ProjectDialog.Label.AutoExportMetadataFilename=Auto-export metadata filename
(relative to project home)
ProjectDialog.Label.ParentProject=Parent project to inherit from
ProjectDialog.Label.ProjectName=Name
+ProjectDialog.Label.ProjectId=Project id
+ProjectDialog.Label.ProjectId.Tooltip=Stored in project-config.json and copied
onto execution information so a shared database or search index can be filtered
to this project. Leave empty to keep the previous behavior (no filtering). The
project name is a readable value; a UUID also works.
ProjectDialog.Label.ReadOnly=This project is read only
ProjectDialog.Label.UnitTestBaseFolder=Unit tests base path
(HOP_UNIT_TESTS_FOLDER)
ProjectDialog.Label.Version=Version
diff --git
a/plugins/misc/projects/src/test/java/org/apache/hop/projects/project/ProjectTest.java
b/plugins/misc/projects/src/test/java/org/apache/hop/projects/project/ProjectTest.java
index 44e663ac69..1653b8beec 100644
---
a/plugins/misc/projects/src/test/java/org/apache/hop/projects/project/ProjectTest.java
+++
b/plugins/misc/projects/src/test/java/org/apache/hop/projects/project/ProjectTest.java
@@ -19,6 +19,7 @@ package org.apache.hop.projects.project;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.File;
@@ -115,6 +116,84 @@ public class ProjectTest {
}
}
+ @Test
+ public void testProjectIdRoundTripAndOmittedWhenEmpty() throws Exception {
+ File tempFile = Files.createTempFile("project-config-id",
".json").toFile();
+ tempFile.deleteOnExit();
+ try {
+ Project project = new Project(tempFile.getAbsolutePath());
+ project.saveToFile();
+ String json = Files.readString(tempFile.toPath(),
StandardCharsets.UTF_8);
+ assertFalse(json.contains("projectId"));
+
+ Project readProject = new Project(tempFile.getAbsolutePath());
+ readProject.readFromFile();
+ assertNull(readProject.getProjectId());
+
+ project.setProjectId("sales");
+ project.saveToFile();
+ readProject.readFromFile();
+ assertEquals("sales", readProject.getProjectId());
+ assertTrue(Files.readString(tempFile.toPath(),
StandardCharsets.UTF_8).contains("sales"));
+ } finally {
+ tempFile.delete();
+ }
+ }
+
+ @Test
+ public void testLegacyConfigWithoutProjectIdLeavesVariableEmpty() throws
Exception {
+ tempRoot = Files.createTempDirectory("hop-project-id-legacy");
+ Path home = tempRoot.resolve("child");
+ Files.createDirectories(home);
+ writeMinimalConfig(home, null);
+
+ ProjectConfig projectConfig =
+ new ProjectConfig("sales", home.toString(),
ProjectsConfig.DEFAULT_PROJECT_CONFIG_FILENAME);
+ registerProject(projectConfig);
+
+ Project project = projectConfig.loadProject(new Variables());
+ assertNull(project.getProjectId());
+
+ IVariables variables = new Variables();
+ variables.setVariable(Defaults.VARIABLE_HOP_PROJECT_ID, "stale");
+ project.modifyVariables(variables, projectConfig, new ArrayList<>(), null);
+ assertEquals("", variables.getVariable(Defaults.VARIABLE_HOP_PROJECT_ID));
+ }
+
+ @Test
+ public void testProjectIdVariableDoesNotInheritFromParent() throws Exception
{
+ tempRoot = Files.createTempDirectory("hop-project-id-parent");
+ Path parentHome = tempRoot.resolve("parent");
+ Path childHome = tempRoot.resolve("child");
+ Files.createDirectories(parentHome);
+ Files.createDirectories(childHome);
+ writeMinimalConfig(parentHome, null);
+ writeMinimalConfig(childHome, "parent-proj");
+
+ ProjectConfig parentConfig =
+ new ProjectConfig(
+ "parent-proj", parentHome.toString(),
ProjectsConfig.DEFAULT_PROJECT_CONFIG_FILENAME);
+ ProjectConfig childConfig =
+ new ProjectConfig(
+ "child-proj", childHome.toString(),
ProjectsConfig.DEFAULT_PROJECT_CONFIG_FILENAME);
+ registerProject(parentConfig);
+ registerProject(childConfig);
+
+ Project parent = parentConfig.loadProject(new Variables());
+ parent.setProjectId("parent-proj");
+ parent.saveToFile();
+
+ Project child = childConfig.loadProject(new Variables());
+ assertNull(child.getProjectId());
+ IVariables variables = new Variables();
+ child.modifyVariables(variables, childConfig, new ArrayList<>(), null);
+ assertEquals("", variables.getVariable(Defaults.VARIABLE_HOP_PROJECT_ID));
+
+ child.setProjectId("child-proj");
+ child.modifyVariables(variables, childConfig, new ArrayList<>(), null);
+ assertEquals("child-proj",
variables.getVariable(Defaults.VARIABLE_HOP_PROJECT_ID));
+ }
+
@Test
public void testParentProjectVariablesWhenNoParent() throws Exception {
tempRoot = Files.createTempDirectory("hop-project-no-parent");
diff --git
a/plugins/misc/projects/src/test/java/org/apache/hop/projects/util/PathVariableReplacerTest.java
b/plugins/misc/projects/src/test/java/org/apache/hop/projects/util/PathVariableReplacerTest.java
index 83a41b1f6a..9c64e21ac3 100644
---
a/plugins/misc/projects/src/test/java/org/apache/hop/projects/util/PathVariableReplacerTest.java
+++
b/plugins/misc/projects/src/test/java/org/apache/hop/projects/util/PathVariableReplacerTest.java
@@ -128,6 +128,7 @@ class PathVariableReplacerTest {
@Test
void ignoresNonPathManagedNames() {
assertFalse(PathVariableReplacer.isCandidateVariableName(Defaults.VARIABLE_HOP_PROJECT_NAME));
+
assertFalse(PathVariableReplacer.isCandidateVariableName(Defaults.VARIABLE_HOP_PROJECT_ID));
assertFalse(
PathVariableReplacer.isCandidateVariableName(Defaults.VARIABLE_HOP_ENVIRONMENT_NAME));
assertFalse(
diff --git
a/plugins/tech/elastic/src/main/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocation.java
b/plugins/tech/elastic/src/main/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocation.java
index 8bf0a3e580..523ff36736 100644
---
a/plugins/tech/elastic/src/main/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocation.java
+++
b/plugins/tech/elastic/src/main/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocation.java
@@ -135,6 +135,7 @@ public class ElasticExecutionInfoLocation extends
BaseCachingExecutionInfoLocati
@Override
protected void persistCacheEntry(CacheEntry cacheEntry) throws HopException {
try {
+ cacheEntry.prepareForPersist();
// Before writing to disk, we calculate some summaries for convenience
of other tools.
cacheEntry.calculateSummary();
@@ -352,22 +353,7 @@ public class ElasticExecutionInfoLocation extends
BaseCachingExecutionInfoLocati
URI uri = URI.create(actualUrl);
URI postUri = uri.resolve(actualIndexName + "/_search");
- String body =
- """
- {
- __LIMIT_CLAUSE__
- "from": 0,
- "query" : { "match_all" : {} },
- "fields": [ "id", "execution.executionStartDate" ],
- "sort" : [ { "execution.executionStartDate" : {"order" : "desc"
}} ],
- "_source": false
- }
- """;
- String limitClause = "";
- if (limit > 0) {
- limitClause = "\"size\": " + limit + ",";
- }
- body = body.replace("__LIMIT_CLAUSE__", limitClause);
+ String body = listQuery(getActiveProjectId(), limit);
HttpRequest request =
HttpRequest.newBuilder()
@@ -456,31 +442,7 @@ public class ElasticExecutionInfoLocation extends
BaseCachingExecutionInfoLocati
// We add a bunch of settings when we create the index
//
- String createBody =
- """
- {
- "mappings" : {
- "properties": {
- "id": { "type" : "text"},
- "name": { "type" : "text"},
- "execution.id": { "type" : "text"},
- "execution.name": { "type" : "text"},
- "execution.filename": { "type" : "text"},
- "execution.executionType": { "type" : "text"},
- "execution.parentId": { "type" : "text"},
- "execution.registrationDate": { "type": "date" },
- "execution.executionStartDate": { "type": "date" },
- "executionState.updateTime": { "type": "date" },
- "executionState.executionEndDate": { "type": "date" },
- "childExecutions": { "type": "object", "enabled" : false },
- "childExecutionStates": { "type": "object", "enabled" :
false },
- "childExecutionData": { "type": "object", "enabled" : false }
- }
- }, "settings": {
- "index.mapping.total_fields.limit": 500
- }
- }
- """;
+ String createBody = createIndexBody();
HttpRequest createRequest =
HttpRequest.newBuilder()
@@ -491,15 +453,32 @@ public class ElasticExecutionInfoLocation extends
BaseCachingExecutionInfoLocati
.PUT(HttpRequest.BodyPublishers.ofString(createBody))
.build();
- // Send to Elastic and we don't care about the response.
- // A 400 usually means that the index already exists.
- //
HttpResponse<String> createResponse =
client.send(createRequest, HttpResponse.BodyHandlers.ofString());
- // Verify the 200 from Elastic
+ // A 400 with resource_already_exists_exception means the index is
already there.
+ // Add the projectId keyword if it is missing. Do not recreate the index.
//
- if (createResponse.statusCode() != 200) {
+ if (isIndexAlreadyExists(createResponse.statusCode(),
createResponse.body())) {
+ URI mappingUri = uri.resolve(location.actualIndexName + "/_mapping");
+ HttpRequest mappingRequest =
+ HttpRequest.newBuilder()
+ .uri(mappingUri)
+ .header("Content-Type", "application/json")
+ .header("Accept", "application/json")
+ .header("Authorization", "ApiKey " + location.actualApiKey)
+
.PUT(HttpRequest.BodyPublishers.ofString(projectIdMappingBody()))
+ .build();
+ HttpResponse<String> mappingResponse =
+ client.send(mappingRequest, HttpResponse.BodyHandlers.ofString());
+ if (mappingResponse.statusCode() != 200) {
+ throw new HopException(
+ "Status code "
+ + mappingResponse.statusCode()
+ + " received from Elastic while adding the projectId
mapping, response: "
+ + mappingResponse.body());
+ }
+ } else if (createResponse.statusCode() != 200) {
throw new HopException(
"Status code "
+ createResponse.statusCode()
@@ -524,6 +503,84 @@ public class ElasticExecutionInfoLocation extends
BaseCachingExecutionInfoLocati
}
}
+ /** List query. An empty project id keeps {@code match_all}, the 2.19.0
query. */
+ static String listQuery(String activeProjectId, int limit) {
+ String limitClause = "";
+ if (limit > 0) {
+ limitClause = "\"size\": " + limit + ",";
+ }
+ return """
+ {
+ %s
+ "from": 0,
+ "query" : %s,
+ "fields": [ "id", "execution.executionStartDate" ],
+ "sort" : [ { "execution.executionStartDate" : {"order" : "desc" }} ],
+ "_source": false
+ }
+ """
+ .formatted(limitClause, listQueryClause(activeProjectId));
+ }
+
+ static String listQueryClause(String activeProjectId) {
+ if (StringUtils.isEmpty(activeProjectId)) {
+ return "{ \"match_all\" : {} }";
+ }
+ return """
+ { "bool": {
+ "minimum_should_match": 1,
+ "should": [
+ { "term": { "projectId": "@@PROJECT_ID@@" } },
+ { "bool": { "must_not": { "exists": { "field": "projectId" } } }
}
+ ]
+ }
+ }
+ """
+ .replace("@@PROJECT_ID@@", jsonEscape(activeProjectId));
+ }
+
+ static String createIndexBody() {
+ return """
+ {
+ "mappings" : {
+ "properties": {
+ "id": { "type" : "text"},
+ "name": { "type" : "text"},
+ "projectId": { "type": "keyword" },
+ "execution.id": { "type" : "text"},
+ "execution.name": { "type" : "text"},
+ "execution.filename": { "type" : "text"},
+ "execution.executionType": { "type" : "text"},
+ "execution.parentId": { "type" : "text"},
+ "execution.registrationDate": { "type": "date" },
+ "execution.executionStartDate": { "type": "date" },
+ "executionState.updateTime": { "type": "date" },
+ "executionState.executionEndDate": { "type": "date" },
+ "childExecutions": { "type": "object", "enabled" : false },
+ "childExecutionStates": { "type": "object", "enabled" : false },
+ "childExecutionData": { "type": "object", "enabled" : false }
+ }
+ }, "settings": {
+ "index.mapping.total_fields.limit": 500
+ }
+ }
+ """;
+ }
+
+ static String projectIdMappingBody() {
+ return """
+ { "properties": { "projectId": { "type": "keyword" } } }
+ """;
+ }
+
+ static boolean isIndexAlreadyExists(int statusCode, String body) {
+ return statusCode == 400 && body != null &&
body.contains("resource_already_exists_exception");
+ }
+
+ static String jsonEscape(String value) {
+ return value.replace("\\", "\\\\").replace("\"", "\\\"");
+ }
+
@Override
public String getPluginId() {
return "elastic-location";
diff --git
a/plugins/tech/elastic/src/test/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocationProjectIdTest.java
b/plugins/tech/elastic/src/test/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocationProjectIdTest.java
new file mode 100644
index 0000000000..02a4bef08d
--- /dev/null
+++
b/plugins/tech/elastic/src/test/java/org/apache/hop/execution/elastic/ElasticExecutionInfoLocationProjectIdTest.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.execution.elastic;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+class ElasticExecutionInfoLocationProjectIdTest {
+
+ @Test
+ void listQueryIsMatchAllWhenProjectIdIsEmpty() {
+ String query = ElasticExecutionInfoLocation.listQuery(null, 50);
+ assertTrue(query.contains("match_all"));
+ assertFalse(query.contains("projectId"));
+
+ String unlimited = ElasticExecutionInfoLocation.listQuery("", 0);
+ assertTrue(unlimited.contains("match_all"));
+ assertFalse(unlimited.contains("\"size\""));
+ }
+
+ @Test
+ void listQueryFiltersByProjectIdAndKeepsLegacyDocuments() {
+ String query = ElasticExecutionInfoLocation.listQuery("sales", 50);
+ assertTrue(query.contains("\"term\""));
+ assertTrue(query.contains("\"projectId\": \"sales\""));
+ assertTrue(query.contains("must_not"));
+ assertTrue(query.contains("exists"));
+ assertFalse(query.contains("match_all"));
+ }
+
+ @Test
+ void listQueryEscapesQuotesInTheProjectId() {
+ String query = ElasticExecutionInfoLocation.listQuery("sa\"les", 10);
+ assertTrue(query.contains("sa\\\"les"));
+ }
+
+ @Test
+ void createIndexDeclaresProjectIdAsKeyword() {
+ String body = ElasticExecutionInfoLocation.createIndexBody();
+ assertTrue(body.contains("\"projectId\": { \"type\": \"keyword\" }"));
+
assertTrue(ElasticExecutionInfoLocation.projectIdMappingBody().contains("keyword"));
+ }
+
+ @Test
+ void alreadyExistsIsOnlyTheElastic400() {
+ assertTrue(
+ ElasticExecutionInfoLocation.isIndexAlreadyExists(
+ 400,
"{\"error\":{\"type\":\"resource_already_exists_exception\"}}"));
+ assertFalse(ElasticExecutionInfoLocation.isIndexAlreadyExists(400,
"other"));
+ assertFalse(
+ ElasticExecutionInfoLocation.isIndexAlreadyExists(
+ 200, "resource_already_exists_exception"));
+ }
+}
diff --git
a/plugins/tech/opensearch/src/main/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocation.java
b/plugins/tech/opensearch/src/main/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocation.java
index 994d394a3e..bce37555c3 100644
---
a/plugins/tech/opensearch/src/main/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocation.java
+++
b/plugins/tech/opensearch/src/main/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocation.java
@@ -173,6 +173,7 @@ public class OpenSearchExecutionInfoLocation extends
BaseCachingExecutionInfoLoc
@Override
protected void persistCacheEntry(CacheEntry cacheEntry) throws HopException {
try {
+ cacheEntry.prepareForPersist();
// Before writing to disk, we calculate some summaries for convenience
of other tools.
cacheEntry.calculateSummary();
@@ -430,6 +431,10 @@ public class OpenSearchExecutionInfoLocation extends
BaseCachingExecutionInfoLoc
body = body.replace("__FROM_CLAUSE__", "FROM " + actualIndexName);
String whereClause = "";
+ String projectClause = projectIdWhereClause(getActiveProjectId());
+ if (StringUtils.isNotEmpty(projectClause)) {
+ whereClause = addToWhereClause(whereClause, projectClause);
+ }
LastPeriod dateFilter = selector != null ? selector.startDateFilter() :
null;
if (dateFilter != null && dateFilter != LastPeriod.NONE) {
// OpenSearch uses UTC and the GUI runs in local time.
@@ -511,6 +516,77 @@ public class OpenSearchExecutionInfoLocation extends
BaseCachingExecutionInfoLoc
}
}
+ /**
+ * SQL fragment without {@code WHERE}. Empty when the active project id is
empty, so the list
+ * query stays the 2.19.0 shape. Single quotes in the id are escaped.
+ */
+ static String projectIdWhereClause(String activeProjectId) {
+ if (StringUtils.isEmpty(activeProjectId)) {
+ return "";
+ }
+ String escaped = activeProjectId.replace("'", "''");
+ return "(projectId = '" + escaped + "' OR projectId IS NULL OR projectId =
'')";
+ }
+
+ static String createIndexBody() {
+ return """
+ {
+ "mappings" : {
+ "properties": {
+ "id" : { "type": "text"},
+ "name": { "type": "text"},
+ "projectId": { "type": "keyword" },
+ "creationDate": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
+ "summary": {
+ "type" : "nested",
+ "properties": {
+ "startDate" : { "type": "date", "format": "yyyy/MM/dd
HH:mm:ss.SSS" },
+ "endDate" : { "type": "date", "format": "yyyy/MM/dd
HH:mm:ss.SSS" },
+ "durationMs" : { "type": "long" }
+ }
+ },
+ "execution": {
+ "type" : "nested",
+ "properties": {
+ "id" : { "type": "text"} ,
+ "name" : { "type": "text" },
+ "filename" : { "type": "text" },
+ "executionType" : { "type": "text" },
+ "parentId" : { "type": "text" },
+ "projectId" : { "type": "keyword" },
+ "registrationDate": { "type": "long" }
+ }
+ },
+ "executionState": {
+ "type" : "nested",
+ "properties": {
+ "executionStartDate": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
+ "executionEndDate": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
+ "updateTime": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
+ "statusDescription": { "type": "text" }
+ }
+ },
+ "childExecutions" : { "type": "object", "enabled": false },
+ "childExecutionStates": { "type": "object", "enabled": false },
+ "childExecutionData" : { "type": "object", "enabled": false }
+ }
+ }, "settings": {
+ "index.mapping.total_fields.limit": 500
+ }
+ }
+ """;
+ }
+
+ static String projectIdMappingBody() {
+ return """
+ { "properties": { "projectId": { "type": "keyword" } } }
+ """;
+ }
+
+ static boolean isIndexAlreadyExists(int statusCode, String body) {
+ return statusCode == 400 && body != null &&
body.contains("resource_already_exists_exception");
+ }
+
private String addToWhereClause(String whereClause, String clause) {
String result = whereClause;
if (StringUtils.isEmpty(whereClause)) {
@@ -536,51 +612,7 @@ public class OpenSearchExecutionInfoLocation extends
BaseCachingExecutionInfoLoc
try {
location.initialize(hopGui.getVariables(), hopGui.getMetadataProvider());
- String putBody =
- """
- {
- "mappings" : {
- "properties": {
- "id" : { "type": "text"},
- "name": { "type": "text"},
- "creationDate": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
- "summary": {
- "type" : "nested",
- "properties": {
- "startDate" : { "type": "date", "format": "yyyy/MM/dd
HH:mm:ss.SSS" },
- "endDate" : { "type": "date", "format": "yyyy/MM/dd
HH:mm:ss.SSS" },
- "durationMs" : { "type": "long" }
- }
- },
- "execution": {
- "type" : "nested",
- "properties": {
- "id" : { "type": "text"} ,
- "name" : { "type": "text" },
- "filename" : { "type": "text" },
- "executionType" : { "type": "text" },
- "parentId" : { "type": "text" },
- "registrationDate": { "type": "long" }
- }
- },
- "executionState": {
- "type" : "nested",
- "properties": {
- "executionStartDate": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
- "executionEndDate": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
- "updateTime": { "type": "date", "format":
"epoch_millis||yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time" },
- "statusDescription": { "type": "text" }
- }
- },
- "childExecutions" : { "type": "object", "enabled":
false },
- "childExecutionStates": { "type": "object", "enabled":
false },
- "childExecutionData" : { "type": "object", "enabled":
false }
- }
- }, "settings": {
- "index.mapping.total_fields.limit": 500
- }
- }
- """;
+ String putBody = createIndexBody();
String result;
RestCaller restCaller =
new RestCaller(
@@ -593,6 +625,21 @@ public class OpenSearchExecutionInfoLocation extends
BaseCachingExecutionInfoLoc
location.ignoreSsl,
getHeaders());
result = getResultFromPipeline(restCaller);
+ if (isIndexAlreadyExists(
+ restCaller.getStatusCode() == null ? 0 :
restCaller.getStatusCode().intValue(), result)) {
+ putBody = projectIdMappingBody();
+ restCaller =
+ new RestCaller(
+ new MemoryMetadataProvider(),
+ location.actualUrl + "/" + location.actualIndexName +
"/_mapping",
+ location.actualUsername,
+ location.actualPassword,
+ "PUT",
+ putBody,
+ location.ignoreSsl,
+ getHeaders());
+ result = getResultFromPipeline(restCaller);
+ }
if (restCaller.getResult() == null ||
restCaller.getResult().getNrErrors() > 0) {
result += Const.CR + "Logging: " + restCaller.getLoggingText();
}
diff --git
a/plugins/tech/opensearch/src/test/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocationProjectIdTest.java
b/plugins/tech/opensearch/src/test/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocationProjectIdTest.java
new file mode 100644
index 0000000000..8618061ede
--- /dev/null
+++
b/plugins/tech/opensearch/src/test/java/org/apache/hop/execution/opensearch/OpenSearchExecutionInfoLocationProjectIdTest.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.execution.opensearch;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+class OpenSearchExecutionInfoLocationProjectIdTest {
+
+ @Test
+ void whereClauseIsEmptyWhenProjectIdIsEmpty() {
+ assertEquals("",
OpenSearchExecutionInfoLocation.projectIdWhereClause(null));
+ assertEquals("", OpenSearchExecutionInfoLocation.projectIdWhereClause(""));
+ }
+
+ @Test
+ void whereClauseMatchesTheProjectAndLegacyRows() {
+ assertEquals(
+ "(projectId = 'sales' OR projectId IS NULL OR projectId = '')",
+ OpenSearchExecutionInfoLocation.projectIdWhereClause("sales"));
+ }
+
+ @Test
+ void whereClauseEscapesQuotes() {
+ assertEquals(
+ "(projectId = 'o''brien' OR projectId IS NULL OR projectId = '')",
+ OpenSearchExecutionInfoLocation.projectIdWhereClause("o'brien"));
+ }
+
+ @Test
+ void createIndexDeclaresRootProjectIdAsKeyword() {
+ String body = OpenSearchExecutionInfoLocation.createIndexBody();
+ assertTrue(body.contains("\"projectId\": { \"type\": \"keyword\" }"));
+
assertTrue(OpenSearchExecutionInfoLocation.projectIdMappingBody().contains("keyword"));
+ }
+}
