This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new e91f7c8b8e Row normaliser: key fields kept on every row when every
other field is normalised, fixes #9398 (#9449)
e91f7c8b8e is described below
commit e91f7c8b8e7e10a4af953b31d9cd88a306de206c
Author: Hans Van Akelyen <[email protected]>
AuthorDate: Tue Nov 17 10:24:12 2026 +0100
Row normaliser: key fields kept on every row when every other field is
normalised, fixes #9398 (#9449)
---
.../pages/pipeline/transforms/rownormaliser.adoc | 34 ++-
.../transforms/0020-row-normaliser-key-fields.hpl | 180 +++++++++++++
.../datasets/golden-row-normaliser-key-fields.csv | 5 +
.../transforms/main-0020-row-normaliser.hwf | 3 +
.../dataset/golden-row-normaliser-key-fields.json | 40 +++
.../0020-row-normaliser-key-fields UNIT.json | 38 +++
.../pipeline/transforms/normaliser/Normaliser.java | 15 +-
.../transforms/normaliser/NormaliserDialog.java | 284 +++++++--------------
.../transforms/normaliser/NormaliserField.java | 114 +++------
.../transforms/normaliser/NormaliserKeyField.java | 56 ++++
.../transforms/normaliser/NormaliserMeta.java | 149 +++++++++--
.../normaliser/messages/messages_en_US.properties | 12 +-
.../normaliser/NormaliserDialogTest.java | 235 +++++++++++++++++
.../normaliser/NormaliserMetaInjectionTest.java | 1 +
.../transforms/normaliser/NormaliserMetaTest.java | 61 +++++
.../transforms/normaliser/NormaliserTest.java | 71 ++++++
.../src/test/resources/normaliser-transform.xml | 5 +
17 files changed, 1004 insertions(+), 299 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
index 3fe51a73b3..b865f99399 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rownormaliser.adoc
@@ -33,9 +33,13 @@ A type can fill each new field from only one input field:
listing two fields wit
== Normalising every input field
-When the **Fields table** is empty, the transform normalises every field it
receives. Each input field is a type of its own, named after the field, and its
value goes into the **Value field**. This works when the input fields are only
known when the pipeline runs, for example in a pipeline that profiles any file
or table it is given.
+When the **Fields table** is empty, the transform normalises every field it
receives, except the **Key fields**. Each input field is a type of its own,
named after the field, and its value goes into the **Value field**. This works
when the input fields are only known when the pipeline runs, for example in a
pipeline that profiles any file or table it is given.
-Every input field is normalised this way, so no field is passed on unchanged:
the output has only the type field and the value field. When the input fields
have different data types the value field is a String, as described below. The
value field keeps a length or precision only when all input fields have the
same one. While you design a pipeline whose input fields are not known yet, the
transform shows the value field to the transforms after it as a String.
+The key fields are passed on unchanged on every row, in the order they have in
the input. Without key fields the output has only the type field and the value
field. When the normalised fields have different data types the value field is
a String, as described below. The value field keeps a length or precision only
when all normalised fields have the same one. While you design a pipeline whose
input fields are not known yet, the transform shows the value field to the
transforms after it a [...]
+
+This is what the `Crosstable` prefix does in a Qlik Sense load script, with
the key fields as its qualifier fields.
+
+The key fields are only used when the **Fields table** is empty. With listed
fields, every field that is not listed is passed on unchanged anyway;
**Verify** warns when key fields are set but not used.
== Data types of the new fields
@@ -56,6 +60,7 @@ NOTE: Before Hop 2.20, a new field took the data type of the
first input field f
|Transform name|Name of the transform, this name has to be unique in a single
pipeline.
|Typefield|The name of the type field (**key** in the example below).
|Value field|The name of the field that holds the values when the **Fields
table** is empty and every input field is normalised. The default is **value**.
Give it a name other than the type field's: otherwise it is renamed in the
output, to **value_1** for example.
+|Key fields|The fields that are passed on unchanged on every row when the
**Fields table** is empty. Every other input field is normalised. The names can
contain variables.
|Fields table|
A list of the fields you want to normalize. Leave it empty to normalise every
input field into the **Value field**. You must set the following properties for
each listed field:
@@ -142,3 +147,28 @@ The result has one row per product, each value in the new
field it names:
|2003-01-01|Product1|100|5
|2003-01-01|Product2|250|10
|===
+
+=== Normalized data (example 4)
+Months as columns, kept per region and year. With this input:
+
+[options="header"]
+|===
+|Region|Year|Jan|Feb|Mar
+|North|2026|10|20|30
+|South|2026|15|25|35
+|===
+
+set **Typefield** = "Month", **Value field** = "Sales", **Key fields** =
"Region" and "Year", and leave the **Fields table** empty. The result is:
+
+[options="header"]
+|===
+|Region|Year|Month|Sales
+|North|2026|Jan|10
+|North|2026|Feb|20
+|North|2026|Mar|30
+|South|2026|Jan|15
+|South|2026|Feb|25
+|South|2026|Mar|35
+|===
+
+A column added to the input, `Apr` for example, is normalised as well without
changing the transform.
diff --git a/integration-tests/transforms/0020-row-normaliser-key-fields.hpl
b/integration-tests/transforms/0020-row-normaliser-key-fields.hpl
new file mode 100644
index 0000000000..429044b649
--- /dev/null
+++ b/integration-tests/transforms/0020-row-normaliser-key-fields.hpl
@@ -0,0 +1,180 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>0020-row-normaliser-key-fields</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <pipeline_version/>
+ <pipeline_type>Normal</pipeline_type>
+ <parameters>
+ </parameters>
+ <capture_transform_performance>N</capture_transform_performance>
+
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <created_user>-</created_user>
+ <created_date>2021/05/31 11:24:13.758</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2021/05/31 11:24:13.758</modified_date>
+ <key_for_session_key>H4sIAAAAAAAAAAMAAAAAAAAAAAA=</key_for_session_key>
+ <is_key_private>N</is_key_private>
+ </info>
+ <notepads>
+ </notepads>
+ <order>
+ <hop>
+ <from>Sample data</from>
+ <to>Normalise months</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Normalise months</from>
+ <to>Verify</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <transform>
+ <name>Normalise months</name>
+ <type>Normaliser</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <typefield>Month</typefield>
+ <valuefield>Sales</valuefield>
+ <keys>
+ <key>
+ <name>Region</name>
+ </key>
+ <key>
+ <name>Year</name>
+ </key>
+ </keys>
+ <fields>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>272</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Sample data</name>
+ <type>DataGrid</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ <field>
+ <name>Region</name>
+ <type>String</type>
+ <format/>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <name>Jan</name>
+ <type>Integer</type>
+ <format/>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <name>Year</name>
+ <type>Integer</type>
+ <format/>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <name>Feb</name>
+ <type>Integer</type>
+ <format/>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <data>
+ <line>
+ <item>North</item>
+ <item>10</item>
+ <item>2026</item>
+ <item>20</item>
+ </line>
+ <line>
+ <item>South</item>
+ <item>30</item>
+ <item>2026</item>
+ <item/>
+ </line>
+ </data>
+ <attributes/>
+ <GUI>
+ <xloc>128</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Verify</name>
+ <type>Dummy</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ <GUI>
+ <xloc>416</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform_error_handling>
+ </transform_error_handling>
+ <attributes/>
+</pipeline>
diff --git
a/integration-tests/transforms/datasets/golden-row-normaliser-key-fields.csv
b/integration-tests/transforms/datasets/golden-row-normaliser-key-fields.csv
new file mode 100644
index 0000000000..8fc5cc75d6
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-row-normaliser-key-fields.csv
@@ -0,0 +1,5 @@
+Region,Year,Month,Sales
+North,2026,Jan,10
+North,2026,Feb,20
+South,2026,Jan,30
+South,2026,Feb,
diff --git a/integration-tests/transforms/main-0020-row-normaliser.hwf
b/integration-tests/transforms/main-0020-row-normaliser.hwf
index f0a0c7a785..7d8593139d 100644
--- a/integration-tests/transforms/main-0020-row-normaliser.hwf
+++ b/integration-tests/transforms/main-0020-row-normaliser.hwf
@@ -66,6 +66,9 @@ limitations under the License.
<test_name>
<name>0020-row-normaliser-all-fields UNIT</name>
</test_name>
+ <test_name>
+ <name>0020-row-normaliser-key-fields UNIT</name>
+ </test_name>
</test_names>
<parallel>N</parallel>
<xloc>272</xloc>
diff --git
a/integration-tests/transforms/metadata/dataset/golden-row-normaliser-key-fields.json
b/integration-tests/transforms/metadata/dataset/golden-row-normaliser-key-fields.json
new file mode 100644
index 0000000000..ee681281ae
--- /dev/null
+++
b/integration-tests/transforms/metadata/dataset/golden-row-normaliser-key-fields.json
@@ -0,0 +1,40 @@
+{
+ "base_filename": "golden-row-normaliser-key-fields.csv",
+ "name": "golden-row-normaliser-key-fields",
+ "description": "",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "Region"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "####0;-####0",
+ "field_name": "Year"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "Month"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "####0;-####0",
+ "field_name": "Sales"
+ }
+ ],
+ "folder_name": ""
+}
\ No newline at end of file
diff --git
a/integration-tests/transforms/metadata/unit-test/0020-row-normaliser-key-fields
UNIT.json
b/integration-tests/transforms/metadata/unit-test/0020-row-normaliser-key-fields
UNIT.json
new file mode 100644
index 0000000000..f6d38b6bbc
--- /dev/null
+++
b/integration-tests/transforms/metadata/unit-test/0020-row-normaliser-key-fields
UNIT.json
@@ -0,0 +1,38 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": true,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "Region",
+ "data_set_field": "Region"
+ },
+ {
+ "transform_field": "Year",
+ "data_set_field": "Year"
+ },
+ {
+ "transform_field": "Month",
+ "data_set_field": "Month"
+ },
+ {
+ "transform_field": "Sales",
+ "data_set_field": "Sales"
+ }
+ ],
+ "field_order": [],
+ "transform_name": "Verify",
+ "data_set_name": "golden-row-normaliser-key-fields"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0020-row-normaliser-key-fields UNIT",
+ "description": "",
+ "trans_test_tweaks": [],
+ "persist_filename": "",
+ "pipeline_filename": "./0020-row-normaliser-key-fields.hpl",
+ "test_type": "UNIT_TEST"
+}
\ No newline at end of file
diff --git
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/Normaliser.java
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/Normaliser.java
index 4ff54446a5..bfc1561b5c 100644
---
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/Normaliser.java
+++
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/Normaliser.java
@@ -110,7 +110,20 @@ public class Normaliser extends
BaseTransform<NormaliserMeta, NormaliserData> {
data.outputRowMeta = data.inputRowMeta.clone();
meta.getFields(data.outputRowMeta, getTransformName(), null, null, this,
metadataProvider);
- List<NormaliserField> fields =
meta.getFieldsToNormalise(data.inputRowMeta);
+ List<NormaliserField> fields =
meta.getFieldsToNormalise(data.inputRowMeta, this);
+
+ // The key fields are kept when every other field is normalised, so they
have to be there.
+ //
+ if (meta.isNormalisingAllFields()) {
+ for (String key : meta.getKeyFieldNames(this)) {
+ if (data.inputRowMeta.indexOfValue(key) < 0) {
+ logError(BaseMessages.getString(PKG,
"Normaliser.Log.CouldNotFindKeyFieldInRow", key));
+ setErrors(1);
+ stopAll();
+ return false;
+ }
+ }
+ }
String duplicate = NormaliserMeta.getDuplicateMapping(fields);
if (duplicate != null) {
diff --git
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialog.java
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialog.java
index 16ee7ec63b..2c342a7d89 100644
---
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialog.java
+++
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialog.java
@@ -18,53 +18,37 @@
package org.apache.hop.pipeline.transforms.normaliser;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
+import java.util.Map;
+import java.util.Set;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.TransformMeta;
import org.apache.hop.ui.core.ConstUi;
-import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
import org.apache.hop.ui.core.dialog.ErrorDialog;
-import org.apache.hop.ui.core.widget.ColumnInfo;
-import org.apache.hop.ui.core.widget.NamingSchemeTypes;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgetsAdapter;
import org.apache.hop.ui.core.widget.TableView;
import org.apache.hop.ui.hopgui.BackgroundThreadFacade;
import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
-import org.eclipse.swt.SWT;
-import org.eclipse.swt.events.ModifyListener;
-import org.eclipse.swt.layout.FormAttachment;
-import org.eclipse.swt.layout.FormData;
-import org.eclipse.swt.widgets.Label;
+import org.eclipse.swt.widgets.Control;
import org.eclipse.swt.widgets.Shell;
-import org.eclipse.swt.widgets.TableItem;
-import org.eclipse.swt.widgets.Text;
public class NormaliserDialog extends BaseTransformDialog {
private static final Class<?> PKG = NormaliserMeta.class;
- private static final int NAME_INDEX = 1;
-
- private static final int VALUE_INDEX = 2;
-
- private static final int NORM_INDEX = 3;
-
- private Text wTypefield;
-
- private Label wlValuefield;
-
- private Text wValuefield;
-
- private TableView wFields;
-
private final NormaliserMeta input;
- private ColumnInfo[] colinf;
+ private GuiCompositeWidgets widgets;
+ /** The names of the input fields, offered in the field name columns. */
private final List<String> inputFields = new ArrayList<>();
public NormaliserDialog(
@@ -79,167 +63,79 @@ public class NormaliserDialog extends BaseTransformDialog {
buildButtonBar().ok(e -> ok()).get(e -> get()).cancel(e ->
cancel()).build();
- ModifyListener lsMod = e -> input.setChanged();
changed = input.hasChanged();
- // Typefield line
- Label wlTypefield = new Label(shell, SWT.RIGHT);
- wlTypefield.setText(BaseMessages.getString(PKG,
"NormaliserDialog.TypeField.Label"));
- PropsUi.setLook(wlTypefield);
- FormData fdlTypefield = new FormData();
- fdlTypefield.left = new FormAttachment(0, 0);
- fdlTypefield.right = new FormAttachment(middle, -margin);
- fdlTypefield.top = new FormAttachment(wSpacer, margin);
- wlTypefield.setLayoutData(fdlTypefield);
- wTypefield = new Text(shell, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
- wTypefield.setText("");
- PropsUi.setLook(wTypefield);
- wTypefield.addModifyListener(lsMod);
- FormData fdTypefield = new FormData();
- fdTypefield.left = new FormAttachment(middle, 0);
- fdTypefield.top = new FormAttachment(wSpacer, margin);
- fdTypefield.right = new FormAttachment(100, 0);
- wTypefield.setLayoutData(fdTypefield);
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell, variables, wSpacer, wOk,
NormaliserMeta.GUI_PLUGIN_ELEMENT_PARENT_ID, input);
+ widgets.setWidgetsListener(
+ new GuiCompositeWidgetsAdapter() {
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control changedWidget,
String widgetId) {
+ input.setChanged();
+ if (NormaliserMeta.WIDGET_FIELDS.equals(widgetId)) {
+ enableKeyFields();
+ }
+ }
+ });
- // Value field line: used when no fields are listed
- wlValuefield = new Label(shell, SWT.RIGHT);
- wlValuefield.setText(BaseMessages.getString(PKG,
"NormaliserDialog.ValueField.Label"));
- wlValuefield.setToolTipText(BaseMessages.getString(PKG,
"NormaliserDialog.ValueField.Tooltip"));
- PropsUi.setLook(wlValuefield);
- FormData fdlValuefield = new FormData();
- fdlValuefield.left = new FormAttachment(0, 0);
- fdlValuefield.right = new FormAttachment(middle, -margin);
- fdlValuefield.top = new FormAttachment(wTypefield, margin);
- wlValuefield.setLayoutData(fdlValuefield);
- wValuefield = new Text(shell, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
- wValuefield.setToolTipText(BaseMessages.getString(PKG,
"NormaliserDialog.ValueField.Tooltip"));
- PropsUi.setLook(wValuefield);
- wValuefield.addModifyListener(lsMod);
- FormData fdValuefield = new FormData();
- fdValuefield.left = new FormAttachment(middle, 0);
- fdValuefield.top = new FormAttachment(wTypefield, margin);
- fdValuefield.right = new FormAttachment(100, 0);
- wValuefield.setLayoutData(fdValuefield);
+ for (TableView table :
+ List.of(table(NormaliserMeta.WIDGET_KEY_FIELDS),
table(NormaliserMeta.WIDGET_FIELDS))) {
+ table.getColumns()[0].setComboValueSupplier(this::inputFieldNames);
+ }
+ enableKeyFields();
+ searchInputFields();
- Label wlFields = new Label(shell, SWT.NONE);
- wlFields.setText(BaseMessages.getString(PKG,
"NormaliserDialog.Fields.Label"));
- PropsUi.setLook(wlFields);
- FormData fdlFields = new FormData();
- fdlFields.left = new FormAttachment(0, 0);
- fdlFields.top = new FormAttachment(wValuefield, margin);
- wlFields.setLayoutData(fdlFields);
+ input.setChanged(changed);
+ focusTransformName();
+ BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
- final int fieldsCols = 3;
- final int fieldsRows = input.getNormaliserFields().size();
- colinf = new ColumnInfo[fieldsCols];
- colinf[0] =
- new ColumnInfo(
- BaseMessages.getString(PKG,
"NormaliserDialog.ColumnInfo.Fieldname"),
- ColumnInfo.COLUMN_TYPE_CCOMBO,
- new String[] {""},
- false);
- colinf[1] =
- new ColumnInfo(
- BaseMessages.getString(PKG, "NormaliserDialog.ColumnInfo.Type"),
- ColumnInfo.COLUMN_TYPE_TEXT,
- false);
- colinf[2] =
- new ColumnInfo(
- BaseMessages.getString(PKG,
"NormaliserDialog.ColumnInfo.NewField"),
- ColumnInfo.COLUMN_TYPE_TEXT,
- false);
- colinf[2].setNamingSchemeType(NamingSchemeTypes.HOP_FIELD);
+ return transformName;
+ }
- // The value field is only used while no fields are listed
- ModifyListener lsFieldsMod =
- e -> {
- input.setChanged();
- enableValueField();
- };
- wFields =
- new TableView(
- variables,
- shell,
- SWT.BORDER | SWT.FULL_SELECTION | SWT.MULTI,
- colinf,
- fieldsRows,
- lsFieldsMod,
- props);
- wFields.setContentListener(e -> enableValueField());
+ private String[] inputFieldNames() {
+ synchronized (inputFields) {
+ return ConstUi.sortFieldNames(new ArrayList<>(inputFields));
+ }
+ }
- FormData fdFields = new FormData();
- fdFields.left = new FormAttachment(0, 0);
- fdFields.top = new FormAttachment(wlFields, margin);
- fdFields.right = new FormAttachment(100, 0);
- fdFields.bottom = new FormAttachment(wOk, -margin);
- wFields.setLayoutData(fdFields);
+ private TableView table(String widgetId) {
+ return (TableView) widgets.getWidgetsMap().get(widgetId);
+ }
- //
- // Search the fields in the background
+ /** The value field and the key fields only count when no fields are listed
to normalise. */
+ private void enableKeyFields() {
+ boolean noFieldsListed = table(NormaliserMeta.WIDGET_FIELDS).nrNonEmpty()
== 0;
+ table(NormaliserMeta.WIDGET_KEY_FIELDS).setEnabled(noFieldsListed);
+ for (Map<String, Control> controls : List.of(widgets.getLabelsMap(),
widgets.getWidgetsMap())) {
+ Control control = controls.get(NormaliserMeta.WIDGET_VALUE_FIELD);
+ if (control != null) {
+ control.setEnabled(noFieldsListed);
+ }
+ }
+ }
- final Runnable runnable =
+ private void searchInputFields() {
+ BackgroundThreadFacade.start(
() -> {
TransformMeta transformMeta =
pipelineMeta.findTransform(transformName);
if (transformMeta != null) {
try {
IRowMeta row = pipelineMeta.getPrevTransformFields(variables,
transformMeta);
-
- // Remember these fields...
+ List<String> names = new ArrayList<>();
for (int i = 0; i < row.size(); i++) {
- inputFields.add(row.getValueMeta(i).getName());
+ names.add(row.getValueMeta(i).getName());
+ }
+ synchronized (inputFields) {
+ inputFields.clear();
+ inputFields.addAll(names);
}
- setComboBoxes();
} catch (HopException e) {
logError(BaseMessages.getString(PKG,
"System.Dialog.GetFieldsFailed.Message"));
}
}
- };
- BackgroundThreadFacade.start(runnable);
-
- getData();
- focusTransformName();
- BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
-
- return transformName;
- }
-
- protected void setComboBoxes() {
- // Something was changed in the row.
- //
- String[] fieldNames = ConstUi.sortFieldNames(inputFields);
- colinf[0].setComboValues(fieldNames);
- }
-
- public void getData() {
- if (input.getTypeField() != null) {
- wTypefield.setText(input.getTypeField());
- }
- wValuefield.setText(input.getValueFieldName());
-
- for (int i = 0; i < input.getNormaliserFields().size(); i++) {
- TableItem item = wFields.table.getItem(i);
- NormaliserField field = input.getNormaliserFields().get(i);
- if (field.getName() != null) {
- item.setText(NAME_INDEX, field.getName());
- }
- if (field.getValue() != null) {
- item.setText(VALUE_INDEX, field.getValue());
- }
- if (field.getNorm() != null) {
- item.setText(NORM_INDEX, field.getNorm());
- }
- }
-
- wFields.setRowNums();
- wFields.optWidth(true);
- enableValueField();
- }
-
- /** The value field is only used when the fields table is empty. */
- private void enableValueField() {
- boolean noFieldsListed = wFields.nrNonEmpty() == 0;
- wlValuefield.setEnabled(noFieldsListed);
- wValuefield.setEnabled(noFieldsListed);
+ });
}
private void cancel() {
@@ -252,40 +148,40 @@ public class NormaliserDialog extends BaseTransformDialog
{
if (Utils.isEmpty(wTransformName.getText())) {
return;
}
-
- transformName = wTransformName.getText(); // return value
-
- input.setTypeField(wTypefield.getText());
-
- int i;
-
- int nrFields = wFields.nrNonEmpty();
- if (nrFields == 0) {
- // Listed fields leave the value field unused: keep what it was
- input.setValueField(wValuefield.getText());
- }
- input.getNormaliserFields().clear();
-
- for (i = 0; i < nrFields; i++) {
- TableItem item = wFields.getNonEmpty(i);
- NormaliserField field = new NormaliserField();
- field.setName(item.getText(NAME_INDEX));
- field.setValue(item.getText(VALUE_INDEX));
- field.setNorm(item.getText(NORM_INDEX));
- input.getNormaliserFields().add(field);
- }
-
+ transformName = wTransformName.getText();
+ applyWidgets();
dispose();
}
+ private void applyWidgets() {
+ widgets.getWidgetsContents(input,
NormaliserMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ input.getNormaliserFields().removeIf(field -> field == null ||
Utils.isEmpty(field.getName()));
+ input.getKeyFields().removeIf(field -> field == null ||
Utils.isEmpty(field.getName()));
+ }
+
+ /** Adds the input fields that are not listed yet, each under its own name
as type. */
private void get() {
try {
- IRowMeta r = pipelineMeta.getPrevTransformFields(variables,
transformName);
- if (r != null && !r.isEmpty()) {
- BaseTransformDialog.getFieldsFromPrevious(
- r, wFields, 1, new int[] {1, 2}, new int[] {}, -1, -1, null);
- enableValueField();
+ applyWidgets();
+ IRowMeta previous = pipelineMeta.getPrevTransformFields(variables,
transformName);
+ if (previous == null || previous.isEmpty()) {
+ return;
+ }
+ Set<String> present = new HashSet<>();
+ for (NormaliserField field : input.getNormaliserFields()) {
+ present.add(field.getName());
+ }
+ for (IValueMeta valueMeta : previous.getValueMetaList()) {
+ if (present.add(valueMeta.getName())) {
+ NormaliserField field = new NormaliserField();
+ field.setName(valueMeta.getName());
+ field.setValue(valueMeta.getName());
+ input.getNormaliserFields().add(field);
+ }
}
+ input.setChanged();
+ widgets.setWidgetsContents(input, shell,
NormaliserMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ enableKeyFields();
} catch (HopException ke) {
new ErrorDialog(
shell,
diff --git
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserField.java
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserField.java
index abf161d021..eb4daf05dc 100644
---
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserField.java
+++
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserField.java
@@ -17,23 +17,52 @@
package org.apache.hop.pipeline.transforms.normaliser;
-import java.util.Objects;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.hop.core.gui.plugin.GuiTableColumn;
+import org.apache.hop.core.gui.plugin.GuiTableColumnType;
import org.apache.hop.metadata.api.HopMetadataProperty;
+/** One input field to normalise: the type it is listed under and the new
field it fills. */
+@Getter
+@Setter
+@EqualsAndHashCode
public class NormaliserField {
+ @GuiTableColumn(
+ id = "name",
+ order = "10",
+ type = GuiTableColumnType.COMBO,
+ label = "i18n::NormaliserDialog.ColumnInfo.Fieldname",
+ variables = false,
+ width = 200)
@HopMetadataProperty(
key = "name",
injectionKey = "NAME",
injectionKeyDescription = "NormaliserMeta.Injection.NAME")
private String name;
+ @GuiTableColumn(
+ id = "value",
+ order = "20",
+ type = GuiTableColumnType.TEXT,
+ label = "i18n::NormaliserDialog.ColumnInfo.Type",
+ variables = false,
+ width = 160)
@HopMetadataProperty(
key = "value",
injectionKey = "VALUE",
injectionKeyDescription = "NormaliserMeta.Injection.VALUE")
private String value;
+ @GuiTableColumn(
+ id = "norm",
+ order = "30",
+ type = GuiTableColumnType.TEXT,
+ label = "i18n::NormaliserDialog.ColumnInfo.NewField",
+ variables = false,
+ width = 160)
@HopMetadataProperty(
key = "norm",
injectionKey = "NORMALISED",
@@ -49,87 +78,4 @@ public class NormaliserField {
this.value = field.value;
this.norm = field.norm;
}
-
- /**
- * @return the name
- */
- public String getName() {
- return name;
- }
-
- /**
- * @param name the name to set
- */
- public void setName(String name) {
- this.name = name;
- }
-
- /**
- * @return the value
- */
- public String getValue() {
- return value;
- }
-
- /**
- * @param value the value to set
- */
- public void setValue(String value) {
- this.value = value;
- }
-
- /**
- * @return the norm
- */
- public String getNorm() {
- return norm;
- }
-
- /**
- * @param norm the norm to set
- */
- public void setNorm(String norm) {
- this.norm = norm;
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(name, norm, value);
- }
-
- @Override
- public boolean equals(Object obj) {
- if (this == obj) {
- return true;
- }
- if (obj == null) {
- return false;
- }
- if (getClass() != obj.getClass()) {
- return false;
- }
- NormaliserField other = (NormaliserField) obj;
- if (name == null) {
- if (other.name != null) {
- return false;
- }
- } else if (!name.equals(other.name)) {
- return false;
- }
- if (norm == null) {
- if (other.norm != null) {
- return false;
- }
- } else if (!norm.equals(other.norm)) {
- return false;
- }
- if (value == null) {
- if (other.value != null) {
- return false;
- }
- } else if (!value.equals(other.value)) {
- return false;
- }
- return true;
- }
}
diff --git
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserKeyField.java
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserKeyField.java
new file mode 100644
index 0000000000..ac28177d1a
--- /dev/null
+++
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserKeyField.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.normaliser;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.hop.core.gui.plugin.GuiTableColumn;
+import org.apache.hop.core.gui.plugin.GuiTableColumnType;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+
+/**
+ * An input field that is kept as it is on every output row when no fields are
listed and every
+ * other input field is normalised.
+ */
+@Getter
+@Setter
+@EqualsAndHashCode
+public class NormaliserKeyField {
+
+ @GuiTableColumn(
+ id = "name",
+ order = "10",
+ type = GuiTableColumnType.COMBO,
+ label = "i18n::NormaliserDialog.ColumnInfo.Fieldname",
+ variables = true,
+ width = 200)
+ @HopMetadataProperty(
+ key = "name",
+ injectionKey = "KEY_NAME",
+ injectionKeyDescription = "NormaliserMeta.Injection.KEY_NAME")
+ private String name;
+
+ public NormaliserKeyField() {
+ // Do nothing
+ }
+
+ public NormaliserKeyField(String name) {
+ this.name = name;
+ }
+}
diff --git
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMeta.java
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMeta.java
index 5b21328231..5b02494c03 100644
---
a/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMeta.java
+++
b/plugins/transforms/normaliser/src/main/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMeta.java
@@ -18,8 +18,10 @@
package org.apache.hop.pipeline.transforms.normaliser;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
import java.util.Objects;
+import java.util.Set;
import lombok.Getter;
import lombok.Setter;
import org.apache.hop.core.CheckResult;
@@ -27,6 +29,10 @@ import org.apache.hop.core.Const;
import org.apache.hop.core.ICheckResult;
import org.apache.hop.core.annotations.Transform;
import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.value.ValueMetaString;
@@ -71,6 +77,7 @@ DATE PRODUCT Sales Number
categoryDescription =
"i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.Transform",
keywords = "i18n::NormaliserMeta.keyword",
documentationUrl = "/pipeline/transforms/rownormaliser.html")
+@GuiPlugin
@Getter
@Setter
public class NormaliserMeta extends BaseTransformMeta<Normaliser,
NormaliserData> {
@@ -78,30 +85,91 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
public static final String DEFAULT_VALUE_FIELD = "value";
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = "NormaliserDialog";
+ public static final String WIDGET_TYPE_FIELD = "typeField";
+ public static final String WIDGET_VALUE_FIELD = "valueField";
+ public static final String WIDGET_KEY_FIELDS = "keyFields";
+ public static final String WIDGET_FIELDS = "normaliserFields";
+
+ private static final String GROUP_OUTPUT =
"i18n::NormaliserMeta.Group.Output";
+ private static final String GROUP_KEYS =
"i18n::NormaliserMeta.Group.KeyFields";
+ private static final String GROUP_FIELDS =
"i18n::NormaliserMeta.Group.Fields";
+
/** Name of the new type-field */
+ @GuiWidgetElement(
+ id = WIDGET_TYPE_FIELD,
+ order = "0100",
+ type = GuiElementType.TEXT,
+ label = "i18n::NormaliserDialog.TypeField.Label",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_OUTPUT,
+ groupOrder = "0100")
@HopMetadataProperty(
key = "typefield",
injectionKey = "TYPEFIELD",
injectionKeyDescription = "NormaliserMeta.Injection.TYPEFIELD")
private String typeField;
- @HopMetadataProperty(
- groupKey = "fields",
- key = "field",
- injectionGroupKey = "FIELDS",
- injectionGroupDescription = "NormaliserMeta.Injection.FIELDS")
- private List<NormaliserField> normaliserFields;
-
/** Name of the field that holds the values when every input field is
normalised. */
+ @GuiWidgetElement(
+ id = WIDGET_VALUE_FIELD,
+ order = "0200",
+ type = GuiElementType.TEXT,
+ label = "i18n::NormaliserDialog.ValueField.Label",
+ toolTip = "i18n::NormaliserDialog.ValueField.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_OUTPUT,
+ groupOrder = "0100")
@HopMetadataProperty(
key = "valuefield",
injectionKey = "VALUEFIELD",
injectionKeyDescription = "NormaliserMeta.Injection.VALUEFIELD")
private String valueField;
+ /**
+ * The fields kept as they are on every row when no fields are listed and
every other input field
+ * is normalised.
+ */
+ @GuiWidgetElement(
+ id = WIDGET_KEY_FIELDS,
+ order = "0300",
+ type = GuiElementType.TABLE,
+ toolTip = "i18n::NormaliserDialog.KeyFields.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_KEYS,
+ groupOrder = "0200",
+ tableRows = 3)
+ @HopMetadataProperty(
+ groupKey = "keys",
+ key = "key",
+ injectionGroupKey = "KEYS",
+ injectionGroupDescription = "NormaliserMeta.Injection.KEYS")
+ private List<NormaliserKeyField> keyFields;
+
+ @GuiWidgetElement(
+ id = WIDGET_FIELDS,
+ order = "0400",
+ type = GuiElementType.TABLE,
+ toolTip = "i18n::NormaliserDialog.Fields.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FIELDS,
+ groupOrder = "0300",
+ tableRows = 8)
+ @HopMetadataProperty(
+ groupKey = "fields",
+ key = "field",
+ injectionGroupKey = "FIELDS",
+ injectionGroupDescription = "NormaliserMeta.Injection.FIELDS")
+ private List<NormaliserField> normaliserFields;
+
public NormaliserMeta() {
super();
this.normaliserFields = new ArrayList<>();
+ this.keyFields = new ArrayList<>();
}
@Override
@@ -109,6 +177,7 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
this.typeField = "typefield";
this.valueField = DEFAULT_VALUE_FIELD;
this.normaliserFields = new ArrayList<>();
+ this.keyFields = new ArrayList<>();
}
/**
@@ -130,19 +199,45 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
}
/**
- * The fields to normalise: the listed fields, or with an empty list every
input field. An input
- * field normalised that way gets its own name as type and puts its value in
the value field.
+ * The names of the key fields, with variables resolved, in the order they
are listed.
+ *
+ * @param variables to resolve the names with
+ */
+ public List<String> getKeyFieldNames(IVariables variables) {
+ List<String> names = new ArrayList<>();
+ if (keyFields != null) {
+ for (NormaliserKeyField keyField : keyFields) {
+ if (!Utils.isEmpty(keyField.getName())) {
+ names.add(variables.resolve(keyField.getName()));
+ }
+ }
+ }
+ return names;
+ }
+
+ /**
+ * The fields to normalise: the listed fields, or with an empty list every
input field that is not
+ * a key field. An input field normalised that way gets its own name as type
and puts its value in
+ * the value field.
*
* @param inputRowMeta the fields entering the transform
+ * @param variables to resolve the names of the key fields with
* @return the fields to normalise
*/
- public List<NormaliserField> getFieldsToNormalise(IRowMeta inputRowMeta) {
+ public List<NormaliserField> getFieldsToNormalise(IRowMeta inputRowMeta,
IVariables variables) {
if (!isNormalisingAllFields()) {
return normaliserFields;
}
+ Set<String> keys = new HashSet<>();
+ for (String key : getKeyFieldNames(variables)) {
+ keys.add(key.toLowerCase());
+ }
String norm = getValueFieldName();
List<NormaliserField> fields = new ArrayList<>(inputRowMeta.size());
for (IValueMeta valueMeta : inputRowMeta.getValueMetaList()) {
+ if (keys.contains(valueMeta.getName().toLowerCase())) {
+ continue;
+ }
NormaliserField field = new NormaliserField();
field.setName(valueMeta.getName());
field.setValue(valueMeta.getName());
@@ -155,7 +250,7 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
/**
* The names of the normalised fields, in output order: in the order they
first appear.
*
- * @param fields the fields to normalise, see {@link
#getFieldsToNormalise(IRowMeta)}
+ * @param fields the fields to normalise, see {@link
#getFieldsToNormalise(IRowMeta, IVariables)}
*/
public static List<String> getNormalisedFieldNames(List<NormaliserField>
fields) {
List<String> names = new ArrayList<>();
@@ -177,13 +272,14 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
* transform that serializes or renders it. See issue #3636.
*
* @param inputRowMeta the fields entering the transform
+ * @param variables to resolve the names of the key fields with
* @return one value metadata per name of {@link
#getNormalisedFieldNames(List)}, in the same
* order
* @throws HopTransformException when the first input field of a normalised
field is missing
*/
- public List<IValueMeta> getNormalisedValueMetas(IRowMeta inputRowMeta)
+ public List<IValueMeta> getNormalisedValueMetas(IRowMeta inputRowMeta,
IVariables variables)
throws HopTransformException {
- return getNormalisedValueMetas(inputRowMeta,
getFieldsToNormalise(inputRowMeta));
+ return getNormalisedValueMetas(inputRowMeta,
getFieldsToNormalise(inputRowMeta, variables));
}
private List<IValueMeta> getNormalisedValueMetas(
@@ -256,7 +352,7 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
throws HopTransformException {
boolean allFields = isNormalisingAllFields();
- List<NormaliserField> fields = getFieldsToNormalise(row);
+ List<NormaliserField> fields = getFieldsToNormalise(row, variables);
int maxlen = 0;
for (NormaliserField field : fields) {
@@ -334,7 +430,8 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
transformMeta);
remarks.add(cr);
- List<NormaliserField> fields = getFieldsToNormalise(prev);
+ List<NormaliserField> fields = getFieldsToNormalise(prev, variables);
+ List<String> keyNames = getKeyFieldNames(variables);
if (isNormalisingAllFields()) {
remarks.add(
new CheckResult(
@@ -342,9 +439,27 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
BaseMessages.getString(
PKG,
"NormaliserMeta.CheckResult.AllInputFieldsNormalised",
- prev.size() + "",
+ fields.size() + "",
getValueFieldName()),
transformMeta));
+ List<String> missingKeys =
+ keyNames.stream().filter(key -> prev.searchValueMeta(key) ==
null).toList();
+ if (!missingKeys.isEmpty()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
+ "NormaliserMeta.CheckResult.KeyFieldsNotFound",
+ String.join(", ", missingKeys)),
+ transformMeta));
+ }
+ } else if (!keyNames.isEmpty()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"NormaliserMeta.CheckResult.KeyFieldsIgnored"),
+ transformMeta));
}
boolean first = true;
@@ -453,7 +568,7 @@ public class NormaliserMeta extends
BaseTransformMeta<Normaliser, NormaliserData
* Two input fields filling the same normalised field on the same row leave
no room for one of
* them.
*
- * @param fields the fields to normalise, see {@link
#getFieldsToNormalise(IRowMeta)}
+ * @param fields the fields to normalise, see {@link
#getFieldsToNormalise(IRowMeta, IVariables)}
* @return a description of the first such pair, or null when there is none
*/
public static String getDuplicateMapping(List<NormaliserField> fields) {
diff --git
a/plugins/transforms/normaliser/src/main/resources/org/apache/hop/pipeline/transforms/normaliser/messages/messages_en_US.properties
b/plugins/transforms/normaliser/src/main/resources/org/apache/hop/pipeline/transforms/normaliser/messages/messages_en_US.properties
index a5a8d7dabe..c10146d837 100644
---
a/plugins/transforms/normaliser/src/main/resources/org/apache/hop/pipeline/transforms/normaliser/messages/messages_en_US.properties
+++
b/plugins/transforms/normaliser/src/main/resources/org/apache/hop/pipeline/transforms/normaliser/messages/messages_en_US.properties
@@ -17,6 +17,7 @@
Normaliser.Description=De-normalised information can be normalised using this
transform type.
Normaliser.Log.CouldNotFindFieldInRow=Couldn''t find field [{0} in row!
+Normaliser.Log.CouldNotFindKeyFieldInRow=Couldn''t find key field [{0}] in row!
Normaliser.Log.LineNumber=linenr
Normaliser.Name=Row normaliser
NormaliserDialog.ColumnInfo.Fieldname=Fieldname
@@ -25,24 +26,33 @@ NormaliserDialog.ColumnInfo.Type=Type
NormaliserDialog.FailedToGetFields.DialogMessage=Unable to get fields from
previous transforms because of an error
NormaliserDialog.FailedToGetFields.DialogTitle=Get fields failed
NormaliserDialog.Fields.Label=Fields
+NormaliserDialog.Fields.Tooltip=The fields to normalise, each under a type and
into a new field. Leave this empty to normalise every input field that is not a
key field into the value field.
NormaliserDialog.GetFields.Button=\ &Get Fields
+NormaliserDialog.KeyFields.Tooltip=Used when no fields are listed below: these
fields are kept as they are on every row, and every other input field is
normalised.
NormaliserDialog.Shell.Title=Row normaliser
NormaliserDialog.TransformName.Label=Transform name
NormaliserDialog.TypeField.Label=Type field
NormaliserDialog.ValueField.Label=Value field
NormaliserDialog.ValueField.Tooltip=Used when the fields table is empty: every
input field is normalised then, with its name in the type field and its value
in this field.
NormaliserMeta.CheckResult.AllFieldsFound=All fields to normalise are found in
the input stream.
-NormaliserMeta.CheckResult.AllInputFieldsNormalised=No fields are listed, so
all {0} input fields are normalised into field {1}.
+NormaliserMeta.CheckResult.AllInputFieldsNormalised=No fields are listed, so
{0} input fields are normalised into field {1}. Key fields are kept.
NormaliserMeta.CheckResult.CouldNotReadFieldsFromPreviousTransform=Couldn''t
read fields from the previous transform.
NormaliserMeta.CheckResult.DuplicateMapping=Fields {0} and {1} both fill
normalised field {2} for type {3}. Every row has room for only one of them.
NormaliserMeta.CheckResult.FieldsNotFound=Fields to normalise, not found in
input stream:
+NormaliserMeta.CheckResult.KeyFieldsIgnored=Key fields are only used when no
fields are listed to normalise. They are ignored now.
+NormaliserMeta.CheckResult.KeyFieldsNotFound=Key fields not found in the input
stream: {0}
NormaliserMeta.CheckResult.MixedTypes=Normalised field {0} is filled from
fields of different types, so it is a String holding the text of each value.
NormaliserMeta.CheckResult.NoInputReceivedError=No input received from other
transforms!
NormaliserMeta.CheckResult.SameNameAsTypeField=Normalised field {0} has the
same name as the type field, so it is renamed in the output, to {0}_1 for
example.
NormaliserMeta.CheckResult.TransformReceivingFieldsOK=Transform is connected
to previous one, receiving {0} fields
NormaliserMeta.CheckResult.TransformReceivingInfoOK=Transform is receiving
info from other transforms.
NormaliserMeta.Exception.UnableToFindField=Unable to find field from upstream
transform: {0}
+NormaliserMeta.Group.Fields=Fields to normalise
+NormaliserMeta.Group.KeyFields=Key fields
+NormaliserMeta.Group.Output=Output
NormaliserMeta.Injection.FIELDS=All the fields
+NormaliserMeta.Injection.KEYS=The key fields, kept on every row when no fields
are listed
+NormaliserMeta.Injection.KEY_NAME=Key field name
NormaliserMeta.Injection.NAME=Input field name
NormaliserMeta.Injection.NORMALISED=Normalised field name
NormaliserMeta.Injection.TYPEFIELD=The name of the type field
diff --git
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialogTest.java
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialogTest.java
new file mode 100644
index 0000000000..c252daec85
--- /dev/null
+++
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserDialogTest.java
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.normaliser;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.hop.core.gui.plugin.GuiRegistry;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.plugins.PluginRegistry;
+import org.apache.hop.core.plugins.TransformPluginType;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.ui.testing.SwtBotTestBase;
+import org.eclipse.swt.graphics.Rectangle;
+import org.eclipse.swt.widgets.Control;
+import org.eclipse.swtbot.swt.finder.SWTBot;
+import org.eclipse.swtbot.swt.finder.widgets.SWTBotButton;
+import org.eclipse.swtbot.swt.finder.widgets.SWTBotTable;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+/** SWTBot coverage for the grouped {@link NormaliserDialog} (#9398). */
+@Tag("uitest")
+class NormaliserDialogTest extends SwtBotTestBase {
+
+ private static final String TRANSFORM_NAME = "row normaliser";
+
+ private static final String SHELL_TITLE =
+ BaseMessages.getString(NormaliserMeta.class,
"NormaliserDialog.Shell.Title");
+
+ /**
+ * The dialog builds its boxes from the annotated widgets on {@link
NormaliserMeta}. The unit-test
+ * JVM does not always run the {@code GuiPluginType} scan, so register them
the way it does.
+ */
+ @BeforeAll
+ static void registerMetaWidgets() {
+ GuiRegistry registry = GuiRegistry.getInstance();
+ if (registry.findGuiElements(
+ NormaliserMeta.class.getName(),
NormaliserMeta.GUI_PLUGIN_ELEMENT_PARENT_ID)
+ != null) {
+ return;
+ }
+ for (Field field : NormaliserMeta.class.getDeclaredFields()) {
+ GuiWidgetElement element = field.getAnnotation(GuiWidgetElement.class);
+ if (element != null) {
+ registry.addGuiWidgetElement(NormaliserMeta.class.getName(), element,
field);
+ }
+ }
+ }
+
+ /** Opening the dialog and pressing OK keeps every setting of the "all
fields" mode. */
+ @Test
+ void okKeepsTheTypeValueAndKeyFields() {
+ NormaliserMeta meta = allFieldsMeta();
+
+ withDialog(
+ parent -> new NormaliserDialog(parent, new Variables(), meta,
pipelineWith(meta)).open(),
+ bot -> {
+ SWTBot dialog = bot.shell(SHELL_TITLE).activate().bot();
+ assertEquals(
+ "Month",
dialog.textWithLabel(label("NormaliserDialog.TypeField.Label")).getText());
+ assertEquals(
+ "Sales",
dialog.textWithLabel(label("NormaliserDialog.ValueField.Label")).getText());
+ assertEquals("Region", keyTable(dialog).cell(0, 1));
+ ok(dialog);
+ });
+
+ assertEquals("Month", meta.getTypeField());
+ assertEquals("Sales", meta.getValueField());
+ assertEquals(
+ List.of("Region", "Year"),
+
meta.getKeyFields().stream().map(NormaliserKeyField::getName).toList());
+ assertTrue(meta.getNormaliserFields().isEmpty());
+ }
+
+ /** Listed fields come back unchanged after OK. */
+ @Test
+ void okKeepsTheListedFields() {
+ NormaliserMeta meta = new NormaliserMeta();
+ meta.setDefault();
+ meta.setNormaliserFields(
+ new ArrayList<>(
+ List.of(field("PR1_SL", "Product1", "Sales"), field("PR1_NR",
"Product1", "Number"))));
+
+ withDialog(
+ parent -> new NormaliserDialog(parent, new Variables(), meta,
pipelineWith(meta)).open(),
+ bot -> ok(bot.shell(SHELL_TITLE).activate().bot()));
+
+ assertEquals(
+ List.of(field("PR1_SL", "Product1", "Sales"), field("PR1_NR",
"Product1", "Number")),
+ meta.getNormaliserFields());
+ }
+
+ /** An edited value field is stored. */
+ @Test
+ void okStoresAnEditedValueField() {
+ NormaliserMeta meta = allFieldsMeta();
+
+ withDialog(
+ parent -> new NormaliserDialog(parent, new Variables(), meta,
pipelineWith(meta)).open(),
+ bot -> {
+ SWTBot dialog = bot.shell(SHELL_TITLE).activate().bot();
+
dialog.textWithLabel(label("NormaliserDialog.ValueField.Label")).setText("Amount");
+ ok(dialog);
+ });
+
+ assertEquals("Amount", meta.getValueField());
+ }
+
+ /** The value and key fields only count without listed fields, so they
follow the fields table. */
+ @Test
+ void theValueAndKeyFieldsAreOnlyEnabledWhileNoFieldsAreListed() {
+ NormaliserMeta empty = allFieldsMeta();
+ withDialog(
+ parent -> new NormaliserDialog(parent, new Variables(), empty,
pipelineWith(empty)).open(),
+ bot -> {
+ SWTBot dialog = bot.shell(SHELL_TITLE).activate().bot();
+ assertTrue(keyTable(dialog).isEnabled(), "no fields listed: key
fields in use");
+ assertEquals(
+ true,
+
dialog.textWithLabel(label("NormaliserDialog.ValueField.Label")).isEnabled(),
+ "no fields listed: value field in use");
+ ok(dialog);
+ });
+
+ NormaliserMeta listed = allFieldsMeta();
+ listed.setNormaliserFields(new ArrayList<>(List.of(field("a", "A",
"value"))));
+ withDialog(
+ parent ->
+ new NormaliserDialog(parent, new Variables(), listed,
pipelineWith(listed)).open(),
+ bot -> {
+ SWTBot dialog = bot.shell(SHELL_TITLE).activate().bot();
+ assertFalse(keyTable(dialog).isEnabled(), "fields listed: key fields
not used");
+ assertEquals(
+ false,
+
dialog.textWithLabel(label("NormaliserDialog.ValueField.Label")).isEnabled(),
+ "fields listed: value field not used");
+ ok(dialog);
+ });
+ }
+
+ /** The OK button stays below the fields table, the last box, rather than on
top of it. */
+ @Test
+ void theButtonsStayBelowTheLastTable() {
+ NormaliserMeta meta = allFieldsMeta();
+ AtomicReference<Rectangle> table = new AtomicReference<>();
+ AtomicReference<Rectangle> button = new AtomicReference<>();
+
+ withDialog(
+ parent -> new NormaliserDialog(parent, new Variables(), meta,
pipelineWith(meta)).open(),
+ bot -> {
+ SWTBot dialog = bot.shell(SHELL_TITLE).activate().bot();
+ SWTBotTable fields = dialog.table(1);
+ SWTBotButton okButton =
dialog.button(buttonLabel("System.Button.OK"));
+ display.syncExec(
+ () -> {
+ table.set(onDisplay(fields.widget));
+ button.set(onDisplay(okButton.widget));
+ });
+ ok(dialog);
+ });
+
+ assertNotNull(table.get());
+ assertTrue(
+ table.get().y + table.get().height <= button.get().y,
+ "fields table " + table.get() + " must end above the OK button " +
button.get());
+ }
+
+ private static Rectangle onDisplay(Control control) {
+ return control.getDisplay().map(control.getParent(), null,
control.getBounds());
+ }
+
+ private static SWTBotTable keyTable(SWTBot dialog) {
+ return dialog.table(0);
+ }
+
+ private static void ok(SWTBot dialog) {
+ dialog.button(buttonLabel("System.Button.OK")).click();
+ }
+
+ private static String label(String key) {
+ return BaseMessages.getString(NormaliserMeta.class, key);
+ }
+
+ private static NormaliserMeta allFieldsMeta() {
+ NormaliserMeta meta = new NormaliserMeta();
+ meta.setDefault();
+ meta.setTypeField("Month");
+ meta.setValueField("Sales");
+ meta.setKeyFields(
+ new ArrayList<>(List.of(new NormaliserKeyField("Region"), new
NormaliserKeyField("Year"))));
+ return meta;
+ }
+
+ private static NormaliserField field(String name, String type, String norm) {
+ NormaliserField field = new NormaliserField();
+ field.setName(name);
+ field.setValue(type);
+ field.setNorm(norm);
+ return field;
+ }
+
+ private static PipelineMeta pipelineWith(NormaliserMeta meta) {
+ String pluginId =
PluginRegistry.getInstance().getPluginId(TransformPluginType.class, meta);
+ assertNotNull(pluginId, "Row normaliser transform plugin must be
registered");
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.addTransform(new TransformMeta(pluginId, TRANSFORM_NAME,
meta));
+ return pipelineMeta;
+ }
+}
diff --git
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaInjectionTest.java
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaInjectionTest.java
index 23efd9acc9..56159ae932 100644
---
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaInjectionTest.java
+++
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaInjectionTest.java
@@ -36,6 +36,7 @@ class NormaliserMetaInjectionTest extends
BaseMetadataInjectionTestJunit5<Normal
void test() throws Exception {
check("TYPEFIELD", () -> meta.getTypeField());
check("VALUEFIELD", () -> meta.getValueField());
+ check("KEY_NAME", () -> meta.getKeyFields().get(0).getName());
check("NAME", () -> meta.getNormaliserFields().get(0).getName());
check("VALUE", () -> meta.getNormaliserFields().get(0).getValue());
check("NORMALISED", () -> meta.getNormaliserFields().get(0).getNorm());
diff --git
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaTest.java
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaTest.java
index 1fe2c09fa8..38539b434b 100644
---
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaTest.java
+++
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserMetaTest.java
@@ -63,6 +63,8 @@ class NormaliserMetaTest {
assertEquals("type", meta.getTypeField());
assertEquals("content", meta.getValueField());
+ assertEquals(1, meta.getKeyFields().size());
+ assertEquals("id", meta.getKeyFields().get(0).getName());
assertEquals(4, meta.getNormaliserFields().size());
NormaliserField last = meta.getNormaliserFields().get(3);
assertEquals("fieldD", last.getName());
@@ -360,6 +362,65 @@ class NormaliserMetaTest {
remarks.toString());
}
+ /** Issue #9398: key fields stay as they are, every other input field is
normalised. */
+ @Test
+ void keyFieldsAreKeptWhenEveryOtherFieldIsNormalised() throws Exception {
+ IRowMeta row = new RowMeta();
+ row.addValueMeta(new ValueMetaString("Region"));
+ row.addValueMeta(new ValueMetaInteger("Jan"));
+ row.addValueMeta(new ValueMetaInteger("Year"));
+ row.addValueMeta(new ValueMetaInteger("Feb"));
+
+ NormaliserMeta meta = meta();
+ meta.setTypeField("Month");
+ meta.setValueField("Sales");
+ meta.setKeyFields(new ArrayList<>(List.of(new NormaliserKeyField("year"),
key("${KEY}"))));
+ Variables variables = new Variables();
+ variables.setVariable("KEY", "Region");
+ meta.getFields(row, "normaliser", null, null, variables, null);
+
+ assertArrayEquals(new String[] {"Region", "Year", "Month", "Sales"},
row.getFieldNames());
+ assertEquals(IValueMeta.TYPE_INTEGER, row.getValueMeta(1).getType());
+ assertEquals(IValueMeta.TYPE_INTEGER, row.getValueMeta(3).getType());
+ }
+
+ @Test
+ void checkReportsMissingKeyFields() {
+ IRowMeta prev = new RowMeta();
+ prev.addValueMeta(new ValueMetaInteger("i1"));
+
+ NormaliserMeta meta = meta();
+ meta.setKeyFields(new ArrayList<>(List.of(key("missing"))));
+ List<ICheckResult> remarks = check(meta, prev);
+
+ assertTrue(
+ remarks.stream()
+ .anyMatch(
+ r ->
+ r.getType() == ICheckResult.TYPE_RESULT_ERROR
+ && r.getText().contains("missing")),
+ remarks.toString());
+ }
+
+ @Test
+ void checkWarnsThatKeyFieldsAreIgnoredWhenFieldsAreListed() {
+ IRowMeta prev = new RowMeta();
+ prev.addValueMeta(new ValueMetaInteger("i1"));
+ prev.addValueMeta(new ValueMetaInteger("i2"));
+
+ NormaliserMeta meta = meta(field("i1", "A", "X"));
+ meta.setKeyFields(new ArrayList<>(List.of(key("i2"))));
+ List<ICheckResult> remarks = check(meta, prev);
+
+ assertTrue(
+ remarks.stream().anyMatch(r -> r.getType() ==
ICheckResult.TYPE_RESULT_WARNING),
+ remarks.toString());
+ }
+
+ private static NormaliserKeyField key(String name) {
+ return new NormaliserKeyField(name);
+ }
+
private static List<ICheckResult> check(NormaliserMeta meta, IRowMeta prev) {
List<ICheckResult> remarks = new ArrayList<>();
meta.check(
diff --git
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserTest.java
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserTest.java
index a91e3cc0cb..7a85488512 100644
---
a/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserTest.java
+++
b/plugins/transforms/normaliser/src/test/java/org/apache/hop/pipeline/transforms/normaliser/NormaliserTest.java
@@ -503,6 +503,77 @@ class NormaliserTest {
assertEquals(declared.getValueMeta(1).getType(),
output.rowMeta.getValueMeta(1).getType());
}
+ /**
+ * Issue #9398, the Qlik Sense crosstable: the key fields stay on every row,
in input order, and
+ * every other input field becomes a row of its own.
+ */
+ @Test
+ void keyFieldsAreKeptOnEveryRowAndTheOtherFieldsAreNormalised() throws
Exception {
+ IRowMeta input = new RowMeta();
+ input.addValueMeta(new ValueMetaString("Region"));
+ input.addValueMeta(new ValueMetaInteger("Jan"));
+ input.addValueMeta(new ValueMetaInteger("Year"));
+ input.addValueMeta(new ValueMetaInteger("Feb"));
+
+ NormaliserMeta meta = meta("Month");
+ meta.setValueField("Sales");
+ meta.setKeyFields(
+ new ArrayList<>(List.of(new NormaliserKeyField("Year"), new
NormaliserKeyField("region"))));
+
+ Output output = normalise(meta, input, "North", 10L, 2026L, 20L);
+
+ assertArrayEquals(
+ new String[] {"Region", "Year", "Month", "Sales"},
output.rowMeta.getFieldNames());
+ assertEquals(IValueMeta.TYPE_INTEGER,
output.rowMeta.getValueMeta(3).getType());
+ assertEquals(2, output.rows.size());
+ assertArrayEquals(new Object[] {"North", 2026L, "Jan", 10L},
output.rows.get(0));
+ assertArrayEquals(new Object[] {"North", 2026L, "Feb", 20L},
output.rows.get(1));
+ }
+
+ /** Key fields only count when no fields are listed: listed fields work as
they always did. */
+ @Test
+ void keyFieldsAreIgnoredWhenFieldsAreListed() throws Exception {
+ IRowMeta input = new RowMeta();
+ for (String name : new String[] {"id", "a", "b"}) {
+ input.addValueMeta(new ValueMetaString(name));
+ }
+
+ NormaliserMeta meta = meta("type", field("a", "A", "value"));
+ meta.setKeyFields(new ArrayList<>(List.of(new NormaliserKeyField("b"))));
+
+ Output output = normalise(meta, input, "1", "a", "b");
+
+ assertArrayEquals(new String[] {"id", "b", "type", "value"},
output.rowMeta.getFieldNames());
+ assertArrayEquals(new Object[] {"1", "b", "A", "a"}, output.rows.get(0));
+ }
+
+ /** A key field that is not in the input stops the transform with an error.
*/
+ @Test
+ void aKeyFieldMissingFromTheInputStopsTheTransform() throws Exception {
+ IRowMeta input = new RowMeta();
+ input.addValueMeta(new ValueMetaString("text"));
+
+ NormaliserMeta meta = meta("type");
+ meta.setKeyFields(new ArrayList<>(List.of(new
NormaliserKeyField("missing"))));
+ when(helper.transformMeta.getTransform()).thenReturn(meta);
+ Normaliser transform =
+ new Normaliser(
+ helper.transformMeta,
+ meta,
+ new NormaliserData(),
+ 0,
+ helper.pipelineMeta,
+ helper.pipeline);
+ transform.init();
+ transform.setInputRowMeta(input);
+ transform.setOutputRowSets(Collections.singletonList(new
BlockingRowSet(10)));
+ Normaliser spied = spy(transform);
+ doReturn(new Object[] {"a"}).when(spied).getRow();
+
+ assertFalse(spied.processRow());
+ assertEquals(1, spied.getErrors());
+ }
+
/** Without input rows the transform writes nothing and finishes without an
error. */
@Test
void noInputRowsWriteNothing() throws Exception {
diff --git
a/plugins/transforms/normaliser/src/test/resources/normaliser-transform.xml
b/plugins/transforms/normaliser/src/test/resources/normaliser-transform.xml
index 47598c3476..0693fa18b3 100644
--- a/plugins/transforms/normaliser/src/test/resources/normaliser-transform.xml
+++ b/plugins/transforms/normaliser/src/test/resources/normaliser-transform.xml
@@ -25,6 +25,11 @@
</partitioning>
<typefield>type</typefield>
<valuefield>content</valuefield>
+ <keys>
+ <key>
+ <name>id</name>
+ </key>
+ </keys>
<fields>
<field>
<name>fieldA</name>