This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new c6b2ffe78f Issue #3460 : Add hash-code checksum and read it in
Combination lookup (#8681)
c6b2ffe78f is described below
commit c6b2ffe78fc1c0a3f0ba95798b0359e5ebfc9ebb
Author: Matt Casters <[email protected]>
AuthorDate: Thu Oct 1 09:11:09 2026 +0200
Issue #3460 : Add hash-code checksum and read it in Combination lookup
(#8681)
* Issue #3460 : Add hash-code checksum and read it in Combination lookup
* Issue #3460 : Point checksum documentation links at the transform pages
---
.../pages/pipeline/transforms/addchecksum.adoc | 11 +-
.../pipeline/transforms/combinationlookup.adoc | 6 +
.../hop/pipeline/transforms/checksum/CheckSum.java | 27 +++-
.../transforms/checksum/CheckSumDialog.java | 34 +++--
.../pipeline/transforms/checksum/CheckSumMeta.java | 13 +-
.../checksum/messages/messages_en_US.properties | 4 +-
.../transforms/checksum/CheckSumMetaTest.java | 26 ++++
.../pipeline/transforms/checksum/CheckSumTest.java | 28 ++++
.../combinationlookup/CombinationLookup.java | 24 ++-
.../combinationlookup/CombinationLookupData.java | 1 +
.../combinationlookup/CombinationLookupDialog.java | 46 +++++-
.../combinationlookup/CombinationLookupMeta.java | 46 ++++++
.../messages/messages_en_US.properties | 6 +
.../CombinationLookupHashTest.java | 167 +++++++++++++++++++++
.../CombinationLookupMetaTest.java | 28 +++-
15 files changed, 434 insertions(+), 33 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
index cb7e71508c..1806e7041d 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/addchecksum.adoc
@@ -40,14 +40,22 @@ These are the types that are available:
* MD5 : Message Digest algorithm 5 : http://en.wikipedia.org/wiki/MD5
* SHA-1 : Secure Hash Algorithm 1 : http://en.wikipedia.org/wiki/SHA-1
* SHA-256 : Secure Hash Algorithm 2 : http://en.wikipedia.org/wiki/SHA-256
-|Result Type|Some checksum types allow to set different result types: String,
Hexadecimal and Binary
+* Hash code : The same signed integer that
xref:pipeline/transforms/combinationlookup.adoc[Combination lookup/update]
calculates from the native field values.
+The result is that 32-bit hash stored in a 64-bit integer, so negative values
are possible.
+Nulls are included and field order matters.
+Prefix, separator and suffix are ignored, because they are not part of the
combination-lookup calculation.
+|Result Type|Some checksum types allow to set different result types: String,
Hexadecimal and Binary.
+CRC32, ADLER32 and Hash code always return an Integer. Result type does not
apply to them.
|Result field|The name of the result field containing the checksum
|Prefix|An optional string to prepend before the field values when building
the input for the checksum calculation.
This is useful to distinguish hash keys from different topics, for example
`CUST` for customer keys and `PROD` for product keys.
+Ignored for Hash code.
|Separator|An optional string to insert between the field values when building
the input for the checksum calculation.
This helps generate more unique hash codes when multiple fields are
concatenated.
+Ignored for Hash code.
|Suffix|An optional string to append after the field values when building the
input for the checksum calculation.
Like the prefix, this can be used to distinguish hash keys from different
topics.
+Ignored for Hash code.
|Fields used in the checksum|The names of the fields to include in the
checksum calculation.
Note: You can use the "Get Fields" button to insert all input fields from
previous transforms.
|===
@@ -55,4 +63,5 @@ Note: You can use the "Get Fields" button to insert all input
fields from previo
== See also
* xref:how-to-guides/cdc-snapshot-comparison.adoc[CDC: comparing snapshots] —
storing an MD5 or SHA hash of a row and comparing it on the next run
+* xref:pipeline/transforms/combinationlookup.adoc[Combination lookup/update] —
store this hash code on a junk dimension, or let that transform read a hash you
calculated here
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
index 55507e4dd8..72f498ca47 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/combinationlookup.adoc
@@ -73,6 +73,12 @@ Note that the new maximum is always cached, so that the
maximum does not need to
The only extra field added is then the technical key.
|Use hashcode|This option allows you to generate a hash code, representing all
values in the key fields in a numerical form (a signed 64 bit integer).
This hash code has to be stored in the table.
+The key fields are still compared, so a hash collision does not return a
different row.
+|Hashcode field in table|Name of the table column that stores the hash code.
+|Hash field in stream|Optional Integer field calculated earlier, for example
by xref:pipeline/transforms/addchecksum.adoc[Add a checksum].
+When set, that value is written to the hashcode field and used for the lookup
instead of calculating the hash in this transform.
+Leave it empty to keep calculating the hash from the key fields.
+To reproduce that calculated value, use Add a checksum with type Hash code on
the same key fields in the same order.
|Date of last update field|When required, specify the date of last update
field (timestamp) from the source system to be copied to the data warehouse.
For example, when you have an address without a primary key.
The field will not be part of the lookup fields (nor be part in the hash code
calculation).
diff --git
a/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSum.java
b/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSum.java
index 0522333aff..e528714ff5 100644
---
a/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSum.java
+++
b/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSum.java
@@ -24,8 +24,10 @@ import java.util.zip.CRC32;
import org.apache.commons.codec.binary.Hex;
import org.apache.hop.core.Const;
import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopValueException;
import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.RowDataUtil;
+import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.util.Utils;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.Pipeline;
@@ -90,12 +92,7 @@ public class CheckSum extends BaseTransform<CheckSumMeta,
CheckSumData> {
data.suffix = Const.NVL(resolve(meta.getSuffix()), "");
try {
- if (meta.getCheckSumType() == CheckSumMeta.CheckSumType.MD5
- || meta.getCheckSumType() == CheckSumMeta.CheckSumType.SHA1
- || meta.getCheckSumType() == CheckSumMeta.CheckSumType.SHA256
- || meta.getCheckSumType() == CheckSumMeta.CheckSumType.SHA384
- || meta.getCheckSumType() == CheckSumMeta.CheckSumType.SHA512) {
- // Case sensitive
+ if (meta.getCheckSumType() != null &&
meta.getCheckSumType().isDigest()) {
data.digest =
MessageDigest.getInstance(meta.getCheckSumType().getCode());
}
} catch (Exception e) {
@@ -106,8 +103,7 @@ public class CheckSum extends BaseTransform<CheckSumMeta,
CheckSumData> {
Object[] outputRowData = null;
try {
- if (meta.getCheckSumType() == CheckSumMeta.CheckSumType.ADLER32
- || meta.getCheckSumType() == CheckSumMeta.CheckSumType.CRC32) {
+ if (meta.getCheckSumType() != null &&
meta.getCheckSumType().isIntegerResult()) {
// get checksum
Long checksum = calculCheckSum(r);
outputRowData = RowDataUtil.addValueData(r, data.nrInfields, checksum);
@@ -236,6 +232,10 @@ public class CheckSum extends BaseTransform<CheckSumMeta,
CheckSumData> {
}
private Long calculCheckSum(Object[] r) throws Exception {
+ if (meta.getCheckSumType() == CheckSumMeta.CheckSumType.HASHCODE) {
+ return calculateHashCode(r);
+ }
+
Long retval;
byte[] byteArray = buildCheckSumInputBytes(r);
@@ -257,6 +257,17 @@ public class CheckSum extends BaseTransform<CheckSumMeta,
CheckSumData> {
return retval;
}
+ /** Native field values, including nulls. Prefix, separator and suffix are
not included. */
+ private Long calculateHashCode(Object[] r) throws HopValueException {
+ RowMeta hashRowMeta = new RowMeta();
+ Object[] hashRow = new Object[data.fieldnr];
+ for (int i = 0; i < data.fieldnr; i++) {
+
hashRowMeta.addValueMeta(getInputRowMeta().getValueMeta(data.fieldnrs[i]));
+ hashRow[i] = r[data.fieldnrs[i]];
+ }
+ return (long) hashRowMeta.hashCode(hashRow);
+ }
+
@Override
public boolean init() {
if (super.init()) {
diff --git
a/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumDialog.java
b/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumDialog.java
index bacd68043b..eb07743e78 100644
---
a/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumDialog.java
+++
b/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumDialog.java
@@ -73,6 +73,10 @@ public class CheckSumDialog extends BaseTransformDialog {
private Label wlResultType;
private CCombo wResultType;
+ private Label wlPrefix;
+ private Label wlSeparator;
+ private Label wlSuffix;
+
public CheckSumDialog(
Shell parent, IVariables variables, CheckSumMeta transformMeta,
PipelineMeta pipelineMeta) {
super(parent, variables, transformMeta, pipelineMeta);
@@ -103,6 +107,7 @@ public class CheckSumDialog extends BaseTransformDialog {
fdType.top = new FormAttachment(wSpacer, margin);
fdType.right = new FormAttachment(100, 0);
wType.setLayoutData(fdType);
+ wType.setToolTipText(BaseMessages.getString(PKG,
"CheckSumDialog.Type.ToolTip"));
wType.addSelectionListener(
new SelectionAdapter() {
@Override
@@ -157,7 +162,7 @@ public class CheckSumDialog extends BaseTransformDialog {
wResult.setLayoutData(fdResult);
// Prefix
- Label wlPrefix = new Label(shell, SWT.RIGHT);
+ wlPrefix = new Label(shell, SWT.RIGHT);
wlPrefix.setText(BaseMessages.getString(PKG,
"CheckSumDialog.Prefix.Label"));
PropsUi.setLook(wlPrefix);
FormData fdlPrefix = new FormData();
@@ -174,7 +179,7 @@ public class CheckSumDialog extends BaseTransformDialog {
wPrefix.setLayoutData(fdPrefix);
// Separator
- Label wlSeparator = new Label(shell, SWT.RIGHT);
+ wlSeparator = new Label(shell, SWT.RIGHT);
wlSeparator.setText(BaseMessages.getString(PKG,
"CheckSumDialog.Separator.Label"));
PropsUi.setLook(wlSeparator);
FormData fdlSeparator = new FormData();
@@ -191,7 +196,7 @@ public class CheckSumDialog extends BaseTransformDialog {
wSeparator.setLayoutData(fdSeparator);
// Suffix
- Label wlSuffix = new Label(shell, SWT.RIGHT);
+ wlSuffix = new Label(shell, SWT.RIGHT);
wlSuffix.setText(BaseMessages.getString(PKG,
"CheckSumDialog.Suffix.Label"));
PropsUi.setLook(wlSuffix);
FormData fdlSuffix = new FormData();
@@ -274,16 +279,19 @@ public class CheckSumDialog extends BaseTransformDialog {
}
private void activeResultType() {
- int currentType = wType.getSelectionIndex();
- // Only available for type MD5 and SHA
- boolean active =
- currentType == 2
- || currentType == 3
- || currentType == 4
- || currentType == 5
- || currentType == 6;
- wlResultType.setEnabled(active);
- wResultType.setEnabled(active);
+ CheckSumMeta.CheckSumType type =
+ CheckSumMeta.CheckSumType.getTypeFromDescription(wType.getText());
+ boolean digest = type.isDigest();
+ wlResultType.setEnabled(digest);
+ wResultType.setEnabled(digest);
+
+ boolean useFieldDecorations = type != CheckSumMeta.CheckSumType.HASHCODE;
+ wlPrefix.setEnabled(useFieldDecorations);
+ wPrefix.setEnabled(useFieldDecorations);
+ wlSeparator.setEnabled(useFieldDecorations);
+ wSeparator.setEnabled(useFieldDecorations);
+ wlSuffix.setEnabled(useFieldDecorations);
+ wSuffix.setEnabled(useFieldDecorations);
}
protected void setComboBoxes() {
diff --git
a/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMeta.java
b/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMeta.java
index f62528bbb5..112316d2ce 100644
---
a/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMeta.java
+++
b/plugins/transforms/checksum/src/main/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMeta.java
@@ -60,6 +60,7 @@ public class CheckSumMeta extends BaseTransformMeta<CheckSum,
CheckSumData> {
SHA256("SHA-256", BaseMessages.getString(PKG, "CheckSumMeta.Type.SHA256")),
SHA384("SHA-384", BaseMessages.getString(PKG, "CheckSumMeta.Type.SHA384")),
SHA512("SHA-512", BaseMessages.getString(PKG, "CheckSumMeta.Type.SHA512")),
+ HASHCODE("HASHCODE", BaseMessages.getString(PKG,
"CheckSumMeta.Type.HASHCODE")),
;
private String code;
@@ -105,6 +106,16 @@ public class CheckSumMeta extends
BaseTransformMeta<CheckSum, CheckSumData> {
public String getDescription() {
return description;
}
+
+ /** CRC32, ADLER32 and the Combination lookup/update hash are returned as
an integer. */
+ public boolean isIntegerResult() {
+ return this == CRC32 || this == ADLER32 || this == HASHCODE;
+ }
+
+ /** Message digests can be returned as a string, hexadecimal or binary
value. */
+ public boolean isDigest() {
+ return this == MD5 || this == SHA1 || this == SHA256 || this == SHA384
|| this == SHA512;
+ }
}
public enum ResultType implements IEnumHasCode {
@@ -224,7 +235,7 @@ public class CheckSumMeta extends
BaseTransformMeta<CheckSum, CheckSumData> {
// Output field (String)
if (!Utils.isEmpty(resultFieldName)) {
IValueMeta v = null;
- if (checkSumType == CheckSumType.CRC32 || checkSumType ==
CheckSumType.ADLER32) {
+ if (checkSumType != null && checkSumType.isIntegerResult()) {
v = new ValueMetaInteger(variables.resolve(resultFieldName));
} else {
v =
diff --git
a/plugins/transforms/checksum/src/main/resources/org/apache/hop/pipeline/transforms/checksum/messages/messages_en_US.properties
b/plugins/transforms/checksum/src/main/resources/org/apache/hop/pipeline/transforms/checksum/messages/messages_en_US.properties
index 1fb85c126c..9d85b50682 100644
---
a/plugins/transforms/checksum/src/main/resources/org/apache/hop/pipeline/transforms/checksum/messages/messages_en_US.properties
+++
b/plugins/transforms/checksum/src/main/resources/org/apache/hop/pipeline/transforms/checksum/messages/messages_en_US.properties
@@ -39,6 +39,7 @@ CheckSumDialog.ResultType.Label=Result type
CheckSumDialog.Shell.Title=Add a checksum
CheckSumDialog.TransformName.Label=Transform name
CheckSumDialog.Type.Label=Type
+CheckSumDialog.Type.ToolTip=Hash code returns the same signed integer as
Combination lookup/update. Prefix, separator and suffix are ignored for that
type. Select the same fields, in the same order, as the key fields.
CheckSumMeta.CheckResult.AllFieldsFound=All fields are found in the input
stream.
CheckSumMeta.CheckResult.FieldsFound=Fields that were not found in input
stream:\n\n{0}
CheckSumMeta.CheckResult.NoFieldsEntered=No fields are entered.
@@ -48,12 +49,13 @@ CheckSumMeta.CheckResult.ResultFieldMissing=Result
fieldname is missing!
CheckSumMeta.CheckResult.ResultFieldOK=Result fieldname is provided.
CheckSumMeta.CheckResult.TransformRecevingData=Transform is connected to
previous one, receiving {0} fields
CheckSumMeta.CheckResult.TransformRecevingData2=Transform is receiving info
from other transforms.
-CheckSumMeta.keyword=checksum,hash,md5,sha,crc32
+CheckSumMeta.keyword=checksum,hash,hashcode,md5,sha,crc32
CheckSumMeta.ResultType.Binary=Binary
CheckSumMeta.ResultType.Hexadecimal=Hexadecimal
CheckSumMeta.ResultType.String=String
CheckSumMeta.Type.ADLER32=ADLER 32
CheckSumMeta.Type.CRC32=CRC 32
+CheckSumMeta.Type.HASHCODE=Hash code
CheckSumMeta.Type.MD5=MD5
CheckSumMeta.Type.SHA1=SHA-1
CheckSumMeta.Type.SHA256=SHA-256
diff --git
a/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMetaTest.java
b/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMetaTest.java
index 8669d6424e..9f483e076f 100644
---
a/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMetaTest.java
+++
b/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumMetaTest.java
@@ -18,6 +18,8 @@
package org.apache.hop.pipeline.transforms.checksum;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.Arrays;
import java.util.HashMap;
@@ -25,6 +27,9 @@ import java.util.List;
import java.util.Map;
import java.util.UUID;
import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.variables.Variables;
import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
import org.apache.hop.pipeline.transforms.loadsave.LoadSaveTester;
import org.apache.hop.pipeline.transforms.loadsave.initializer.IInitializer;
@@ -53,6 +58,27 @@ class CheckSumMetaTest implements IInitializer<CheckSumMeta>
{
assertEquals("SHA-256", CheckSumMeta.CheckSumType.SHA256.getCode());
assertEquals("SHA-384", CheckSumMeta.CheckSumType.SHA384.getCode());
assertEquals("SHA-512", CheckSumMeta.CheckSumType.SHA512.getCode());
+ assertEquals("HASHCODE", CheckSumMeta.CheckSumType.HASHCODE.getCode());
+ assertTrue(CheckSumMeta.CheckSumType.HASHCODE.isIntegerResult());
+ assertTrue(CheckSumMeta.CheckSumType.CRC32.isIntegerResult());
+ assertTrue(CheckSumMeta.CheckSumType.ADLER32.isIntegerResult());
+ assertFalse(CheckSumMeta.CheckSumType.HASHCODE.isDigest());
+ assertTrue(CheckSumMeta.CheckSumType.MD5.isDigest());
+ assertFalse(CheckSumMeta.CheckSumType.MD5.isIntegerResult());
+ }
+
+ @Test
+ void hashCodeResultFieldIsAnInteger() throws HopException {
+ CheckSumMeta meta = new CheckSumMeta();
+ meta.setCheckSumType(CheckSumMeta.CheckSumType.HASHCODE);
+ meta.setResultFieldName("row_hash");
+ RowMeta rowMeta = new RowMeta();
+
+ meta.getFields(rowMeta, "checksum", null, null, new Variables(), null);
+
+ assertEquals(1, rowMeta.size());
+ assertEquals("row_hash", rowMeta.getValueMeta(0).getName());
+ assertEquals(IValueMeta.TYPE_INTEGER, rowMeta.getValueMeta(0).getType());
}
@Test
diff --git
a/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumTest.java
b/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumTest.java
index 436f3ca8c2..b373c41201 100644
---
a/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumTest.java
+++
b/plugins/transforms/checksum/src/test/java/org/apache/hop/pipeline/transforms/checksum/CheckSumTest.java
@@ -36,6 +36,7 @@ import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.row.value.ValueMetaBinary;
+import org.apache.hop.core.row.value.ValueMetaInteger;
import org.apache.hop.core.row.value.ValueMetaNumber;
import org.apache.hop.core.row.value.ValueMetaString;
import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
@@ -378,6 +379,33 @@ class CheckSumTest {
results.getWritten().get(0)[1]);
}
+ @Test
+ void testHashCodeMatchesRowMetaHash() throws Exception {
+ RowMeta inputRowMeta = new RowMeta();
+ inputRowMeta.addValueMeta(new ValueMetaString("name"));
+ inputRowMeta.addValueMeta(new ValueMetaInteger("id"));
+ inputRowMeta.addValueMeta(new ValueMetaNumber("amount"));
+ inputRowMeta.addValueMeta(new ValueMetaBinary("payload"));
+ Object[] input = new Object[] {"abc", 5L, 10.8d, new byte[] {1, 2, 3}};
+
+ CheckSumMeta checkSumMeta = new CheckSumMeta();
+ checkSumMeta.setPrefix("CUST");
+ checkSumMeta.setSeparator("|");
+ checkSumMeta.setSuffix("END");
+ checkSumMeta.setFields(
+ List.of(new Field("name"), new Field("id"), new Field("amount"), new
Field("payload")));
+
+ MockRowListener results =
+ executeHexTest(CheckSumMeta.CheckSumType.HASHCODE, input,
inputRowMeta, checkSumMeta);
+ assertEquals(1, results.getWritten().size());
+ assertEquals((long) inputRowMeta.hashCode(input),
results.getWritten().get(0)[4]);
+
+ Object[] nulls = new Object[] {null, null, null, null};
+ results = executeHexTest(CheckSumMeta.CheckSumType.HASHCODE, nulls,
inputRowMeta, checkSumMeta);
+ assertEquals(1, results.getWritten().size());
+ assertEquals((long) inputRowMeta.hashCode(nulls),
results.getWritten().get(0)[4]);
+ }
+
@Test
void testHexOutput_adler32() throws Exception {
MockRowListener results =
diff --git
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookup.java
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookup.java
index 339fa49ff0..eafbf4ee13 100644
---
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookup.java
+++
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookup.java
@@ -222,7 +222,7 @@ public class CombinationLookup extends
BaseTransform<CombinationLookupMeta, Comb
}
if (meta.isUseHash()) {
- valHash = (long) data.hashRowMeta.hashCode(hashRow);
+ valHash = hashValue(rowMeta, row, hashRow);
lookupRow[lookupIndex] = valHash;
lookupIndex++;
}
@@ -329,6 +329,26 @@ public class CombinationLookup extends
BaseTransform<CombinationLookupMeta, Comb
return outputRow;
}
+ Long hashValue(IRowMeta rowMeta, Object[] row, Object[] hashRow) throws
HopValueException {
+ if (data.hashFieldNr >= 0) {
+ return rowMeta.getInteger(row, data.hashFieldNr);
+ }
+ return (long) data.hashRowMeta.hashCode(hashRow);
+ }
+
+ void resolveHashField(IRowMeta inputRowMeta) throws HopTransformException {
+ data.hashFieldNr = -1;
+ if (!meta.isUseHash() ||
Utils.isEmpty(resolve(meta.getHashFieldInStream()))) {
+ return;
+ }
+ String hashFieldName = resolve(meta.getHashFieldInStream());
+ data.hashFieldNr = inputRowMeta.indexOfValue(hashFieldName);
+ if (data.hashFieldNr < 0) {
+ throw new HopTransformException(
+ BaseMessages.getString(PKG,
"CombinationLookup.Exception.FieldNotFound", hashFieldName));
+ }
+ }
+
@Override
public boolean processRow() throws HopException {
Object[] r = getRow(); // Get row from input rowset & set row busy!
@@ -389,6 +409,8 @@ public class CombinationLookup extends
BaseTransform<CombinationLookupMeta, Comb
data.hashRowMeta.addValueMeta(getInputRowMeta().getValueMeta(data.keynrs[i]));
// KEYi = ?
}
+ resolveHashField(getInputRowMeta());
+
setCombiLookup(getInputRowMeta());
preloadCache(data.hashRowMeta);
}
diff --git
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupData.java
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupData.java
index 5e36a41afa..1c3e1a2313 100644
---
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupData.java
+++
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupData.java
@@ -29,6 +29,7 @@ import org.apache.hop.pipeline.transform.ITransformData;
public class CombinationLookupData extends BaseTransformData implements
ITransformData {
public Database db;
public int[] keynrs; // nrs in row of the keys
+ public int hashFieldNr = -1;
public Map<RowMetaAndData, Long> cache;
diff --git
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupDialog.java
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupDialog.java
index 8124d0a998..2b3c800d74 100644
---
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupDialog.java
+++
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupDialog.java
@@ -41,6 +41,7 @@ import org.apache.hop.ui.core.dialog.ErrorDialog;
import org.apache.hop.ui.core.dialog.MessageBox;
import org.apache.hop.ui.core.gui.GuiResource;
import org.apache.hop.ui.core.widget.ColumnInfo;
+import org.apache.hop.ui.core.widget.ComboVar;
import org.apache.hop.ui.core.widget.MetaSelectionLine;
import org.apache.hop.ui.core.widget.NamingSchemeTypes;
import org.apache.hop.ui.core.widget.TableView;
@@ -106,6 +107,9 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
private Label wlHashfield;
private TextVar wHashfield;
+ private Label wlHashFieldInStream;
+ private ComboVar wHashFieldInStream;
+
private Text wLastUpdateField;
private ColumnInfo[] ciKey;
@@ -392,6 +396,28 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
fdHashfield.right = new FormAttachment(100, 0);
wHashfield.setLayoutData(fdHashfield);
+ wlHashFieldInStream = new Label(wGeneralComp, SWT.RIGHT);
+ wlHashFieldInStream.setText(
+ BaseMessages.getString(PKG,
"CombinationLookupDialog.HashFieldInStream.Label"));
+ wlHashFieldInStream.setToolTipText(
+ BaseMessages.getString(PKG,
"CombinationLookupDialog.HashFieldInStream.ToolTip"));
+ PropsUi.setLook(wlHashFieldInStream);
+ FormData fdlHashFieldInStream = new FormData();
+ fdlHashFieldInStream.left = new FormAttachment(0, 0);
+ fdlHashFieldInStream.right = new FormAttachment(middle, -margin);
+ fdlHashFieldInStream.top = new FormAttachment(wHashfield, margin);
+ wlHashFieldInStream.setLayoutData(fdlHashFieldInStream);
+ wHashFieldInStream = new ComboVar(variables, wGeneralComp, SWT.SINGLE |
SWT.LEFT | SWT.BORDER);
+ wHashFieldInStream.setToolTipText(
+ BaseMessages.getString(PKG,
"CombinationLookupDialog.HashFieldInStream.ToolTip"));
+ PropsUi.setLook(wHashFieldInStream);
+ wHashFieldInStream.addModifyListener(lsMod);
+ FormData fdHashFieldInStream = new FormData();
+ fdHashFieldInStream.left = new FormAttachment(middle, 0);
+ fdHashFieldInStream.top = new FormAttachment(wHashfield, margin);
+ fdHashFieldInStream.right = new FormAttachment(100, 0);
+ wHashFieldInStream.setLayoutData(fdHashFieldInStream);
+
// Last update field:
Label wlLastUpdateField = new Label(wGeneralComp, SWT.RIGHT);
wlLastUpdateField.setText(
@@ -400,14 +426,14 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
FormData fdlLastUpdateField = new FormData();
fdlLastUpdateField.left = new FormAttachment(0, 0);
fdlLastUpdateField.right = new FormAttachment(middle, -margin);
- fdlLastUpdateField.top = new FormAttachment(wHashfield, margin);
+ fdlLastUpdateField.top = new FormAttachment(wHashFieldInStream, margin);
wlLastUpdateField.setLayoutData(fdlLastUpdateField);
wLastUpdateField = new Text(wGeneralComp, SWT.SINGLE | SWT.LEFT |
SWT.BORDER);
PropsUi.setLook(wLastUpdateField);
wLastUpdateField.addModifyListener(lsMod);
FormData fdLastUpdateField = new FormData();
fdLastUpdateField.left = new FormAttachment(middle, 0);
- fdLastUpdateField.top = new FormAttachment(wHashfield, margin);
+ fdLastUpdateField.top = new FormAttachment(wHashFieldInStream, margin);
fdLastUpdateField.right = new FormAttachment(100, 0);
wLastUpdateField.setLayoutData(fdLastUpdateField);
@@ -633,11 +659,17 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
//
String[] fieldNames = ConstUi.sortFieldNames(inputFields);
ciKey[1].setComboValues(fieldNames);
+ if (wHashFieldInStream != null && !wHashFieldInStream.isDisposed()) {
+ wHashFieldInStream.setItems(fieldNames);
+ }
}
public void enableFields() {
- wHashfield.setEnabled(wHashcode.getSelection());
- wlHashfield.setEnabled(wHashcode.getSelection());
+ boolean useHash = wHashcode.getSelection();
+ wHashfield.setEnabled(useHash);
+ wlHashfield.setEnabled(useHash);
+ wHashFieldInStream.setEnabled(useHash);
+ wlHashFieldInStream.setEnabled(useHash);
}
private void setTableFieldCombo() {
@@ -734,8 +766,6 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
wPreloadCache.setSelection(input.isPreloadCache());
wReplace.setSelection(input.isReplaceFields());
wHashcode.setSelection(input.isUseHash());
- wHashfield.setEnabled(input.isUseHash());
- wlHashfield.setEnabled(input.isUseHash());
String techKeyCreation = returnFields.getTechKeyCreation();
if (techKeyCreation == null) {
@@ -785,6 +815,8 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
wConnection.setText(input.getConnectionName());
}
wHashfield.setText(Const.NVL(input.getHashField(), ""));
+ wHashFieldInStream.setText(Const.NVL(input.getHashFieldInStream(), ""));
+ enableFields();
wCommit.setText("" + input.getCommitSize());
wCachesize.setText("" + input.getCacheSize());
@@ -834,6 +866,8 @@ public class CombinationLookupDialog extends
BaseTransformDialog {
in.setReplaceFields(wReplace.getSelection());
in.setUseHash(wHashcode.getSelection());
in.setHashField(wHashfield.getText());
+ String hashFieldInStream = wHashFieldInStream.getText();
+ in.setHashFieldInStream(Utils.isEmpty(hashFieldInStream) ? null :
hashFieldInStream);
in.setSchemaName(wSchema.getText());
in.setTableName(wTable.getText());
returnFields.setTechnicalKeyField(wTk.getText());
diff --git
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMeta.java
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMeta.java
index 4f9b7f2fee..bcddb7b2d8 100644
---
a/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMeta.java
+++
b/plugins/transforms/combinationlookup/src/main/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMeta.java
@@ -117,6 +117,15 @@ public class CombinationLookupMeta
injectionKeyDescription = "CombinationLookup.Injection.HASH_FIELD")
private String hashField;
+ /**
+ * Optional input field that already contains the hash. Empty means this
transform calculates it.
+ */
+ @HopMetadataProperty(
+ key = "hashfield_stream",
+ injectionKey = "HASH_FIELD_IN_STREAM",
+ injectionKeyDescription =
"CombinationLookup.Injection.HASH_FIELD_IN_STREAM")
+ private String hashFieldInStream;
+
/** Commit size for insert / update */
@HopMetadataProperty(
key = "commit",
@@ -325,6 +334,7 @@ public class CombinationLookupMeta
transformMeta);
}
remarks.add(cr);
+ checkHashField(remarks, variables, transformMeta, prev);
} else {
errorMessage =
BaseMessages.getString(PKG,
"CombinationLookupMeta.CheckResult.CouldNotReadFields")
@@ -413,6 +423,42 @@ public class CombinationLookupMeta
}
}
+ void checkHashField(
+ List<ICheckResult> remarks,
+ IVariables variables,
+ TransformMeta transformMeta,
+ IRowMeta prev) {
+ if (!useHash || Utils.isEmpty(variables.resolve(hashFieldInStream))) {
+ return;
+ }
+ String hashName = variables.resolve(hashFieldInStream);
+ IValueMeta hashValue = prev.searchValueMeta(hashName);
+ CheckResult cr;
+ if (hashValue == null) {
+ cr =
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG, "CombinationLookupMeta.CheckResult.HashFieldNotFound",
hashName),
+ transformMeta);
+ } else if (hashValue.getType() != IValueMeta.TYPE_INTEGER) {
+ cr =
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
"CombinationLookupMeta.CheckResult.HashFieldNotInteger", hashName),
+ transformMeta);
+ } else {
+ cr =
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_OK,
+ BaseMessages.getString(
+ PKG, "CombinationLookupMeta.CheckResult.HashFieldFound",
hashName),
+ transformMeta);
+ }
+ remarks.add(cr);
+ }
+
@Override
public SqlStatement getSqlStatements(
IVariables variables,
diff --git
a/plugins/transforms/combinationlookup/src/main/resources/org/apache/hop/pipeline/transforms/combinationlookup/messages/messages_en_US.properties
b/plugins/transforms/combinationlookup/src/main/resources/org/apache/hop/pipeline/transforms/combinationlookup/messages/messages_en_US.properties
index 0985af1e7b..5966b3bf72 100644
---
a/plugins/transforms/combinationlookup/src/main/resources/org/apache/hop/pipeline/transforms/combinationlookup/messages/messages_en_US.properties
+++
b/plugins/transforms/combinationlookup/src/main/resources/org/apache/hop/pipeline/transforms/combinationlookup/messages/messages_en_US.properties
@@ -23,6 +23,7 @@ CombinationLookup.Injection.CACHE_SIZE=The size of the cache.
CombinationLookup.Injection.COMMIT_SIZE=The number of rows to commit at a time.
CombinationLookup.Injection.CONNECTION_NAME=The name of the database
connection.
CombinationLookup.Injection.HASH_FIELD=The name of the field containing
hashcodes.
+CombinationLookup.Injection.HASH_FIELD_IN_STREAM=The optional input field that
already contains the hash code. When empty, the hash is calculated from the key
fields.
CombinationLookup.Injection.KEY_FIELD=The table field to use to lookup rows.
CombinationLookup.Injection.KEY_FIELDS=The key fields to use to lookup rows.
CombinationLookup.Injection.KEY_LOOKUP=The stream field to use to lookup rows.
@@ -60,6 +61,8 @@ CombinationLookupDialog.GetFields.Button=\ &Get Fields
CombinationLookupDialog.GetSchemas.Error=ERROR
CombinationLookupDialog.Hashcode.Label=Use hashcode
CombinationLookupDialog.Hashfield.Label=Hashcode field in table
+CombinationLookupDialog.HashFieldInStream.Label=Hash field in stream
+CombinationLookupDialog.HashFieldInStream.ToolTip=Optional Integer field
calculated earlier, for example by Add a checksum. When set, this value is
stored in the hashcode field and used for lookup. Leave empty to calculate the
hash from the key fields. Use checksum type Hash code on the same key fields,
in the same order, to match that calculation.
CombinationLookupDialog.KeyFieldsTab.TabTitle=Key fields
CombinationLookupDialog.Keyfields.Label=Key fields (to look up row in table)\:
CombinationLookupDialog.LastUpdateField.Label=Date of last update field
(optional)
@@ -99,6 +102,9 @@ CombinationLookupMeta.CheckResult.ErrorNoSequenceName=No
sequence name was speci
CombinationLookupMeta.CheckResult.ErrorOccurred=An error occurred\:
CombinationLookupMeta.CheckResult.ErrorReadingSequence=Error reading sequence
CombinationLookupMeta.CheckResult.ErrorTechKeyCreation=Technical key creation
method invalid
+CombinationLookupMeta.CheckResult.HashFieldFound=Hash field [{0}] was found in
the input stream.
+CombinationLookupMeta.CheckResult.HashFieldNotFound=Hash field [{0}] was not
found in the input stream.
+CombinationLookupMeta.CheckResult.HashFieldNotInteger=Hash field [{0}] must be
an Integer.
CombinationLookupMeta.CheckResult.InvalidConnection=Please select or create a
connection\!
CombinationLookupMeta.CheckResult.MissingCompareFields=Missing compare fields
in target table\:
CombinationLookupMeta.CheckResult.MissingFields=Missing fields, not found in
input from previous transforms\:
diff --git
a/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupHashTest.java
b/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupHashTest.java
new file mode 100644
index 0000000000..0806a1d187
--- /dev/null
+++
b/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupHashTest.java
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.combinationlookup;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.plugins.PluginRegistry;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaInteger;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+class CombinationLookupHashTest {
+
+ @BeforeAll
+ static void setUp() throws Exception {
+ HopEnvironment.init();
+ PluginRegistry.init();
+ }
+
+ @Test
+ void calculatedHashMatchesTheRowHash() throws Exception {
+ CombinationLookup transform = transform();
+ RowMeta hashRowMeta = new RowMeta();
+ hashRowMeta.addValueMeta(new ValueMetaString("key"));
+ Object[] hashRow = new Object[] {"abc"};
+ transform.getData().hashRowMeta = hashRowMeta;
+ transform.getData().hashFieldNr = -1;
+
+ RowMeta rowMeta = rowMeta();
+ long expected = hashRowMeta.hashCode(hashRow);
+
+ assertEquals(expected, transform.hashValue(rowMeta, new Object[] {"abc",
42L}, hashRow));
+ assertNotEquals(42L, expected);
+ }
+
+ @Test
+ void streamFieldReplacesTheCalculatedHash() throws Exception {
+ CombinationLookup transform = transform();
+ transform.getMeta().setUseHash(true);
+ transform.getMeta().setHashFieldInStream("${HASH_FIELD}");
+ transform.setVariable("HASH_FIELD", "hash");
+
+ RowMeta rowMeta = rowMeta();
+ transform.resolveHashField(rowMeta);
+ assertEquals(1, transform.getData().hashFieldNr);
+
+ RowMeta hashRowMeta = new RowMeta();
+ hashRowMeta.addValueMeta(new ValueMetaString("key"));
+ Object[] hashRow = new Object[] {"abc"};
+ transform.getData().hashRowMeta = hashRowMeta;
+
+ assertEquals(42L, transform.hashValue(rowMeta, new Object[] {"abc", 42L},
hashRow));
+ assertEquals(-15L, transform.hashValue(rowMeta, new Object[] {"abc",
-15L}, hashRow));
+ assertNull(transform.hashValue(rowMeta, new Object[] {"abc", null},
hashRow));
+ }
+
+ @Test
+ void emptyStreamFieldKeepsTheCalculatedHash() throws Exception {
+ CombinationLookup transform = transform();
+ transform.getMeta().setUseHash(true);
+ transform.getMeta().setHashFieldInStream("");
+ transform.resolveHashField(rowMeta());
+ assertEquals(-1, transform.getData().hashFieldNr);
+
+ transform.getMeta().setUseHash(false);
+ transform.getMeta().setHashFieldInStream("hash");
+ transform.resolveHashField(rowMeta());
+ assertEquals(-1, transform.getData().hashFieldNr);
+ }
+
+ @Test
+ void missingStreamFieldFails() {
+ CombinationLookup transform = transform();
+ transform.getMeta().setUseHash(true);
+ transform.getMeta().setHashFieldInStream("missing");
+
+ HopTransformException exception =
+ assertThrows(HopTransformException.class, () ->
transform.resolveHashField(rowMeta()));
+ assertTrue(exception.getMessage().contains("missing"));
+ }
+
+ @Test
+ void checkAcceptsAnIntegerStreamField() {
+ CombinationLookupMeta meta = new CombinationLookupMeta();
+ meta.setUseHash(true);
+ meta.setHashFieldInStream("${HASH_FIELD}");
+ Variables variables = new Variables();
+ variables.setVariable("HASH_FIELD", "row_hash");
+ RowMeta prev = new RowMeta();
+ prev.addValueMeta(new ValueMetaInteger("row_hash"));
+ List<ICheckResult> remarks = new ArrayList<>();
+
+ meta.checkHashField(remarks, variables, new TransformMeta("lookup", meta),
prev);
+
+ assertEquals(1, remarks.size());
+ assertEquals(ICheckResult.TYPE_RESULT_OK, remarks.get(0).getType());
+ }
+
+ @Test
+ void checkRejectsAMissingOrNonIntegerStreamField() {
+ CombinationLookupMeta meta = new CombinationLookupMeta();
+ meta.setUseHash(true);
+ meta.setHashFieldInStream("row_hash");
+ RowMeta prev = new RowMeta();
+ prev.addValueMeta(new ValueMetaString("row_hash"));
+ List<ICheckResult> remarks = new ArrayList<>();
+ TransformMeta transformMeta = new TransformMeta("lookup", meta);
+
+ meta.checkHashField(remarks, new Variables(), transformMeta, prev);
+ assertEquals(ICheckResult.TYPE_RESULT_ERROR, remarks.get(0).getType());
+
+ remarks.clear();
+ prev.clear();
+ meta.checkHashField(remarks, new Variables(), transformMeta, prev);
+ assertEquals(ICheckResult.TYPE_RESULT_ERROR, remarks.get(0).getType());
+
+ remarks.clear();
+ meta.setUseHash(false);
+ meta.checkHashField(remarks, new Variables(), transformMeta, prev);
+ assertTrue(remarks.isEmpty());
+ }
+
+ private static CombinationLookup transform() {
+ CombinationLookupMeta meta = new CombinationLookupMeta();
+ CombinationLookupData data = new CombinationLookupData();
+ TransformMeta transformMeta = new TransformMeta("lookup", meta);
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.addTransform(transformMeta);
+ return new CombinationLookup(transformMeta, meta, data, 0, pipelineMeta,
null);
+ }
+
+ private static RowMeta rowMeta() {
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("key"));
+ rowMeta.addValueMeta(new ValueMetaInteger("hash"));
+ return rowMeta;
+ }
+}
diff --git
a/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMetaTest.java
b/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMetaTest.java
index 6cf97afd9f..c4a6065921 100644
---
a/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMetaTest.java
+++
b/plugins/transforms/combinationlookup/src/test/java/org/apache/hop/pipeline/transforms/combinationlookup/CombinationLookupMetaTest.java
@@ -31,14 +31,18 @@ import org.apache.hop.core.plugins.PluginRegistry;
import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.row.value.ValueMetaString;
import org.apache.hop.core.variables.Variables;
+import org.apache.hop.core.xml.XmlHandler;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.apache.hop.metadata.serializer.xml.XmlMetadataUtil;
import org.apache.hop.pipeline.DatabaseImpact;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.TransformMeta;
import org.apache.hop.pipeline.transform.TransformSerializationTestUtil;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.w3c.dom.Document;
+import org.w3c.dom.Node;
class CombinationLookupMetaTest {
private static final Class<?> PKG = CombinationLookupMeta.class;
@@ -53,8 +57,28 @@ class CombinationLookupMetaTest {
@Test
void testSerialization() throws Exception {
- TransformSerializationTestUtil.testSerialization(
- "/combination-lookup-transform.xml", CombinationLookupMeta.class);
+ CombinationLookupMeta meta =
+ TransformSerializationTestUtil.testSerialization(
+ "/combination-lookup-transform.xml", CombinationLookupMeta.class);
+ assertNull(meta.getHashFieldInStream());
+ }
+
+ @Test
+ void hashFieldInStreamRoundTrips() throws Exception {
+ CombinationLookupMeta meta = new CombinationLookupMeta();
+ meta.setHashFieldInStream("row_hash");
+ String xml =
+ XmlHandler.openTag(TransformMeta.XML_TAG)
+ + meta.getXml()
+ + XmlHandler.closeTag(TransformMeta.XML_TAG);
+
+ Document document = XmlHandler.loadXmlString(xml);
+ Node node = XmlHandler.getSubNode(document, TransformMeta.XML_TAG);
+ CombinationLookupMeta copy =
+ XmlMetadataUtil.deSerializeFromXml(
+ node, CombinationLookupMeta.class, new MemoryMetadataProvider());
+
+ assertEquals("row_hash", copy.getHashFieldInStream());
}
/**
