This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 4940b06181 Issue #8592 : Read and write cube files in parallel (#8830)
4940b06181 is described below
commit 4940b0618152ac8579f6d7aca66ef7bb81a248c9
Author: Matt Casters <[email protected]>
AuthorDate: Sat Oct 10 10:37:25 2026 +0200
Issue #8592 : Read and write cube files in parallel (#8830)
* Issue #8592 : Read and write cube files in parallel
Serialize to file and De-serialize from file can give each transform copy
its own cube file, and De-serialize can read filenames from an upstream field.
The copy number is resolved when the layout is looked up, including when it is
nested in another variable.
The two dialogs are built from annotated widgets.
* Issue #8592 : Keep each cube copy on its own file when a pipeline is
exported
Resource export mapped every copy onto the copy-0 file and cleared Include
transform nr. Map the parent folder and keep the stored file name so each copy
still opens its own file. Filenames from a field are checked against the sample
layout, including the first file. The output filename tooltip now says each
copy writes the rows it receives.
---
.../transforms/serialize-de-from-file.adoc | 37 ++-
.../pipeline/transforms/serialize-to-file.adoc | 31 ++
.../hop/pipeline/transforms/cube/CubeFilename.java | 148 ++++++++++
.../pipeline/transforms/cubeinput/CubeInput.java | 225 ++++++++++----
.../transforms/cubeinput/CubeInputData.java | 6 +
.../transforms/cubeinput/CubeInputDialog.java | 173 +++++------
.../transforms/cubeinput/CubeInputMeta.java | 269 ++++++++++-------
.../pipeline/transforms/cubeoutput/CubeOutput.java | 11 +-
.../transforms/cubeoutput/CubeOutputDialog.java | 207 +++----------
.../transforms/cubeoutput/CubeOutputMeta.java | 184 ++++++------
.../cubeinput/messages/messages_en_US.properties | 12 +
.../cubeoutput/messages/messages_en_US.properties | 3 +
.../pipeline/transforms/cube/CubeFilenameTest.java | 65 ++++
.../transforms/cubeinput/CubeInputMetaTest.java | 167 +++++++++++
.../transforms/cubeoutput/CubeOutputMetaTest.java | 90 ++++++
.../cubeoutput/CubeParallelFileTest.java | 327 +++++++++++++++++++++
.../src/test/resources/de-serialize-from-field.xml | 44 +++
.../src/test/resources/serialize-with-copy-nr.xml | 42 +++
18 files changed, 1521 insertions(+), 520 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
index 8701494846..71c6ffb089 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-de-from-file.adoc
@@ -30,9 +30,34 @@ The De-serialize From File transform reads rows of data from
a binary Hop file c
[options="header"]
|===
|Option|Description
-|Transform Name|Name of the transform; this name has to be unique in a single
pipeline.
-|Filename|The name of the Hop cube file to be generated.
-|Limit|Allows you to limit the number of rows written to
-|Size|the cube file.
-A value of zero (0) indicates no size limit (optional).
-|===
\ No newline at end of file
+
+|Transform name
+|Name of the transform. This name has to be unique in a single pipeline.
+
+|Filename
+|Cube file to read. The path can contain variables. The layout of the output
fields is read from this file. When filenames come from a field, this file is
only the sample used to determine that layout, and the files you read must use
the same fields.
+
+|Include transform nr in filename
+|Reads a different file in each transform copy. The copy number is inserted
before the extension, starting at 0. With this option, `data.cube` is read as
`data_0.cube` in copy 0, `data_1.cube` in copy 1, and so on. You can instead
put `'${Internal.Transform.CopyNr}'` in the filename. Use one of the two. Using
both inserts the number twice. This option is not used when the filename comes
from a field.
+
+|Filename is defined in a field
+|Read one cube file for each incoming row. The field below holds the full
filename. Rows from the previous transform are not copied to the output. The
output is the rows stored in each cube file.
+
+|Filename field
+|Field that contains the cube filename.
+
+|Limit size
+|Maximum number of rows to read, across every file. `0` reads every row.
+
+|Add filename to result
+|Add each file that is opened to the list of result filenames.
+|===
+
+== Several files at once
+
+Two setups read more than one cube file.
+
+* The same files Serialize to file wrote with *Include transform nr in
filename*. Set the same number of copies here and enable the same option. Copy
0 reads `data_0.cube`, which is also the file used to determine the layout.
+* A previous transform that produces one filename per row, for example Get
file names. Enable *Filename is defined in a field*, choose that field, and set
the filename above to any one of the cube files so the layout can be read. More
than one copy of this transform splits those filenames across the copies.
+
+Every file in one run has to have the same field names and types. A file that
does not fails the transform. The error names that file and the first file that
was read.
\ No newline at end of file
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
index d89b563d2f..008feae235 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/serialize-to-file.adoc
@@ -28,3 +28,34 @@ The binary format has an advantage over a text (flat) file
because the content d
The metadata is stored in the file as well.
The Serialize to file transform supports a write-once access pattern, and does
not support appending to existing files.
+
+== Options
+
+[options="header"]
+|===
+|Option|Description
+
+|Transform name
+|Name of the transform. This name has to be unique in a single pipeline.
+
+|Filename
+|Name of the cube file to write. The path can contain variables.
+
+|Include transform nr in filename
+|Gives each transform copy its own file. The copy number is inserted before
the extension, starting at 0. With this option, `data.cube` is written as
`data_0.cube`, `data_1.cube`, and so on. You can instead put
`'${Internal.Transform.CopyNr}'` in the filename. Use one of the two. Using
both inserts the number twice.
+
+|Create parent folders
+|Create the parent folders when they do not exist.
+
+|Do not create file at start
+|Wait for the first row before creating the file. Otherwise the file is
created when the pipeline starts, including the layout of an empty stream.
+
+|Add filenames to result
+|Add the written filename to the list of result filenames.
+|===
+
+== Several files at once
+
+Set the number of copies on the transform and enable *Include transform nr in
filename*. Each copy writes the rows it receives to its own file. De-serialize
from file reads those files back with the same option and the same number of
copies.
+
+A single copy writes one file. Several copies that share one filename
overwrite that file.
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cube/CubeFilename.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cube/CubeFilename.java
new file mode 100644
index 0000000000..8d60349d89
--- /dev/null
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cube/CubeFilename.java
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.cube;
+
+import java.util.regex.Pattern;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.core.Const;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.core.vfs.HopVfs;
+import org.apache.hop.resource.IResourceNaming;
+
+/**
+ * Resolves a cube filename the same way at design time and at runtime.
+ *
+ * <p>{@code Internal.Transform.CopyNr} exists only on a running transform
copy. Field lookup and
+ * export run with the pipeline variables, so an unresolved token would be
left in the path and the
+ * file would not be found. The copy number passed in is applied even when the
variable is missing,
+ * nested inside another variable, written with {@code %%}, or spelled in a
different case.
+ */
+public final class CubeFilename {
+
+ /**
+ * Quoted {@code \Q} blocks are not case-folded by {@code (?i)}, so the dots
are escaped by hand.
+ * The variable name has no other regex metacharacters.
+ */
+ private static final Pattern COPY_NR =
+ Pattern.compile(
+ "(?i)\\$\\{"
+ + Const.INTERNAL_VARIABLE_TRANSFORM_COPYNR.replace(".", "\\.")
+ + "\\}|%%"
+ + Const.INTERNAL_VARIABLE_TRANSFORM_COPYNR.replace(".", "\\.")
+ + "%%");
+
+ private CubeFilename() {}
+
+ /**
+ * Resolve {@code filename} for the given 0-based copy.
+ *
+ * @param includeTransformNr when true, insert {@code _} and the copy number
before the extension
+ */
+ public static String resolve(
+ IVariables variables, String filename, boolean includeTransformNr, int
copyNr) {
+ String copy = Integer.toString(copyNr);
+ Variables scoped = new Variables();
+ if (variables != null) {
+ scoped.initializeFrom(variables);
+ }
+ // Override a stale copy number inherited from a parent pipeline or the
GUI.
+ scoped.setVariable(Const.INTERNAL_VARIABLE_TRANSFORM_COPYNR, copy);
+
+ String name = filename == null ? "" : filename;
+ name = replaceCopyNr(name, copy);
+ name = scoped.resolve(name);
+ name = replaceCopyNr(name, copy);
+ name = scoped.resolve(name);
+ if (includeTransformNr) {
+ name = insertCopyBeforeExtension(name, copy);
+ }
+ return name;
+ }
+
+ /**
+ * Point an exported pipeline at the folder of the copy-0 file and keep the
stored file name.
+ *
+ * <p>Copy 0 is opened only to find that folder. The stored base name is
appended unchanged, so a
+ * copy-number variable and the include-transform-nr option still select a
file per copy. Returns
+ * null when that copy-0 file does not exist.
+ */
+ public static String exportResourceName(
+ IVariables variables, String filename, boolean includeTransformNr,
IResourceNaming naming)
+ throws HopException {
+ try {
+ String resolved = resolve(variables, filename, includeTransformNr, 0);
+ FileObject fileObject = HopVfs.getFileObject(resolved, variables);
+ if (!fileObject.exists()) {
+ return null;
+ }
+ FileObject parent = fileObject.getParent();
+ if (parent == null) {
+ return naming.nameResource(fileObject, variables, true);
+ }
+ // The flag means "include the file name" in SimpleResourceNaming,
despite the interface
+ // calling it pathOnly. false maps the folder and leaves the name for us
to append.
+ String folder = naming.nameResource(parent, variables, false);
+ String baseName = baseName(filename);
+ if (baseName.isEmpty()) {
+ baseName = fileObject.getName().getBaseName();
+ }
+ return appendBaseName(folder, baseName);
+ } catch (HopException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new HopException(e);
+ }
+ }
+
+ private static String baseName(String filename) {
+ if (filename == null || filename.isEmpty()) {
+ return "";
+ }
+ int slash = Math.max(filename.lastIndexOf('/'),
filename.lastIndexOf('\\'));
+ if (slash < 0) {
+ return filename;
+ }
+ return filename.substring(slash + 1);
+ }
+
+ private static String appendBaseName(String folder, String baseName) {
+ if (folder == null || folder.isEmpty()) {
+ return baseName;
+ }
+ if (folder.endsWith("/") || folder.endsWith("\\")) {
+ return folder + baseName;
+ }
+ return folder + "/" + baseName;
+ }
+
+ private static String replaceCopyNr(String name, String copy) {
+ return COPY_NR.matcher(name).replaceAll(copy);
+ }
+
+ static String insertCopyBeforeExtension(String path, String copy) {
+ int slash = Math.max(path.lastIndexOf('/'), path.lastIndexOf('\\'));
+ int dot = path.lastIndexOf('.');
+ String suffix = "_" + copy;
+ if (dot > slash) {
+ return path.substring(0, dot) + suffix + path.substring(dot);
+ }
+ return path + suffix;
+ }
+}
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInput.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInput.java
index 06998a138b..7d3ee62b0b 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInput.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInput.java
@@ -19,6 +19,7 @@ package org.apache.hop.pipeline.transforms.cubeinput;
import java.io.DataInputStream;
import java.io.IOException;
+import java.io.InputStream;
import java.net.SocketTimeoutException;
import java.util.zip.GZIPInputStream;
import org.apache.hop.core.Const;
@@ -27,13 +28,17 @@ import org.apache.hop.core.exception.HopEofException;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopFileException;
import org.apache.hop.core.io.CountingInputStream;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.util.Utils;
import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.Pipeline;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.BaseTransform;
import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.cube.CubeFilename;
public class CubeInput extends BaseTransform<CubeInputMeta, CubeInputData> {
@@ -59,26 +64,137 @@ public class CubeInput extends
BaseTransform<CubeInputMeta, CubeInputData> {
realRowLimit = Const.toIntExpanded(resolve(meta.getRowLimit()), 0);
}
- try {
- Object[] r = data.meta.readData(data.dis);
- putRow(data.meta, r); // fill the rowset(s). (sleeps if full)
- incrementLinesInput();
+ while (true) {
+ if (data.dis == null) {
+ if (!meta.isFilenameInField() || !openNextFileFromRow()) {
+ setOutputDone();
+ return false;
+ }
+ }
- if (realRowLimit > 0 && getLinesInput() >= realRowLimit) { // finished!
- setOutputDone();
- return false;
+ try {
+ Object[] r = data.meta.readData(data.dis);
+ putRow(data.meta, r);
+ incrementLinesInput();
+
+ if (realRowLimit > 0 && getLinesInput() >= realRowLimit) {
+ setOutputDone();
+ return false;
+ }
+ } catch (HopEofException eof) {
+ closeFile();
+ if (!meta.isFilenameInField()) {
+ setOutputDone();
+ return false;
+ }
+ continue;
+ } catch (SocketTimeoutException e) {
+ throw new HopException(e); // shouldn't happen on files
}
- } catch (HopEofException eof) {
- setOutputDone();
+
+ if (checkFeedback(getLinesInput()) && isBasic()) {
+ logBasic(BaseMessages.getString(PKG, "CubeInput.Log.LineNumber") +
getLinesInput());
+ }
+
+ return true;
+ }
+ }
+
+ /**
+ * @return false when the upstream row stream is finished
+ */
+ private boolean openNextFileFromRow() throws HopException {
+ Object[] row = getRow();
+ if (row == null) {
return false;
- } catch (SocketTimeoutException e) {
- throw new HopException(e); // shouldn't happen on files
}
+ String fieldName = resolve(meta.getFilenameField());
+ int index = Utils.isEmpty(fieldName) ? -1 :
getInputRowMeta().indexOfValue(fieldName);
+ if (index < 0) {
+ throw new HopException(
+ BaseMessages.getString(
+ PKG, "CubeInputMeta.Exception.FilenameFieldNotFound",
Const.NVL(fieldName, "")));
+ }
+ String filename = getInputRowMeta().getString(row, index);
+ // Names from a field are complete paths. The transform-nr suffix is not
applied to them.
+ filename = CubeFilename.resolve(this, filename, false, getCopy());
+ if (Utils.isEmpty(filename)) {
+ throw new HopException(BaseMessages.getString(PKG,
"CubeInputMeta.Exception.EmptyFilename"));
+ }
+ openFile(filename);
+ return true;
+ }
- if (checkFeedback(getLinesInput()) && isBasic()) {
- logBasic(BaseMessages.getString(PKG, "CubeInput.Log.LineNumber") +
getLinesInput());
+ private void openFile(String filename) throws HopException {
+ closeFile();
+ InputStream input = null;
+ DataInputStream dataInput = null;
+ try {
+ input = HopVfs.getInputStream(filename, variables);
+ input = new CountingInputStream(input);
+ GZIPInputStream gzip = new GZIPInputStream(input);
+ dataInput = new DataInputStream(gzip);
+ IRowMeta layout = new RowMeta(dataInput);
+ if (data.meta != null && !sameLayout(data.meta, layout)) {
+ throw new HopException(
+ BaseMessages.getString(
+ PKG,
+ "CubeInputMeta.Exception.LayoutMismatch",
+ filename,
+ Const.NVL(data.referenceFilename, "")));
+ }
+ data.fis = input;
+ data.zip = gzip;
+ data.dis = dataInput;
+ input = null;
+ dataInput = null;
+ if (data.meta == null) {
+ data.meta = layout;
+ data.referenceFilename = filename;
+ }
+ if (meta.isAddFilenameResult()) {
+ ResultFile resultFile =
+ new ResultFile(
+ ResultFile.FILE_TYPE_GENERAL,
+ HopVfs.getFileObject(filename, variables),
+ getPipelineMeta().getName(),
+ toString());
+ resultFile.setComment("File was read by a Cube Input transform");
+ addResultFile(resultFile);
+ }
+ } catch (HopException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new HopException(
+ BaseMessages.getString(PKG,
"CubeInput.Log.ErrorReadingFromDataCube") + filename, e);
+ } finally {
+ if (dataInput != null) {
+ try {
+ dataInput.close();
+ } catch (IOException ignored) {
+ // The transform is already failing. dispose() closes a file that
was stored.
+ }
+ } else if (input != null) {
+ try {
+ input.close();
+ } catch (IOException ignored) {
+ // See above.
+ }
+ }
}
+ }
+ private static boolean sameLayout(IRowMeta left, IRowMeta right) {
+ if (left.size() != right.size()) {
+ return false;
+ }
+ for (int i = 0; i < left.size(); i++) {
+ IValueMeta a = left.getValueMeta(i);
+ IValueMeta b = right.getValueMeta(i);
+ if (!a.getName().equals(b.getName()) || a.getType() != b.getType()) {
+ return false;
+ }
+ }
return true;
}
@@ -87,32 +203,20 @@ public class CubeInput extends
BaseTransform<CubeInputMeta, CubeInputData> {
if (super.init()) {
try {
- String filename = resolve(meta.getFile().getName());
-
- // Add filename to result filenames ?
- if (meta.isAddFilenameResult()) {
- ResultFile resultFile =
- new ResultFile(
- ResultFile.FILE_TYPE_GENERAL,
- HopVfs.getFileObject(filename, variables),
- getPipelineMeta().getName(),
- toString());
- resultFile.setComment("File was read by a Cube Input transform");
- addResultFile(resultFile);
- }
-
- data.fis = HopVfs.getInputStream(filename, variables);
- data.fis = new CountingInputStream(data.fis);
- data.zip = new GZIPInputStream(data.fis);
- data.dis = new DataInputStream(data.zip);
-
- try {
- data.meta = new RowMeta(data.dis);
+ if (meta.isFilenameInField()) {
+ // Downstream fields come from the sample file. Keep that layout as
the reference so the
+ // first file named in the field is checked against it too.
+ rememberSampleLayout();
return true;
- } catch (HopFileException kfe) {
- logError(BaseMessages.getString(PKG,
"CubeInput.Log.UnableToReadMetadata"), kfe);
- return false;
}
+ String filename =
+ CubeFilename.resolve(
+ this, meta.getFilename(), meta.usesTransformNrInFilename(),
getCopy());
+ openFile(filename);
+ return true;
+ } catch (HopFileException kfe) {
+ logError(BaseMessages.getString(PKG,
"CubeInput.Log.UnableToReadMetadata"), kfe);
+ return false;
} catch (Exception e) {
logError(BaseMessages.getString(PKG,
"CubeInput.Log.ErrorReadingFromDataCube"), e);
}
@@ -120,30 +224,49 @@ public class CubeInput extends
BaseTransform<CubeInputMeta, CubeInputData> {
return false;
}
- @Override
- public void dispose() {
+ /** Read the dialog filename's layout and close it. Rows still come from the
filename field. */
+ private void rememberSampleLayout() throws HopException {
+ // Copy 0 matches getFields(), which is the layout downstream transforms
already have.
+ String filename = CubeFilename.resolve(this, meta.getFilename(), false, 0);
+ if (Utils.isEmpty(filename)) {
+ throw new HopException(
+ BaseMessages.getString(PKG,
"CubeInputMeta.Exception.NoFilenameSpecified"));
+ }
+ try (InputStream is = HopVfs.getInputStream(filename, variables);
+ GZIPInputStream fis = new GZIPInputStream(is);
+ DataInputStream dis = new DataInputStream(fis)) {
+ data.meta = new RowMeta(dis);
+ data.referenceFilename = filename;
+ } catch (HopException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new HopException(
+ BaseMessages.getString(PKG,
"CubeInput.Log.ErrorReadingFromDataCube") + filename, e);
+ }
+ }
+
+ private void closeFile() {
+ if (data.fis instanceof CountingInputStream cis) {
+ dataVolumeIn = (dataVolumeIn != null ? dataVolumeIn : 0L) +
cis.getCount();
+ }
try {
if (data.dis != null) {
data.dis.close();
- data.dis = null;
- }
- if (data.zip != null) {
- data.zip.close();
- data.zip = null;
- }
- if (data.fis != null) {
- if (data.fis instanceof CountingInputStream cis) {
- dataVolumeIn = (dataVolumeIn != null ? dataVolumeIn : 0L) +
cis.getCount();
- }
- data.fis.close();
- data.fis = null;
}
} catch (IOException e) {
logError(BaseMessages.getString(PKG, "CubeInput.Log.ErrorClosingCube") +
e.toString());
setErrors(1);
stopAll();
+ } finally {
+ data.dis = null;
+ data.zip = null;
+ data.fis = null;
}
+ }
+ @Override
+ public void dispose() {
+ closeFile();
super.dispose();
}
}
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputData.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputData.java
index e00beb60fe..153c8607dc 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputData.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputData.java
@@ -32,6 +32,12 @@ public class CubeInputData extends BaseTransformData
implements ITransformData {
public IRowMeta meta;
+ /**
+ * File whose layout {@link #meta} was read from. Every file opened after
that, including the
+ * first name from a field, is compared with it.
+ */
+ public String referenceFilename;
+
public CubeInputData() {
super();
}
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputDialog.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputDialog.java
index bfffe61619..125e4a480d 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputDialog.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputDialog.java
@@ -17,33 +17,23 @@
package org.apache.hop.pipeline.transforms.cubeinput;
-import org.apache.hop.core.Const;
import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.PipelineMeta;
-import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
-import org.apache.hop.ui.core.widget.TextVar;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
-import org.apache.hop.ui.pipeline.transform.ComponentSelectionListener;
-import org.eclipse.swt.SWT;
-import org.eclipse.swt.events.ModifyListener;
-import org.eclipse.swt.layout.FormAttachment;
-import org.eclipse.swt.layout.FormData;
import org.eclipse.swt.widgets.Button;
-import org.eclipse.swt.widgets.Label;
+import org.eclipse.swt.widgets.Control;
import org.eclipse.swt.widgets.Shell;
-import org.eclipse.swt.widgets.Text;
public class CubeInputDialog extends BaseTransformDialog {
private static final Class<?> PKG = CubeInputMeta.class;
- private TextVar wFilename;
-
- private Text wLimit;
- private Button wAddResult;
private final CubeInputMeta input;
+ private GuiCompositeWidgets widgets;
public CubeInputDialog(
Shell parent, IVariables variables, CubeInputMeta transformMeta,
PipelineMeta pipelineMeta) {
@@ -57,97 +47,80 @@ public class CubeInputDialog extends BaseTransformDialog {
buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
- ModifyListener lsMod = e -> input.setChanged();
changed = input.hasChanged();
- // Filename line
- Label wlFilename = new Label(shell, SWT.RIGHT);
- wlFilename.setText(BaseMessages.getString(PKG,
"CubeInputDialog.Filename.Label"));
- PropsUi.setLook(wlFilename);
- FormData fdlFilename = new FormData();
- fdlFilename.left = new FormAttachment(0, 0);
- fdlFilename.top = new FormAttachment(wSpacer, margin + 5);
- fdlFilename.right = new FormAttachment(middle, -margin);
- wlFilename.setLayoutData(fdlFilename);
- Button wbFilename = new Button(shell, SWT.PUSH | SWT.CENTER);
- PropsUi.setLook(wbFilename);
- wbFilename.setText(BaseMessages.getString(PKG,
"CubeInputDialog.FilenameButton.Label"));
- FormData fdbFilename = new FormData();
- fdbFilename.right = new FormAttachment(100, 0);
- fdbFilename.top = new FormAttachment(wSpacer, margin + 5);
- wbFilename.setLayoutData(fdbFilename);
- wFilename = new TextVar(variables, shell, SWT.SINGLE | SWT.LEFT |
SWT.BORDER);
- PropsUi.setLook(wFilename);
- wFilename.addModifyListener(lsMod);
- FormData fdFilename = new FormData();
- fdFilename.left = new FormAttachment(middle, 0);
- fdFilename.top = new FormAttachment(wSpacer, margin + 5);
- fdFilename.right = new FormAttachment(wbFilename, -margin);
- wFilename.setLayoutData(fdFilename);
-
- // Limit input ...
- Label wlLimit = new Label(shell, SWT.RIGHT);
- wlLimit.setText(BaseMessages.getString(PKG,
"CubeInputDialog.Limit.Label"));
- PropsUi.setLook(wlLimit);
- FormData fdlLimit = new FormData();
- fdlLimit.left = new FormAttachment(0, 0);
- fdlLimit.right = new FormAttachment(middle, -margin);
- fdlLimit.top = new FormAttachment(wFilename, margin);
- wlLimit.setLayoutData(fdlLimit);
- wLimit = new Text(shell, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
- PropsUi.setLook(wLimit);
- wLimit.addModifyListener(lsMod);
- FormData fdLimit = new FormData();
- fdLimit.left = new FormAttachment(middle, 0);
- fdLimit.top = new FormAttachment(wFilename, margin);
- fdLimit.right = new FormAttachment(100, 0);
- wLimit.setLayoutData(fdLimit);
-
- // Add filename to result filenames
- Label wlAddResult = new Label(shell, SWT.RIGHT);
- wlAddResult.setText(BaseMessages.getString(PKG,
"CubeInputDialog.AddResult.Label"));
- PropsUi.setLook(wlAddResult);
- FormData fdlAddResult = new FormData();
- fdlAddResult.left = new FormAttachment(0, 0);
- fdlAddResult.top = new FormAttachment(wLimit, margin);
- fdlAddResult.right = new FormAttachment(middle, -margin);
- wlAddResult.setLayoutData(fdlAddResult);
- wAddResult = new Button(shell, SWT.CHECK);
- PropsUi.setLook(wAddResult);
- wAddResult.setToolTipText(BaseMessages.getString(PKG,
"CubeInputDialog.AddResult.Tooltip"));
- FormData fdAddResult = new FormData();
- fdAddResult.left = new FormAttachment(middle, 0);
- fdAddResult.top = new FormAttachment(wlAddResult, 0, SWT.CENTER);
- wAddResult.setLayoutData(fdAddResult);
- wAddResult.addSelectionListener(new ComponentSelectionListener(input));
-
- wbFilename.addListener(
- SWT.Selection,
- e ->
- BaseDialog.presentFileDialog(
- shell,
- wFilename,
- variables,
- new String[] {"*.cube", "*"},
- new String[] {
- BaseMessages.getString(PKG,
"CubeInputDialog.FilterNames.CubeFiles"),
- BaseMessages.getString(PKG,
"CubeInputDialog.FilterNames.AllFiles")
- },
- false));
-
- getData();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ CubeInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ setFilenameFieldChoices();
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets compositeWidgets) {
+ // Fields come from the metadata annotations.
+ }
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets compositeWidgets) {
+ updateFieldMode();
+ }
+
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control changedWidget,
String widgetId) {
+ input.setChanged();
+ if (CubeInputMeta.WIDGET_FILENAME_IN_FIELD.equals(widgetId)) {
+ updateFieldMode();
+ }
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets compositeWidgets) {
+ // OK reads the widgets itself.
+ }
+ });
+ updateFieldMode();
+
focusTransformName();
BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
return transformName;
}
- /** Copy information from the meta-data input to the dialog fields. */
- public void getData() {
+ private void setFilenameFieldChoices() {
+ String[] fieldNames;
+ try {
+ fieldNames = pipelineMeta.getPrevTransformFields(variables,
transformName).getFieldNames();
+ } catch (Exception e) {
+ fieldNames = new String[] {};
+ }
+ widgets.setComboValues(CubeInputMeta.WIDGET_FILENAME_FIELD, fieldNames);
+ }
- wFilename.setText(Const.NVL(input.getFile().getName(), ""));
- wLimit.setText("" + input.getRowLimit());
- wAddResult.setSelection(input.isAddFilenameResult());
+ /**
+ * Filenames from a field and the transform-number suffix are alternatives.
The static filename
+ * stays available: it is the file the layout is read from.
+ */
+ private void updateFieldMode() {
+ Control filenameInField =
widgets.getWidgetsMap().get(CubeInputMeta.WIDGET_FILENAME_IN_FIELD);
+ Control includeTransformNr =
+ widgets.getWidgetsMap().get(CubeInputMeta.WIDGET_INCLUDE_TRANSFORM_NR);
+ Control filenameField =
widgets.getWidgetsMap().get(CubeInputMeta.WIDGET_FILENAME_FIELD);
+ boolean fromField = filenameInField instanceof Button button &&
button.getSelection();
+ if (fromField && includeTransformNr instanceof Button includeButton) {
+ includeButton.setSelection(false);
+ }
+ if (includeTransformNr != null) {
+ includeTransformNr.setEnabled(!fromField);
+ }
+ if (filenameField != null) {
+ filenameField.setEnabled(fromField);
+ }
}
private void cancel() {
@@ -161,12 +134,8 @@ public class CubeInputDialog extends BaseTransformDialog {
return;
}
- transformName = wTransformName.getText(); // return value
- // copy info to Meta class (input)
- input.getFile().setName(wFilename.getText());
- input.setRowLimit(wLimit.getText());
- input.setAddFilenameResult(wAddResult.getSelection());
-
+ widgets.getWidgetsContents(input,
CubeInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
dispose();
}
}
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMeta.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMeta.java
index 1741bda7ed..b22cffb285 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMeta.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMeta.java
@@ -23,15 +23,23 @@ import java.io.InputStream;
import java.util.List;
import java.util.Map;
import java.util.zip.GZIPInputStream;
-import org.apache.commons.vfs2.FileObject;
+import lombok.AccessLevel;
+import lombok.Getter;
+import lombok.Setter;
import org.apache.hop.core.CheckResult;
import org.apache.hop.core.ICheckResult;
import org.apache.hop.core.annotations.Transform;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopFileException;
import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.gui.plugin.ITypeFilename;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.i18n.BaseMessages;
@@ -40,6 +48,7 @@ import org.apache.hop.metadata.api.IHopMetadataProvider;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.BaseTransformMeta;
import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.cube.CubeFilename;
import org.apache.hop.resource.IResourceNaming;
import org.apache.hop.resource.ResourceDefinition;
@@ -51,15 +60,98 @@ import org.apache.hop.resource.ResourceDefinition;
categoryDescription =
"i18n:org.apache.hop.pipeline.transform:BaseTransform.Category.Input",
keywords = "i18n::CubeInputMeta.keyword",
documentationUrl = "/pipeline/transforms/serialize-de-from-file.html")
+@GuiPlugin
+@Getter
+@Setter
public class CubeInputMeta extends BaseTransformMeta<CubeInput, CubeInputData>
{
private static final Class<?> PKG = CubeInputMeta.class;
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID =
"CubeInputDialog.File";
+ public static final String WIDGET_FILENAME = "filename";
+ public static final String WIDGET_INCLUDE_TRANSFORM_NR =
"includeTransformNr";
+ public static final String WIDGET_FILENAME_IN_FIELD = "filenameInField";
+ public static final String WIDGET_FILENAME_FIELD = "filenameField";
+
+ public static final String GROUP_FILE = "File";
+
@HopMetadataProperty(key = "file")
private CubeFile file;
+ /**
+ * Dialog value for the cube filename. Existing pipelines store the name on
{@link CubeFile}, so
+ * this field is not serialized. The accessors keep it aligned with {@link
CubeFile#getName()}.
+ */
+ @Getter(AccessLevel.NONE)
+ @Setter(AccessLevel.NONE)
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME,
+ order = "0100",
+ type = GuiElementType.FILENAME,
+ typeFilename = CubeFileType.class,
+ label = "i18n::CubeInputDialog.Filename.Label",
+ toolTip = "i18n::CubeInputDialog.Filename.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
+ private String filename;
+
+ @GuiWidgetElement(
+ id = WIDGET_INCLUDE_TRANSFORM_NR,
+ order = "0200",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeInputDialog.IncludeTransformNr.Label",
+ toolTip = "i18n::CubeInputDialog.IncludeTransformNr.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
+ @HopMetadataProperty(key = "include_transform_nr")
+ private boolean includeTransformNr;
+
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_IN_FIELD,
+ order = "0300",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeInputDialog.FilenameInField.Label",
+ toolTip = "i18n::CubeInputDialog.FilenameInField.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
+ @HopMetadataProperty(key = "filename_in_field")
+ private boolean filenameInField;
+
+ @GuiWidgetElement(
+ id = WIDGET_FILENAME_FIELD,
+ order = "0400",
+ type = GuiElementType.COMBO,
+ label = "i18n::CubeInputDialog.FilenameField.Label",
+ toolTip = "i18n::CubeInputDialog.FilenameField.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
+ @HopMetadataProperty(key = "filename_field")
+ private String filenameField;
+
+ @GuiWidgetElement(
+ id = "rowLimit",
+ order = "0500",
+ type = GuiElementType.TEXT,
+ label = "i18n::CubeInputDialog.Limit.Label",
+ toolTip = "i18n::CubeInputDialog.Limit.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
@HopMetadataProperty(key = "limit")
private String rowLimit;
+ @GuiWidgetElement(
+ id = "addFilenameResult",
+ order = "0600",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeInputDialog.AddResult.Label",
+ toolTip = "i18n::CubeInputDialog.AddResult.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
@HopMetadataProperty(key = "addfilenameresult")
private boolean addFilenameResult;
@@ -68,21 +160,58 @@ public class CubeInputMeta extends
BaseTransformMeta<CubeInput, CubeInputData> {
file = new CubeFile();
}
+ /**
+ * Filename shown in the dialog and stored as {@code <file><name>}.
+ *
+ * @return the cube filename, or null when none is set
+ */
+ public String getFilename() {
+ if (file != null && file.getName() != null) {
+ return file.getName();
+ }
+ return filename;
+ }
+
+ /**
+ * @param filename the cube filename. Kept on {@link CubeFile} so existing
XML stays valid.
+ */
+ public void setFilename(String filename) {
+ this.filename = filename;
+ if (file == null) {
+ file = new CubeFile();
+ }
+ file.setName(filename);
+ }
+
+ /** The copy-number suffix applies to the static filename, not to names read
from a field. */
+ public boolean usesTransformNrInFilename() {
+ return includeTransformNr && !filenameInField;
+ }
+
@Override
public boolean consumesMainInput() {
- return false;
+ return filenameInField;
}
@Override
public boolean canStartWithoutInput() {
- return true;
+ return !filenameInField;
+ }
+
+ @Override
+ public String getMainInputRequirementHint() {
+ return BaseMessages.getString(PKG, "CubeInput.FileInField.Label");
}
@Override
public void setDefault() {
this.file = new CubeFile();
+ this.filename = null;
this.rowLimit = "0";
this.addFilenameResult = false;
+ this.includeTransformNr = false;
+ this.filenameInField = false;
+ this.filenameField = null;
}
@Override
@@ -94,16 +223,15 @@ public class CubeInputMeta extends
BaseTransformMeta<CubeInput, CubeInputData> {
IVariables variables,
IHopMetadataProvider metadataProvider)
throws HopTransformException {
- if (file == null || file.getName() == null) {
+ if (file == null || Utils.isEmpty(file.getName())) {
throw new HopTransformException(
BaseMessages.getString(PKG,
"CubeInputMeta.Exception.NoFilenameSpecified"));
}
+ // The layout of a cube lives in the file itself, so we have to open it.
Copy 0 is the file
+ // that carries the layout when each copy has its own file. Pass the
variables through so named
+ // VFS connections work here the same way they do at runtime.
String filename =
-
variables.resolve(file.getName().replace("${Internal.Transform.CopyNr}", "0"));
- // The layout of a cube lives in the file itself, so we have to open it.
Resolve the name
- // through the variables so that named VFS connections work here the same
way they do at
- // runtime in CubeInput.init(), and close the raw stream too when the GZIP
header can't be read.
- //
+ CubeFilename.resolve(variables, file.getName(),
usesTransformNrInFilename(), 0);
try (InputStream is = HopVfs.getInputStream(filename, variables);
GZIPInputStream fis = new GZIPInputStream(is);
DataInputStream dis = new DataInputStream(fis)) {
@@ -159,29 +287,42 @@ public class CubeInputMeta extends
BaseTransformMeta<CubeInput, CubeInputData> {
IResourceNaming iResourceNaming,
IHopMetadataProvider metadataProvider)
throws HopException {
- try {
- // The object that we're modifying here is a copy of the original!
- // So let's change the filename from relative to absolute by grabbing
the file object...
- //
- // From : ${Internal.Pipeline.Filename.Directory}/../foo/bar.data
- // To : /home/matt/test/files/foo/bar.data
- //
- FileObject fileObject =
HopVfs.getFileObject(variables.resolve(file.getName()));
-
- // If the file doesn't exist, forget about this effort too!
- //
- if (fileObject.exists()) {
- // Convert to an absolute path...
- //
- file.name = iResourceNaming.nameResource(fileObject, variables, true);
- return file.name;
- }
+ // The object that we're modifying here is a copy of the original.
+ // Map the folder of copy 0 and keep the stored file name, so each copy
still opens its own
+ // file after export.
+ String exported =
+ CubeFilename.exportResourceName(
+ variables, getFilename(), usesTransformNrInFilename(),
iResourceNaming);
+ if (exported == null) {
return null;
- } catch (Exception e) {
- throw new HopException(e);
}
+ setFilename(exported);
+ return exported;
}
+ /** Browse filter for {@code *.cube} files. */
+ public static class CubeFileType implements ITypeFilename {
+ @Override
+ public String getDefaultFileExtension() {
+ return ".cube";
+ }
+
+ @Override
+ public String[] getFilterExtensions() {
+ return new String[] {"*.cube", "*"};
+ }
+
+ @Override
+ public String[] getFilterNames() {
+ return new String[] {
+ BaseMessages.getString(PKG, "CubeInputDialog.FilterNames.CubeFiles"),
+ BaseMessages.getString(PKG, "CubeInputDialog.FilterNames.AllFiles")
+ };
+ }
+ }
+
+ @Getter
+ @Setter
public static class CubeFile {
@HopMetadataProperty private String name;
@@ -194,77 +335,5 @@ public class CubeInputMeta extends
BaseTransformMeta<CubeInput, CubeInputData> {
public CubeFile(CubeFile f) {
this.name = f.name;
}
-
- /**
- * Gets name
- *
- * @return value of name
- */
- public String getName() {
- return name;
- }
-
- /**
- * Sets name
- *
- * @param name value of name
- */
- public void setName(String name) {
- this.name = name;
- }
- }
-
- /**
- * Gets file
- *
- * @return value of file
- */
- public CubeFile getFile() {
- return file;
- }
-
- /**
- * Sets file
- *
- * @param file value of file
- */
- public void setFile(CubeFile file) {
- this.file = file;
- }
-
- /**
- * Gets rowLimit
- *
- * @return value of rowLimit
- */
- public String getRowLimit() {
- return rowLimit;
- }
-
- /**
- * Sets rowLimit
- *
- * @param rowLimit value of rowLimit
- */
- public void setRowLimit(String rowLimit) {
- this.rowLimit = rowLimit;
- }
-
- /**
- * Gets addFilenameResult
- *
- * @return value of addFilenameResult
- */
- public boolean isAddFilenameResult() {
- return addFilenameResult;
- }
-
- /**
- * Sets addFilenameResult
- *
- * @param addFilenameResult value of addFilenameResult
- */
- public void setAddFilenameResult(boolean addFilenameResult) {
- this.addFilenameResult = addFilenameResult;
}
}
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutput.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutput.java
index 59de22d5de..e2b687b410 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutput.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutput.java
@@ -34,6 +34,7 @@ import org.apache.hop.pipeline.Pipeline;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.BaseTransform;
import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.cube.CubeFilename;
/**
* Outputs a stream/series of rows to a file, effectively building a sort of
(compressed) microcube.
@@ -69,9 +70,7 @@ public class CubeOutput extends BaseTransform<CubeOutputMeta,
CubeOutputData> {
if (data.outputMeta == null) {
logBasic(
BaseMessages.getString(
- PKG,
- "CubeOutput.Log.UnableToDetermineEmptyStreamLayout",
- resolve(meta.getFilename())));
+ PKG, "CubeOutput.Log.UnableToDetermineEmptyStreamLayout",
resolvedFilename()));
data.outputMeta = new RowMeta();
}
}
@@ -199,9 +198,13 @@ public class CubeOutput extends
BaseTransform<CubeOutputMeta, CubeOutputData> {
return false;
}
+ private String resolvedFilename() {
+ return CubeFilename.resolve(this, meta.getFilename(),
meta.isIncludeTransformNr(), getCopy());
+ }
+
private void prepareFile() throws HopFileException {
try {
- String filename = resolve(meta.getFilename());
+ String filename = resolvedFilename();
FileObject fileObject = HopVfs.getFileObject(filename, variables);
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputDialog.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputDialog.java
index fe6ca6e757..915d999967 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputDialog.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputDialog.java
@@ -21,32 +21,18 @@ import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.pipeline.PipelineMeta;
-import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
-import org.apache.hop.ui.core.widget.TextVar;
+import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
+import org.apache.hop.ui.core.gui.IGuiPluginCompositeWidgetsListener;
import org.apache.hop.ui.pipeline.transform.BaseTransformDialog;
-import org.eclipse.swt.SWT;
-import org.eclipse.swt.custom.ScrolledComposite;
-import org.eclipse.swt.events.ModifyListener;
-import org.eclipse.swt.graphics.Rectangle;
-import org.eclipse.swt.layout.FillLayout;
-import org.eclipse.swt.layout.FormAttachment;
-import org.eclipse.swt.layout.FormData;
-import org.eclipse.swt.layout.FormLayout;
-import org.eclipse.swt.widgets.Button;
-import org.eclipse.swt.widgets.Composite;
-import org.eclipse.swt.widgets.Label;
+import org.eclipse.swt.widgets.Control;
import org.eclipse.swt.widgets.Shell;
public class CubeOutputDialog extends BaseTransformDialog {
private static final Class<?> PKG = CubeOutputMeta.class;
- private TextVar wFilename;
- private Button wCreatingParentFolders;
- private Button wAddToResult;
- private Button wDoNotOpenNewFileInit;
-
private final CubeOutputMeta input;
+ private GuiCompositeWidgets widgets;
public CubeOutputDialog(
Shell parent, IVariables variables, CubeOutputMeta transformMeta,
PipelineMeta pipelineMeta) {
@@ -60,162 +46,49 @@ public class CubeOutputDialog extends BaseTransformDialog {
buildButtonBar().ok(e -> ok()).cancel(e -> cancel()).build();
- ModifyListener lsMod = e -> input.setChanged();
changed = input.hasChanged();
- ScrolledComposite scrolledComposite = new ScrolledComposite(shell,
SWT.V_SCROLL | SWT.H_SCROLL);
- PropsUi.setLook(scrolledComposite);
- FormData fdScrolledComposite = new FormData();
- fdScrolledComposite.left = new FormAttachment(0, 0);
- fdScrolledComposite.top = new FormAttachment(wSpacer, 0);
- fdScrolledComposite.right = new FormAttachment(100, 0);
- fdScrolledComposite.bottom = new FormAttachment(wOk, -margin);
- scrolledComposite.setLayoutData(fdScrolledComposite);
- scrolledComposite.setLayout(new FillLayout());
-
- Composite wContent = new Composite(scrolledComposite, SWT.NONE);
- PropsUi.setLook(wContent);
- FormLayout contentLayout = new FormLayout();
- contentLayout.marginWidth = PropsUi.getFormMargin();
- contentLayout.marginHeight = PropsUi.getFormMargin();
- wContent.setLayout(contentLayout);
-
- // Filename line
- Label wlFilename = new Label(wContent, SWT.RIGHT);
- wlFilename.setText(BaseMessages.getString(PKG,
"CubeOutputDialog.Filename.Label"));
- PropsUi.setLook(wlFilename);
- FormData fdlFilename = new FormData();
- fdlFilename.left = new FormAttachment(0, 0);
- fdlFilename.top = new FormAttachment(0, margin);
- fdlFilename.right = new FormAttachment(middle, -margin);
- wlFilename.setLayoutData(fdlFilename);
- Button wbFilename = new Button(wContent, SWT.PUSH | SWT.CENTER);
- PropsUi.setLook(wbFilename);
- wbFilename.setText(BaseMessages.getString(PKG,
"CubeOutputDialog.Browse.Button"));
- FormData fdbFilename = new FormData();
- fdbFilename.right = new FormAttachment(100, 0);
- fdbFilename.top = new FormAttachment(0, margin);
- wbFilename.setLayoutData(fdbFilename);
-
- wFilename = new TextVar(variables, wContent, SWT.SINGLE | SWT.LEFT |
SWT.BORDER);
- PropsUi.setLook(wFilename);
- wFilename.addModifyListener(lsMod);
- FormData fdFilename = new FormData();
- fdFilename.left = new FormAttachment(middle, 0);
- fdFilename.top = new FormAttachment(0, margin);
- fdFilename.right = new FormAttachment(wbFilename, -margin);
- wFilename.setLayoutData(fdFilename);
-
- // Creating parent folders
- //
- Label wlCreatingParentFolders = new Label(wContent, SWT.RIGHT);
- wlCreatingParentFolders.setText(
- BaseMessages.getString(PKG,
"CubeOutputDialog.CreatingParentFolders.Label"));
- PropsUi.setLook(wlCreatingParentFolders);
- FormData fdlCreatingParentFolders = new FormData();
- fdlCreatingParentFolders.left = new FormAttachment(0, 0);
- fdlCreatingParentFolders.top = new FormAttachment(wFilename, margin);
- fdlCreatingParentFolders.right = new FormAttachment(middle, -margin);
- wlCreatingParentFolders.setLayoutData(fdlCreatingParentFolders);
- wCreatingParentFolders = new Button(wContent, SWT.CHECK);
- wCreatingParentFolders.setToolTipText(
- BaseMessages.getString(PKG,
"CubeOutputDialog.CreatingParentFolders.Tooltip"));
- PropsUi.setLook(wCreatingParentFolders);
- FormData fdCreatingParentFolders = new FormData();
- fdCreatingParentFolders.left = new FormAttachment(middle, 0);
- fdCreatingParentFolders.top = new FormAttachment(wlCreatingParentFolders,
0, SWT.CENTER);
- fdCreatingParentFolders.right = new FormAttachment(100, 0);
- wCreatingParentFolders.setLayoutData(fdCreatingParentFolders);
- wCreatingParentFolders.addListener(SWT.Selection, e -> input.setChanged());
-
- // Open new File at Init
- //
- Label wlDoNotOpenNewFileInit = new Label(wContent, SWT.RIGHT);
- wlDoNotOpenNewFileInit.setText(
- BaseMessages.getString(PKG,
"CubeOutputDialog.DoNotOpenNewFileInit.Label"));
- PropsUi.setLook(wlDoNotOpenNewFileInit);
- FormData fdlDoNotOpenNewFileInit = new FormData();
- fdlDoNotOpenNewFileInit.left = new FormAttachment(0, 0);
- fdlDoNotOpenNewFileInit.top = new FormAttachment(wCreatingParentFolders,
margin);
- fdlDoNotOpenNewFileInit.right = new FormAttachment(middle, -margin);
- wlDoNotOpenNewFileInit.setLayoutData(fdlDoNotOpenNewFileInit);
- wDoNotOpenNewFileInit = new Button(wContent, SWT.CHECK);
- wDoNotOpenNewFileInit.setToolTipText(
- BaseMessages.getString(PKG,
"CubeOutputDialog.DoNotOpenNewFileInit.Tooltip"));
- PropsUi.setLook(wDoNotOpenNewFileInit);
- FormData fdDoNotOpenNewFileInit = new FormData();
- fdDoNotOpenNewFileInit.left = new FormAttachment(middle, 0);
- fdDoNotOpenNewFileInit.top = new FormAttachment(wlDoNotOpenNewFileInit, 0,
SWT.CENTER);
- fdDoNotOpenNewFileInit.right = new FormAttachment(100, 0);
- wDoNotOpenNewFileInit.setLayoutData(fdDoNotOpenNewFileInit);
- wDoNotOpenNewFileInit.addListener(SWT.Selection, e -> input.setChanged());
-
- // Add File to the result files name
- //
- Label wlAddToResult = new Label(wContent, SWT.RIGHT);
- wlAddToResult.setText(BaseMessages.getString(PKG,
"CubeOutputDialog.AddFileToResult.Label"));
- PropsUi.setLook(wlAddToResult);
- FormData fdlAddToResult = new FormData();
- fdlAddToResult.left = new FormAttachment(0, 0);
- fdlAddToResult.top = new FormAttachment(wDoNotOpenNewFileInit, margin);
- fdlAddToResult.right = new FormAttachment(middle, -margin);
- wlAddToResult.setLayoutData(fdlAddToResult);
- wAddToResult = new Button(wContent, SWT.CHECK);
- wAddToResult.setToolTipText(
- BaseMessages.getString(PKG,
"CubeOutputDialog.AddFileToResult.Tooltip"));
- PropsUi.setLook(wAddToResult);
- FormData fdAddToResult = new FormData();
- fdAddToResult.left = new FormAttachment(middle, 0);
- fdAddToResult.top = new FormAttachment(wlAddToResult, 0, SWT.CENTER);
- fdAddToResult.right = new FormAttachment(100, 0);
- wAddToResult.setLayoutData(fdAddToResult);
- wAddToResult.addListener(SWT.Selection, e -> input.setChanged());
-
- wContent.pack();
- Rectangle bounds = wContent.getBounds();
- scrolledComposite.setContent(wContent);
- scrolledComposite.setExpandHorizontal(true);
- scrolledComposite.setExpandVertical(true);
- scrolledComposite.setMinWidth(bounds.width);
- scrolledComposite.setMinHeight(bounds.height);
-
- // Add listeners
- wbFilename.addListener(
- SWT.Selection,
- e ->
- BaseDialog.presentFileDialog(
- true,
- shell,
- wFilename,
- variables,
- new String[] {"*.cube", "*"},
- new String[] {
- BaseMessages.getString(PKG,
"CubeOutputDialog.FilterNames.Options.CubeFiles"),
- BaseMessages.getString(PKG,
"CubeOutputDialog.FilterNames.Options.AllFiles")
- },
- true));
-
- getData();
+ widgets =
+ GuiCompositeWidgets.addScrolledComposite(
+ shell,
+ variables,
+ wTransformName,
+ wOk,
+ CubeOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID,
+ input);
+ widgets.setCompositeWidgetsListener(
+ new IGuiPluginCompositeWidgetsListener() {
+ @Override
+ public void widgetsCreated(GuiCompositeWidgets compositeWidgets) {
+ // Fields come from the metadata annotations.
+ }
+
+ @Override
+ public void widgetsPopulated(GuiCompositeWidgets compositeWidgets) {
+ // Nothing to enable or disable.
+ }
+
+ @Override
+ public void widgetModified(
+ GuiCompositeWidgets compositeWidgets, Control changedWidget,
String widgetId) {
+ input.setChanged();
+ }
+
+ @Override
+ public void persistContents(GuiCompositeWidgets compositeWidgets) {
+ // OK reads the widgets itself.
+ }
+ });
+
focusTransformName();
BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
return transformName;
}
- /** Copy information from the meta-data input to the dialog fields. */
- public void getData() {
- if (input.getFilename() != null) {
- wFilename.setText(input.getFilename());
- }
-
wCreatingParentFolders.setSelection(input.isFilenameCreatingParentFolders());
- wDoNotOpenNewFileInit.setSelection(input.isDoNotOpenNewFileInit());
- wAddToResult.setSelection(input.isAddToResultFilenames());
- }
-
private void cancel() {
transformName = null;
input.setChanged(changed);
-
dispose();
}
@@ -224,12 +97,8 @@ public class CubeOutputDialog extends BaseTransformDialog {
return;
}
- transformName = wTransformName.getText(); // return value
-
input.setFilenameCreatingParentFolders(wCreatingParentFolders.getSelection());
- input.setAddToResultFilenames(wAddToResult.getSelection());
- input.setDoNotOpenNewFileInit(wDoNotOpenNewFileInit.getSelection());
- input.setFilename(wFilename.getText());
-
+ widgets.getWidgetsContents(input,
CubeOutputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
+ transformName = wTransformName.getText();
dispose();
}
}
diff --git
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMeta.java
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMeta.java
index c787d20783..28fe5a2e7e 100644
---
a/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMeta.java
+++
b/plugins/transforms/cube/src/main/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMeta.java
@@ -19,14 +19,19 @@ package org.apache.hop.pipeline.transforms.cubeoutput;
import java.util.List;
import java.util.Map;
-import org.apache.commons.vfs2.FileObject;
+import lombok.Getter;
+import lombok.Setter;
import org.apache.hop.core.CheckResult;
import org.apache.hop.core.ICheckResult;
import org.apache.hop.core.annotations.Transform;
import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.gui.plugin.GuiElementType;
+import org.apache.hop.core.gui.plugin.GuiPlugin;
+import org.apache.hop.core.gui.plugin.GuiWidgetElement;
+import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
+import org.apache.hop.core.gui.plugin.ITypeFilename;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.variables.IVariables;
-import org.apache.hop.core.vfs.HopVfs;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.api.HopMetadataProperty;
import org.apache.hop.metadata.api.HopMetadataWrapper;
@@ -34,6 +39,7 @@ import org.apache.hop.metadata.api.IHopMetadataProvider;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.BaseTransformMeta;
import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.cube.CubeFilename;
import org.apache.hop.resource.IResourceNaming;
import org.apache.hop.resource.ResourceDefinition;
@@ -46,31 +52,90 @@ import org.apache.hop.resource.ResourceDefinition;
keywords = "i18n::CubeOutputMeta.keyword",
documentationUrl = "/pipeline/transforms/serialize-to-file.html")
@HopMetadataWrapper(tag = "file")
+@GuiPlugin
+@Getter
+@Setter
public class CubeOutputMeta extends BaseTransformMeta<CubeOutput,
CubeOutputData> {
private static final Class<?> PKG = CubeOutputMeta.class;
+ public static final String GUI_PLUGIN_ELEMENT_PARENT_ID =
"CubeOutputDialog.File";
+ public static final String WIDGET_INCLUDE_TRANSFORM_NR =
"includeTransformNr";
+
+ public static final String GROUP_FILE = "File";
+
+ @GuiWidgetElement(
+ id = "filename",
+ order = "0100",
+ type = GuiElementType.FILENAME,
+ typeFilename = CubeFileType.class,
+ label = "i18n::CubeOutputDialog.Filename.Label",
+ toolTip = "i18n::CubeOutputDialog.Filename.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
@HopMetadataProperty(key = "name")
private String filename;
+ @GuiWidgetElement(
+ id = WIDGET_INCLUDE_TRANSFORM_NR,
+ order = "0200",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeOutputDialog.IncludeTransformNr.Label",
+ toolTip = "i18n::CubeOutputDialog.IncludeTransformNr.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
+ @HopMetadataProperty(key = "include_transform_nr")
+ private boolean includeTransformNr;
+
+ @GuiWidgetElement(
+ id = "filenameCreatingParentFolders",
+ order = "0300",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeOutputDialog.CreatingParentFolders.Label",
+ toolTip = "i18n::CubeOutputDialog.CreatingParentFolders.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
@HopMetadataProperty(key = "filename_create_parent_folders")
private boolean filenameCreatingParentFolders;
- /** Flag: add the filenames to result filenames */
- @HopMetadataProperty(key = "add_to_result_filenames")
- private boolean addToResultFilenames;
-
/** Flag : Do not open new file when pipeline start */
+ @GuiWidgetElement(
+ id = "doNotOpenNewFileInit",
+ order = "0400",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeOutputDialog.DoNotOpenNewFileInit.Label",
+ toolTip = "i18n::CubeOutputDialog.DoNotOpenNewFileInit.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
@HopMetadataProperty(key = "do_not_open_newfile_init")
private boolean doNotOpenNewFileInit;
+ /** Flag: add the filenames to result filenames */
+ @GuiWidgetElement(
+ id = "addToResultFilenames",
+ order = "0500",
+ type = GuiElementType.CHECKBOX,
+ label = "i18n::CubeOutputDialog.AddFileToResult.Label",
+ toolTip = "i18n::CubeOutputDialog.AddFileToResult.Tooltip",
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_FILE)
+ @HopMetadataProperty(key = "add_to_result_filenames")
+ private boolean addToResultFilenames;
+
public CubeOutputMeta() {
- super(); // allocate BaseTransformMeta
+ super();
}
@Override
public void setDefault() {
addToResultFilenames = false;
doNotOpenNewFileInit = false;
+ filenameCreatingParentFolders = false;
+ includeTransformNr = false;
}
@Override
@@ -119,93 +184,36 @@ public class CubeOutputMeta extends
BaseTransformMeta<CubeOutput, CubeOutputData
IResourceNaming iResourceNaming,
IHopMetadataProvider metadataProvider)
throws HopException {
- try {
- // The object that we're modifying here is a copy of the original!
- // So let's change the filename from relative to absolute by grabbing
the file object...
- //
- // From : ${Internal.Pipeline.Filename.Directory}/../foo/bar.data
- // To : /home/matt/test/files/foo/bar.data
- //
- FileObject fileObject =
HopVfs.getFileObject(variables.resolve(filename));
-
- // If the file doesn't exist, forget about this effort too!
- //
- if (fileObject.exists()) {
- // Convert to an absolute path...
- //
- filename = iResourceNaming.nameResource(fileObject, variables, true);
-
- return filename;
- }
+ // The object that we're modifying here is a copy of the original.
+ // Map the folder of copy 0 and keep the stored file name, so each copy
still opens its own
+ // file after export.
+ String exported =
+ CubeFilename.exportResourceName(variables, filename,
includeTransformNr, iResourceNaming);
+ if (exported == null) {
return null;
- } catch (Exception e) {
- throw new HopException(e);
}
- }
-
- /**
- * Gets filename
- *
- * @return value of filename
- */
- public String getFilename() {
+ filename = exported;
return filename;
}
- /**
- * Sets filename
- *
- * @param filename value of filename
- */
- public void setFilename(String filename) {
- this.filename = filename;
- }
-
- /** Gets filename creating parent folders */
- public boolean isFilenameCreatingParentFolders() {
- return filenameCreatingParentFolders;
- }
-
- /**
- * @param filenameCreatingParentFolders The filenameCreatingParentFolders to
set
- */
- public void setFilenameCreatingParentFolders(boolean
filenameCreatingParentFolders) {
- this.filenameCreatingParentFolders = filenameCreatingParentFolders;
- }
-
- /**
- * Gets addToResultFilenames
- *
- * @return value of addToResultFilenames
- */
- public boolean isAddToResultFilenames() {
- return addToResultFilenames;
- }
-
- /**
- * Sets addToResultFilenames
- *
- * @param addToResultFilenames value of addToResultFilenames
- */
- public void setAddToResultFilenames(boolean addToResultFilenames) {
- this.addToResultFilenames = addToResultFilenames;
- }
+ /** Browse filter for {@code *.cube} files. */
+ public static class CubeFileType implements ITypeFilename {
+ @Override
+ public String getDefaultFileExtension() {
+ return ".cube";
+ }
- /**
- * Gets doNotOpenNewFileInit
- *
- * @return value of doNotOpenNewFileInit
- */
- public boolean isDoNotOpenNewFileInit() {
- return doNotOpenNewFileInit;
- }
+ @Override
+ public String[] getFilterExtensions() {
+ return new String[] {"*.cube", "*"};
+ }
- /**
- * Sets doNotOpenNewFileInit
- *
- * @param doNotOpenNewFileInit value of doNotOpenNewFileInit
- */
- public void setDoNotOpenNewFileInit(boolean doNotOpenNewFileInit) {
- this.doNotOpenNewFileInit = doNotOpenNewFileInit;
+ @Override
+ public String[] getFilterNames() {
+ return new String[] {
+ BaseMessages.getString(PKG,
"CubeOutputDialog.FilterNames.Options.CubeFiles"),
+ BaseMessages.getString(PKG,
"CubeOutputDialog.FilterNames.Options.AllFiles")
+ };
+ }
}
}
diff --git
a/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeinput/messages/messages_en_US.properties
b/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeinput/messages/messages_en_US.properties
index d6b9d056d8..1ad602d00b 100644
---
a/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeinput/messages/messages_en_US.properties
+++
b/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeinput/messages/messages_en_US.properties
@@ -19,19 +19,31 @@ CubeInput.Description=Read rows of data from a data cube.
CubeInput.Log.ErrorClosingCube=Error closing cube input file:
CubeInput.Log.ErrorReadingFromDataCube=Error reading from data cube :
CubeInput.Log.LineNumber=linenr
+CubeInput.FileInField.Label=To read filenames from the previous transform,
enable "Filename is defined in a field".
CubeInput.Log.UnableToReadMetadata=INIT: Unable to read metadata from cube
file:
CubeInput.Name=De-serialize from file
CubeInputDialog.AddResult.Label=Add filename to result
CubeInputDialog.AddResult.Tooltip=Add filename to result filenames
CubeInputDialog.Filename.Label=Filename
+CubeInputDialog.Filename.Tooltip=Cube file to read. When filenames come from a
field, this file is only used to determine the layout and must match the files
you read.
CubeInputDialog.FilenameButton.Label=&Browse...
+CubeInputDialog.FilenameField.Label=Filename field
+CubeInputDialog.FilenameField.Tooltip=Field that contains the full name of the
cube file to read
+CubeInputDialog.FilenameInField.Label=Filename is defined in a field
+CubeInputDialog.FilenameInField.Tooltip=Read one cube file for each row from
the previous transform. The filename above is then only used to determine the
layout.
+CubeInputDialog.IncludeTransformNr.Label=Include transform nr in filename
+CubeInputDialog.IncludeTransformNr.Tooltip=Add the transform copy number
before the file extension. Copy 1 of data.cube is data_1.cube. You can instead
put '${Internal.Transform.CopyNr}' in the filename. Using both inserts the
number twice.
+CubeInputDialog.Limit.Tooltip=Maximum number of rows to read from the cube
file or files. 0 reads every row. A variable such as '${ROW_LIMIT}' is allowed.
CubeInputDialog.FilterNames.AllFiles=All files
CubeInputDialog.FilterNames.CubeFiles=Cube files
CubeInputDialog.Limit.Label=Limit size
CubeInputDialog.Shell.Title=De-serialize from file
CubeInputDialog.TransformName.Label=Transform name
CubeInputMeta.CheckResult.FileSpecificationsNotChecked=File specifications are
not checked.
+CubeInputMeta.Exception.EmptyFilename=The filename field is empty
CubeInputMeta.Exception.ErrorOpeningOrReadingCubeFile=Error opening/reading
cube file [{0}]
+CubeInputMeta.Exception.FilenameFieldNotFound=Filename field [{0}] could not
be found in the input row
+CubeInputMeta.Exception.LayoutMismatch=Cube file [{0}] has a different layout
than [{1}]
CubeInputMeta.Exception.NoFilenameSpecified=No cube file name has been
specified for this transform
CubeInputMeta.Exception.UnableToCloseCubeFile=Unable to close cube file
CubeInputMeta.Exception.UnableToLoadTransformMeta=Unable to load transform
info from XML
diff --git
a/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeoutput/messages/messages_en_US.properties
b/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeoutput/messages/messages_en_US.properties
index 7fdc7fe97b..47a7ee9060 100644
---
a/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeoutput/messages/messages_en_US.properties
+++
b/plugins/transforms/cube/src/main/resources/org/apache/hop/pipeline/transforms/cubeoutput/messages/messages_en_US.properties
@@ -32,7 +32,10 @@ CubeOutputDialog.DefaultTransformName=write to cube file
CubeOutputDialog.DoNotOpenNewFileInit.Label=Do not create file at start
CubeOutputDialog.DoNotOpenNewFileInit.Tooltip=Check this if you don''t want to
create file at pipeline starts.\n Apache Hop will create file at first row
received.
CubeOutputDialog.Filename.Label=Filename
+CubeOutputDialog.Filename.Tooltip=Cube file to write. Each copy writes the
rows it receives. Copies that share this name overwrite the file.
CubeOutputDialog.FilterNames.Options.AllFiles=All files
+CubeOutputDialog.IncludeTransformNr.Label=Include transform nr in filename
+CubeOutputDialog.IncludeTransformNr.Tooltip=Add the transform copy number
before the file extension. Copy 1 of data.cube is written as data_1.cube. You
can instead put '${Internal.Transform.CopyNr}' in the filename. Using both
inserts the number twice.
CubeOutputDialog.FilterNames.Options.CubeFiles=Cube files
CubeOutputDialog.Shell.Text=Serialize to file
CubeOutputDialog.TransformName.Label=Transform name
diff --git
a/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cube/CubeFilenameTest.java
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cube/CubeFilenameTest.java
new file mode 100644
index 0000000000..bdb7317de0
--- /dev/null
+++
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cube/CubeFilenameTest.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.cube;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import org.apache.hop.core.Const;
+import org.apache.hop.core.variables.Variables;
+import org.junit.jupiter.api.Test;
+
+class CubeFilenameTest {
+
+ @Test
+ void insertCopyNumberBeforeTheExtension() {
+ Variables variables = new Variables();
+
+ assertEquals(
+ "/tmp/dir.v1/data_1.cube",
+ CubeFilename.resolve(variables, "/tmp/dir.v1/data.cube", true, 1));
+ assertEquals(
+ "C:\\data\\file_2.cube", CubeFilename.resolve(variables,
"C:\\data\\file.cube", true, 2));
+ assertEquals("/tmp/data_0", CubeFilename.resolve(variables, "/tmp/data",
true, 0));
+ }
+
+ @Test
+ void copyVariableReplacesAStaleValueAndANestedVariable() {
+ Variables variables = new Variables();
+ variables.setVariable(Const.INTERNAL_VARIABLE_TRANSFORM_COPYNR, "9");
+ variables.setVariable("FILE",
"/tmp/nested_${Internal.Transform.CopyNr}.cube");
+
+ assertEquals(
+ "/tmp/direct_0.cube",
+ CubeFilename.resolve(variables,
"/tmp/direct_${Internal.Transform.CopyNr}.cube", false, 0));
+ assertEquals("/tmp/nested_0.cube", CubeFilename.resolve(variables,
"${FILE}", false, 0));
+ assertEquals(
+ "/tmp/case_3.cube",
+ CubeFilename.resolve(variables,
"/tmp/case_${internal.transform.copynr}.cube", false, 3));
+ assertEquals(
+ "/tmp/win_4.cube",
+ CubeFilename.resolve(variables,
"/tmp/win_%%Internal.Transform.CopyNr%%.cube", false, 4));
+ }
+
+ @Test
+ void checkboxAndVariableBothInsertTheNumber() {
+ assertEquals(
+ "/tmp/data_1_1.cube",
+ CubeFilename.resolve(
+ new Variables(), "/tmp/data_${Internal.Transform.CopyNr}.cube",
true, 1));
+ }
+}
diff --git
a/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMetaTest.java
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMetaTest.java
index 405f04d87d..32d27cf4bc 100644
---
a/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMetaTest.java
+++
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeinput/CubeInputMetaTest.java
@@ -17,24 +17,45 @@
package org.apache.hop.pipeline.transforms.cubeinput;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.io.DataOutputStream;
import java.io.EOFException;
+import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
+import java.util.zip.GZIPOutputStream;
+import org.apache.hop.core.Const;
+import org.apache.hop.core.HopEnvironment;
import org.apache.hop.core.exception.HopTransformException;
+import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaString;
import org.apache.hop.core.variables.Variables;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
import org.apache.hop.pipeline.transform.TransformSerializationTestUtil;
+import org.apache.hop.resource.SimpleResourceNaming;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
import org.junit.jupiter.api.io.TempDir;
class CubeInputMetaTest {
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
@TempDir Path tempDir;
+ @BeforeAll
+ static void setUpBeforeClass() throws Exception {
+ HopEnvironment.init();
+ }
+
@Test
void testRoundTrip() throws Exception {
CubeInputMeta meta =
@@ -42,6 +63,23 @@ class CubeInputMetaTest {
"/de-serialize-transform.xml", CubeInputMeta.class);
assertNotNull(meta.getFile());
assertNotNull(meta.getFile().getName());
+ assertFalse(meta.isIncludeTransformNr());
+ assertFalse(meta.isFilenameInField());
+ }
+
+ @Test
+ void filenameFromAFieldRoundTrips() throws Exception {
+ CubeInputMeta meta =
+ TransformSerializationTestUtil.testSerialization(
+ "/de-serialize-from-field.xml", CubeInputMeta.class);
+
+ assertTrue(meta.isFilenameInField());
+ assertFalse(meta.isIncludeTransformNr());
+ assertEquals("filename", meta.getFilenameField());
+ assertEquals("5", meta.getRowLimit());
+ assertTrue(meta.isAddFilenameResult());
+ assertTrue(meta.consumesMainInput());
+ assertFalse(meta.canStartWithoutInput());
}
/**
@@ -79,4 +117,133 @@ class CubeInputMetaTest {
assertTrue(e.getMessage().contains(cubeFile.toString()), e.getMessage());
assertTrue(e.getCause() instanceof EOFException,
String.valueOf(e.getCause()));
}
+
+ @Test
+ void
getFieldsWithoutASampleNameStillReportsTheMisconfigurationWhenReadingFromAField()
{
+ CubeInputMeta meta = new CubeInputMeta();
+ meta.setFilenameInField(true);
+ meta.setFilenameField("filename");
+
+ HopTransformException e =
+ assertThrows(
+ HopTransformException.class,
+ () -> meta.getFields(new RowMeta(), "cube input", null, null, new
Variables(), null));
+ assertTrue(e.getMessage().contains("cube file name"), e.getMessage());
+ }
+
+ @Test
+ void getFieldsOpensCopyZeroWhenTheCopyVariableIsMissingOrStale() throws
Exception {
+ writeCube(tempDir.resolve("direct_0.cube"));
+ writeCube(tempDir.resolve("case_0.cube"));
+ writeCube(tempDir.resolve("win_0.cube"));
+ writeCube(tempDir.resolve("nested_0.cube"));
+ writeCube(tempDir.resolve("stale_0.cube"));
+ writeCube(tempDir.resolve("data_0.cube"));
+
+ Variables variables = new Variables();
+ variables.setVariable(Const.INTERNAL_VARIABLE_TRANSFORM_COPYNR, "9");
+ variables.setVariable(
+ "FILE",
tempDir.resolve("nested_${Internal.Transform.CopyNr}.cube").toString());
+
+ assertField(
+ readFields(
+
tempDir.resolve("direct_${Internal.Transform.CopyNr}.cube").toString(),
variables));
+ assertField(
+ readFields(
+
tempDir.resolve("case_${internal.transform.copynr}.cube").toString(),
variables));
+ assertField(
+ readFields(tempDir.toString() +
"/win_%%Internal.Transform.CopyNr%%.cube", variables));
+ assertField(readFields("${FILE}", variables));
+ assertField(
+ readFields(
+
tempDir.resolve("stale_${Internal.Transform.CopyNr}.cube").toString(),
variables));
+
+ CubeInputMeta numbered = new CubeInputMeta();
+ numbered.setDefault();
+ numbered.setIncludeTransformNr(true);
+ numbered.getFile().setName(tempDir.resolve("data.cube").toString());
+ IRowMeta fields = new RowMeta();
+ numbered.getFields(fields, "cube input", null, null, variables, null);
+ assertField(fields);
+ }
+
+ @Test
+ void getFieldsNamesTheResolvedCopyZeroFileWhenItIsMissing() {
+ CubeInputMeta meta = new CubeInputMeta();
+ meta.setDefault();
+
meta.getFile().setName(tempDir.resolve("missing_${Internal.Transform.CopyNr}.cube").toString());
+
+ HopTransformException e =
+ assertThrows(
+ HopTransformException.class,
+ () -> meta.getFields(new RowMeta(), "cube input", null, null, new
Variables(), null));
+ assertTrue(e.getMessage().contains("missing_0.cube"), e.getMessage());
+ assertFalse(e.getMessage().contains("${"), e.getMessage());
+ }
+
+ @Test
+ void exportResourcesKeepsANamePerCopy() throws Exception {
+ writeCube(tempDir.resolve("export_0.cube"));
+ Variables variables = new Variables();
+ variables.setVariable("DIR", tempDir.toString());
+ CubeInputMeta meta = new CubeInputMeta();
+ meta.setDefault();
+ meta.getFile().setName("${DIR}/export_${Internal.Transform.CopyNr}.cube");
+
+ String exported = meta.exportResources(variables, null, new
SimpleResourceNaming(), null);
+
+ assertNotNull(exported);
+ assertTrue(exported.contains("/export_${Internal.Transform.CopyNr}.cube"),
exported);
+ assertFalse(exported.contains("export_0"), exported);
+ assertTrue(meta.getFilename().contains("Internal.Transform.CopyNr"),
meta.getFilename());
+ assertFalse(meta.isIncludeTransformNr());
+
+ writeCube(tempDir.resolve("numbered_0.cube"));
+ CubeInputMeta numbered = new CubeInputMeta();
+ numbered.setDefault();
+ numbered.setIncludeTransformNr(true);
+ numbered.getFile().setName("${DIR}/numbered.cube");
+
+ String numberedExport =
+ numbered.exportResources(variables, null, new SimpleResourceNaming(),
null);
+
+ assertNotNull(numberedExport);
+ assertTrue(numberedExport.contains("/numbered.cube"), numberedExport);
+ assertFalse(numberedExport.contains("numbered_0"), numberedExport);
+ assertTrue(numbered.isIncludeTransformNr());
+
+ writeCube(tempDir.resolve("plain.cube"));
+ CubeInputMeta plain = new CubeInputMeta();
+ plain.setDefault();
+ plain.getFile().setName(tempDir.resolve("plain.cube").toString());
+
+ String plainExport =
+ plain.exportResources(new Variables(), null, new
SimpleResourceNaming(), null);
+
+ assertNotNull(plainExport);
+ assertTrue(plainExport.endsWith("/plain.cube"), plainExport);
+ }
+
+ private IRowMeta readFields(String filename, Variables variables) throws
Exception {
+ CubeInputMeta meta = new CubeInputMeta();
+ meta.setDefault();
+ meta.getFile().setName(filename);
+ IRowMeta fields = new RowMeta();
+ meta.getFields(fields, "cube input", null, null, variables, null);
+ return fields;
+ }
+
+ private static void assertField(IRowMeta fields) {
+ assertEquals(1, fields.size());
+ assertEquals("name", fields.getValueMeta(0).getName());
+ }
+
+ private static void writeCube(Path file) throws Exception {
+ IRowMeta layout = new RowMeta();
+ layout.addValueMeta(new ValueMetaString("name"));
+ try (OutputStream os = Files.newOutputStream(file);
+ DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os)))
{
+ layout.writeMeta(dos);
+ }
+ }
}
diff --git
a/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMetaTest.java
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMetaTest.java
index 2fe59631da..bf9c41f651 100644
---
a/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMetaTest.java
+++
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeOutputMetaTest.java
@@ -17,13 +17,41 @@
package org.apache.hop.pipeline.transforms.cubeoutput;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.io.DataOutputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.zip.GZIPOutputStream;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
import org.apache.hop.pipeline.transform.TransformSerializationTestUtil;
+import org.apache.hop.resource.SimpleResourceNaming;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
class CubeOutputMetaTest {
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ @TempDir Path tempDir;
+
+ @BeforeAll
+ static void setUpBeforeClass() throws Exception {
+ HopEnvironment.init();
+ }
+
@Test
void testRoundTrip() throws Exception {
CubeOutputMeta meta =
@@ -31,5 +59,67 @@ class CubeOutputMetaTest {
"/serialize-transform.xml", CubeOutputMeta.class);
assertNotNull(meta.getFilename());
+ assertFalse(meta.isIncludeTransformNr());
+ }
+
+ @Test
+ void includeTransformNrRoundTrips() throws Exception {
+ CubeOutputMeta meta =
+ TransformSerializationTestUtil.testSerialization(
+ "/serialize-with-copy-nr.xml", CubeOutputMeta.class);
+
+ assertTrue(meta.isIncludeTransformNr());
+ assertTrue(meta.isFilenameCreatingParentFolders());
+ }
+
+ @Test
+ void exportResourcesKeepsANamePerCopy() throws Exception {
+ writeCube(tempDir.resolve("export_0.cube"));
+ Variables variables = new Variables();
+ variables.setVariable("DIR", tempDir.toString());
+
+ CubeOutputMeta meta = new CubeOutputMeta();
+ meta.setDefault();
+ meta.setFilename("${DIR}/export_${Internal.Transform.CopyNr}.cube");
+
+ String exported = meta.exportResources(variables, null, new
SimpleResourceNaming(), null);
+
+ assertNotNull(exported);
+ assertTrue(exported.contains("/export_${Internal.Transform.CopyNr}.cube"),
exported);
+ assertFalse(exported.contains("export_0"), exported);
+ assertTrue(meta.getFilename().contains("Internal.Transform.CopyNr"),
meta.getFilename());
+ assertFalse(meta.isIncludeTransformNr());
+
+ writeCube(tempDir.resolve("numbered_0.cube"));
+ CubeOutputMeta numbered = new CubeOutputMeta();
+ numbered.setDefault();
+ numbered.setIncludeTransformNr(true);
+ numbered.setFilename("${DIR}/numbered.cube");
+
+ String numberedExport =
+ numbered.exportResources(variables, null, new SimpleResourceNaming(),
null);
+
+ assertNotNull(numberedExport);
+ assertTrue(numberedExport.contains("/numbered.cube"), numberedExport);
+ assertFalse(numberedExport.contains("numbered_0"), numberedExport);
+ assertTrue(numbered.isIncludeTransformNr());
+
+ CubeOutputMeta missing = new CubeOutputMeta();
+ missing.setDefault();
+ missing.setIncludeTransformNr(true);
+ missing.setFilename(tempDir.resolve("missing.cube").toString());
+
+ assertNull(missing.exportResources(new Variables(), null, new
SimpleResourceNaming(), null));
+ assertTrue(missing.isIncludeTransformNr());
+ assertTrue(missing.getFilename().endsWith("missing.cube"),
missing.getFilename());
+ }
+
+ private static void writeCube(Path file) throws Exception {
+ IRowMeta layout = new RowMeta();
+ layout.addValueMeta(new ValueMetaString("name"));
+ try (OutputStream os = Files.newOutputStream(file);
+ DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os)))
{
+ layout.writeMeta(dos);
+ }
}
}
diff --git
a/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeParallelFileTest.java
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeParallelFileTest.java
new file mode 100644
index 0000000000..6b495b8074
--- /dev/null
+++
b/plugins/transforms/cube/src/test/java/org/apache/hop/pipeline/transforms/cubeoutput/CubeParallelFileTest.java
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.cubeoutput;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.exception.HopEofException;
+import org.apache.hop.core.logging.HopLogStore;
+import org.apache.hop.core.logging.LoggingObject;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.pipeline.PipelineHopMeta;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.RowProducer;
+import org.apache.hop.pipeline.engines.local.LocalPipelineEngine;
+import org.apache.hop.pipeline.transform.RowAdapter;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.cubeinput.CubeInputMeta;
+import org.apache.hop.pipeline.transforms.injector.InjectorField;
+import org.apache.hop.pipeline.transforms.injector.InjectorMeta;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+/** Several cube files written and read by transform copies, and by a filename
field. */
+class CubeParallelFileTest {
+
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ @TempDir Path tempDir;
+
+ @BeforeAll
+ static void setUpBeforeClass() throws Exception {
+ HopEnvironment.init();
+ }
+
+ @Test
+ void twoCopiesWriteAndReadTheirOwnFiles() throws Exception {
+ Path base = tempDir.resolve("data.cube");
+ writeTwoValues(base);
+
+ List<String> written = new ArrayList<>();
+ written.addAll(readValues(tempDir.resolve("data_0.cube")));
+ written.addAll(readValues(tempDir.resolve("data_1.cube")));
+ assertEquals(Set.of("one", "two"), new HashSet<>(written));
+ assertEquals(2, written.size());
+
+ CubeInputMeta inputMeta = new CubeInputMeta();
+ inputMeta.setDefault();
+ inputMeta.setIncludeTransformNr(true);
+ inputMeta.getFile().setName(base.toString());
+
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.setName("read copies");
+ TransformMeta input = new TransformMeta("read", inputMeta);
+ input.setCopies(2);
+ pipelineMeta.addTransform(input);
+
+ LocalPipelineEngine pipeline =
+ new LocalPipelineEngine(pipelineMeta, new Variables(), new
LoggingObject("read copies"));
+ pipeline.prepareExecution();
+ List<String> read = new CopyOnWriteArrayList<>();
+ for (int copy = 0; copy < 2; copy++) {
+ pipeline
+ .getTransform("read", copy)
+ .addRowListener(
+ new RowAdapter() {
+ @Override
+ public void rowWrittenEvent(IRowMeta rowMeta, Object[] row) {
+ read.add((String) row[0]);
+ }
+ });
+ }
+ pipeline.startThreads();
+ pipeline.waitUntilFinished();
+
+ assertEquals(0, pipeline.getErrors(), "the read reported errors");
+ assertEquals(Set.of("one", "two"), new HashSet<>(read));
+ assertEquals(2, read.size());
+ }
+
+ @Test
+ void filenamesFromAFieldAreReadInOrder() throws Exception {
+ Path first = tempDir.resolve("alpha.cube");
+ Path second = tempDir.resolve("beta.cube");
+ writeCube(first, "value", "alpha");
+ writeCube(second, "value", "beta");
+
+ List<String> read = runFilenameField(first, List.of(first, second));
+
+ assertEquals(List.of("alpha", "beta"), read);
+ }
+
+ @Test
+ void aDifferentLayoutFailsAndNamesTheFile() throws Exception {
+ Path first = tempDir.resolve("same.cube");
+ Path different = tempDir.resolve("different.cube");
+ writeCube(first, "value", "alpha");
+ writeCube(different, "other", "beta");
+
+ InjectorMeta injectorMeta = new InjectorMeta();
+ injectorMeta.getInjectorFields().add(new InjectorField("filename",
"String", "-1", "-1"));
+ CubeInputMeta inputMeta = new CubeInputMeta();
+ inputMeta.setDefault();
+ inputMeta.setFilenameInField(true);
+ inputMeta.setFilenameField("filename");
+ inputMeta.getFile().setName(first.toString());
+
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.setName("mismatched cubes");
+ TransformMeta injector = new TransformMeta("injector", injectorMeta);
+ TransformMeta input = new TransformMeta("read", inputMeta);
+ pipelineMeta.addTransform(injector);
+ pipelineMeta.addTransform(input);
+ pipelineMeta.addPipelineHop(new PipelineHopMeta(injector, input));
+
+ LocalPipelineEngine pipeline =
+ new LocalPipelineEngine(pipelineMeta, new Variables(), new
LoggingObject("mismatch"));
+ pipeline.prepareExecution();
+ RowProducer producer = pipeline.addRowProducer("injector", 0);
+ pipeline.startThreads();
+ IRowMeta names = filenameRowMeta();
+ producer.putRow(names, new Object[] {first.toString()});
+ producer.putRow(names, new Object[] {different.toString()});
+ producer.finished();
+ pipeline.waitUntilFinished();
+
+ assertTrue(pipeline.getErrors() > 0, "a different layout must fail the
transform");
+ String log =
+ HopLogStore.getAppender()
+ .getBuffer(pipeline.getTransform("read",
0).getLogChannel().getLogChannelId(), false)
+ .toString();
+ assertTrue(log.contains(different.getFileName().toString()), log);
+ assertTrue(log.contains(first.getFileName().toString()), log);
+ }
+
+ @Test
+ void theFirstFilenameFromAFieldIsCheckedAgainstTheSample() throws Exception {
+ Path sample = tempDir.resolve("sample.cube");
+ Path first = tempDir.resolve("first.cube");
+ writeCube(sample, "value", "sample-row");
+ writeCube(first, "other", "beta");
+
+ InjectorMeta injectorMeta = new InjectorMeta();
+ injectorMeta.getInjectorFields().add(new InjectorField("filename",
"String", "-1", "-1"));
+ CubeInputMeta inputMeta = new CubeInputMeta();
+ inputMeta.setDefault();
+ inputMeta.setFilenameInField(true);
+ inputMeta.setFilenameField("filename");
+ inputMeta.getFile().setName(sample.toString());
+
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.setName("sample layout");
+ TransformMeta injector = new TransformMeta("injector", injectorMeta);
+ TransformMeta input = new TransformMeta("read", inputMeta);
+ pipelineMeta.addTransform(injector);
+ pipelineMeta.addTransform(input);
+ pipelineMeta.addPipelineHop(new PipelineHopMeta(injector, input));
+
+ LocalPipelineEngine pipeline =
+ new LocalPipelineEngine(pipelineMeta, new Variables(), new
LoggingObject("sample layout"));
+ pipeline.prepareExecution();
+ List<String> read = new CopyOnWriteArrayList<>();
+ pipeline
+ .getTransform("read", 0)
+ .addRowListener(
+ new RowAdapter() {
+ @Override
+ public void rowWrittenEvent(IRowMeta rowMeta, Object[] row) {
+ read.add(String.valueOf(row[0]));
+ }
+ });
+ RowProducer producer = pipeline.addRowProducer("injector", 0);
+ pipeline.startThreads();
+ producer.putRow(filenameRowMeta(), new Object[] {first.toString()});
+ producer.finished();
+ pipeline.waitUntilFinished();
+
+ assertTrue(pipeline.getErrors() > 0, "the first file must be checked
against the sample");
+ assertTrue(read.isEmpty(), "rows from a mismatched first file must not go
downstream");
+ String log =
+ HopLogStore.getAppender()
+ .getBuffer(pipeline.getTransform("read",
0).getLogChannel().getLogChannelId(), false)
+ .toString();
+ assertTrue(log.contains(first.getFileName().toString()), log);
+ assertTrue(log.contains(sample.getFileName().toString()), log);
+ }
+
+ private void writeTwoValues(Path base) throws Exception {
+ InjectorMeta injectorMeta = new InjectorMeta();
+ injectorMeta.getInjectorFields().add(new InjectorField("value", "String",
"-1", "-1"));
+ CubeOutputMeta outputMeta = new CubeOutputMeta();
+ outputMeta.setDefault();
+ outputMeta.setIncludeTransformNr(true);
+ outputMeta.setFilename(base.toString());
+
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.setName("write copies");
+ TransformMeta injector = new TransformMeta("injector", injectorMeta);
+ TransformMeta output = new TransformMeta("write", outputMeta);
+ output.setCopies(2);
+ pipelineMeta.addTransform(injector);
+ pipelineMeta.addTransform(output);
+ pipelineMeta.addPipelineHop(new PipelineHopMeta(injector, output));
+
+ LocalPipelineEngine pipeline =
+ new LocalPipelineEngine(pipelineMeta, new Variables(), new
LoggingObject("write copies"));
+ pipeline.prepareExecution();
+ RowProducer producer = pipeline.addRowProducer("injector", 0);
+ pipeline.startThreads();
+ IRowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("value"));
+ producer.putRow(rowMeta, new Object[] {"one"});
+ producer.putRow(rowMeta, new Object[] {"two"});
+ producer.finished();
+ pipeline.waitUntilFinished();
+ assertEquals(0, pipeline.getErrors(), "the write reported errors");
+ }
+
+ private List<String> runFilenameField(Path sample, List<Path> files) throws
Exception {
+ InjectorMeta injectorMeta = new InjectorMeta();
+ injectorMeta.getInjectorFields().add(new InjectorField("filename",
"String", "-1", "-1"));
+ CubeInputMeta inputMeta = new CubeInputMeta();
+ inputMeta.setDefault();
+ inputMeta.setFilenameInField(true);
+ inputMeta.setFilenameField("filename");
+ inputMeta.getFile().setName(sample.toString());
+
+ PipelineMeta pipelineMeta = new PipelineMeta();
+ pipelineMeta.setName("filenames from a field");
+ TransformMeta injector = new TransformMeta("injector", injectorMeta);
+ TransformMeta input = new TransformMeta("read", inputMeta);
+ pipelineMeta.addTransform(injector);
+ pipelineMeta.addTransform(input);
+ pipelineMeta.addPipelineHop(new PipelineHopMeta(injector, input));
+
+ LocalPipelineEngine pipeline =
+ new LocalPipelineEngine(pipelineMeta, new Variables(), new
LoggingObject("field names"));
+ pipeline.prepareExecution();
+ List<String> read = new CopyOnWriteArrayList<>();
+ pipeline
+ .getTransform("read", 0)
+ .addRowListener(
+ new RowAdapter() {
+ @Override
+ public void rowWrittenEvent(IRowMeta rowMeta, Object[] row) {
+ read.add((String) row[0]);
+ }
+ });
+ RowProducer producer = pipeline.addRowProducer("injector", 0);
+ pipeline.startThreads();
+ IRowMeta names = filenameRowMeta();
+ for (Path file : files) {
+ producer.putRow(names, new Object[] {file.toString()});
+ }
+ producer.finished();
+ pipeline.waitUntilFinished();
+ assertEquals(0, pipeline.getErrors(), "reading filenames from a field
reported errors");
+ return read;
+ }
+
+ private static IRowMeta filenameRowMeta() {
+ IRowMeta names = new RowMeta();
+ names.addValueMeta(new ValueMetaString("filename"));
+ return names;
+ }
+
+ private static void writeCube(Path file, String fieldName, String value)
throws Exception {
+ IRowMeta layout = new RowMeta();
+ layout.addValueMeta(new ValueMetaString(fieldName));
+ try (OutputStream os = Files.newOutputStream(file);
+ DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os)))
{
+ layout.writeMeta(dos);
+ layout.writeData(dos, new Object[] {value});
+ }
+ }
+
+ private static List<String> readValues(Path file) throws Exception {
+ List<String> values = new ArrayList<>();
+ try (InputStream is = Files.newInputStream(file);
+ DataInputStream dis = new DataInputStream(new GZIPInputStream(is))) {
+ IRowMeta rowMeta = new RowMeta(dis);
+ while (true) {
+ try {
+ values.add((String) rowMeta.readData(dis)[0]);
+ } catch (HopEofException e) {
+ return values;
+ }
+ }
+ }
+ }
+}
diff --git
a/plugins/transforms/cube/src/test/resources/de-serialize-from-field.xml
b/plugins/transforms/cube/src/test/resources/de-serialize-from-field.xml
new file mode 100644
index 0000000000..12d9e70a97
--- /dev/null
+++ b/plugins/transforms/cube/src/test/resources/de-serialize-from-field.xml
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<transform>
+ <name>De-serialize from file</name>
+ <type>CubeInput</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <file>
+ <name>${java.io.tmpdir}/sample.cube</name>
+ </file>
+ <limit>5</limit>
+ <addfilenameresult>Y</addfilenameresult>
+ <include_transform_nr>N</include_transform_nr>
+ <filename_in_field>Y</filename_in_field>
+ <filename_field>filename</filename_field>
+ <attributes></attributes>
+ <GUI>
+ <xloc>160</xloc>
+ <yloc>80</yloc>
+ </GUI>
+</transform>
diff --git
a/plugins/transforms/cube/src/test/resources/serialize-with-copy-nr.xml
b/plugins/transforms/cube/src/test/resources/serialize-with-copy-nr.xml
new file mode 100644
index 0000000000..45b0420b85
--- /dev/null
+++ b/plugins/transforms/cube/src/test/resources/serialize-with-copy-nr.xml
@@ -0,0 +1,42 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+<transform>
+ <name>Serialize to file</name>
+ <type>CubeOutput</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <file>
+ <name>${java.io.tmpdir}/file.cube</name>
+ <include_transform_nr>Y</include_transform_nr>
+ <filename_create_parent_folders>Y</filename_create_parent_folders>
+ <add_to_result_filenames>N</add_to_result_filenames>
+ <do_not_open_newfile_init>N</do_not_open_newfile_init>
+ </file>
+ <attributes></attributes>
+ <GUI>
+ <xloc>416</xloc>
+ <yloc>64</yloc>
+ </GUI>
+</transform>
