GitHub user lionants02 reopened a discussion: I want to write/read multiple
files simultaneously with Specify copies
I want to write/read multiple files simultaneously with Specify copies feature.
Someone suggested me to try Hop to migrate from my old PDI system, but I have
problems with "Serialize To File" for multiple writes and "De-Serialized From
File". I have hundreds of millions of rows processed almost every 3-4 hours.
One of the bottlenecks is file read/write. In PDI, I increased the speed by
adding Specify copies for write and read, but in Apache, I can't find any
documentation about the reference to ${Internal.Step.CopyNr} that can be used
instead.
Can anyone advise me? If I have to do it manually by copying and pasting
multiple file writers with about 30-40 files without any parameters, it will be
quite difficult for me.
GitHub link: https://github.com/apache/hop/discussions/5499
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
