Hello Apache Hop users, We often get this error in the DOS console despite (Java Heap Space): 2023/11/17 12:14:29 - Hop - Workflow execution has ended Exception in thread "Timer-4" java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3745) at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120) at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95) at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156) at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:262) at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
We have also observed that for a complex pipeline having >15000 rows in a json: - It works well (whole pipeline could successful be executed) on the developer machine (has >32MB RAM), this despite the fact that in the Dos console you got the Java heap Space Error, Data looks also ok. - It will never ends on a “standard” machine (has<12MB RAM), also on the UI you wont find any error message (you might get in the DOS console the Java Heap Space Error) We played with the Row set size in the Meta-->Pipeline Run Configuration-->Local but without success. Kindly ask if there are good practices when we shall deal with "huge" json (having >15000 rows/items) Thank you + Cheers Michel