Rookie625 opened a new issue, #7752:
URL: https://github.com/apache/hop/issues/7752
### Apache Hop version?
2.18.0
### Java version?
21
### Operating system
Windows
### What happened?
### Problem Statement
The Stream Lookup transform is known to cause pipeline deadlocks when
processing large datasets with split/rejoin streams [0][1]. The root cause is
that when one input stream is blocked due to a full rowset buffer (e.g., after
a Group By transform), Stream Lookup waits for that stream while the upstream
transform cannot proceed, creating a cyclic dependency.
Current official solutions [2] require users to manually adjust pipeline
design (increase rowset size, separate streams, split pipelines, or use
Blocking transforms). These are workarounds, not root-cause fixes.
### Proposed Solution
Use **separate threads** (regular threads or virtual threads, depending on
the runtime environment) to read each info stream independently and
concurrently, with a `CountDownLatch` to coordinate the main thread. This
approach eliminates deadlocks because each stream is consumed in its own
thread, preventing buffer contention between upstream transforms.
**Pseudo‑code (using regular threads for compatibility with Java 11):**
```java
private boolean loadInfoStream(IStream stream, CountDownLatch latch,
List<Exception> errors) {
Thread loader = new Thread(() -> {
try {
IRowSet rowSet = findInputRowSet(stream.getTransformName());
Object[] row = getRowFrom(rowSet);
while (row != null && !isStopped()) {
// Add to cache (needs thread-safe collection)
data.cache.add(row);
row = getRowFrom(rowSet);
}
} catch (Exception e) {
errors.add(e);
} finally {
latch.countDown();
}
});
loader.setName("StreamLookup-Loader-" + stream.getTransformName());
loader.setUncaughtExceptionHandler((t, e) -> errors.add(e));
loader.start();
return true;
}
```
```java
// In init() or before processing rows:
CountDownLatch latch = new CountDownLatch(infoStreams.size());
List<Exception> errors = Collections.synchronizedList(new ArrayList<>());
for (IStream stream : infoStreams) {
loadInfoStream(stream, latch, errors);
}
latch.await(); // wait for all loading threads to finish
if (!errors.isEmpty()) {
throw new HopException("Failed to load lookup data", errors.get(0));
}
```
Key benefits:
No deadlock – each stream is read independently, no circular blocking.
Transparent to users – no pipeline redesign needed.
Compatible with Java 11 – regular threads work, virtual threads can be used
as an optimisation if Java 21+ is available.
References
[0]
https://hop.apache.org/manual/2.18.0/how-to-guides/avoiding-deadlocks.html
[1] https://github.com/apache/hop/issues/3740
[2]
https://hop.incubator.apache.org/manual/2.18.0/pipeline/transforms/streamlookup.html
### My Understanding of Apache Hop
I have been working with Apache Hop for some time and have developed a solid
understanding of its pipeline execution model — particularly how transforms
exchange data via rowsets, how scheduling and blocking work, and where
concurrency bottlenecks can arise. For example, I analysed the Stream Lookup
deadlock issue and proposed a practical solution using virtual threads and
CountDownLatch to read info streams concurrently, eliminating deadlocks without
forcing users to redesign their pipelines. This demonstrates my ability to dive
into Hop’s internals and propose viable fixes.
### What I Can Do for Your Team
- Design and optimise Hop pipelines for large-scale data processing
- Develop custom transforms and plugins to extend Hop’s capabilities
- Debug and resolve performance issues, including deadlocks and memory
bottlenecks
- Write clean, maintainable Java code that fits Hop’s architecture
- Collaborate with cross-functional teams and contribute to open-source
projects
### What I Am Looking For
I am actively seeking a **paid position** (full-time or long-term contract)
and I am **open to both remote and on-site opportunities**. I would love to
apply my skills to real-world data engineering challenges — whether that’s
building Hop-based ETL solutions, improving the Hop codebase, or integrating
Hop with other data platforms. I am eager to work with an experienced team,
learn from senior developers, and deliver tangible value from day one. I am
flexible, open to feedback, and ready to contribute.
### A Note on My Mindset
I do not claim to be an expert yet, but I am highly motivated and have
already proven that I can understand and improve complex parts of the system. I
am confident that I can grow quickly and become a productive member of any
Hop-focused engineering team. I would be grateful for any opportunity or
referral — wherever the role is located.
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
