mattcasters commented on issue #8321:
URL: https://github.com/apache/hop/issues/8321#issuecomment-5662412108
Thanks Romain for taking the time to spell out the journey: OTel Java agent
/ K8s auto-instrumentation, then making that work across Spark, Flink, Beam,
and a standalone collector without a Hop-specific logging dialect.
That helps a lot. I want to separate two things that I think have been
getting collapsed under the word “logging”:
1. **Text logs** — lines with level, timestamp, subject, and the
transform/action/parent context. This is what SLF4J/Log4j2/OTel are good at
shipping.
2. **Execution information** — run state, metrics, parent/child hierarchy,
data profiles, “give me the complete picture of this run when it finishes.”
That is a Hop feature on purpose. Spark/Flink/Beam are weak at this, and it is
one of the reasons people run Hop on those engines instead of only using the
engine UI.
I am not opposed to (1) riding a standard logger. I *am* opposed to (2)
becoming an afterthought of an async appender, or to pipelines waiting on a
remote backend before they can finish.
Proposed path that I think gets you the embedding story without throwing
away what Hop already does:
**Keep Hop as the producer of truth, make the standard logger a first-class
sink.**
- Transforms/actions keep talking to `LogChannel` (public API stays).
- We enrich SLF4J MDC at that point (channel id, parent,
pipeline/workflow/transform/action names, execution id). That is the missing
piece for structured logs and agents.
- Default `log4j2.xml` (or equivalent) ships with:
- a **synchronous Hop appender** that feeds `LoggingBuffer` / GUI /
hop-server / “grab the log when work is done”
- console / file as today
- Extra appenders (JSON, Kafka, OTel, collector stdout) are config. No
jar-swapping as the documented path.
- Async is allowed on *those* extra appenders. The Hop buffer path stays
synchronous so we do not recreate the incomplete-log problem we already had.
- Execution Information Location stays its own channel. It is not “the log
file in Elastic.”
That is close to “put the GUI/server buffer behind an appender” without
making Log4j the only place a finished run can be reconstructed.
What I would want before merging a large swap:
- A short design note (this issue is fine) that states the two layers above
and the sync requirement.
- No silent drop of Hop levels (`DETAILED` / `ROWLEVEL` must survive a
round-trip, even if SLF4J only has five levels — marker, MDC, or a Hop event
wrapper).
- Stack traces still attached.
- Public logging API not broken in a patch release; if we break it, next
minor and a migration note.
- At least a few tests: buffer still complete at pipeline end; two
concurrent hop-server executions stay split; a custom `log4j2.xml` can add
JSON/stdout without emptying the GUI; MDC keys documented.
- Hot path cost measured on a real pipeline, not only 20k `logBasic` in a
vacuum. If we pay 4× on the log call, that has to be justified by I/O we were
going to do anyway.
If the current PR is “replace the store with SLF4J and hope the buffer is
just another appender,” I would rather split it:
1. Small PR: real Log4j2 binding + default config + MDC on the existing
`Slf4jLoggingEventListener` (Hans’s narrower change). That already unblocks
agent / collector / Kafka for you.
2. Follow-up: move file writer / console into appenders, keep the sync Hop
appender, document how a packager turns the buffer off for pure engine embeds.
3. Only then discuss shrinking `HopLogStore` further.
(1) gets you 80% of the ops story with a fraction of the compatibility risk.
(2) is the design cleanup. (3) is optional and should not be a prerequisite for
OTel.
Does that match the embed you actually need — Hop as an engine inside
Spark/Flink/K8s, logs looking like every other Java service, while
GUI/server/execution information keep a complete, synchronous view of the run?
If yes, I am happy to work from that list instead of debating the whole stack
at once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
