shunping commented on issue #37143: URL: https://github.com/apache/beam/issues/37143#issuecomment-3675811505
I think this is ultimately a design question regarding the role and scope of logging in the Go SDK. First, the `beam/log` package is an abstraction layer designed to support swappable logging backends. Currently, there are three primary implementations (two of which are mentioned in this issue): * FnAPI (Remote): This is the backend used by the worker harness (e.g., [default_remote_logging](https://github.com/apache/beam/blob/dfe01c0f0926f1837d5700c53bbf2f2a278f8721/sdks/go/pkg/beam/core/runtime/harness/logging.go#L104)) to stream logs back to the runner's logging service. * `log/slog`: The modern structured logging standard. * Standard `log`: The legacy Go logging backend. Second, backend selection for `beam/log` is context-dependent. Remote runners (like Dataflow) hook into the FnAPI backend to ensure log reaches the runner. In contrast, local runners (and direct calls to slog or log) are typically configured for local output only. This dynamic is at the heart of the current issue. Third, I feel we lack a guideline on when to use Beam's log / slog / standard log. I propose the following distinction: * `beam/log`: Should be used for information that the worker needs to communicate back to the runner(e.g., job progress, high-level errors, or actionable events meant for the job UI), * `slog` or `log`: Should be used for local worker-specific diagnostics and debugging. There may be places in the current codebase where we have misuse one for the other. However, I believe keeping this distinction is important for managing log egress volume and ensuring that runner-side logs remain focused on high-value job telemetry. @lostluck, what's your thought on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
