cozos commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-4285399174
Hi @tvalentyn I am not sure I fully understand your question. So just to
clarify
## The current state:
In order to get Python logs, we need this filter in Google Logs Explorer.
This will NOT include stdout/stderr logs from extensions.
```
logName="projects/cruise-mlp-prod-13d0/logs/dataflow.googleapis.com%2Fworker"
jsonPayload.worker = "${MY_DATAFLOW_JOBNAME}-04020129-nd2t-harness-svpn"
jsonPayload.portability_worker_id" = "sdk-0-0_sibling_5"
```
In order to get the stdout/stderr logs from extensions, you need this query:
```
logName="projects/cruise-mlp-prod-13d0/logs/dataflow.googleapis.com%2Fworker"
jsonPayload.worker = "${MY_DATAFLOW_JOBNAME}-04020129-nd2t-harness-svpn"
jsonPayload.portability_worker_id" = "sdk-0-0"
```
This has two downsides:
1. All of the extension stdout/stderr logs from ALL the sibling worker are
mixed together, we don't know which sibling an individual log entry came from.
2. We can't correlate the Python and C++ extensions logs from any individual
sibling worker
## The proposed state:
For this feature request, I am asking that this query would return BOTH
Python logs and C++ extension logs, which would be achieved by having a
separate logger with a separate `workerCtx` for each sibling worker.
```
logName="projects/cruise-mlp-prod-13d0/logs/dataflow.googleapis.com%2Fworker"
jsonPayload.worker = "${MY_DATAFLOW_JOBNAME}-04020129-nd2t-harness-svpn"
jsonPayload.portability_worker_id" = "sdk-0-0_sibling_5"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
