johannaojeling commented on issue #29180: URL: https://github.com/apache/beam/issues/29180#issuecomment-1783734391
Referencing my [comment](https://github.com/apache/beam/issues/28981#issuecomment-1783727490) from the other issue here, since we might be referring to the same problem of the pipeline hanging, but it may be related to excessive splitting rather than failures caused by unregistered functions. It also happens in other `textio.Read` pipelines, e.g. the standard wordcount example. Dumping some logs below. `go run ./go/examples/wordcount/wordcount.go --output wordcount.txt` <details> <summary>Success</summary> ``` [...] 2023/10/28 09:16:19 Prepared job with id: job-001 and staging token: job-001 2023/10/28 09:16:19 Staged binary artifact with token: job-001 2023/10/28 09:16:19 Submitted job: job-001 2023/10/28 09:16:19 (): starting job-001[go-job-1-1698477379421403000] 2023/10/28 09:16:19 (): running job-001[go-job-1-1698477379421403000] 2023/10/28 09:16:19 Job[job-001] state: RUNNING 2023/10/28 09:16:19 starting worker job-001[go-job-1-1698477379421403000]_go 2023/10/28 09:16:20 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:16:20.504Z worker.ID=job-001[go-job-1-1698477379421403000]_go worker.endpoint=localhost:64061 2023/10/28 09:16:20 INFO Writing to wordcount.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:343 time=2023-10-28T07:16:20.978Z worker.ID=job-001[go-job-1-1698477379421403000]_go worker.endpoint=localhost:64061 2023/10/28 09:16:20 INFO (): pipeline completed job-001[go-job-1-1698477379421403000] source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/runners/universal/runnerlib/execute.go:109 time=2023-10-28T07:16:20.979Z worker.ID=job-001[go-job-1-1698477379421403000]_go worker.endpoint=localhost:64061 2023/10/28 09:16:20 INFO (): terminating job-001[go-job-1-1698477379421403000] source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/runners/universal/runnerlib/execute.go:109 time=2023-10-28T07:16:20.979Z worker.ID=job-001[go-job-1-1698477379421403000]_go worker.endpoint=localhost:64061 2023/10/28 09:16:20 INFO Job[job-001] state: DONE source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/runners/universal/runnerlib/job.go:126 time=2023-10-28T07:16:20.979Z worker.ID=job-001[go-job-1-1698477379421403000]_go worker.endpoint=localhost:64061 ``` </details> <details> <summary>Unexpected behavior</summary> ``` [...] 2023/10/28 09:17:36 Prepared job with id: job-001 and staging token: job-001 2023/10/28 09:17:36 Staged binary artifact with token: job-001 2023/10/28 09:17:36 Submitted job: job-001 2023/10/28 09:17:36 (): starting job-001[go-job-1-1698477456790632000] 2023/10/28 09:17:36 (): running job-001[go-job-1-1698477456790632000] 2023/10/28 09:17:36 Job[job-001] state: RUNNING 2023/10/28 09:17:36 starting worker job-001[go-job-1-1698477456790632000]_go 2023/10/28 09:17:37 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:37.781Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:38 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:38.083Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:38 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:38.285Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:38 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:38.488Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:38 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:38.690Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:38 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:38.891Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.093Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.295Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.496Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.597Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.697Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.697Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.797Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.898Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 2023/10/28 09:17:39 INFO Reading from gs://apache-beam-samples/shakespeare/kinglear.txt source=/Users/johannaojeling/repo/johannaojeling/beam/sdks/go/pkg/beam/io/textio/textio.go:226 time=2023-10-28T07:17:39.998Z worker.ID=job-001[go-job-1-1698477456790632000]_go worker.endpoint=localhost:64078 [...] ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
