pricealexandra opened a new issue, #35348:
URL: https://github.com/apache/beam/issues/35348
### What happened?
I've been trying out the WriteToPubSub option in a python Dataflow job, with
a formatting step like this:
```
yield PubsubMessage(
data=json.dumps(dataclasses.asdict(my_data)).encode('utf-8'),
ordering_key=ordering_key,
attributes={
"timestamp": str(my_data.timestamp),
}
)
```
right before the write step, where the call to WriteToPubSub looks as
follows:
```
WriteToPubSub(
topic='projects/my_project/topics/my_topic_name',
with_attributes=True)
```
What I observe is that my job succeeds, but no messages are published. There
are also no errors displayed in either the job logs or worker logs.
I've confirmed I can publish the messages in the same Dataflow job if I
write my own Pub/Sub writer transform that uses the python client library for
Pub/Sub, so I believe that rules out issues related to service account
permissions, networking, the size of the messages, or a misconfigured topic.
That's fine for now but I'd prefer to use the official WriteToPubSub transform.
If my code above looks correct, is there a way to expose errors that occur
in WriteToPubSub so I can debug this more effectively?
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
