[ https://issues.apache.org/jira/browse/BEAM-7819?focusedWorklogId=292115&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292115 ]
ASF GitHub Bot logged work on BEAM-7819: ---------------------------------------- Author: ASF GitHub Bot Created on: 09/Aug/19 15:28 Start Date: 09/Aug/19 15:28 Worklog Time Spent: 10m Work Description: udim commented on pull request #9232: [BEAM-7819] Python - parse PubSub message_id into attributes property URL: https://github.com/apache/beam/pull/9232#discussion_r312533330 ########## File path: sdks/python/apache_beam/io/gcp/pubsub.py ########## @@ -127,6 +130,10 @@ def _from_message(msg): """ # Convert ScalarMapContainer to dict. attributes = dict((key, msg.attributes[key]) for key in msg.attributes) + # Parse the PubSub message_id and add to attributes + sysattribs = dict(message_id=msg.message_id) Review comment: I investigated the issue yesterday and found the issue was with the Windmill service leaving out message id and publish_time. We can go ahead with this PR, and leave a note saying message_ids will work once Windmill service is updated (point to BEAM-7819 for details). FYI, I believe PR #8370 will also require the Windmill update for message ids to work on Dataflow (but I'm not sure when Java SDK uses Windmill). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 292115) Time Spent: 3h 40m (was: 3.5h) > PubsubMessage message parsing is lacking non-attribute fields > ------------------------------------------------------------- > > Key: BEAM-7819 > URL: https://issues.apache.org/jira/browse/BEAM-7819 > Project: Beam > Issue Type: Bug > Components: io-py-gcp > Reporter: Ahmet Altay > Assignee: Udi Meiri > Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > User reported issue: > https://lists.apache.org/thread.html/139b0c15abc6471a2e2202d76d915c645a529a23ecc32cd9cfecd315@%3Cuser.beam.apache.org%3E > """ > Looking at the source code, with my untrained python eyes, I think if the > intention is to include the message id and the publish time in the attributes > attribute of the PubSubMessage type, then the protobuf mapping is missing > something:- > @staticmethod > def _from_proto_str(proto_msg): > """Construct from serialized form of ``PubsubMessage``. > Args: > proto_msg: String containing a serialized protobuf of type > https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage > Returns: > A new PubsubMessage object. > """ > msg = pubsub.types.pubsub_pb2.PubsubMessage() > msg.ParseFromString(proto_msg) > # Convert ScalarMapContainer to dict. > attributes = dict((key, msg.attributes[key]) for key in msg.attributes) > return PubsubMessage(msg.data, attributes) > The protobuf definition is here:- > https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage > and so it looks as if the message_id and publish_time are not being parsed as > they are seperate from the attributes. Perhaps the PubsubMessage class needs > expanding to include these as attributes, or they would need adding to the > dictionary for attributes. This would only need doing for the _from_proto_str > as obviously they would not need to be populated when transmitting a message > to PubSub. > My python is not great, I'm assuming the latter option would need to look > something like this? > attributes = dict((key, msg.attributes[key]) for key in msg.attributes) > attributes.update({'message_id': msg.message_id, 'publish_time': > msg.publish_time}) > return PubsubMessage(msg.data, attributes) > """ -- This message was sent by Atlassian JIRA (v7.6.14#76016)