[ https://issues.apache.org/jira/browse/BEAM-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107778#comment-17107778 ]
Brian Hulette commented on BEAM-9975: ------------------------------------- It looks like there are (at least) two problems here: - get_all_options relies on [__subclasses__|https://github.com/apache/beam/blob/5d00ccba5b905584f07c1b0275841113d4921a8c/sdks/python/apache_beam/options/pipeline_options.py#L283] to find every PipelineOptions subclass, which finds all the [subclasses that have had their definition executed|https://stackoverflow.com/questions/3862310/how-to-find-all-the-subclasses-of-a-class-given-its-name]. It seems when running tests it's possible for this to pull in definitions from previously executed tests. I tried to repro this locally by running two tests with pytest and I couldn't do it. I'm not sure what's different on jenkins. - I'm pretty sure we should check for instances of ValueProvider and call get() before trying to convert to a proto struct. > PortableRunnerTest flake "ParseError: Unexpected type for Value message." > ------------------------------------------------------------------------- > > Key: BEAM-9975 > URL: https://issues.apache.org/jira/browse/BEAM-9975 > Project: Beam > Issue Type: Bug > Components: sdk-py-core > Reporter: Brian Hulette > Assignee: Brian Hulette > Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Error looks similar to the one in BEAM-9907. Example from > https://builds.apache.org/job/beam_PreCommit_Python_Cron/2732 > {code} > apache_beam/runners/portability/fn_api_runner/fn_runner_test.py:569: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > apache_beam/pipeline.py:550: in __exit__ > self.run().wait_until_finish() > apache_beam/pipeline.py:529: in run > return self.runner.run_pipeline(self, self._options) > apache_beam/runners/portability/portable_runner.py:426: in run_pipeline > job_service_handle.submit(proto_pipeline) > apache_beam/runners/portability/portable_runner.py:107: in submit > prepare_response = self.prepare(proto_pipeline) > apache_beam/runners/portability/portable_runner.py:184: in prepare > pipeline_options=self.get_pipeline_options()), > apache_beam/runners/portability/portable_runner.py:174: in > get_pipeline_options > return job_utils.dict_to_struct(p_options) > apache_beam/runners/job/utils.py:33: in dict_to_struct > return json_format.ParseDict(dict_obj, struct_pb2.Struct()) > target/.tox-py36-cython/py36-cython/lib/python3.6/site-packages/google/protobuf/json_format.py:450: > in ParseDict > parser.ConvertMessage(js_dict, message) > target/.tox-py36-cython/py36-cython/lib/python3.6/site-packages/google/protobuf/json_format.py:479: > in ConvertMessage > methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self) > target/.tox-py36-cython/py36-cython/lib/python3.6/site-packages/google/protobuf/json_format.py:667: > in _ConvertStructMessage > self._ConvertValueMessage(value[key], message.fields[key]) > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > self = <google.protobuf.json_format._Parser object at 0x7f69eb5b59e8> > value = <apache_beam.options.value_provider.RuntimeValueProvider object at > 0x7f69eb7b3ac8> > message = > def _ConvertValueMessage(self, value, message): > """Convert a JSON representation into Value message.""" > if isinstance(value, dict): > self._ConvertStructMessage(value, message.struct_value) > elif isinstance(value, list): > self. _ConvertListValueMessage(value, message.list_value) > elif value is None: > message.null_value = 0 > elif isinstance(value, bool): > message.bool_value = value > elif isinstance(value, six.string_types): > message.string_value = value > elif isinstance(value, _INT_OR_FLOAT): > message.number_value = value > else: > > raise ParseError('Unexpected type for Value message.') > E google.protobuf.json_format.ParseError: Unexpected type for Value > message. > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)