[
https://issues.apache.org/jira/browse/BEAM-5788?focusedWorklogId=165034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-165034
]
ASF GitHub Bot logged work on BEAM-5788:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Nov/18 17:37
Start Date: 12/Nov/18 17:37
Worklog Time Spent: 10m
Work Description: markflyhigh opened a new pull request #7010:
[BEAM-5788] Fix storage client encoding issue in Python3
URL: https://github.com/apache/beam/pull/7010
DataflowRunner needs to specify response encoding when running in Python 3.
This is required when using `google-apitools`.
+R: @aaltay @tvalentyn
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
| --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 165034)
Time Spent: 10m
Remaining Estimate: 0h
> wordcount_fnapi_it failed on TestDataflowRunner because of JSON string
> decoding error
> -------------------------------------------------------------------------------------
>
> Key: BEAM-5788
> URL: https://issues.apache.org/jira/browse/BEAM-5788
> Project: Beam
> Issue Type: Sub-task
> Components: test-failures
> Reporter: Mark Liu
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Similar to BEAM-5785, wordcount_fnapi_it failed on Python 3 when running with
> TestDataflowRunner. Got TypeError: the JSON object must be str, not 'bytes'.
> This error cause infinite retry before job could submitted to service.
> More details about my env and test:
> Python version: 3.5.3
> Test:
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it
> Command:
> {code}
> python setup.py nosetests \
> --tests
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it \
> --nocapture \
> --nologcapture \
> --test-pipeline-options=" \
> --runner=TestDataflowRunner \
> --project=<my_project> \
> --staging_location=<my_staging> \
> --temp_location=<my_temp> \
> --output=<my_output> \
>
>
> --sdk_location=.../beam/sdks/python/dist/apache-beam-2.9.0.dev0.tar.gz \
> --num_workers=1"
> {code}
> Stacktrace:
> {code}
> WARNING:root:Retry with exponential backoff: waiting for 7.661876827680761
> seconds before retrying exists because we caught exception: TypeError: the
> JSON object must be str, not 'bytes'
> Traceback for above exception (most recent call last):
> File ".../beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
> return fun(*args, **kwargs)
> File ".../beam/sdks/python/apache_beam/io/gcp/gcsio.py", line 375, in exists
> self.client.objects.Get(request) # metadata
> File
> ".../beam/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
> line 955, in Get
> download=download)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 722, in _RunMethod
> return self.ProcessHttpResponse(method_config, http_response, request)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 728, in ProcessHttpResponse
> self.__ProcessHttpResponse(method_config, http_response, request))
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 611, in __ProcessHttpResponse
> response_type, http_response.content)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py",
> line 442, in DeserializeMessage
> message = encoding.JsonToMessage(response_type, data)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py",
> line 104, in JsonToMessage
> return _ProtoJsonApiTools.Get().decode_message(message_type, message)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py",
> line 290, in decode_message
> message_type, result)
> File
> ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/protorpclite/protojson.py",
> line 210, in decode_message
> dictionary = json.loads(encoded_message)
> File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
> s.__class__.__name__))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
