[
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=121189&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-121189
]
ASF GitHub Bot logged work on BEAM-1251:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jul/18 03:55
Start Date: 10/Jul/18 03:55
Worklog Time Spent: 10m
Work Description: charlesccychen closed pull request #5911: [BEAM-1251]
Upgrade snappy and use a memoryview
URL: https://github.com/apache/beam/pull/5911
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/io/avroio.py
b/sdks/python/apache_beam/io/avroio.py
index 9b86b58982b..f90dc3c6833 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -341,8 +341,8 @@ def _decompress_bytes(data, codec):
# Compressed data includes a 4-byte CRC32 checksum which we verify.
# We take care to avoid extra copies of data while slicing large objects
- # by use of a buffer.
- result = snappy.decompress(buffer(data)[:-4])
+ # by use of a memoryview.
+ result = snappy.decompress(memoryview(data)[:-4])
avroio.BinaryDecoder(io.BytesIO(data[-4:])).check_crc32(result)
return result
else:
diff --git a/sdks/python/apache_beam/io/tfrecordio.py
b/sdks/python/apache_beam/io/tfrecordio.py
index 989247a96ee..2ef7c5b4c72 100644
--- a/sdks/python/apache_beam/io/tfrecordio.py
+++ b/sdks/python/apache_beam/io/tfrecordio.py
@@ -43,7 +43,7 @@ def _default_crc32c_fn(value):
if not _default_crc32c_fn.fn:
try:
import snappy # pylint: disable=import-error
- _default_crc32c_fn.fn = snappy._crc32c # pylint:
disable=protected-access
+ _default_crc32c_fn.fn = snappy._snappy._crc32c # pylint:
disable=protected-access
except ImportError:
logging.warning('Couldn\'t find python-snappy so the implementation of '
'_TFRecordUtil._masked_crc32c is not as fast as it could
'
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py
b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index c31e43f78a5..fb4643fe0a1 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -42,7 +42,7 @@
# Update this version to the next version whenever there is a change that will
# require changes to legacy Dataflow worker execution environment.
-BEAM_CONTAINER_VERSION = 'beam-master-20180619'
+BEAM_CONTAINER_VERSION = 'beam-master-20180709'
# Update this version to the next version whenever there is a change that
# requires changes to SDK harness container or SDK harness launcher.
BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20180619'
diff --git a/sdks/python/container/Dockerfile b/sdks/python/container/Dockerfile
index 90348c6e231..afb6b43f938 100644
--- a/sdks/python/container/Dockerfile
+++ b/sdks/python/container/Dockerfile
@@ -70,7 +70,7 @@ RUN \
# Optional packages
pip install "cython == 0.28.1" && \
pip install "guppy == 0.1.10" && \
- pip install "python-snappy == 0.5.1" && \
+ pip install "python-snappy == 0.5.3" && \
# These are additional packages likely to be used by customers.
pip install "numpy == 1.13.3" --no-binary=:all: && \
pip install "pandas == 0.18.1" && \
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 121189)
Time Spent: 16h 50m (was: 16h 40m)
> Python 3 Support
> ----------------
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Eyad Sibai
> Assignee: Robbe
> Priority: Trivial
> Time Spent: 16h 50m
> Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are
> several packages that does not support python3 yet which google datalab
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
