[
https://issues.apache.org/jira/browse/BEAM-778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940890#comment-15940890
]
Konstantinos Katsiapis commented on BEAM-778:
---------------------------------------------
Yes, when I had looked at that in the past, it was very easy to use
gzip.GzipFile(..., fileobj = self._file, ...) [1] but unfortunately not so for
Bzip2 [2] or Snappy [3]. And we wanted to share as much implementation as
possible (as opposed to have completely different codepaths for each
compression type).
Provided that we can have a single interface that allows us to handle
Gzip/Bzip2 (and ideally in the future Snappy and other whole-file compression
techniques) with minimal diffs, changing the underlying implementation is I
think fair game.
[1] https://docs.python.org/2/library/gzip.html#gzip.GzipFile and
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filesystem.py#L103
[2] https://docs.python.org/2/library/bz2.html#bz2.BZ2File
[3] https://pypi.python.org/pypi/python-snappy
> Make fileio._CompressedFile seekable.
> -------------------------------------
>
> Key: BEAM-778
> URL: https://issues.apache.org/jira/browse/BEAM-778
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py
> Reporter: Chamikara Jayalath
> Assignee: Tibor Kiss
> Fix For: Not applicable
>
>
> We have a TODO to make fileio._CompressedFile seekable.
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L692
> Without this, compressed file objects produce for FileBasedSource
> implementations may not be able to use libraries that utilize methods seek()
> and tell().
> For example tarfile.open().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
