GitHub user dhalperi opened a pull request:
https://github.com/apache/incubator-beam/pull/794
Compressed source test and cleanups
R: @jkff @bjchambers
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dhalperi/incubator-beam compressed-source-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/794.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #794
----
commit 960e8db24aa96be812d81dc86c75ededddcc9a0d
Author: Dan Halperin <dhalp...@google.com>
Date: 2016-08-05T15:51:15Z
TextIO: test compression with larger files
It is important for correctness that we test with large files
because otherwise the compressed file may be larger than the
uncompressed file, which could mask bugs
commit 8a83165fff9e419fd828499243b3055079791bed
Author: Dan Halperin <dhalp...@google.com>
Date: 2016-08-05T16:05:55Z
CompressedSource: CompressedReader is never splittable
The only way it's safe to split a compressed file is if the file is not
compressed. This can
only happen when the source itself is splittable, and that in turn will
result in the inner
source's reader being returned. A CompressedReader will only be created in
the event that
the file is NOT splittable. So remove all the logic handling splittable
compressed readers,
and instead go with the logic when we know/assume the file is compressed.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
