[jira] [Commented] (BEAM-2081) I/O Authoring overview - better clarify how to read from files

2017-04-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985581#comment-15985581 ] Eugene Kirpichov commented on BEAM-2081: I agree with this guidance. The only downside is it

[jira] [Closed] (BEAM-1573) KafkaIO does not allow using Kafka serializers and deserializers

2017-04-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1573. -- Resolution: Fixed Fix Version/s: (was: Not applicable) First stable

[jira] [Closed] (BEAM-1414) CountingInput should comply with PTransform style guide

2017-04-21 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1414. -- Resolution: Fixed > CountingInput should comply with PTransform style guide >

[jira] [Closed] (BEAM-1914) XML IO should comply with PTransform style guide

2017-04-21 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1914. -- > XML IO should comply with PTransform style guide >

[jira] [Commented] (BEAM-673) Data locality for Read.Bounded

2017-04-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983364#comment-15983364 ] Eugene Kirpichov commented on BEAM-673: --- [~iemejia] No - the fact that a DoFn's efficiency depends on

[jira] [Commented] (BEAM-2114) KafkaIO broken with CoderException

2017-04-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989264#comment-15989264 ] Eugene Kirpichov commented on BEAM-2114: There's still another issue here: the display data

[jira] [Commented] (BEAM-2060) XmlIO use harcoded Charset

2017-04-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987600#comment-15987600 ] Eugene Kirpichov commented on BEAM-2060: We could support multi-byte encodings when reading, but at

[jira] [Created] (BEAM-2120) TestDataflowRunner prints all messages from the job, repeatedly

2017-04-28 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2120: -- Summary: TestDataflowRunner prints all messages from the job, repeatedly Key: BEAM-2120 URL: https://issues.apache.org/jira/browse/BEAM-2120 Project: Beam

[jira] [Commented] (BEAM-2120) TestDataflowRunner prints all messages from the job, repeatedly

2017-04-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989569#comment-15989569 ] Eugene Kirpichov commented on BEAM-2120: Each call to waitToFinish() prints all messages the job

[jira] [Created] (BEAM-2108) Integration tests for PubsubIO

2017-04-27 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2108: -- Summary: Integration tests for PubsubIO Key: BEAM-2108 URL: https://issues.apache.org/jira/browse/BEAM-2108 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115164#comment-16115164 ] Eugene Kirpichov commented on BEAM-2671: 20820fa5477ffcdd4a9ef2e9340353ed3c5691a9, mentioned above,

[jira] [Commented] (BEAM-2734) Dataflow ValidatesRunner broken at HEAD

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115114#comment-16115114 ] Eugene Kirpichov commented on BEAM-2734: Another issue is also something I broke with

[jira] [Assigned] (BEAM-2734) Dataflow ValidatesRunner broken at HEAD

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2734: -- Assignee: Eugene Kirpichov (was: Thomas Groh) > Dataflow ValidatesRunner broken at

[jira] [Closed] (BEAM-2512) TextIO should support watching for new files

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2512. -- Resolution: Fixed Fix Version/s: 2.2.0 > TextIO should support watching for new files >

[jira] [Updated] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2671: --- Fix Version/s: (was: 2.1.0) > CreateStreamTest.testFirstElementLate validatesRunner test

[jira] [Created] (BEAM-2734) Dataflow ValidatesRunner broken at HEAD

2017-08-04 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2734: -- Summary: Dataflow ValidatesRunner broken at HEAD Key: BEAM-2734 URL: https://issues.apache.org/jira/browse/BEAM-2734 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-2734) Dataflow ValidatesRunner broken at HEAD

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2734: --- Description:

[jira] [Closed] (BEAM-2677) AvroIO.read without specifying a schema

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2677. -- Resolution: Fixed Fix Version/s: 2.2.0 > AvroIO.read without specifying a schema >

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105918#comment-16105918 ] Eugene Kirpichov commented on BEAM-2671: Any updates? > CreateStreamTest.testFirstElementLate

[jira] [Commented] (BEAM-2699) AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105917#comment-16105917 ] Eugene Kirpichov commented on BEAM-2699: By the way, this was found when testing a PR where

[jira] [Created] (BEAM-2699) AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable

2017-07-28 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2699: -- Summary: AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable Key: BEAM-2699 URL: https://issues.apache.org/jira/browse/BEAM-2699

[jira] [Created] (BEAM-2754) Simplify DefaultCoder

2017-08-08 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2754: -- Summary: Simplify DefaultCoder Key: BEAM-2754 URL: https://issues.apache.org/jira/browse/BEAM-2754 Project: Beam Issue Type: Bug Components:

[jira] [Commented] (BEAM-2754) Simplify DefaultCoder

2017-08-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119049#comment-16119049 ] Eugene Kirpichov commented on BEAM-2754: cc: [~tgroh] [~kenn] > Simplify DefaultCoder >

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

2017-08-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109527#comment-16109527 ] Eugene Kirpichov commented on BEAM-1323: This may be addressed by creating JdbcIO.readAll() that

[jira] [Commented] (BEAM-92) Data-dependent sinks

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105942#comment-16105942 ] Eugene Kirpichov commented on BEAM-92: -- I'm not sure whether having the current JIRA still makes sense.

[jira] [Commented] (BEAM-2699) AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105927#comment-16105927 ] Eugene Kirpichov commented on BEAM-2699: Relevant code:

[jira] [Commented] (BEAM-2699) AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105929#comment-16105929 ] Eugene Kirpichov commented on BEAM-2699: Note that hashing this in general is fine, because this

[jira] [Commented] (BEAM-2699) AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106003#comment-16106003 ] Eugene Kirpichov commented on BEAM-2699: You're right, this is another bug with the way things are

[jira] [Created] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

2017-08-08 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2753: -- Summary: File DynamicDestinations side inputs don't work with sharding Key: BEAM-2753 URL: https://issues.apache.org/jira/browse/BEAM-2753 Project: Beam

[jira] [Updated] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

2017-08-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2753: --- Fix Version/s: 2.2.0 > File DynamicDestinations side inputs don't work with sharding >

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-08-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117469#comment-16117469 ] Eugene Kirpichov commented on BEAM-2671: Aviem - my apologies for misunderstanding previous

[jira] [Updated] (BEAM-1868) CreateStreamTest testMultiOutputParDo is flaky on the Spark runner

2017-08-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1868: --- Fix Version/s: 2.1.0 > CreateStreamTest testMultiOutputParDo is flaky on the Spark runner >

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103907#comment-16103907 ] Eugene Kirpichov commented on BEAM-2671: It's interesting that testMultiOutputParDo succeeds if you

[jira] [Commented] (BEAM-2641) Improve discoverability of TextIO.readAll() as a replacement of TextIO.read() for large globs

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104019#comment-16104019 ] Eugene Kirpichov commented on BEAM-2641: The PR https://github.com/apache/beam/pull/3639 introduces

[jira] [Closed] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2640. -- Resolution: Fixed Fix Version/s: 2.2.0 > Introduce Create.ofProvider(ValueProvider) >

[jira] [Closed] (BEAM-2641) Improve discoverability of TextIO.readAll() as a replacement of TextIO.read() for large globs

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2641. -- Resolution: Fixed Fix Version/s: 2.2.0 > Improve discoverability of TextIO.readAll() as a

[jira] [Closed] (BEAM-2656) Introduce AvroIO.readAll()

2017-07-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2656. -- Resolution: Fixed Fix Version/s: 2.2.0 > Introduce AvroIO.readAll() >

[jira] [Commented] (BEAM-2774) Add I/O source for VCF files (python)

2017-08-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133395#comment-16133395 ] Eugene Kirpichov commented on BEAM-2774: Related issue:

[jira] [Updated] (BEAM-2776) TextIO should support reading header lines

2017-08-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2776: --- Component/s: sdk-py > TextIO should support reading header lines >

[jira] [Created] (BEAM-2776) TextIO should support reading header lines

2017-08-17 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2776: -- Summary: TextIO should support reading header lines Key: BEAM-2776 URL: https://issues.apache.org/jira/browse/BEAM-2776 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-2776) TextIO should support reading header lines

2017-08-17 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2776: --- Description: Users frequently request the ability to skip some header rows when reading text

[jira] [Created] (BEAM-2781) Should have a canonical Compression enum

2017-08-18 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2781: -- Summary: Should have a canonical Compression enum Key: BEAM-2781 URL: https://issues.apache.org/jira/browse/BEAM-2781 Project: Beam Issue Type: Bug

[jira] [Closed] (BEAM-1353) Beam should comply with PTransform style guide

2017-05-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1353. -- Resolution: Fixed > Beam should comply with PTransform style guide >

[jira] [Closed] (BEAM-1415) PubsubIO should comply with PTransform style guide

2017-05-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1415. -- Resolution: Fixed > PubsubIO should comply with PTransform style guide >

[jira] [Commented] (BEAM-638) Add sink transform to write bounded data per window, pane, [and key] even when PCollection is unbounded

2017-05-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995167#comment-15995167 ] Eugene Kirpichov commented on BEAM-638: --- TextIO and AvroIO can now write unbounded data, producing one

[jira] [Closed] (BEAM-1402) Make TextIO and AvroIO use best-practice types.

2017-05-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1402. -- Resolution: Fixed Assignee: Eugene Kirpichov (was: Reuven Lax) > Make TextIO and AvroIO

[jira] [Closed] (BEAM-2023) BigQueryIO.Write needs a way of dynamically specifying table schemas

2017-05-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2023. -- Resolution: Fixed > BigQueryIO.Write needs a way of dynamically specifying table schemas >

[jira] [Closed] (BEAM-2221) Make KafkaIO coder specification less awkward

2017-05-09 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2221. -- Resolution: Fixed > Make KafkaIO coder specification less awkward >

[jira] [Assigned] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

2017-05-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1013: -- Assignee: Melissa Pashniak (was: Eugene Kirpichov) > Recheck all existing programming

[jira] [Commented] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

2017-05-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011160#comment-16011160 ] Eugene Kirpichov commented on BEAM-1013: The style-guide part of this is done. Melissa, is there

[jira] [Created] (BEAM-2301) Standard expansion of SDF should be in runners-core-construction

2017-05-15 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2301: -- Summary: Standard expansion of SDF should be in runners-core-construction Key: BEAM-2301 URL: https://issues.apache.org/jira/browse/BEAM-2301 Project: Beam

[jira] [Closed] (BEAM-2258) BigtableIO should use AutoValue for read and write

2017-05-12 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2258. -- Resolution: Fixed Fix Version/s: 2.1.0 > BigtableIO should use AutoValue for read and

[jira] [Created] (BEAM-2284) Completely remove OldDoFn from Beam

2017-05-12 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2284: -- Summary: Completely remove OldDoFn from Beam Key: BEAM-2284 URL: https://issues.apache.org/jira/browse/BEAM-2284 Project: Beam Issue Type: Task

[jira] [Comment Edited] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-05-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005465#comment-16005465 ] Eugene Kirpichov edited comment on BEAM-2140 at 5/10/17 9:22 PM: - Aljoscha

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-05-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005465#comment-16005465 ] Eugene Kirpichov commented on BEAM-2140: Aljoscha - SDF code does not inspect watermarks. Here's

[jira] [Closed] (BEAM-2052) Windowed file sinks should support dynamic sharding

2017-05-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2052. -- Resolution: Fixed This is now fixed when applying the sink to bounded collections. For

[jira] [Closed] (BEAM-1377) Support Splittable DoFn in Dataflow streaming runner

2017-06-20 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1377. -- Resolution: Fixed Fix Version/s: 2.1.0 > Support Splittable DoFn in Dataflow streaming

[jira] [Commented] (BEAM-1620) Add streaming Dataflow ValidatesRunner coverage

2017-06-20 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056663#comment-16056663 ] Eugene Kirpichov commented on BEAM-1620: Note: there are tests like SplittableDoFnTest that, in

[jira] [Updated] (BEAM-2483) SplittableDoFnTest should not explicitly setStreaming(true)

2017-06-20 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2483: --- Description: https://github.com/apache/beam/pull/1898 adds Splittable DoFn to Dataflow

[jira] [Created] (BEAM-2476) Dataflow streaming runner fails SDF testWindowedSideInputWithCheckpoints

2017-06-19 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2476: -- Summary: Dataflow streaming runner fails SDF testWindowedSideInputWithCheckpoints Key: BEAM-2476 URL: https://issues.apache.org/jira/browse/BEAM-2476 Project:

[jira] [Created] (BEAM-2483) SplittableDoFnTest should not explicitly setStreaming(true)

2017-06-20 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2483: -- Summary: SplittableDoFnTest should not explicitly setStreaming(true) Key: BEAM-2483 URL: https://issues.apache.org/jira/browse/BEAM-2483 Project: Beam

[jira] [Created] (BEAM-2513) TextIO should support watching files for new entries

2017-06-24 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2513: -- Summary: TextIO should support watching files for new entries Key: BEAM-2513 URL: https://issues.apache.org/jira/browse/BEAM-2513 Project: Beam Issue

[jira] [Created] (BEAM-2512) TextIO should support watching for new files

2017-06-24 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2512: -- Summary: TextIO should support watching for new files Key: BEAM-2512 URL: https://issues.apache.org/jira/browse/BEAM-2512 Project: Beam Issue Type: New

[jira] [Created] (BEAM-2511) TextIO should support reading a PCollection of filenames

2017-06-24 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2511: -- Summary: TextIO should support reading a PCollection of filenames Key: BEAM-2511 URL: https://issues.apache.org/jira/browse/BEAM-2511 Project: Beam

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063548#comment-16063548 ] Eugene Kirpichov commented on BEAM-2140: On 1: the timer should not be dropped because ProcessFn is

[jira] [Commented] (BEAM-2447) Reintroduce DoFn.ProcessContinuation

2017-06-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064054#comment-16064054 ] Eugene Kirpichov commented on BEAM-2447: If the runner did not already take a checkpoint, and the

[jira] [Closed] (BEAM-2301) Standard expansion of SDF should be in runners-core-construction

2017-05-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2301. -- Resolution: Fixed Fix Version/s: 2.1.0 > Standard expansion of SDF should be in

[jira] [Commented] (BEAM-788) Execute ReduceFn directly, not via OldDoFn wrapper

2017-05-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016539#comment-16016539 ] Eugene Kirpichov commented on BEAM-788: --- Yes. > Execute ReduceFn directly, not via OldDoFn wrapper >

[jira] [Closed] (BEAM-2284) Completely remove OldDoFn from Beam

2017-05-16 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2284. -- Resolution: Fixed Fix Version/s: 2.1.0 > Completely remove OldDoFn from Beam >

[jira] [Closed] (BEAM-2357) Add HCatalogIO (Hive)

2017-06-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2357. -- Resolution: Fixed Fix Version/s: 2.1.0 > Add HCatalogIO (Hive) > - >

[jira] [Created] (BEAM-2447) Reintroduce DoFn.ProcessContinuation

2017-06-13 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2447: -- Summary: Reintroduce DoFn.ProcessContinuation Key: BEAM-2447 URL: https://issues.apache.org/jira/browse/BEAM-2447 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-1824) Adapter for running SDF on a statically known input as a Source

2017-05-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1824: --- Description: [~bchambers] suggested the following idea: while the runner implementation of

[jira] [Reopened] (BEAM-2052) Windowed file sinks should support dynamic sharding

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reopened BEAM-2052: Apologies, for some reason I thought the PR was merged, and didn't check. > Windowed file sinks

[jira] [Closed] (BEAM-2218) PubsubIO.readPubsubMessages function names are too long

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2218. -- Resolution: Duplicate didn't notice the jira. > PubsubIO.readPubsubMessages function names are

[jira] [Closed] (BEAM-2052) Windowed file sinks should support dynamic sharding

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2052. -- Resolution: Fixed > Windowed file sinks should support dynamic sharding >

[jira] [Commented] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001197#comment-16001197 ] Eugene Kirpichov commented on BEAM-1013: Many of them are most likely out of date due to

[jira] [Created] (BEAM-2218) PubsubIO.readPubsubMessages function names are too long

2017-05-08 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2218: -- Summary: PubsubIO.readPubsubMessages function names are too long Key: BEAM-2218 URL: https://issues.apache.org/jira/browse/BEAM-2218 Project: Beam Issue

[jira] [Updated] (BEAM-2221) Make KafkaIO coder specification less awkward

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2221: --- Fix Version/s: 2.0.0 > Make KafkaIO coder specification less awkward >

[jira] [Created] (BEAM-2221) Make KafkaIO coder specification less awkward

2017-05-08 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2221: -- Summary: Make KafkaIO coder specification less awkward Key: BEAM-2221 URL: https://issues.apache.org/jira/browse/BEAM-2221 Project: Beam Issue Type:

[jira] [Updated] (BEAM-2221) Make KafkaIO coder specification less awkward

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2221: --- Description: readWithCoders and writeWithCoders functions are awkward because they don't

[jira] [Closed] (BEAM-2210) PubsubIO.readPubsubMessagesWithoutAttributes is awkward

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2210. -- Resolution: Fixed > PubsubIO.readPubsubMessagesWithoutAttributes is awkward >

[jira] [Commented] (BEAM-2170) PubsubIO.readStrings should handle messages without metadata

2017-05-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996945#comment-15996945 ] Eugene Kirpichov commented on BEAM-2170: Sorry for the trouble, I'll fix this today around 1pm PST.

[jira] [Assigned] (BEAM-2170) PubsubIO.readStrings should handle messages without metadata

2017-05-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2170: -- Assignee: Eugene Kirpichov > PubsubIO.readStrings should handle messages without

[jira] [Closed] (BEAM-2170) PubsubIO.readStrings should handle messages without metadata

2017-05-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2170. -- Resolution: Fixed > PubsubIO.readStrings should handle messages without metadata >

[jira] [Created] (BEAM-2172) ProcessBundleHandlerTest.testCreatingAndProcessingDoFn fails at HEAD

2017-05-04 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2172: -- Summary: ProcessBundleHandlerTest.testCreatingAndProcessingDoFn fails at HEAD Key: BEAM-2172 URL: https://issues.apache.org/jira/browse/BEAM-2172 Project: Beam

[jira] [Closed] (BEAM-2114) KafkaIO broken with CoderException

2017-04-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2114. -- Resolution: Fixed > KafkaIO broken with CoderException > -- > >

[jira] [Closed] (BEAM-2154) Writing to large numbers of BigQuery tables causes out-of-memory

2017-05-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2154. -- Resolution: Fixed > Writing to large numbers of BigQuery tables causes out-of-memory >

[jira] [Commented] (BEAM-2414) Add TwitterIO

2017-06-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039463#comment-16039463 ] Eugene Kirpichov commented on BEAM-2414: Please clarify what you mean by this. > Add TwitterIO >

[jira] [Commented] (BEAM-2415) Add FacebookIO

2017-06-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039460#comment-16039460 ] Eugene Kirpichov commented on BEAM-2415: Please clarify what you mean by this? > Add FacebookIO >

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065661#comment-16065661 ] Eugene Kirpichov commented on BEAM-2140: Working backwards from that, in the "read Pubsub topic

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-09-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170471#comment-16170471 ] Eugene Kirpichov commented on BEAM-2140: Thanks Aljoscha! It would be awesome to get this working

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-09-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170472#comment-16170472 ] Eugene Kirpichov commented on BEAM-2140: Also note that this support is critical for portable

[jira] [Closed] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure

2017-09-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2837. -- Resolution: Fixed Fix Version/s: 2.2.0 > Writing To Spanner From Google Cloud DataFlow -

[jira] [Closed] (BEAM-407) Inconsistent synchronization in OffsetRangeTracker.copy

2017-09-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-407. - Resolution: Fixed Fix Version/s: 2.2.0 > Inconsistent synchronization in

[jira] [Commented] (BEAM-2826) Need to generate a single XML file when write is performed on small amount of data

2017-09-20 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172818#comment-16172818 ] Eugene Kirpichov commented on BEAM-2826: This will be addressed as part of the FileIO.write()

[jira] [Assigned] (BEAM-2834) NullPointerException @ BigQueryServicesImpl.java:759

2017-09-14 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2834: -- Assignee: Reuven Lax (was: Thomas Groh) > NullPointerException @

[jira] [Commented] (BEAM-2981) Unable to deserialize ProtoCoder in Dataflow, serialVersionUID mismatch

2017-09-22 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176837#comment-16176837 ] Eugene Kirpichov commented on BEAM-2981: This should be fixed by a recent fix in the Dataflow

[jira] [Commented] (BEAM-2879) Implement and use an Avro coder rather than the JSON one for intermediary files to be loaded in BigQuery

2017-09-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168669#comment-16168669 ] Eugene Kirpichov commented on BEAM-2879: Actually, we already go through a GenericRecord ->

[jira] [Commented] (BEAM-2879) Implement and use an Avro coder rather than the JSON one for intermediary files to be loaded in BigQuery

2017-09-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168666#comment-16168666 ] Eugene Kirpichov commented on BEAM-2879: There is no reason why we didn't go Avro in the first

[jira] [Created] (BEAM-3030) watchForNewFiles() can emit a file multiple times if it's growing

2017-10-06 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3030: -- Summary: watchForNewFiles() can emit a file multiple times if it's growing Key: BEAM-3030 URL: https://issues.apache.org/jira/browse/BEAM-3030 Project: Beam

<    1   2   3   4   5   6   7   >