[jira] [Commented] (BEAM-2949) Initial splitting

2017-10-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188816#comment-16188816 ] Eugene Kirpichov commented on BEAM-2949: Per discussion with Luke, the design can be simply: expand

[jira] [Commented] (BEAM-2993) AvroIO.write without specifying a schema

2017-10-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190169#comment-16190169 ] Eugene Kirpichov commented on BEAM-2993: (I replied in email and it didn't get posted here, so

[jira] [Closed] (BEAM-3009) Implement context access from user code closures

2017-10-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3009. -- Resolution: Fixed Fix Version/s: 2.3.0 > Implement context access from user code closures

[jira] [Assigned] (BEAM-3054) org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest is flaky

2017-10-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3054: -- Assignee: Etienne Chauchot (was: Reuven Lax) >

[jira] [Created] (BEAM-2844) Support implicit side inputs

2017-09-05 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2844: -- Summary: Support implicit side inputs Key: BEAM-2844 URL: https://issues.apache.org/jira/browse/BEAM-2844 Project: Beam Issue Type: Bug

[jira] [Closed] (BEAM-2860) SDF blog post discusses Match, which no longer exists

2017-09-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2860. -- Resolution: Fixed Fix Version/s: Not applicable > SDF blog post discusses Match, which no

[jira] [Commented] (BEAM-2826) Need to generate a single XML file when write is performed on small amount of data

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147720#comment-16147720 ] Eugene Kirpichov commented on BEAM-2826: The solution to this bug would be either augmenting

[jira] [Closed] (BEAM-2644) Make it easier to test runtime-accessible ValueProvider's

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2644. -- Resolution: Fixed Fix Version/s: 2.2.0 > Make it easier to test runtime-accessible

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148245#comment-16148245 ] Eugene Kirpichov commented on BEAM-2803: The conclusion of my experiments is that the combined

[jira] [Assigned] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2803: -- Assignee: Eugene Kirpichov (was: Jean-Baptiste Onofré) > JdbcIO read is very slow when

[jira] [Updated] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2803: --- Fix Version/s: (was: Not applicable) 2.2.0 > JdbcIO read is very slow

[jira] [Closed] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2753. -- Resolution: Fixed > File DynamicDestinations side inputs don't work with sharding >

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148145#comment-16148145 ] Eugene Kirpichov commented on BEAM-2803: Side inputs (at least in Dataflow) have the disadvantage

[jira] [Assigned] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure

2017-09-05 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2837: -- Assignee: Mairbek Khadikov (was: Thomas Groh) > Writing To Spanner From Google Cloud

[jira] [Commented] (BEAM-2841) NPE in Dataflow job

2017-09-05 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154497#comment-16154497 ] Eugene Kirpichov commented on BEAM-2841: This is a duplicate of BEAM-2834. > NPE in Dataflow job >

[jira] [Commented] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure

2017-09-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155516#comment-16155516 ] Eugene Kirpichov commented on BEAM-2837: I've passed this on to a member of the Spanner team and

[jira] [Created] (BEAM-2864) Support backfill deduplication in BigQueryIO.write()

2017-09-07 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2864: -- Summary: Support backfill deduplication in BigQueryIO.write() Key: BEAM-2864 URL: https://issues.apache.org/jira/browse/BEAM-2864 Project: Beam Issue

[jira] [Created] (BEAM-2857) Create FileIO in Python

2017-09-06 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2857: -- Summary: Create FileIO in Python Key: BEAM-2857 URL: https://issues.apache.org/jira/browse/BEAM-2857 Project: Beam Issue Type: New Feature

[jira] [Updated] (BEAM-2865) Implement FileIO.write()

2017-09-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2865: --- Issue Type: New Feature (was: Bug) > Implement FileIO.write() > > >

[jira] [Created] (BEAM-2865) Implement FileIO.write()

2017-09-07 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2865: -- Summary: Implement FileIO.write() Key: BEAM-2865 URL: https://issues.apache.org/jira/browse/BEAM-2865 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2950) Provide implicit access to State

2017-09-12 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2950: -- Summary: Provide implicit access to State Key: BEAM-2950 URL: https://issues.apache.org/jira/browse/BEAM-2950 Project: Beam Issue Type: Bug

[jira] [Assigned] (BEAM-2883) Poor error message when forgetting to specify a Datastore project.

2017-09-11 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2883: -- Assignee: Eugene Kirpichov (was: Chamikara Jayalath) > Poor error message when

[jira] [Assigned] (BEAM-2706) Create JdbcIO.readAll()

2017-09-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2706: -- Assignee: Eugene Kirpichov (was: Jean-Baptiste Onofré) > Create JdbcIO.readAll() >

[jira] [Closed] (BEAM-2706) Create JdbcIO.readAll()

2017-09-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2706. -- Resolution: Fixed Fix Version/s: 2.2.0 Closed by https://github.com/apache/beam/pull/3800

[jira] [Closed] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-09-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2803. -- Resolution: Fixed > JdbcIO read is very slow when query return a lot of rows >

[jira] [Closed] (BEAM-2467) KinesisIO watermark based on approximateArrivalTimestamp

2017-09-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2467. -- Resolution: Fixed Fix Version/s: 2.2.0 > KinesisIO watermark based on

[jira] [Commented] (BEAM-2857) Create FileIO in Python

2017-09-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181788#comment-16181788 ] Eugene Kirpichov commented on BEAM-2857: I don't think you need to be formally added to anything in

[jira] [Commented] (BEAM-2986) Support reading avro GenericRecords with BigQueryIO

2017-09-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179175#comment-16179175 ] Eugene Kirpichov commented on BEAM-2986: To do this change in a compatible way, you may want to

[jira] [Commented] (BEAM-2993) AvroIO.write without specifying a schema

2017-09-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182818#comment-16182818 ] Eugene Kirpichov commented on BEAM-2993: Do you have a more concrete use case? I don't think it's

[jira] [Assigned] (BEAM-2994) Refactor TikaIO

2017-09-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2994: -- Assignee: Sergey Beryozkin (was: Reuven Lax) > Refactor TikaIO > --- > >

[jira] [Closed] (BEAM-2986) Support reading avro GenericRecords with BigQueryIO

2017-09-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2986. -- Resolution: Fixed Fix Version/s: 2.2.0 > Support reading avro GenericRecords with

[jira] [Commented] (BEAM-2993) AvroIO.write without specifying a schema

2017-10-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188412#comment-16188412 ] Eugene Kirpichov commented on BEAM-2993: But when you do a schemaless read, you don't get a

[jira] [Created] (BEAM-3009) Implement context access from user code closures

2017-10-02 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3009: -- Summary: Implement context access from user code closures Key: BEAM-3009 URL: https://issues.apache.org/jira/browse/BEAM-3009 Project: Beam Issue Type:

[jira] [Commented] (BEAM-2993) AvroIO.write without specifying a schema

2017-09-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186163#comment-16186163 ] Eugene Kirpichov commented on BEAM-2993: The error message says that your inner class

[jira] [Closed] (BEAM-2455) Backlog size retrieval for Kinesis source

2017-09-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2455. -- Resolution: Fixed Fix Version/s: 2.2.0 > Backlog size retrieval for Kinesis source >

[jira] [Created] (BEAM-2991) BigQueryIO.read().fromQuery() should set a TTL on the temporary tables with query results

2017-09-26 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2991: -- Summary: BigQueryIO.read().fromQuery() should set a TTL on the temporary tables with query results Key: BEAM-2991 URL: https://issues.apache.org/jira/browse/BEAM-2991

[jira] [Created] (BEAM-2992) Remove codepaths for reading unsplit BigQuery sources

2017-09-26 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2992: -- Summary: Remove codepaths for reading unsplit BigQuery sources Key: BEAM-2992 URL: https://issues.apache.org/jira/browse/BEAM-2992 Project: Beam Issue

[jira] [Closed] (BEAM-2991) BigQueryIO.read().fromQuery() should set a TTL on the temporary tables with query results

2017-09-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2991. -- Resolution: Fixed > BigQueryIO.read().fromQuery() should set a TTL on the temporary tables with

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-24 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140190#comment-16140190 ] Eugene Kirpichov commented on BEAM-2803: Could you quantify "very slow" - what performance are you

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

2017-08-24 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140180#comment-16140180 ] Eugene Kirpichov commented on BEAM-2802: Please see BEAM-2586 and previous discussion of this on

[jira] [Assigned] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

2017-08-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2753: -- Assignee: Eugene Kirpichov (was: Reuven Lax) > File DynamicDestinations side inputs

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

2017-08-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142139#comment-16142139 ] Eugene Kirpichov commented on BEAM-2802: What does the custom delimiter look like in practice? E.g.

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142875#comment-16142875 ] Eugene Kirpichov commented on BEAM-2803: Hmm, indeed, seems that shuffle is being quite slow here.

[jira] [Comment Edited] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142875#comment-16142875 ] Eugene Kirpichov edited comment on BEAM-2803 at 8/26/17 5:43 PM: - Hmm,

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

2017-08-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142905#comment-16142905 ] Eugene Kirpichov commented on BEAM-2803: We can try another way to break fusion I guess: pass the

[jira] [Created] (BEAM-2810) Consider a faster Avro library in Python

2017-08-27 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2810: -- Summary: Consider a faster Avro library in Python Key: BEAM-2810 URL: https://issues.apache.org/jira/browse/BEAM-2810 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2810) Consider a faster Avro library in Python

2017-08-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143372#comment-16143372 ] Eugene Kirpichov commented on BEAM-2810: It might be a good idea to fix fastavro then. Or to fix

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

2017-08-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143916#comment-16143916 ] Eugene Kirpichov commented on BEAM-2802: Hmm, I have a hard time thinking why somebody would think

[jira] [Updated] (BEAM-2834) NullPointerException @ BigQueryServicesImpl.java:759

2017-09-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2834: --- Fix Version/s: 2.2.0 > NullPointerException @ BigQueryServicesImpl.java:759 >

[jira] [Closed] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

2017-09-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2624. -- Resolution: Fixed > File-based sinks should produce a PCollection of written filenames >

[jira] [Closed] (BEAM-2802) TextIO should allow specifying a custom delimiter

2017-09-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2802. -- Resolution: Fixed Fix Version/s: 2.2.0 > TextIO should allow specifying a custom

[jira] [Closed] (BEAM-2828) Create FileIO

2017-09-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2828. -- Resolution: Fixed FileIO has been created. > Create FileIO > - > >

[jira] [Assigned] (BEAM-2750) Read whole files as one PCollection element each

2017-09-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2750: -- Assignee: Eugene Kirpichov (was: Christopher Hebert) > Read whole files as one

[jira] [Closed] (BEAM-2750) Read whole files as one PCollection element each

2017-09-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2750. -- Resolution: Fixed Fix Version/s: 2.2.0 This has been fixed to a sufficient extent by

[jira] [Closed] (BEAM-2781) Should have a canonical Compression enum

2017-08-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2781. -- Resolution: Fixed Fix Version/s: 2.2.0 > Should have a canonical Compression enum >

[jira] [Assigned] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

2017-08-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2390: -- Assignee: Reuven Lax (was: Eric Johston) > allow user to use .setTimePartitioning in

[jira] [Commented] (BEAM-2993) AvroIO.write without specifying a schema

2017-10-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192129#comment-16192129 ] Eugene Kirpichov commented on BEAM-2993: OK, thanks for the explanations. A couple more questions:

[jira] [Reopened] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reopened BEAM-3169: Reopening, since it makes more sense to close this when it has been cherrypicked into 2.2.0 ,

[jira] [Commented] (BEAM-3195) Failure in Java postcommit & nightly - WriteFiles - "When using windowed writes, must specify number of output shards explicitly [WriteFiles]"

2017-11-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253950#comment-16253950 ] Eugene Kirpichov commented on BEAM-3195: Fix in https://github.com/apache/beam/pull/4137 > Failure

[jira] [Closed] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-14 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3169. -- Resolution: Fixed > WriteFiles data loss with some triggers >

[jira] [Updated] (BEAM-2870) BQ Partitioned Table Write Fails When Destination has Partition Decorator

2017-11-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2870: --- Fix Version/s: 2.3.0 > BQ Partitioned Table Write Fails When Destination has Partition

[jira] [Reopened] (BEAM-2870) BQ Partitioned Table Write Fails When Destination has Partition Decorator

2017-11-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reopened BEAM-2870: The issue is also present in the batch codepath:

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

2017-11-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269000#comment-16269000 ] Eugene Kirpichov commented on BEAM-3201: Brief comment: "The deserialized object cannot be jackson

[jira] [Commented] (BEAM-3268) getPerDestinationOutputFilenames() is getting processed before write is finished on dataflow runner

2017-11-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269877#comment-16269877 ] Eugene Kirpichov commented on BEAM-3268: cc: [~reuvenlax] > getPerDestinationOutputFilenames() is

[jira] [Commented] (BEAM-3268) getPerDestinationOutputFilenames() is getting processed before write is finished on dataflow runner

2017-11-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269876#comment-16269876 ] Eugene Kirpichov commented on BEAM-3268: Yeah this is a bug, because the transforms that produce

[jira] [Assigned] (BEAM-3267) Return file names from TFRecordIO write

2017-11-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3267: -- Assignee: Eugene Kirpichov (was: Kenneth Knowles) > Return file names from TFRecordIO

[jira] [Commented] (BEAM-3267) Return file names from TFRecordIO write

2017-11-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271612#comment-16271612 ] Eugene Kirpichov commented on BEAM-3267: This should be solved by FileIO.write() and the followup

[jira] [Created] (BEAM-3273) ArtifactServiceStagerTest flaky

2017-11-29 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3273: -- Summary: ArtifactServiceStagerTest flaky Key: BEAM-3273 URL: https://issues.apache.org/jira/browse/BEAM-3273 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-3261) Apex runner does not detect pipeline failure

2017-11-27 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3261: -- Summary: Apex runner does not detect pipeline failure Key: BEAM-3261 URL: https://issues.apache.org/jira/browse/BEAM-3261 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

2017-11-29 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3272: -- Summary: ParDoTranslatorTest: Error creating local cluster while creating checkpoint file Key: BEAM-3272 URL: https://issues.apache.org/jira/browse/BEAM-3272

[jira] [Commented] (BEAM-3261) Apex runner does not detect pipeline failure

2017-11-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269755#comment-16269755 ] Eugene Kirpichov commented on BEAM-3261: It looks like a separate issue to me. I filed

[jira] [Created] (BEAM-3269) Occasional ClassCastException in ParDoTranslatorTest.testAssertionFailure

2017-11-28 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3269: -- Summary: Occasional ClassCastException in ParDoTranslatorTest.testAssertionFailure Key: BEAM-3269 URL: https://issues.apache.org/jira/browse/BEAM-3269 Project:

[jira] [Commented] (BEAM-3030) watchForNewFiles() can emit a file multiple times if it's growing

2017-11-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269863#comment-16269863 ] Eugene Kirpichov commented on BEAM-3030: Fix in https://github.com/apache/beam/pull/4190 >

[jira] [Commented] (BEAM-3030) watchForNewFiles() can emit a file multiple times if it's growing

2017-11-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269802#comment-16269802 ] Eugene Kirpichov commented on BEAM-3030: This also happens in FileIOTest:

[jira] [Created] (BEAM-3288) Guard against unsafe triggers at construction time

2017-12-04 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3288: -- Summary: Guard against unsafe triggers at construction time Key: BEAM-3288 URL: https://issues.apache.org/jira/browse/BEAM-3288 Project: Beam Issue

[jira] [Created] (BEAM-3285) Switch Beam to Java 8 only

2017-12-04 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3285: -- Summary: Switch Beam to Java 8 only Key: BEAM-3285 URL: https://issues.apache.org/jira/browse/BEAM-3285 Project: Beam Issue Type: Task

[jira] [Created] (BEAM-3353) Prohibit stacked GBKs with accumulating mode

2017-12-14 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3353: -- Summary: Prohibit stacked GBKs with accumulating mode Key: BEAM-3353 URL: https://issues.apache.org/jira/browse/BEAM-3353 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-3353) Prohibit stacked GBKs with accumulating mode

2017-12-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293058#comment-16293058 ] Eugene Kirpichov commented on BEAM-3353: An analysis of existing Dataflow job graphs involving this

[jira] [Closed] (BEAM-2865) Implement FileIO.write()

2017-12-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2865. -- Resolution: Fixed > Implement FileIO.write() > > > Key:

[jira] [Commented] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247890#comment-16247890 ] Eugene Kirpichov commented on BEAM-3169: With the user's configuration (windowed writes, fixed

[jira] [Updated] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-3169: --- Affects Version/s: 2.2.0 2.0.0 2.1.0 >

[jira] [Commented] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247876#comment-16247876 ] Eugene Kirpichov commented on BEAM-3169: User reads data from pubsub and windows it into 1-minute

[jira] [Comment Edited] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247891#comment-16247891 ] Eugene Kirpichov edited comment on BEAM-3169 at 11/10/17 6:41 PM: -- I think

[jira] [Updated] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-3169: --- Fix Version/s: 2.2.0 > WriteFiles data loss with some triggers >

[jira] [Created] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3169: -- Summary: WriteFiles data loss with some triggers Key: BEAM-3169 URL: https://issues.apache.org/jira/browse/BEAM-3169 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247891#comment-16247891 ] Eugene Kirpichov commented on BEAM-3169: I think the proper fix to this is to either make that "GBK

[jira] [Commented] (BEAM-3169) WriteFiles data loss with some triggers

2017-11-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247982#comment-16247982 ] Eugene Kirpichov commented on BEAM-3169: I was able to reproduce this in a few seconds in direct

[jira] [Closed] (BEAM-3137) BigQueryIO.write() should better verify user schemas

2017-11-20 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3137. -- Resolution: Fixed Assignee: Eugene Kirpichov (was: Reuven Lax) > BigQueryIO.write()

[jira] [Closed] (BEAM-3195) Failure in Java postcommit & nightly - WriteFiles - "When using windowed writes, must specify number of output shards explicitly [WriteFiles]"

2017-11-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3195. -- Resolution: Fixed Fix Version/s: 2.2.0 > Failure in Java postcommit & nightly -

[jira] [Commented] (BEAM-3200) Streaming Pipeline throws RuntimeException when using DynamicDestinations and Method.FILE_LOADS

2017-11-17 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257715#comment-16257715 ] Eugene Kirpichov commented on BEAM-3200: Hmm, I'd expect that the code you linked would have the

[jira] [Created] (BEAM-3145) Improve cleanup of zombie temporary files in WriteFiles

2017-11-06 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3145: -- Summary: Improve cleanup of zombie temporary files in WriteFiles Key: BEAM-3145 URL: https://issues.apache.org/jira/browse/BEAM-3145 Project: Beam Issue

[jira] [Created] (BEAM-3137) BigQueryIO.write() should better verify user schemas

2017-11-02 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-3137: -- Summary: BigQueryIO.write() should better verify user schemas Key: BEAM-3137 URL: https://issues.apache.org/jira/browse/BEAM-3137 Project: Beam Issue

[jira] [Updated] (BEAM-3137) BigQueryIO.write() should better verify user schemas

2017-11-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-3137: --- Fix Version/s: 2.3.0 > BigQueryIO.write() should better verify user schemas >

[jira] [Commented] (BEAM-3158) DoFnTester should call close() in catch bloc and then re-throw exception to allow using @Rule ExpectedException

2017-11-08 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245032#comment-16245032 ] Eugene Kirpichov commented on BEAM-3158: Sorry I'm not following: what exactly goes wrong

[jira] [Commented] (BEAM-3067) BigQueryIO.Write fails on empty PCollection with DirectRunner (batch job)

2017-11-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238341#comment-16238341 ] Eugene Kirpichov commented on BEAM-3067: This was fixed as a side effect of

[jira] [Closed] (BEAM-3067) BigQueryIO.Write fails on empty PCollection with DirectRunner (batch job)

2017-11-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3067. -- Resolution: Fixed Assignee: Reuven Lax (was: Thomas Groh) Fix Version/s: 2.2.0

[jira] [Commented] (BEAM-3067) BigQueryIO.Write fails on empty PCollection with DirectRunner (batch job)

2017-11-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238282#comment-16238282 ] Eugene Kirpichov commented on BEAM-3067: Acknowledged, this is a bug, I'm investigating. >

[jira] [Commented] (BEAM-3070) Add support for windowed filenames in Python SDK

2017-12-05 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278890#comment-16278890 ] Eugene Kirpichov commented on BEAM-3070: Thanks! However, I strongly recommend to wait until

[jira] [Closed] (BEAM-3030) watchForNewFiles() can emit a file multiple times if it's growing

2017-12-05 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3030. -- Resolution: Fixed > watchForNewFiles() can emit a file multiple times if it's growing >

[jira] [Closed] (BEAM-1834) Bigquery Write validation doesn't work well with ValueInSingleWindow

2017-12-11 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1834. -- Resolution: Fixed Fix Version/s: 2.2.0 Support for data-dependent schemas has been added

<    1   2   3   4   5   6   7   >