[jira] [Assigned] (BEAM-2588) Portable Flink Runner Job API

[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2588: -- Assignee: Robert Bradshaw (was: Axel Magnuson) > Portable Flink Runner Job API > --

[jira] [Commented] (BEAM-2588) Portable Flink Runner Job API

[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479875#comment-16479875 ] Eugene Kirpichov commented on BEAM-2588: Axel said Robert is going to work on this.

[jira] [Created] (BEAM-4375) Gradle doesn't run tests under RunWith(Enclosed.class)

Eugene Kirpichov created BEAM-4375: -- Summary: Gradle doesn't run tests under RunWith(Enclosed.class) Key: BEAM-4375 URL: https://issues.apache.org/jira/browse/BEAM-4375 Project: Beam Issue T

[jira] [Commented] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ https://issues.apache.org/jira/browse/BEAM-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484476#comment-16484476 ] Eugene Kirpichov commented on BEAM-4267: Woohoo! Now I think it just remains to hoo

[jira] [Closed] (BEAM-4473) Flaky org.apache.beam.runners.direct.portable.ReferenceRunnerTest.pipelineExecution

[ https://issues.apache.org/jira/browse/BEAM-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4473. -- Resolution: Fixed Fix Version/s: 2.6.0 Fixed in https://github.com/apache/beam/pull/5585

[jira] [Closed] (BEAM-4281) GrpcDataServiceTest.testMessageReceivedBySingleClientWhenThereAreMultipleClients is flaky

[ https://issues.apache.org/jira/browse/BEAM-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4281. -- Resolution: Fixed Fix Version/s: 2.6.0 > GrpcDataServiceTest.testMessageReceivedBySingleC

[jira] [Closed] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ https://issues.apache.org/jira/browse/BEAM-4291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4291. -- Resolution: Fixed Fix Version/s: 2.6.0 > ArtifactRetrievalService that retrieves artifact

[jira] [Closed] (BEAM-4216) Flink: Staged artifacts are delivered to the SDK container

[ https://issues.apache.org/jira/browse/BEAM-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4216. -- Resolution: Fixed Fix Version/s: 2.6.0 > Flink: Staged artifacts are delivered to the SDK

[jira] [Closed] (BEAM-2588) Portable Flink Runner Job API

[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2588. -- Resolution: Fixed Fix Version/s: 2.6.0 > Portable Flink Runner Job API >

[jira] [Closed] (BEAM-4149) Java SDK Harness should populate worker id in control plane headers

[ https://issues.apache.org/jira/browse/BEAM-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4149. -- Resolution: Duplicate Fix Version/s: Not applicable > Java SDK Harness should populate wo

[jira] [Assigned] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ https://issues.apache.org/jira/browse/BEAM-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4145: -- Assignee: Eugene Kirpichov (was: Thomas Groh) > Java SDK Harness populates control req

[jira] [Commented] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ https://issues.apache.org/jira/browse/BEAM-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516491#comment-16516491 ] Eugene Kirpichov commented on BEAM-4145: I'll take over tgroh's PR since he's curr

[jira] [Closed] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ https://issues.apache.org/jira/browse/BEAM-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4145. -- Resolution: Fixed Fix Version/s: 2.6.0 > Java SDK Harness populates control request heade

[jira] [Closed] (BEAM-3833) Java SDK harness should detect SDF ProcessFn and proactively checkpoint it

[ https://issues.apache.org/jira/browse/BEAM-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3833. -- Resolution: Fixed Fix Version/s: 2.6.0 https://github.com/apache/beam/pull/5566 > Java S

[jira] [Closed] (BEAM-3743) Support for SDF splitting protocol in ULR

[ https://issues.apache.org/jira/browse/BEAM-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3743. -- Resolution: Fixed Fix Version/s: 2.6.0 https://github.com/apache/beam/pull/5566 > Suppor

[jira] [Closed] (BEAM-3741) Proto changes for splitting over Fn API

[ https://issues.apache.org/jira/browse/BEAM-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3741. -- Resolution: Fixed Fix Version/s: 2.5.0 > Proto changes for splitting over Fn API > --

[jira] [Closed] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ https://issues.apache.org/jira/browse/BEAM-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4267. -- Resolution: Fixed Fix Version/s: 2.6.0 I think we're good here :) > Implement a reusable

[jira] [Assigned] (BEAM-4205) Java: WordCount runs against manually started Flink at master

[ https://issues.apache.org/jira/browse/BEAM-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4205: -- Assignee: Ankur Goenka > Java: WordCount runs against manually started Flink at master

[jira] [Commented] (BEAM-4206) Python: WordCount runs against manually started Flink at master

[ https://issues.apache.org/jira/browse/BEAM-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522982#comment-16522982 ] Eugene Kirpichov commented on BEAM-4206: Robert and Axel are working on this, and

[jira] [Assigned] (BEAM-4206) Python: WordCount runs against manually started Flink at master

[ https://issues.apache.org/jira/browse/BEAM-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4206: -- Assignee: Axel Magnuson (was: Aljoscha Krettek) > Python: WordCount runs against manu

[jira] [Closed] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ https://issues.apache.org/jira/browse/BEAM-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3883. -- Resolution: Fixed > Python SDK stages artifacts when talking to job server > ---

[jira] [Closed] (BEAM-4689) Dataflow cannot deserialize SplittableParDo DoFns

[ https://issues.apache.org/jira/browse/BEAM-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4689. -- Resolution: Fixed Fix Version/s: 2.6.0 > Dataflow cannot deserialize SplittableParDo DoFn

[jira] [Closed] (BEAM-1841) FileBasedSource should have safeguards for when set of files grows while job is running

[ https://issues.apache.org/jira/browse/BEAM-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1841. -- Resolution: Won't Fix Fix Version/s: Not applicable Probably not worth investing more int

[jira] [Commented] (BEAM-1841) FileBasedSource should have safeguards for when set of files grows while job is running

[ https://issues.apache.org/jira/browse/BEAM-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530528#comment-16530528 ] Eugene Kirpichov commented on BEAM-1841: (especially because there is already a tr

[jira] [Closed] (BEAM-1824) Adapter for running SDF on a statically known input as a Source

[ https://issues.apache.org/jira/browse/BEAM-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1824. -- Resolution: Won't Fix Fix Version/s: Not applicable At this point it's better to just imp

[jira] [Closed] (BEAM-217) BoundedSource.splitAtFraction should be splitAfterFraction

[ https://issues.apache.org/jira/browse/BEAM-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-217. - Resolution: Won't Fix Not worth investing into BoundedSource anymore. But SDF will get this right.

[jira] [Closed] (BEAM-1190) FileBasedSource should ignore files that matched the glob but don't exist

[ https://issues.apache.org/jira/browse/BEAM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1190. -- Resolution: Won't Fix Fix Version/s: Not applicable This is an old issue, proposing a con

[jira] [Closed] (BEAM-2751) Write PCollection elements to individual files

[ https://issues.apache.org/jira/browse/BEAM-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2751. -- Resolution: Fixed Fix Version/s: 2.2.0 This can be done closely enough using FileIO.write

[jira] [Closed] (BEAM-2682) Merge AvroIOTest and AvroIOTransformTest

[ https://issues.apache.org/jira/browse/BEAM-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2682. -- Resolution: Fixed Fix Version/s: Not applicable Was fixed a long time ago. > Merge AvroI

[jira] [Closed] (BEAM-2883) Poor error message when forgetting to specify a Datastore project.

[ https://issues.apache.org/jira/browse/BEAM-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2883. -- Resolution: Fixed Fix Version/s: Not applicable Fixed a long time ago. > Poor error mess

[jira] [Closed] (BEAM-2826) Need to generate a single XML file when write is performed on small amount of data

[ https://issues.apache.org/jira/browse/BEAM-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2826. -- Resolution: Fixed Fix Version/s: 2.2.0 This is indeed addressed by FileIO.write which I t

[jira] [Assigned] (BEAM-3834) Python SDK harness should detect SDF ProcessFn and proactively checkpoint it

[ https://issues.apache.org/jira/browse/BEAM-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3834: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > Python SDK harness should detec

[jira] [Assigned] (BEAM-3837) Python SDK harness should understand a BundleSplitRequest and respond with a BundleSplit before bundle finishes

[ https://issues.apache.org/jira/browse/BEAM-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3837: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > Python SDK harness should under

[jira] [Closed] (BEAM-3595) Normalize URNs across SDKs and runners.

[ https://issues.apache.org/jira/browse/BEAM-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3595. -- Resolution: Fixed Fix Version/s: 2.5.0 Yeah this was done to a sufficient degree some tim

[jira] [Assigned] (BEAM-3742) Support for running a streaming SDF in Python SDK

[ https://issues.apache.org/jira/browse/BEAM-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3742: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > Support for running a streaming

[jira] [Assigned] (BEAM-3857) Dynamic output support for python

[ https://issues.apache.org/jira/browse/BEAM-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3857: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > Dynamic output support for pyth

[jira] [Assigned] (BEAM-3945) TFRecord Performance Tests doesn't work on hdfs

[ https://issues.apache.org/jira/browse/BEAM-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3945: -- Assignee: Udi Meiri (was: Eugene Kirpichov) > TFRecord Performance Tests doesn't work

[jira] [Closed] (BEAM-3907) Clarify how watermark is estimated for watchForNewFiles() transforms

[ https://issues.apache.org/jira/browse/BEAM-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3907. -- Resolution: Fixed Fix Version/s: 2.6.0 This will not need clarification since I fixed a b

[jira] [Assigned] (BEAM-3874) Switch AvroIO sink default codec to Snappy

[ https://issues.apache.org/jira/browse/BEAM-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3874: -- Assignee: (was: Eugene Kirpichov) > Switch AvroIO sink default codec to Snappy > --

[jira] [Assigned] (BEAM-4064) ClassCastExeption when reading Avro files using specific records with org.apache.avro.util.Utf8 fields

[ https://issues.apache.org/jira/browse/BEAM-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4064: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > ClassCastExeption when reading

[jira] [Closed] (BEAM-4166) FnApiDoFnRunner doesn't invoke setup/teardown

[ https://issues.apache.org/jira/browse/BEAM-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4166. -- Resolution: Fixed Fix Version/s: 2.5.0 > FnApiDoFnRunner doesn't invoke setup/teardown >

[jira] [Closed] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ https://issues.apache.org/jira/browse/BEAM-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4204. -- Resolution: Fixed Fix Version/s: (was: Not applicable) 2.5.0 > Pyt

[jira] [Assigned] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ https://issues.apache.org/jira/browse/BEAM-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4204: -- Assignee: Ankur Goenka (was: Eugene Kirpichov) > Python: PortableRunner - p.run() via

[jira] [Closed] (BEAM-3268) getPerDestinationOutputFilenames() is getting processed before write is finished on dataflow runner

[ https://issues.apache.org/jira/browse/BEAM-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3268. -- Resolution: Fixed Fix Version/s: 2.5.0 > getPerDestinationOutputFilenames() is getting pr

[jira] [Assigned] (BEAM-4288) SplittableDoFn: splitAtFraction() API for Python

[ https://issues.apache.org/jira/browse/BEAM-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4288: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > SplittableDoFn: splitAtFraction

[jira] [Assigned] (BEAM-3194) Support annotating that a DoFn requires stable / deterministic input for replay/retry

[ https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-3194: -- Assignee: Yueyang Qiu (was: Eugene Kirpichov) > Support annotating that a DoFn require

[jira] [Assigned] (BEAM-4372) Need an undeprecated Reshuffle transform

[ https://issues.apache.org/jira/browse/BEAM-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4372: -- Assignee: Yueyang Qiu (was: Eugene Kirpichov) > Need an undeprecated Reshuffle transfo

[jira] [Created] (BEAM-4737) SplittableDoFn dynamic rebalancing in Dataflow

Eugene Kirpichov created BEAM-4737: -- Summary: SplittableDoFn dynamic rebalancing in Dataflow Key: BEAM-4737 URL: https://issues.apache.org/jira/browse/BEAM-4737 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-4745) SDF tests broken by innocent change due to Dataflow worker dependencies

Eugene Kirpichov created BEAM-4745: -- Summary: SDF tests broken by innocent change due to Dataflow worker dependencies Key: BEAM-4745 URL: https://issues.apache.org/jira/browse/BEAM-4745 Project: Beam

[jira] [Assigned] (BEAM-4758) Avro-Protobuf support

[ https://issues.apache.org/jira/browse/BEAM-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-4758: -- Assignee: Chamikara Jayalath (was: Eugene Kirpichov) > Avro-Protobuf support > ---

[jira] [Commented] (BEAM-4758) Avro-Protobuf support

[ https://issues.apache.org/jira/browse/BEAM-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540614#comment-16540614 ] Eugene Kirpichov commented on BEAM-4758: Hi - thanks for the feedback, this seems

[jira] [Created] (BEAM-4775) JobService should support returning metrics

Eugene Kirpichov created BEAM-4775: -- Summary: JobService should support returning metrics Key: BEAM-4775 URL: https://issues.apache.org/jira/browse/BEAM-4775 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-4776) Java PortableRunner should support metrics

Eugene Kirpichov created BEAM-4776: -- Summary: Java PortableRunner should support metrics Key: BEAM-4776 URL: https://issues.apache.org/jira/browse/BEAM-4776 Project: Beam Issue Type: Bug

[jira] [Closed] (BEAM-4205) Java: WordCount runs against manually started Flink at master

[ https://issues.apache.org/jira/browse/BEAM-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4205. -- Resolution: Fixed Fix Version/s: 2.6.0 > Java: WordCount runs against manually started Fl

[jira] [Closed] (BEAM-4206) Python: WordCount runs against manually started Flink at master

[ https://issues.apache.org/jira/browse/BEAM-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-4206. -- Resolution: Fixed Fix Version/s: 2.6.0 > Python: WordCount runs against manually started

[jira] [Updated] (BEAM-4777) Python PortableRunner should support metrics

[ https://issues.apache.org/jira/browse/BEAM-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-4777: --- Component/s: (was: runner-core) sdk-py-core > Python PortableRunner shoul

[jira] [Created] (BEAM-4777) Python PortableRunner should support metrics

Eugene Kirpichov created BEAM-4777: -- Summary: Python PortableRunner should support metrics Key: BEAM-4777 URL: https://issues.apache.org/jira/browse/BEAM-4777 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-4777) Python PortableRunner should support metrics

[ https://issues.apache.org/jira/browse/BEAM-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-4777: --- Description: BEAM-4775 concerns adding metrics to the JobService API; the current issue is abo

[jira] [Created] (BEAM-4778) Less wasteful ArtifactStagingService

Eugene Kirpichov created BEAM-4778: -- Summary: Less wasteful ArtifactStagingService Key: BEAM-4778 URL: https://issues.apache.org/jira/browse/BEAM-4778 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-4780) Entry point for ULR JobService compatible with TestPortableRunner

Eugene Kirpichov created BEAM-4780: -- Summary: Entry point for ULR JobService compatible with TestPortableRunner Key: BEAM-4780 URL: https://issues.apache.org/jira/browse/BEAM-4780 Project: Beam

[jira] [Created] (BEAM-4779) Python PortableTestRunner that runs VR tests against a given portable runner

Eugene Kirpichov created BEAM-4779: -- Summary: Python PortableTestRunner that runs VR tests against a given portable runner Key: BEAM-4779 URL: https://issues.apache.org/jira/browse/BEAM-4779 Project:

[jira] [Created] (BEAM-4792) Add support for bounded SDF to all runners

Eugene Kirpichov created BEAM-4792: -- Summary: Add support for bounded SDF to all runners Key: BEAM-4792 URL: https://issues.apache.org/jira/browse/BEAM-4792 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-4792) Add support for bounded SDF to all runners

[ https://issues.apache.org/jira/browse/BEAM-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544249#comment-16544249 ] Eugene Kirpichov commented on BEAM-4792: https://github.com/apache/beam/pull/5940

[jira] [Updated] (BEAM-1868) CreateStreamTest testMultiOutputParDo is flaky on the Spark runner

[ https://issues.apache.org/jira/browse/BEAM-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1868: --- Fix Version/s: 2.1.0 > CreateStreamTest testMultiOutputParDo is flaky on the Spark runner > ---

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117469#comment-16117469 ] Eugene Kirpichov commented on BEAM-2671: Aviem - my apologies for misunderstanding

[jira] [Updated] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

[ https://issues.apache.org/jira/browse/BEAM-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2753: --- Fix Version/s: 2.2.0 > File DynamicDestinations side inputs don't work with sharding >

[jira] [Created] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

Eugene Kirpichov created BEAM-2753: -- Summary: File DynamicDestinations side inputs don't work with sharding Key: BEAM-2753 URL: https://issues.apache.org/jira/browse/BEAM-2753 Project: Beam

[jira] [Created] (BEAM-2754) Simplify DefaultCoder

Eugene Kirpichov created BEAM-2754: -- Summary: Simplify DefaultCoder Key: BEAM-2754 URL: https://issues.apache.org/jira/browse/BEAM-2754 Project: Beam Issue Type: Bug Components: sd

[jira] [Commented] (BEAM-2754) Simplify DefaultCoder

[ https://issues.apache.org/jira/browse/BEAM-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119049#comment-16119049 ] Eugene Kirpichov commented on BEAM-2754: cc: [~tgroh] [~kenn] > Simplify DefaultCo

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126416#comment-16126416 ] Eugene Kirpichov commented on BEAM-2140: Sorry for the delayed response. The output

[jira] [Closed] (BEAM-2700) BigQueryIO should support using file load jobs when using unbounded collections

[ https://issues.apache.org/jira/browse/BEAM-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2700. -- Resolution: Fixed Fix Version/s: 2.2.0 > BigQueryIO should support using file load jobs wh

[jira] [Assigned] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2768: -- Assignee: Reuven Lax (was: Kenneth Knowles) > Fix bigquery.WriteTables generating non-u

[jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127687#comment-16127687 ] Eugene Kirpichov commented on BEAM-2768: Could you tell more about how you're using

[jira] [Created] (BEAM-2776) TextIO should support reading header lines

Eugene Kirpichov created BEAM-2776: -- Summary: TextIO should support reading header lines Key: BEAM-2776 URL: https://issues.apache.org/jira/browse/BEAM-2776 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-2776) TextIO should support reading header lines

[ https://issues.apache.org/jira/browse/BEAM-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2776: --- Description: Users frequently request the ability to skip some header rows when reading text f

[jira] [Commented] (BEAM-2774) Add I/O source for VCF files (python)

[ https://issues.apache.org/jira/browse/BEAM-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133395#comment-16133395 ] Eugene Kirpichov commented on BEAM-2774: Related issue: https://issues.apache.org/j

[jira] [Updated] (BEAM-2776) TextIO should support reading header lines

[ https://issues.apache.org/jira/browse/BEAM-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2776: --- Component/s: sdk-py > TextIO should support reading header lines >

[jira] [Created] (BEAM-2781) Should have a canonical Compression enum

Eugene Kirpichov created BEAM-2781: -- Summary: Should have a canonical Compression enum Key: BEAM-2781 URL: https://issues.apache.org/jira/browse/BEAM-2781 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140180#comment-16140180 ] Eugene Kirpichov commented on BEAM-2802: Please see BEAM-2586 and previous discussi

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140190#comment-16140190 ] Eugene Kirpichov commented on BEAM-2803: Could you quantify "very slow" - what perf

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142139#comment-16142139 ] Eugene Kirpichov commented on BEAM-2802: What does the custom delimiter look like i

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142141#comment-16142141 ] Eugene Kirpichov commented on BEAM-2803: Thanks, do you have the Dataflow job IDs t

[jira] [Assigned] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

[ https://issues.apache.org/jira/browse/BEAM-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2753: -- Assignee: Eugene Kirpichov (was: Reuven Lax) > File DynamicDestinations side inputs don

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142875#comment-16142875 ] Eugene Kirpichov commented on BEAM-2803: Hmm, indeed, seems that shuffle is being q

[jira] [Comment Edited] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142875#comment-16142875 ] Eugene Kirpichov edited comment on BEAM-2803 at 8/26/17 5:43 PM:

[jira] [Commented] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142905#comment-16142905 ] Eugene Kirpichov commented on BEAM-2803: We can try another way to break fusion I g

[jira] [Created] (BEAM-2810) Consider a faster Avro library in Python

Eugene Kirpichov created BEAM-2810: -- Summary: Consider a faster Avro library in Python Key: BEAM-2810 URL: https://issues.apache.org/jira/browse/BEAM-2810 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2810) Consider a faster Avro library in Python

[ https://issues.apache.org/jira/browse/BEAM-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143372#comment-16143372 ] Eugene Kirpichov commented on BEAM-2810: It might be a good idea to fix fastavro th

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

[ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143916#comment-16143916 ] Eugene Kirpichov commented on BEAM-2802: Hmm, I have a hard time thinking why someb

[jira] [Created] (BEAM-2816) Create SplittableDoFnTester

Eugene Kirpichov created BEAM-2816: -- Summary: Create SplittableDoFnTester Key: BEAM-2816 URL: https://issues.apache.org/jira/browse/BEAM-2816 Project: Beam Issue Type: Bug Componen

[jira] [Assigned] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[ https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2390: -- Assignee: Reuven Lax (was: Eric Johston) > allow user to use .setTimePartitioning in Bi

[jira] [Closed] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[ https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2390. -- Resolution: Fixed This has been submitted. Release note: this is NOT update-compatible for Dataf

[jira] [Commented] (BEAM-2826) Need to generate a single XML file when write is performed on small amount of data

[ https://issues.apache.org/jira/browse/BEAM-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147720#comment-16147720 ] Eugene Kirpichov commented on BEAM-2826: The solution to this bug would be either a

[jira] [Created] (BEAM-2827) Introduce AvroIO.watchForNewFiles

Eugene Kirpichov created BEAM-2827: -- Summary: Introduce AvroIO.watchForNewFiles Key: BEAM-2827 URL: https://issues.apache.org/jira/browse/BEAM-2827 Project: Beam Issue Type: Bug Co

[jira] [Created] (BEAM-2828) Create FileIO

Eugene Kirpichov created BEAM-2828: -- Summary: Create FileIO Key: BEAM-2828 URL: https://issues.apache.org/jira/browse/BEAM-2828 Project: Beam Issue Type: New Feature Components: sd

[jira] [Closed] (BEAM-2827) Introduce AvroIO.watchForNewFiles

[ https://issues.apache.org/jira/browse/BEAM-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2827. -- Resolution: Fixed > Introduce AvroIO.watchForNewFiles > - > >

[jira] [Closed] (BEAM-2644) Make it easier to test runtime-accessible ValueProvider's

[ https://issues.apache.org/jira/browse/BEAM-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2644. -- Resolution: Fixed Fix Version/s: 2.2.0 > Make it easier to test runtime-accessible ValuePr

[jira] [Updated] (BEAM-2516) User reports 4 minutes to process 1 million line CSV in DirectRunner

[ https://issues.apache.org/jira/browse/BEAM-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2516: --- Fix Version/s: 2.2.0 > User reports 4 minutes to process 1 million line CSV in DirectRunner > -

[jira] [Updated] (BEAM-2790) Error while reading from Amazon S3 via Hadoop File System

[ https://issues.apache.org/jira/browse/BEAM-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2790: --- Fix Version/s: 2.2.0 > Error while reading from Amazon S3 via Hadoop File System >

[jira] [Updated] (BEAM-2803) JdbcIO read is very slow when query return a lot of rows

[ https://issues.apache.org/jira/browse/BEAM-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2803: --- Fix Version/s: (was: Not applicable) 2.2.0 > JdbcIO read is very slow wh

<    1   2   3   4   5   6   7   >