[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-23 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-648451444 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-23 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-648451338 One shortcoming of this approach is that component ids are stored in a _global_ cache. I think it would be better if the cache could be scoped to a pipeline, but that would

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-24 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-648889479 CC: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-24 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-649046980 I suppose another place where global state is ill-advised is when running tests, since we run all the tests in the same process and many of them create pipelines. Py

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-24 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-649168647 I have a POC working that solves the problem by fixing all of the component IDs in the external transforms in `to_runner_api()`. It feels pretty error-prone, but it doesn't

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-25 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-649898941 I'll see if I can add a simple unit test tomorrow. This is an automated message from the Apache Git Service.

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-26 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-650452189 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-26 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-650454710 I think something like this will fix the ExternalTransform bug, without breaking all the tests that rely on specific component id assignments. It will still grow witho

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-29 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-651416156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-29 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-651426642 Ok I think I'm finally done churning on this. Sorry for all the changes. What I have now stores the component ids in a `ComponentIdContext` object stored on the Pipeline. Th

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-651988106 retest this please This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652002495 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652002592 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652067737 The only failure in Python precommit seems to be due to BEAM-10007. I think this is safe to merge after an LGTM

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-06-30 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652090261 @lukecwik do you happen to have time to review this? Looks like cham is OOO I'd like to get this in before the release cut tomorrow since it fixes a significant issue

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652700056 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652715930 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-07-06 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-654504237 @kennknowles are you ok with merging this now? This is an automated message from the Apache Git Service. To