[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742841#comment-15742841 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1513 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742820#comment-15742820 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1549 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742700#comment-15742700 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1575 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740320#comment-15740320 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1576 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738829#comment-15738829 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1576 [BEAM-551] Fix toString for FileBasedSource You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/fix_file_based_source_to_string Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1576 commit 015477df98f0378b3ff51a2126e156c7f28d0477 Author: Sam McVeety Date: 2016-12-10T23:59:54Z [BEAM-551] Fix toString for FileBasedSource > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738531#comment-15738531 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1575 [BEAM-551] Fix handling of default values for RVPs Found a bug here, fixed an added a regression test. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/fix_rvp_default_handling Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1575.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1575 commit 9dd250c726e6bfee93d77c67e155faa771d4a526 Author: Sam McVeety Date: 2016-12-10T17:16:57Z [BEAM-551] Fix handling of default for VP > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738327#comment-15738327 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1545 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736504#comment-15736504 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety reopened a pull request: https://github.com/apache/incubator-beam/pull/1513 [BEAM-551] Add support for BigQueryIO.Write R: @davorbonaci Based on https://github.com/apache/incubator-beam/pull/1238 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1513 commit 29ab056136e36204140ded78a4fbf9ac51afca4c Author: Sam McVeety Date: 2016-12-04T22:16:23Z Add schema parameterization commit 5bd20ae13d17be7e810f7e03065b8e6b5c8cc8d5 Author: Sam McVeety Date: 2016-12-05T23:12:14Z update bq commit e530f1fed2b3ab457f55964755668522d2fc450f Author: Sam McVeety Date: 2016-12-05T23:38:04Z Fixes commit 911911fda79cbb5b54b4af5e8d888c0279830211 Author: Sam McVeety Date: 2016-12-06T01:54:48Z Fix issue with overriding project commit fec3ce58f7f382c7f717da96af3d68cc63fb9521 Author: Sam McVeety Date: 2016-12-07T05:06:18Z Fix serialization commit 5dbfeb1590248dfd566a16f220adcfe1d3cd67ea Author: Sam McVeety Date: 2016-12-09T17:58:08Z Minor fixups commit 58fcfa44d164d651e4b49f28aaeebbface9f33a0 Author: Sam McVeety Date: 2016-12-09T18:33:21Z Add more tests and fix merge issues commit a0098357d605521ac59215c5effd6b7ae55a8e03 Author: Sam McVeety Date: 2016-12-09T19:18:22Z Fix style > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736503#comment-15736503 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1513 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736327#comment-15736327 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1513 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736326#comment-15736326 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety reopened a pull request: https://github.com/apache/incubator-beam/pull/1513 [BEAM-551] Add support for BigQueryIO.Write R: @davorbonaci Based on https://github.com/apache/incubator-beam/pull/1238 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1513 commit 29ab056136e36204140ded78a4fbf9ac51afca4c Author: Sam McVeety Date: 2016-12-04T22:16:23Z Add schema parameterization commit 5bd20ae13d17be7e810f7e03065b8e6b5c8cc8d5 Author: Sam McVeety Date: 2016-12-05T23:12:14Z update bq commit e530f1fed2b3ab457f55964755668522d2fc450f Author: Sam McVeety Date: 2016-12-05T23:38:04Z Fixes commit 911911fda79cbb5b54b4af5e8d888c0279830211 Author: Sam McVeety Date: 2016-12-06T01:54:48Z Fix issue with overriding project commit fec3ce58f7f382c7f717da96af3d68cc63fb9521 Author: Sam McVeety Date: 2016-12-07T05:06:18Z Fix serialization commit 5dbfeb1590248dfd566a16f220adcfe1d3cd67ea Author: Sam McVeety Date: 2016-12-09T17:58:08Z Minor fixups commit 58fcfa44d164d651e4b49f28aaeebbface9f33a0 Author: Sam McVeety Date: 2016-12-09T18:33:21Z Add more tests and fix merge issues commit a0098357d605521ac59215c5effd6b7ae55a8e03 Author: Sam McVeety Date: 2016-12-09T19:18:22Z Fix style > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735793#comment-15735793 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1238 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730289#comment-15730289 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1545 [BEAM-551] Fix handling of TextIO.Sink R: @dhalperi Directory needs to be parameterized. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/text_io_write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1545.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1545 commit d017fde2d063765a73b290e1b1e1b849f147910f Author: Sam McVeety Date: 2016-12-07T21:27:53Z [BEAM-551] Fix handling of TextIO.Sink commit 9309c9389d9e9fa2cae3f7378692d0484ddc54b2 Author: Sam McVeety Date: 2016-12-07T22:09:41Z Fix test > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727268#comment-15727268 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1504 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727142#comment-15727142 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1503 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726722#comment-15726722 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1238 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726721#comment-15726721 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety reopened a pull request: https://github.com/apache/incubator-beam/pull/1238 [BEAM-551] BigqueryIO.Read support for ValueProvider R: @dhalperi This is the serialization issue I was referencing, where I think the issue is that there is a deferred translation from String -> TableRef -> String, causing a serialization error. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_vp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1238 commit ced1073597e3fbcc826f10a417e810803c7317ce Author: Sam McVeety Date: 2016-10-30T18:58:44Z Add BQ VP code BQ VP Fix most tests Make serializable Fix BQ tests Fix tests Update API Fix validation case Fix one more query reference commit 2d05f107499ce94bb35f05a4c386625e66fe660c Author: Sam McVeety Date: 2016-12-06T05:24:20Z Minor fixes commit 1ca49af0659fd32ffa06e4e5778e3e4f90d85be6 Author: Sam McVeety Date: 2016-12-06T05:31:34Z Address serialization issue > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723978#comment-15723978 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1513 [BEAM-551] Add support for BigQueryIO.Write R: @davorbonaci Based on https://github.com/apache/incubator-beam/pull/1238 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1513 commit e06fd8ce238971d1b235ce01f527f7cedb058127 Author: sammcveety Date: 2016-10-20T16:29:04Z Add NestedValueProvider commit 2e2aa465295dd141577639098522fdbae359126a Author: Sam McVeety Date: 2016-10-30T18:58:44Z Add BQ VP code commit 6fed9071be77b6b12506ca223a8fb68032887f5e Author: Sam McVeety Date: 2016-10-31T17:39:10Z BQ VP commit b8974403bae0bf50da841f526f01085d99f89f97 Author: Sam McVeety Date: 2016-11-01T23:35:59Z Merge branch 'master' into sgmc/bq_vp commit 61b220f30b0e9fd67f688de7df50fd8909c58a01 Author: Sam McVeety Date: 2016-11-01T23:57:39Z Fix most tests commit 053267a41ffe78180c0bcef8046258314f8970d9 Author: Sam McVeety Date: 2016-11-29T18:16:25Z Make serializable commit 73d248e3e3fdf0d1b15abf60a1763c41e2054940 Author: Sam McVeety Date: 2016-12-01T21:39:39Z Fix BQ tests commit 407ee12e8f454889bc335bf17bb9c37dc0df33e5 Author: Sam McVeety Date: 2016-12-02T23:12:56Z Merge branch 'master' into sgmc/bq_vp commit 4958f8634298e1078a4e20b920dc158e630d6633 Author: Sam McVeety Date: 2016-12-02T23:23:48Z Fix tests commit 9a44144c475e19afa33103c3247672c71f7034be Author: Sam McVeety Date: 2016-12-03T01:29:40Z Update API commit 9c13496df5e4630476925032ce596797471fd396 Author: Sam McVeety Date: 2016-12-03T02:11:19Z BQ Write commit aacb0281fe714e94658001746c97edf37049dd55 Author: Sam McVeety Date: 2016-12-04T22:16:23Z Add schema parameterization commit 99fc371a34ec24e6ee59f852db9a6d327bf5589b Author: Sam McVeety Date: 2016-12-05T23:12:14Z update bq commit a5230670b3f893a0a97ee4eba434eb3bd7ca9ff0 Author: Sam McVeety Date: 2016-12-05T23:38:04Z Fixes > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722650#comment-15722650 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1459 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718784#comment-15718784 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1504 [BEAM-551] Add support for native PubSubIO R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/pubsub_native Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1504.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1504 commit 21d6e45aef7d8d9f93b79e7573952cff68357199 Author: Sam McVeety Date: 2016-12-03T21:40:38Z [BEAM-551] Add support for native PubSubIO > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718757#comment-15718757 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1503 [BEAM-551] Add DisplayData handling of ValueProvider R: @dhalperi Notes: - An alternative would be to accept ValueProvider and call T.toString() if isAccessible - The parameter is not marked @Nullable because the DisplayData code calls checkNotNull You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/vp_displaydata Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1503.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1503 commit 9d4efdec6932e2f0f840d84734cc7dc89f614581 Author: Sam McVeety Date: 2016-12-03T21:20:24Z [BEAM-551] Add DisplayData handling of ValueProvider > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717092#comment-15717092 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1475 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Sam McVeety >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710197#comment-15710197 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1475 [BEAM-551] Add TextIO.Write support for ValueProvider R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/textio Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1475.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1475 commit 18d3ee531c45bf972ca5da701ede0ad43c4760f7 Author: Sam McVeety Date: 2016-11-30T22:06:59Z Add TextIO.Write support > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707098#comment-15707098 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1459 [BEAM-551] Changes to BigQuery validation to support dynamic values R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/bqstream Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1459.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1459 commit e8d5e82748bfa1c2801cd06291c73f7d12799cc8 Author: Sam McVeety Date: 2016-11-30T00:55:48Z Fix tests > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706425#comment-15706425 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1452 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705995#comment-15705995 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1452 [BEAM-551] Add a test to demonstrate NVP serialization R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/nvp_repro Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1452.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1452 commit c9a5f8d8bfbe48921b1327acb4d2bfe8fa31f12e Author: Sam McVeety Date: 2016-11-02T00:58:16Z Demonstrate serializing issue commit 29932b2fa1798df16d7114ed3dd915e0dcf50d93 Author: Sam McVeety Date: 2016-11-29T17:45:10Z Update test > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704249#comment-15704249 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1230 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704186#comment-15704186 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1156 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680416#comment-15680416 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1230 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680417#comment-15680417 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety reopened a pull request: https://github.com/apache/incubator-beam/pull/1230 [BEAM-551] PubsubIO with ValueProvider R: @davorbonaci You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/pubsub_new Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1230.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1230 commit e06fd8ce238971d1b235ce01f527f7cedb058127 Author: sammcveety Date: 2016-10-20T16:29:04Z Add NestedValueProvider commit 54af0b970701115ba3843d02f85cb25086101734 Author: Sam McVeety Date: 2016-10-30T02:02:51Z Demonstrate PubsubIO with NVP commit a3730dcf3c7072e11fe6340dd3b757bf64b18f52 Author: Sam McVeety Date: 2016-10-30T03:47:42Z Fix null pointer issues commit 4a2d2637a4c09049500b22231674536e4ae20706 Author: Sam McVeety Date: 2016-11-01T20:42:11Z Merge branch 'master' into sgmc/pubsub_new commit 6cb4822e18e1f6850e855b0f7ce2d9737cc73002 Author: Sam McVeety Date: 2016-11-01T21:07:36Z Fix up tests commit 57e52a23b46b76a3e45884c33c52a0331f0d4e8a Author: Sam McVeety Date: 2016-11-02T15:48:48Z Address comments. commit a94526af16465be54a9792a41a5ead9162e7c3c7 Author: Sam McVeety Date: 2016-11-03T22:26:06Z Address comments commit a8811ace2cce6f685bb4875562355a8e664fa5bd Author: Sam McVeety Date: 2016-11-14T19:17:50Z Address builder comments > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658324#comment-15658324 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1266 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630570#comment-15630570 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1266 [BEAM-551] Add better toString methods for VPs R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/rvp_string Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1266.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1266 commit 5a7954cfdc5f4bd6954790f9a0ed501451050e33 Author: Sam McVeety Date: 2016-11-02T21:05:04Z [BEAM-551] Add better toString methods for VPs > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627669#comment-15627669 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1249 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625975#comment-15625975 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1134 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625902#comment-15625902 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1249 [BEAM-551] Add --templateRunner option to Dataflow This provides the DataflowRunner with a means of creating templates for use with ValueProvider. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/template_runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1249.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1249 commit 4707450f65b40851918205dd27f0eb0d1cb7348d Author: Sam McVeety Date: 2016-11-01T16:27:44Z Add templateRunner option to Dataflow runner > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624014#comment-15624014 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1146 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612889#comment-15612889 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/1134 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612890#comment-15612890 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety reopened a pull request: https://github.com/apache/incubator-beam/pull/1134 [BEAM-551] Allow TextIO to accept ValueProvider Thinking about testing, is it worth having something sub-integration-test level? We'd likely want to add some convenience methods to RVP for that, then. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/vp_textio Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1134 commit 7310c1f5e77f042aa21db046c91ccdb18edec1d8 Author: sammcveety Date: 2016-10-16T22:37:51Z VP for TextIO commit 37b89c063b2e24507dfd6725cf89668b135b9774 Author: sammcveety Date: 2016-10-17T12:31:51Z Add alternate VP getter commit 9dcf6bfac552737586fc8b0416f1ab0d46291f4b Author: sammcveety Date: 2016-10-17T12:56:41Z Add surface methods for VP commit 9492444b44f331cedd0b8e1a4faec6dc0528d4e2 Author: sammcveety Date: 2016-10-19T13:40:07Z Further improvements to VP handling. commit 9f4022a9ed30ac906415789a0a1d72764df117c9 Author: sammcveety Date: 2016-10-24T15:49:01Z Address comments. commit 4a7067dc5ba5f9dd785b3a7a3aec9d32f19aca2f Author: sammcveety Date: 2016-10-24T17:27:03Z Fix parameter passing > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609083#comment-15609083 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1105 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602276#comment-15602276 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1107 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596351#comment-15596351 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1156 [BEAM-551] Add method to output runtime options Add a method that outputs the dynamic options from PipelineOptions, so an interested consumer could write these to a metadata storage location in order to more intelligently provide information about the parameters that a template supports. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/vp_meta Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1156.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1156 commit 1dcce68db8c2627b96de4e17f91916a79217e098 Author: sammcveety Date: 2016-10-21T16:50:01Z Add method to output runtime options > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592313#comment-15592313 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1146 [BEAM-551] Add NestedValueProvider Wrote this up based on some nascent thoughts, and looking ahead to PubsubIO, which uses a bunch of wrapper classes instead of String. This allows us to nicely turn those into VP instead of deferring all the translation code into the caller, which would be gross. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/nested_vp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1146.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1146 commit e06fd8ce238971d1b235ce01f527f7cedb058127 Author: sammcveety Date: 2016-10-20T16:29:04Z Add NestedValueProvider > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588838#comment-15588838 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1134 [BEAM-551] Allow TextIO to accept ValueProvider Thinking about testing, is it worth having something sub-integration-test level? We'd likely want to add some convenience methods to RVP for that, then. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/vp_textio Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1134 commit 7310c1f5e77f042aa21db046c91ccdb18edec1d8 Author: sammcveety Date: 2016-10-16T22:37:51Z VP for TextIO commit 37b89c063b2e24507dfd6725cf89668b135b9774 Author: sammcveety Date: 2016-10-17T12:31:51Z Add alternate VP getter commit 9dcf6bfac552737586fc8b0416f1ab0d46291f4b Author: sammcveety Date: 2016-10-17T12:56:41Z Add surface methods for VP commit 9492444b44f331cedd0b8e1a4faec6dc0528d4e2 Author: sammcveety Date: 2016-10-19T13:40:07Z Further improvements to VP handling. > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576060#comment-15576060 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1107 [BEAM-551] Add property name to RuntimeValueProvider The intention here is to allow Native IO to perform the same substitution that would otherwise take place at job execution time. R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam prop_name Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1107 commit e84cb0911129757e42716cce5c75b1d64e619e31 Author: sammcveety Date: 2016-10-14T17:41:33Z Add property name to RuntimeValueProvider > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575968#comment-15575968 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1105 [BEAM-551] Add utility to handle JSON option manipulation Provide a utility to be used with RuntimeValueProviders that allows runners to manipulate PipelineOptions by interleaving runtime parameters. Green travis: https://travis-ci.org/sammcveety/incubator-beam/jobs/167702097 R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/json_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1105.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1105 commit 83230a2c47b533162cdbf45580daf59896149487 Author: sammcveety Date: 2016-10-14T17:06:08Z Add utility to handle JSON option manipulation > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574304#comment-15574304 ] ASF GitHub Bot commented on BEAM-551: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1009 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524418#comment-15524418 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1009 [BEAM-551] Add ValueProvider class (Take 2) My branch had become a dumpster fire, so I'm opening a fresh PR (history is here: https://github.com/apache/incubator-beam/pull/827). Add a basic implementation of StaticValueProvider and DynamicValueProvider. @dhalperi , could you please take a look? Hoping the jackson API surface test failures are easy to fix. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/vp2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1009.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1009 commit 3e55e508ede87b92f0d78bc960803bc546a8a354 Author: sammcveety Date: 2016-09-26T22:27:15Z Reset of ValueProvider PR. commit 562baf6ec7a9a77dad797c5a40df1f5122e5d1b8 Author: sammcveety Date: 2016-09-26T22:46:05Z Fix checkstyle, etc., failures. > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524378#comment-15524378 ] ASF GitHub Bot commented on BEAM-551: - Github user sammcveety closed the pull request at: https://github.com/apache/incubator-beam/pull/827 > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421067#comment-15421067 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/827 [BEAM-551] Add ValueProvider class Hi @lukecwik , can you please take a look? Currently implemented for ValueProvider only. Tests demonstrate intended functionality, once the questions below are answered: - What's the best way to avoid threading issues with the static options in test? - Is it reasonable to continue down the road of special-casing getter/setter validation for this class? - What would be the preferred way to annotate T for ValueProvider to work around type erasure? You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/valueprovider Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/827.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #827 commit 867a080b98f585db2687d9aaf7d42de8c813 Author: sammcveety Date: 2016-08-14T00:45:58Z Merge pull request #1 from apache/master Update my fork. commit bb7aea8db07b3a7ae988f75888716dd865e07e08 Author: sammcveety Date: 2016-08-14T00:48:18Z Create ValueProvider.java commit 5330800d9d5b54b897c53cb8ea40c6331fe3c36a Author: sammcveety Date: 2016-08-14T00:48:54Z Update ValueProvider.java commit 9b9b17c2338612b4030d1c4a36f8fb4c5c179c2e Author: sammcveety Date: 2016-08-14T00:50:56Z Update ProxyInvocationHandler.java commit b54832bd0c28691d1e6087b4901fb6ae11ebd2c9 Author: sammcveety Date: 2016-08-14T00:54:33Z Create ValueProviderTest.java commit e2d16a7ad66676a099d4160ecd4c31c369acd46e Author: sammcveety Date: 2016-08-14T00:56:41Z Update ValueProviderTest.java commit afbe8157f711fca00ed37afbe461edd4ac78e30f Author: sammcveety Date: 2016-08-14T16:34:23Z Update ProxyInvocationHandler.java commit e111045ae1125ed49ea1f26cdde31b3c1583ed2d Author: sammcveety Date: 2016-08-14T16:58:29Z Update ValueProviderTest.java commit 0bff463a71c4f4b3ad13edd924fd03226a915a61 Author: sammcveety Date: 2016-08-14T20:11:49Z Add tests. commit c4f3acc427b828f0f0254f83e84690b14c0a0f98 Author: sammcveety Date: 2016-08-14T21:42:56Z Iterations on tests. commit f19435b688fed41d873b85ac7750d923025256a9 Author: sammcveety Date: 2016-08-15T01:32:37Z Proposal for how to handle getters/setters. > Support Dynamic PipelineOptions > --- > > Key: BEAM-551 > URL: https://issues.apache.org/jira/browse/BEAM-551 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sam McVeety >Assignee: Frances Perry >Priority: Minor > > During the graph construction phase, the given SDK generates an initial > execution graph for the program. At execution time, this graph is > executed, either locally or by a service. Currently, Beam only supports > parameterization at graph construction time. Both Flink and Spark supply > functionality that allows a pre-compiled job to be run without SDK > interaction with updated runtime parameters. > In its current incarnation, Dataflow can read values of PipelineOptions at > job submission time, but this requires the presence of an SDK to properly > encode these values into the job. We would like to build a common layer > into the Beam model so that these dynamic options can be properly provided > to jobs. > Please see > https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit > for the high-level model, and > https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit > for > the specific API proposal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)