[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300892#comment-15300892 ] Scott Wegner commented on BEAM-117: --- This can be closed. Assigning back to [~bchambers] > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299100#comment-15299100 ] ASF GitHub Bot commented on BEAM-117: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/375 > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297137#comment-15297137 ] ASF GitHub Bot commented on BEAM-117: - GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/375 [BEAM-117] Evaluate display data from InProcessPipelineRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Display data can be added to any PTransform to be used for display from any runner. Runners are not required to consume display data, and currently many don't. This changes InProcessRunner to consumer display data (and then discard it) in order to validate that display data is properly implemented on transforms within a pipeline. Exceptions thrown within HasDisplayData implementations will cause Pipeline.run() to fail. You can merge this pull request into a Git repository by running: $ git pull https://github.com/swegner/incubator-beam displaydata-directrunner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/375.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #375 commit 698a0af1553279ba48ad2f4768556fbd4810208a Author: Scott Wegner Date: 2016-05-23T21:29:33Z Evaluate display data from InProcessPipelineRunner Display data can be added to any PTransform to be used for display from any runner. Runners are not required to consume display data, and currently many don't. This changes InProcessRunner to consumer display data (and then discard it) in order to validate that display data is properly implemented on transforms within a pipeline. Exceptions thrown within HasDisplayData implementations will cause Pipeline.run() to fail. > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294621#comment-15294621 ] ASF GitHub Bot commented on BEAM-117: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/355 > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291647#comment-15291647 ] ASF GitHub Bot commented on BEAM-117: - GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/355 [BEAM-117] Fix bug in PipelineOptions DisplayData serialization Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- In the code for populating display data for `PipelineOptions` in the serialized JSON, we had a bug in `ProxyInvocationHandler.Serializer` where it was not checking for null values from previously-deserialized JSON values. As a result, a null value in the JSON would cause us to throw and block pipelines from running. This PR fixes a number of issues: * Properly handle null JSON values in `ProxyInvocatinHandler.Serializer` * Establish a common pattern for handling errors during display data collection, via `DisplayData.from(Throwable)` * Implement resiliency pattern in `ProxyInvocationHandler.Serializer` so bugs in `PipelineOptions` serialization do not prevent running pipelines. You can merge this pull request into a Git repository by running: $ git pull https://github.com/swegner/incubator-beam displaydata-resiliency Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/355.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #355 commit 8ffeb3da894ee5e25a54f7faa8faeff8fe1ae98f Author: Scott Wegner Date: 2016-05-19T16:17:37Z Fix bug in PipelineOptions DisplayData serialization > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285185#comment-15285185 ] ASF GitHub Bot commented on BEAM-117: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/338 > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280395#comment-15280395 ] ASF GitHub Bot commented on BEAM-117: - GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/325 [BEAM-117] Validate display data registration in Dataflow Runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- The DataflowRunner (like other runners) transforms pipelines during construction and sends only primitive transforms to its service. This PR validates that primitive transforms are correctly registering display data for IO transforms. You can merge this pull request into a Git repository by running: $ git pull https://github.com/swegner/incubator-beam displaydata-dataflow Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/325.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #325 commit c03af9e254497c9af9674b2b9517fe0b1db5381b Author: Scott Wegner Date: 2016-05-09T20:31:39Z Add overload for DisplayDataMatchers.hasDisplayItem commit 48b30e8e133a1949693fa854883471ca623e05a4 Author: Scott Wegner Date: 2016-05-09T20:50:01Z Correct DisplayDataEvaluator behavior for primitives Previously DisplayDataEvaluator would assume the input PTransform was a composite and if so collect DisplayData for inner primitives. However it incorrectly discarding DisplayData when the input itself is a primitive. This fixes the tracking logic. commit be44a9bdd81e054e7e235eae613b8c35b9f9738b Author: Scott Wegner Date: 2016-05-09T20:52:49Z Implement TransformTreeNode.toString Previously TransformTreeNode took the default Object#toString implementation, which displays the class name and ID number. This change instead uses the fullName property, which is the path of inner tranform within the graph. This is very useful for stepping through graph transformation logic in a debugger. The toString implelementation is not necessary for correctness. commit b5b7c9baeebb27cdc5f7ed5074f1d764299837b2 Author: Scott Wegner Date: 2016-05-09T20:55:07Z Add pinning test to verify BigQueryIO display data commit de888f9b353d6addc86662cd010089dc869a9854 Author: Scott Wegner Date: 2016-05-09T21:06:24Z Add pinning test for DatastoreIO display data commit dd26b46867755612aff4a74b7222ea7435a73966 Author: Scott Wegner Date: 2016-05-09T21:09:34Z Add pinning test for TextIO display data commit 2a3229bcafbdbdcd613fd16b511041f0e01c5700 Author: Scott Wegner Date: 2016-05-09T21:17:50Z Add pinning test for AvroIO display data commit df8b28de636454d4b580898fa8870c0929a3cf8d Author: Scott Wegner Date: 2016-05-09T21:23:38Z Add pinning test for PubsubIO display data > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278633#comment-15278633 ] ASF GitHub Bot commented on BEAM-117: - GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/315 [BEAM-117] Runners should be resilient to DisplayData failure Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Display data is collected from PTransforms at Pipeline construction time. Collecting display data runs user code from provided transforms and fn's. These components should be designed not to throw during pipeline construction, however we also shouldn't fail a pipeline if this code does fail. This PR adds resiliency to the DataflowPipelineTranslator, where we collect display data for the Dataflow runner, and also a RunnableOnService test to verify that all runners are resilient to display data failures. Other runners are not yet using display data, but will get this validation for free when they do. You can merge this pull request into a Git repository by running: $ git pull https://github.com/swegner/incubator-beam displaydata-safe Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #315 commit d6c5025937cbd0016d0a83d08e53902ae4d4519b Author: Scott Wegner Date: 2016-05-10T18:19:14Z Runners should be resilient to DisplayData failure Display data is collected from PTransforms at Pipeline construction time. Collecting display data runs user code from provided transforms and fn's. These components should be designed not to throw during pipeline construction, however we also shouldn't fail a pipeline if this code does fail. This PR adds resiliency to the DataflowPipelineTranslator, where we collect display data for the Dataflow runner, and also a RunnableOnService test to verify that all runners are resilient to display data failures. Other runners are not yet using display data, but will get this validation for free when they do. > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258481#comment-15258481 ] ASF GitHub Bot commented on BEAM-117: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/241 > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258326#comment-15258326 ] ASF GitHub Bot commented on BEAM-117: - GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/241 [BEAM-117] Minor updates for DisplayData usage Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This PR addresses the following JIRA issues: * [[BEAM-195] Rename DisplayDataMatchers#includes](https://issues.apache.org/jira/browse/BEAM-195) * [[BEAM-200] Standardize terminology to "display data" in documentation](https://issues.apache.org/jira/browse/BEAM-200) * [[BEAM-219] Use super.populateDisplayData consistently](https://issues.apache.org/jira/browse/BEAM-219) [[BEAM-199] Improve fluent interface for manipulating DisplayData.Items](https://issues.apache.org/jira/browse/BEAM-199) will be in the next PR as to not bloat this one too much. You can merge this pull request into a Git repository by running: $ git pull https://github.com/swegner/incubator-beam displaydata-tweaks Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/241.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #241 commit 08b475f1971b3f74c9b321941cdbe2818c25c939 Author: Scott Wegner Date: 2016-04-25T16:31:13Z Rename DisplayDataMatchers.includes to includesDisplayData to clarify usage commit 5a1d93ce01000df8577acd66d6e9c6ebf5d6948f Author: Scott Wegner Date: 2016-04-25T16:34:58Z Normalize comments to refer to 'display data' rather than 'display metadata' commit 954e734fbde4a055ee97d84b9f1c91d598fc5de9 Author: Scott Wegner Date: 2016-04-25T16:41:07Z Add super.populateDisplayData() to standard implementations. The current super implementation is a no-op, but this is the recommended way to implement the pattern > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
