[GitHub] beam pull request #4100: Dataflow: Add option to upload heap dumps to GCS
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/4100 Dataflow: Add option to upload heap dumps to GCS This flag needs to go in before backend runner code that reads it. It will have no effect until that code is deployed. Follow this checklist to help us incorporate your contribution quickly and easily: - [*] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [*] Each commit in the pull request should have a meaningful subject line and body. - [*] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [*] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [*] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam heap-dump-flag Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/4100.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4100 commit 2db71dd77d99e0117024658104b1040ec87f896b Author: bchambers <bchamb...@google.com> Date: 2017-11-08T20:50:44Z Dataflow: Add option to upload heap dumps to GCS This flag needs to go in before backend runner code that reads it. It will have no effect until that code is deployed. ---
[GitHub] beam pull request #3914: [BEAM-2996] Ensure metric names are not null or emp...
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3914 [BEAM-2996] Ensure metric names are not null or empty Follow this checklist to help us incorporate your contribution quickly and easily: - [*] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [*] Each commit in the pull request should have a meaningful subject line and body. - [*] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [*] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [*] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam metric-names Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3914.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3914 commit 8635e46ca2be718e9a5a32108be177b2a096ca51 Author: bchambers <bchamb...@google.com> Date: 2017-09-27T17:44:48Z Ensure metric names are not null or empty ---
[GitHub] beam pull request #3670: [BEAM-2708] Configure BZIP2 to read all "streams"
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3670 [BEAM-2708] Configure BZIP2 to read all "streams" Without this, CompressionMode.BZIP2 only supports "standard" bz2 files containing a single stream. With this change, BZIP2 also supports bz2 files containing multiple streams, such as those produced by pbzip2. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam pbzip2-cherrypick Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3670.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3670 commit 371a89ec7bdf4b92ebd7e346b376160253416a32 Author: bchambers <bchamb...@google.com> Date: 2017-08-01T21:11:24Z [BEAM-2708] Configure BZIP2 to read all "streams" Without this, CompressionMode.BZIP2 only supports "standard" bz2 files containing a single stream. With this change, BZIP2 also supports bz2 files containing multiple streams, such as those produced by pbzip2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3669: [BEAM-2708] Configure BZIP2 to read all "streams"
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3669 [BEAM-2708] Configure BZIP2 to read all "streams" Without this, CompressionMode.BZIP2 only supports "standard" bz2 files containing a single stream. With this change, BZIP2 also supports bz2 files containing multiple streams, such as those produced by pbzip2. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam pbzip2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3669.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3669 commit 5f82ea2569db3c2d287344700b212bdf3d9b7ea6 Author: bchambers <bchamb...@google.com> Date: 2017-08-01T21:11:24Z [BEAM-2708] Configure BZIP2 to read all "streams" Without this, CompressionMode.BZIP2 only supports "standard" bz2 files containing a single stream. With this change, BZIP2 also supports bz2 files containing multiple streams, such as those produced by pbzip2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3538: Add a test for Avro write with RVP; fix code
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3538 Add a test for Avro write with RVP; fix code Add a test for AvroIO using RuntimeValueProvider Make AvroIO actually work with RuntimeValueProvider. Previously it caused the code to be non-serializable. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam avro-io Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3538.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3538 commit 08f1adafa2792d09eecc173d05cb2650a4f9eedb Author: bchambers <bchamb...@google.com> Date: 2017-07-10T23:41:15Z Add a test for Avro write with RVP; fix code Add a test for AvroIO using RuntimeValueProvider Make AvroIO actually work with RuntimeValueProvider. Previously it caused the code to be non-serializable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3537: TestPipeline should support errors in expand
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3537 TestPipeline should support errors in expand Writing a test that expects an exception during transform application is currently not possible with TestPipeline in a NeedsRunner or ValidatesRunner test. The exception causes the pipeline to be unrunnable. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam test-pipeline-construction-errors Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3537.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3537 commit 5d820c690540322f2d3c5869061703b8e460069f Author: bchambers <bchamb...@google.com> Date: 2017-07-11T00:29:24Z TestPipeline should support errors in expand Writing a test that expects an exception during transform application is currently not possible with TestPipeline in a NeedsRunner or ValidatesRunner test. The exception causes the pipeline to be unrunnable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3307: Increase visibility of some Metrics methods
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3307 Increase visibility of some Metrics methods Also revise the Javadoc on MetricsContainers. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam visibility Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3307.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3307 commit 39674ca8d0ae1d82bfb5da6a81f26843613d3cd7 Author: bchambers <bchamb...@google.com> Date: 2017-06-06T22:08:46Z Increase visibility of some Metrics methods Also revise the Javadoc on MetricsContainers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3053: Metrics to runners core
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/3053 Metrics to runners core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam metrics-to-runners-core Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3053.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3053 commit 2bb1ef29fce40dff6d26b6d4dd7ae20ce1ec5c45 Author: bchambers <bchamb...@google.com> Date: 2017-05-09T22:45:50Z [BEAM-2244] Move details of Metrics to Runners Core Largeish changes this required were: - splitting the MetricsContainer into an interface in Java Core with an implementation in Runners Core - modifying the various *Cell classes to have a name - cleaning up dependency cross-fire. commit 997440811a4781de2d9489edf50c50ecf9f33536 Author: bchambers <bchamb...@google.com> Date: 2017-05-10T18:43:43Z fixup! Move pieces needed in Dataflow to core-construction-java --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2882: [BEAM-2162] Add logging to long BigQuery jobs
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/2882 [BEAM-2162] Add logging to long BigQuery jobs Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam bq-logging Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2882.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2882 commit ed5ca43d40cb004fc30b17a4fd70664170a2f246 Author: bchambers <bchamb...@google.com> Date: 2017-05-03T23:40:09Z [BEAM-2162] Add logging to long BigQuery jobs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2599: Add javadoc to the Metric interfaces
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/2599 Add javadoc to the Metric interfaces Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- R: @dhalperi @malo-denielou @aviemzur CC: @pabloem You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam document-metrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2599.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2599 commit f4d3baf21593a512cde0560faefff860b2c1ae30 Author: bchambers <bchamb...@google.com> Date: 2017-04-19T21:24:27Z Add javadoc to the Metric interfaces --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2192: DO NOT MERGE: Track the reader in UnboundedReaderEv...
Github user bjchambers closed the pull request at: https://github.com/apache/beam/pull/2192 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2192: Track the reader in UnboundedReaderEvaluator
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/2192 Track the reader in UnboundedReaderEvaluator Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam patch-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2192.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2192 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #1891: Correct Javadoc on accessing windows in DoFn
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/1891 Correct Javadoc on accessing windows in DoFn Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/1891.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1891 commit 00bbb07933fdea17556994ee1686e15a98bec44a Author: Ben Chambers <bjchamb...@users.noreply.github.com> Date: 2017-02-01T23:08:13Z Correct Javadoc on accessing windows in DoFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #1730: [BEAM-370] Remove last mention of PTransform#named
GitHub user bjchambers opened a pull request: https://github.com/apache/beam/pull/1730 [BEAM-370] Remove last mention of PTransform#named Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [*] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [*] Replace `` in the title with the actual Jira issue number, if there is one. - [*] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/bjchambers/beam ptransform-named Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/1730.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1730 commit 175bcbc987cd56370433118f93d3559bdd09b103 Author: bchambers <bchamb...@google.com> Date: 2017-01-03T21:30:45Z [BEAM-370] Remove last mention of PTransform#named --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
