Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Direct Java #1158

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: Core #1159

[Apache Jenkins Server]

See 


--
[...truncated 549 lines...]
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/puppycrawl/tools/checkstyle/6.17/checkstyle-6.17.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/puppycrawl/tools/checkstyle/6.17/checkstyle-6.17.pom
 (70 KB at 1774.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom (632 B 
at 20.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/antlr/antlr4-runtime/4.5.2-1/antlr4-runtime-4.5.2-1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/antlr/antlr4-runtime/4.5.2-1/antlr4-runtime-4.5.2-1.pom
 (3 KB at 77.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/antlr/antlr4-master/4.5.2-1/antlr4-master-4.5.2-1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/antlr/antlr4-master/4.5.2-1/antlr4-master-4.5.2-1.pom
 (4 KB at 133.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.pom
 (14 KB at 517.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.pom
 (11 KB at 390.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/37/commons-parent-37.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/37/commons-parent-37.pom
 (62 KB at 1986.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.3/maven-reporting-impl-2.3.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.3/maven-reporting-impl-2.3.pom
 (5 KB at 174.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/0.6/maven-shared-utils-0.6.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/0.6/maven-shared-utils-0.6.pom
 (5 KB at 176.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.pom
 (2 KB at 47.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.pom
 (2 KB at 55.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/puppycrawl/tools/checkstyle/6.17/checkstyle-6.17.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/antlr/antlr4-runtime/4.5.2-1/antlr4-runtime-4.5.2-1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar
 (52 KB at 1124.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.3/maven-reporting-impl-2.3.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar
 (229 KB at 2854.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/0.6/maven-shared-utils-0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.3/maven-reporting-impl-2.3.jar
 (18 KB at 254.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/antlr/antlr4-runtime/4.5.2-1/antlr4-runtime-4.5.2-1.jar
 (295 KB at 2651.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/0.6/maven-shared-utils-0.6.jar
 (161 KB at 1728.8 KB/sec)
[INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_MavenVerify #1159

[Apache Jenkins Server]

See 

--
[...truncated 2494 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-9/plexus-container-default-1.0-alpha-9.jar
 (191 KB at 664.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar 
(1253 KB at 4231.0 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
 (9 KB at 26.4 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/findbugs/3.0.1/findbugs-3.0.1.jar
 (3711 KB at 11210.4 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.9.4/ant-1.9.4.jar 
(1972 KB at 5508.2 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.4.0/groovy-all-2.4.0.jar
 (6758 KB at 10146.7 KB/sec)
[INFO] Fork Value is true
[INFO] Done FindBugs Analysis
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[INFO] 
[INFO] <<< findbugs-maven-plugin:3.0.1:check (default) < :findbugs @ 
beam-sdks-java-core <<<
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-sdks-java-core ---
[INFO] BugInstance size is 0
[INFO] Error size is 0
[INFO] No errors/warnings found
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ 
beam-sdks-java-core ---
[INFO] Surefire report directory: 

[INFO] Using configured provider 
org.apache.maven.surefire.junitcore.JUnitCoreProvider
[INFO] parallel='none', perCoreThreadCount=true, threadCount=0, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.18.1/surefire-booter-2.18.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.18.1/surefire-booter-2.18.1.pom
 (3 KB at 91.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.18.1/surefire-2.18.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.18.1/surefire-2.18.1.pom
 (17 KB at 585.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/26/maven-parent-26.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/26/maven-parent-26.pom
 (39 KB at 1253.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom (16 KB 
at 556.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.18.1/surefire-api-2.18.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.18.1/surefire-api-2.18.1.pom
 (3 KB at 76.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.18.1/surefire-api-2.18.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.18.1/surefire-api-2.18.1.jar
 (145 KB at 4013.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/2.18.1/surefire-junit47-2.18.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/2.18.1/surefire-junit47-2.18.1.pom
 (5 KB at 163.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.18.1/surefire-providers-2.18.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.18.1/surefire-providers-2.18.1.pom
 (3 KB at 92.0 KB/sec)
[INFO] Downloading: 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Direct Java #1157

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-571) Migrate DatastoreIO from v1beta3 to v1 version

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-571:
--

 Summary: Migrate DatastoreIO from v1beta3 to v1 version
 Key: BEAM-571
 URL: https://issues.apache.org/jira/browse/BEAM-571
 Project: Beam
  Issue Type: Bug
Reporter: Vikas Kedigehalli
Assignee: Vikas Kedigehalli






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-398) JAXBCoder uses incorrect Double-Checked Locking

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429038#comment-15429038
 ] 

ASF GitHub Bot commented on BEAM-398:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/848


> JAXBCoder uses incorrect Double-Checked Locking
> ---
>
> Key: BEAM-398
> URL: https://issues.apache.org/jira/browse/BEAM-398
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> DC_DOUBLECHECK|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L72]:
>  Possible double check of field
> Applies to: 
> [JAXBCoder.getContext|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/JAXBCoder.java#L113].
>  For details on why this is incorrect, see: 
> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-565) Datastore Sink should support deletes

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli resolved BEAM-565.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Datastore Sink should support deletes
> -
>
> Key: BEAM-565
> URL: https://issues.apache.org/jira/browse/BEAM-565
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify #1156

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #845: [BEAM-565] Datastore Sink support for writ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/845


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #845

[dhalperi]

Closes #845


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/78fda451
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/78fda451
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/78fda451

Branch: refs/heads/master
Commit: 78fda4513bd14b57a9a1f8d04f784a6e61001c3e
Parents: 6645dcd 9533065
Author: Dan Halperin 
Authored: Fri Aug 19 15:42:24 2016 -0700
Committer: Dan Halperin 
Committed: Fri Aug 19 15:42:24 2016 -0700

--
 .../beam/sdk/io/gcp/datastore/V1Beta3.java  | 436 +--
 .../beam/sdk/io/gcp/datastore/V1Beta3Test.java  | 237 --
 2 files changed, 508 insertions(+), 165 deletions(-)
--

[jira] [Created] (BEAM-570) Update AvroSource to support more compression types

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-570:
---

 Summary: Update AvroSource to support more compression types
 Key: BEAM-570
 URL: https://issues.apache.org/jira/browse/BEAM-570
 Project: Beam
  Issue Type: Improvement
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Python AvroSource [1] currently only support 'deflate' compression. We should 
update it to support other compression types supported by the Avro library 
(e.g.: snappy, bzip2).

[1] 
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/avroio.py



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-146) WindowFn.AssignContext leaks implementation details about compressed WindowedValue representation

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-146:
-
Issue Type: Improvement  (was: Bug)

> WindowFn.AssignContext leaks implementation details about compressed 
> WindowedValue representation
> -
>
> Key: BEAM-146
> URL: https://issues.apache.org/jira/browse/BEAM-146
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Today, {{WindowFn.AssignContext}} provides simultaneous access to all of the 
> windows that a value has been placed in.
> Providing access to the current window for a value is convenient for, e.g. 
> converting day windows to hour windows for each hour of the assign day. But 
> providing access to all the assigned windows allows spooky action across 
> windows, and is generally not intended to be observable - elements are 
> semantically considered to be "duplicated" into each of the assigned windows.
> This ticket proposes that the {{AssignContext}} should provide only a single 
> window, and that windows should be "exploded" prior to window re-assignment 
> so that elements are only observed within one window at a time. This can be 
> accomplished trivially today via surgical insertion of 
> {{RequiresWindowAccess}} but the {{AssignContext}} should have its API 
> adjusted to be explicit about it, too.
> This will affect only pipelines for which _all_ of the following hold:
>  - assigns to sliding windows (or custom {{WindowFn}} that places each 
> element in multiple windows)
>  - re-assigns to different windows without a {{GroupByKey}} between.
>  - the new window assignment actually does depend on the full set of windows 
> assigned
> I hypothesize the number of such pipelines is zero.
> I expect to address this during the Beam Runner API design.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-142) BigQueryIO: don't unnecessarily initialize an ExecutorService to validate parameters

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-142:
-
Issue Type: Improvement  (was: Bug)

> BigQueryIO: don't unnecessarily initialize an ExecutorService to validate 
> parameters
> 
>
> Key: BEAM-142
> URL: https://issues.apache.org/jira/browse/BEAM-142
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Davor Bonaci
>Assignee: Lucas Amorim
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> See 
> http://stackoverflow.com/questions/36142464/kicking-off-dataflow-jobs-with-app-engine-errors-with-a-securityexception-on-add
> From BigQueryIO$Write$Bound.verifyTableEmpty, we call BigQueryTableInserter, 
> which unnecessarily sets up ExecutorService for this case. As a result, the 
> pipeline cannot be run in the App Engine environment.
> We should fix this by avoid initialization of the ExecutorService.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-463) BoundedHeapCoder should be a StandardCoder and not a CustomCoder

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-463:
-
Fix Version/s: (was: 0.2.0-incubating)

> BoundedHeapCoder should be a StandardCoder and not a CustomCoder
> 
>
> Key: BEAM-463
> URL: https://issues.apache.org/jira/browse/BEAM-463
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>Priority: Minor
>
> The issue is that BoundedHeapCoder does not report component encodings which 
> prevents effective runner inspection of the components.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-286) Reorganize flink runner directories

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-286:
-
Fix Version/s: (was: 0.2.0-incubating)

> Reorganize flink runner directories
> ---
>
> Key: BEAM-286
> URL: https://issues.apache.org/jira/browse/BEAM-286
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> The flink runner Maven module uses two sub-modules: runner and examples. It's 
> the only one which use such layout (compare to spark, dataflow or 
> inprocess/direct runners).
> I will propose a PR to align flink runner module with the other, keeping the 
> examples in a sub-directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-409) Incorrect use of Math.ceil in ApproximateQuantiles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428683#comment-15428683
 ] 

ASF GitHub Bot commented on BEAM-409:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/856

[BEAM-409] Add tests for ApproximateQuantiles calculated buffers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Note: these tests currently fail due to 
[[BEAM-409]](https://issues.apache.org/jira/browse/BEAM-409). #853 fixes them, 
but I wanted to have tests that would verify the fix to go in with it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam approx-quantiles

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #856






> Incorrect use of Math.ceil in ApproximateQuantiles
> --
>
> Key: BEAM-409
> URL: https://issues.apache.org/jira/browse/BEAM-409
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> ICAST_INT_CAST_TO_DOUBLE_PASSED_TO_CEIL|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L274]:
>  Integral value cast to double and then passed to Math.ceil
> Applies to: 
> [ApproximateQuanitles|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java#L349].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #856: [BEAM-409] Add tests for ApproximateQuanti...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/856

[BEAM-409] Add tests for ApproximateQuantiles calculated buffers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Note: these tests currently fail due to 
[[BEAM-409]](https://issues.apache.org/jira/browse/BEAM-409). #853 fixes them, 
but I wanted to have tests that would verify the fix to go in with it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam approx-quantiles

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #856






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Incorporate private IP option for Dataflow runner

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8d31ca0ca -> 6645dcd4a


Incorporate private IP option for Dataflow runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ac7e6f20
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ac7e6f20
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ac7e6f20

Branch: refs/heads/master
Commit: ac7e6f202812cd54ba72dd332c5bf7c2ace4c989
Parents: 8d31ca0
Author: sammcveety 
Authored: Thu Aug 18 18:02:06 2016 -0400
Committer: Dan Halperin 
Committed: Fri Aug 19 11:56:42 2016 -0700

--
 pom.xml   |  2 +-
 .../runners/dataflow/DataflowPipelineTranslator.java  |  7 +++
 .../options/DataflowPipelineWorkerPoolOptions.java| 14 ++
 3 files changed, 22 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ac7e6f20/pom.xml
--
diff --git a/pom.xml b/pom.xml
index f39f94a..58adbe7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -105,7 +105,7 @@
 v2-rev295-1.22.0
 0.0.2
 v2-rev8-1.22.0
-v1b3-rev30-1.22.0
+v1b3-rev36-1.22.0
 0.5.160222
 1.0.0-beta.2
 1.0.0-beta

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ac7e6f20/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 0ca92c8..b8d9445 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -425,6 +425,13 @@ public class DataflowPipelineTranslator {
 workerPool.setMachineType(options.getWorkerMachineType());
   }
 
+  if (options.getUsePublicIps() != null) {
+if (options.getUsePublicIps()) {
+  workerPool.setIpConfiguration("WORKER_IP_PUBLIC");
+} else {
+  workerPool.setIpConfiguration("WORKER_IP_PRIVATE");
+}
+  }
   workerPool.setPackages(packages);
   workerPool.setNumWorkers(options.getNumWorkers());
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ac7e6f20/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
index e3a1a0f..9b7a9ce 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
@@ -29,6 +29,8 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
 
 import java.util.List;
 
+import javax.annotation.Nullable;
+
 /**
  * Options that are used to configure the Dataflow pipeline worker pool.
  */
@@ -260,4 +262,16 @@ public interface DataflowPipelineWorkerPoolOptions extends 
PipelineOptions {
   + "https://cloud.google.com/compute/docs/reference/latest/diskTypes;)
   String getWorkerDiskType();
   void setWorkerDiskType(String value);
+
+  /**
+   * Specifies whether worker pools should be started with public IP addresses.
+   *
+   * WARNING: This feature is experimental.  You must be whitelisted to use 
it.
+   */
+  @Description("Specifies whether worker pools should be started with public 
IP addresses. WARNING:"
++ "This feature is experimental. You must be whitelisted to use it.")
+  @Experimental
+  @JsonIgnore
+  @Nullable Boolean getUsePublicIps();
+  void setUsePublicIps(@Nullable Boolean value);
 }

[jira] [Closed] (BEAM-368) Add getSplitPointsConsumed() to ByteKeyRangeTracker

[Ian Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ian Zhou closed BEAM-368.
-
   Resolution: Fixed
Fix Version/s: 0.2.0-incubating

> Add getSplitPointsConsumed() to ByteKeyRangeTracker
> ---
>
> Key: BEAM-368
> URL: https://issues.apache.org/jira/browse/BEAM-368
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Ian Zhou
>Assignee: Ian Zhou
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> ByteKeyRangeTracker should keep track of the number of split points seen and 
> the number of split points consumed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/4] incubator-beam git commit: Add TestStream to the DirectRunner Package

[lcwik]

Add TestStream to the DirectRunner Package

This closes #817


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8d31ca0c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8d31ca0c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8d31ca0c

Branch: refs/heads/master
Commit: 8d31ca0ca084c37dca5a436fa8b784622b25348d
Parents: bfa3b70 a5ef9a9
Author: Luke Cwik 
Authored: Fri Aug 19 09:06:08 2016 -0700
Committer: Luke Cwik 
Committed: Fri Aug 19 09:06:08 2016 -0700

--
 .../beam/runners/direct/DirectOptions.java  |  45 +--
 .../beam/runners/direct/DirectRunner.java   |  47 ++-
 .../beam/runners/direct/EvaluationContext.java  |  10 +-
 .../FixedThreadPoolExecutorServiceFactory.java  |  45 ---
 .../beam/runners/direct/NanosOffsetClock.java   |  13 -
 .../direct/TestStreamEvaluatorFactory.java  | 204 ++
 .../direct/TransformEvaluatorRegistry.java  |  11 +
 .../direct/WriteWithShardingFactory.java|   2 +-
 .../runners/direct/EvaluationContextTest.java   |   1 +
 .../org/apache/beam/sdk/testing/PAssert.java|  18 +
 .../apache/beam/sdk/testing/PaneExtractors.java |  18 +
 .../org/apache/beam/sdk/testing/TestStream.java | 372 +++
 .../apache/beam/sdk/testing/TestStreamTest.java | 328 
 13 files changed, 1005 insertions(+), 109 deletions(-)
--

[GitHub] incubator-beam pull request #817: Add TestStream to the DirectRunner Package

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/817


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] incubator-beam git commit: Implement TestStream in the DirectRunner

[lcwik]

Implement TestStream in the DirectRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a5ef9a96
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a5ef9a96
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a5ef9a96

Branch: refs/heads/master
Commit: a5ef9a9689147105854f12d3ea054b3887a94e24
Parents: c72d4fc
Author: Thomas Groh 
Authored: Mon Aug 15 19:45:58 2016 -0700
Committer: Luke Cwik 
Committed: Fri Aug 19 09:04:19 2016 -0700

--
 runners/direct-java/pom.xml |   3 -
 .../beam/runners/direct/DirectOptions.java  |  45 +---
 .../beam/runners/direct/DirectRunner.java   |  47 -
 .../beam/runners/direct/EvaluationContext.java  |  10 +-
 .../FixedThreadPoolExecutorServiceFactory.java  |  45 
 .../beam/runners/direct/NanosOffsetClock.java   |  13 --
 .../direct/TestStreamEvaluatorFactory.java  | 204 +++
 .../direct/TransformEvaluatorRegistry.java  |  11 +
 .../direct/WriteWithShardingFactory.java|   2 +-
 .../runners/direct/EvaluationContextTest.java   |   1 +
 .../org/apache/beam/sdk/testing/TestStream.java | 114 +++
 .../apache/beam/sdk/testing/TestStreamTest.java | 159 +++
 12 files changed, 508 insertions(+), 146 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a5ef9a96/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 8b0f91d..e06883f 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -85,9 +85,6 @@
 org.apache.beam:beam-sdks-java-core
 org.apache.beam:beam-runners-java-core
   
-  
-
org/apache/beam/sdk/testing/TestStreamTest.java
-  
   
 
   [

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a5ef9a96/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectOptions.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectOptions.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectOptions.java
index 3901c04..798fda4 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectOptions.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectOptions.java
@@ -17,59 +17,16 @@
  */
 package org.apache.beam.runners.direct;
 
-import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.options.ApplicationNameOptions;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.Description;
-import org.apache.beam.sdk.options.Hidden;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.Validation.Required;
-import org.apache.beam.sdk.transforms.PTransform;
-
-import com.fasterxml.jackson.annotation.JsonIgnore;
-
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
 
 /**
  * Options that can be used to configure the {@link 
org.apache.beam.runners.direct.DirectRunner}.
  */
 public interface DirectOptions extends PipelineOptions, ApplicationNameOptions 
{
-  /**
-   * Gets the {@link ExecutorServiceFactory} to use to create instances of 
{@link ExecutorService}
-   * to execute {@link PTransform PTransforms}.
-   *
-   * Note that {@link ExecutorService ExecutorServices} returned by the 
factory must ensure that
-   * it cannot enter a state in which it will not schedule additional pending 
work unless currently
-   * scheduled work completes, as this may cause the {@link Pipeline} to cease 
processing.
-   *
-   * Defaults to a {@link FixedThreadPoolExecutorServiceFactory}, which 
produces instances of
-   * {@link Executors#newCachedThreadPool()}.
-   */
-  @JsonIgnore
-  @Required
-  @Hidden
-  @Default.InstanceFactory(FixedThreadPoolExecutorServiceFactory.class)
-  ExecutorServiceFactory getExecutorServiceFactory();
-
-  void setExecutorServiceFactory(ExecutorServiceFactory executorService);
-
-  /**
-   * Gets the {@link Clock} used by this pipeline. The clock is used in place 
of accessing the
-   * system time when time values are required by the evaluator.
-   */
-  @Default.InstanceFactory(NanosOffsetClock.Factory.class)
-  @JsonIgnore
-  @Required
-  @Hidden
-  @Description(
-  "The processing time source used by the pipeline. When the current time 
is "
-  + "needed by the evaluator, the result of clock#now() is used.")
-  Clock getClock();
-
-  void setClock(Clock clock);
-
-  @Default.Boolean(false)
+  

[2/4] incubator-beam git commit: Add inEarlyPanesInGlobalWindow as a PAssert Extractor

[lcwik]

Add inEarlyPanesInGlobalWindow as a PAssert Extractor

This is for use in asserting the contents of speculative panes in the
global window.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f15fab8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f15fab8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f15fab8c

Branch: refs/heads/master
Commit: f15fab8ccdb3b40004583e8f7e4e32a0b8ba5121
Parents: bfa3b70
Author: Thomas Groh 
Authored: Thu Aug 11 15:46:10 2016 -0700
Committer: Luke Cwik 
Committed: Fri Aug 19 09:04:18 2016 -0700

--
 .../java/org/apache/beam/sdk/testing/PAssert.java | 18 ++
 .../apache/beam/sdk/testing/PaneExtractors.java   | 18 ++
 2 files changed, 36 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f15fab8c/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index e07ee3d..3f1a741 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -49,6 +49,7 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.Never;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
 import org.apache.beam.sdk.transforms.windowing.Trigger;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.transforms.windowing.Window.ClosingBehavior;
@@ -176,6 +177,13 @@ public class PAssert {
 IterableAssert inCombinedNonLatePanes(BoundedWindow window);
 
 /**
+ * Creates a new {@link IterableAssert} like this one, but with the 
assertion restricted to only
+ * run on panes in the {@link GlobalWindow} that were emitted before the 
{@link GlobalWindow}
+ * closed. These panes have {@link Timing#EARLY}.
+ */
+IterableAssert inEarlyGlobalWindowPanes();
+
+/**
  * Asserts that the iterable in question contains the provided elements.
  *
  * @return the same {@link IterableAssert} builder for further assertions
@@ -381,6 +389,11 @@ public class PAssert {
   return withPane(window, PaneExtractors.nonLatePanes());
 }
 
+@Override
+public IterableAssert inEarlyGlobalWindowPanes() {
+  return withPane(GlobalWindow.INSTANCE, PaneExtractors.earlyPanes());
+}
+
 private PCollectionContentsAssert withPane(
 BoundedWindow window,
 SimpleFunction, Iterable> paneExtractor) 
{
@@ -557,6 +570,11 @@ public class PAssert {
   return withPanes(window, PaneExtractors.nonLatePanes());
 }
 
+@Override
+public IterableAssert inEarlyGlobalWindowPanes() {
+  return withPanes(GlobalWindow.INSTANCE, 
PaneExtractors.earlyPanes());
+}
+
 private PCollectionSingletonIterableAssert withPanes(
 BoundedWindow window,
 SimpleFunction>, 
Iterable> paneExtractor) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f15fab8c/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PaneExtractors.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PaneExtractors.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PaneExtractors.java
index f699bfc..899612b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PaneExtractors.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PaneExtractors.java
@@ -59,6 +59,10 @@ final class PaneExtractors {
 return new ExtractNonLatePanes<>();
   }
 
+  static  SimpleFunction, Iterable> 
earlyPanes() {
+return new ExtractEarlyPanes<>();
+  }
+
   static  SimpleFunction, Iterable> 
allPanes() {
 return new ExtractAllPanes<>();
   }
@@ -137,4 +141,18 @@ final class PaneExtractors {
   return outputs;
 }
   }
+
+  private static class ExtractEarlyPanes
+  extends SimpleFunction, Iterable> {
+@Override
+public Iterable apply(Iterable input) {
+  List outputs = new ArrayList<>();
+  for (WindowedValue value : input) {
+if (value.getPane().getTiming() == PaneInfo.Timing.EARLY) {
+  outputs.add(value.getValue());

[GitHub] incubator-beam pull request #846: Move native TextFileWriter to use GcsIO fo...

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/846


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #849: [BEAM-556] Fix typo in comment

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/849


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Move native TextFileWriter to use GcsIO for writing

[dhalperi]

Move native TextFileWriter to use GcsIO for writing


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7e50b3f1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7e50b3f1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7e50b3f1

Branch: refs/heads/python-sdk
Commit: 7e50b3f1a5da0223576b51570713b7c72750f9f2
Parents: a3de9fb
Author: Charles Chen 
Authored: Thu Aug 18 01:01:06 2016 -0700
Committer: Dan Halperin 
Committed: Fri Aug 19 07:48:23 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 31 +++
 1 file changed, 3 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7e50b3f1/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index a6ce26a..bfa246f 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -25,7 +25,6 @@ from multiprocessing.pool import ThreadPool
 import os
 import re
 import shutil
-import tempfile
 import threading
 import time
 import zlib
@@ -34,8 +33,6 @@ import weakref
 from apache_beam import coders
 from apache_beam.io import iobase
 from apache_beam.io import range_trackers
-from apache_beam.utils import processes
-from apache_beam.utils import retry
 
 
 __all__ = ['TextFileSource', 'TextFileSink']
@@ -43,26 +40,6 @@ __all__ = ['TextFileSource', 'TextFileSink']
 DEFAULT_SHARD_NAME_TEMPLATE = '-S-of-N'
 
 
-# Retrying is needed because there are transient errors that can happen.
-@retry.with_exponential_backoff(num_retries=4, retry_filter=lambda _: True)
-def _gcs_file_copy(from_path, to_path, encoding=''):
-  """Copy a local file to a GCS location with retries for transient errors."""
-  if not encoding:
-command_args = ['gsutil', '-m', '-q', 'cp', from_path, to_path]
-  else:
-encoding = 'Content-Type:' + encoding
-command_args = ['gsutil', '-m', '-q', '-h', encoding, 'cp', from_path,
-to_path]
-  logging.info('Executing command: %s', command_args)
-  popen = processes.Popen(command_args, stdout=processes.PIPE,
-  stderr=processes.PIPE)
-  stdoutdata, stderrdata = popen.communicate()
-  if popen.returncode != 0:
-raise ValueError(
-'Failed to copy GCS file from %s to %s (stdout=%s, stderr=%s).' % (
-from_path, to_path, stdoutdata, stderrdata))
-
-
 # -
 # TextFileSource, TextFileSink.
 
@@ -901,17 +878,15 @@ class TextFileWriter(iobase.NativeSinkWriter):
 
   def __enter__(self):
 if self.sink.is_gcs_sink:
-  # TODO(silviuc): Use the storage library instead of gsutil for writes.
-  self.temp_path = os.path.join(tempfile.mkdtemp(), 'gcsfile')
-  self._file = open(self.temp_path, 'wb')
+  # pylint: disable=wrong-import-order, wrong-import-position
+  from apache_beam.io import gcsio
+  self._file = gcsio.GcsIO().open(self.sink.file_path, 'wb')
 else:
   self._file = open(self.sink.file_path, 'wb')
 return self
 
   def __exit__(self, exception_type, exception_value, traceback):
 self._file.close()
-if hasattr(self, 'temp_path'):
-  _gcs_file_copy(self.temp_path, self.sink.file_path, 'text/plain')
 
   def Write(self, line):
 self._file.write(self.sink.coder.encode(line))

[1/2] incubator-beam git commit: Closes #846

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk a3de9fb6d -> b9a4b6a13


Closes #846


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b9a4b6a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b9a4b6a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b9a4b6a1

Branch: refs/heads/python-sdk
Commit: b9a4b6a13264a9d5b51566a5c66137bf6a10c1b7
Parents: a3de9fb 7e50b3f
Author: Dan Halperin 
Authored: Fri Aug 19 07:48:23 2016 -0700
Committer: Dan Halperin 
Committed: Fri Aug 19 07:48:23 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 31 +++
 1 file changed, 3 insertions(+), 28 deletions(-)
--

[1/2] incubator-beam git commit: Fix typo in comment

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk e39bf3ff4 -> a3de9fb6d


Fix typo in comment


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/52b92029
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/52b92029
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/52b92029

Branch: refs/heads/python-sdk
Commit: 52b92029ec8953f957a4a9cc4476ba41ac9ee5e7
Parents: e39bf3f
Author: Frank Yellin 
Authored: Tue Aug 16 23:03:56 2016 -0700
Committer: Frank Yellin 
Committed: Thu Aug 18 14:46:29 2016 -0700

--
 sdks/python/apache_beam/transforms/ptransform.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/52b92029/sdks/python/apache_beam/transforms/ptransform.py
--
diff --git a/sdks/python/apache_beam/transforms/ptransform.py 
b/sdks/python/apache_beam/transforms/ptransform.py
index 6eb28b0..7fb7caf 100644
--- a/sdks/python/apache_beam/transforms/ptransform.py
+++ b/sdks/python/apache_beam/transforms/ptransform.py
@@ -221,7 +221,7 @@ class PTransform(WithTypeHints):
 Args:
   args: A tuple of position arguments.
   kwargs: A dictionary of keyword arguments.
-  arg_name: The name of the second ergument.
+  arg_name: The name of the second argument.
 
 Returns:
   A (label, value) tuple. The label will be the one passed in or one

[2/2] incubator-beam git commit: Closes #849

[dhalperi]

Closes #849


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a3de9fb6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a3de9fb6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a3de9fb6

Branch: refs/heads/python-sdk
Commit: a3de9fb6d431fcc806e0976c4dda0765db618deb
Parents: e39bf3f 52b9202
Author: Dan Halperin 
Authored: Fri Aug 19 07:45:19 2016 -0700
Committer: Dan Halperin 
Committed: Fri Aug 19 07:45:19 2016 -0700

--
 sdks/python/apache_beam/transforms/ptransform.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-409) Incorrect use of Math.ceil in ApproximateQuantiles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428217#comment-15428217
 ] 

ASF GitHub Bot commented on BEAM-409:
-

GitHub user JasonMWhite opened a pull request:

https://github.com/apache/incubator-beam/pull/853

[BEAM-409] Ensure precision in division of Longs in 
ApproximateQuantilesCombineFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This PR resolves [BEAM-409](https://issues.apache.org/jira/browse/BEAM-409) 
by casting a long to a float before division.

@swegner Would you mind reviewing for me?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JasonMWhite/incubator-beam fix_beam_409

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #853


commit 8a2bc49c1a7ae2467fc2cbbdaad6d10c1411dd16
Author: Jason White 
Date:   2016-08-19T13:46:52Z

cast long to float before division to ensure precision




> Incorrect use of Math.ceil in ApproximateQuantiles
> --
>
> Key: BEAM-409
> URL: https://issues.apache.org/jira/browse/BEAM-409
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> ICAST_INT_CAST_TO_DOUBLE_PASSED_TO_CEIL|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L274]:
>  Integral value cast to double and then passed to Math.ceil
> Applies to: 
> [ApproximateQuanitles|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java#L349].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #853: [BEAM-409] Ensure precision in division of...

[JasonMWhite]

GitHub user JasonMWhite opened a pull request:

https://github.com/apache/incubator-beam/pull/853

[BEAM-409] Ensure precision in division of Longs in 
ApproximateQuantilesCombineFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This PR resolves [BEAM-409](https://issues.apache.org/jira/browse/BEAM-409) 
by casting a long to a float before division.

@swegner Would you mind reviewing for me?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JasonMWhite/incubator-beam fix_beam_409

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #853


commit 8a2bc49c1a7ae2467fc2cbbdaad6d10c1411dd16
Author: Jason White 
Date:   2016-08-19T13:46:52Z

cast long to float before division to ensure precision




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-569) JMS IO should be able to wait for messages (timeout)

[JIRA]

Jean-Baptiste Onofré created BEAM-569:
-

 Summary: JMS IO should be able to wait for messages (timeout)
 Key: BEAM-569
 URL: https://issues.apache.org/jira/browse/BEAM-569
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


Right now, the JMS IO source doesn't wait for new message on the JMS 
destination (it uses {{receiveNoWait()}} method on the consumer).

I think, it's worth to give the possibility for the user to define the behavior.

I propose to introduce {{withTimeout()}} configuration on JMS IO:
- if the user defines {{-1}}, it means infinite timeout, so the reader will use 
{{consumer.receive()}} and the source will wait for new messages on the 
destination
- if the user defines {{0}}, it means the current behavior (and default): the 
reader will use {{consumer.receiveNoWait()}}.
- if the user defines any positive value, it's actually the timeout to wait for 
new messages: the reader will use {{consumer.receive(timeout)}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)