[jira] [Commented] (BEAM-1925) Make DoFn invocation logic of Python SDK more extensible

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984242#comment-15984242
 ] 

ASF GitHub Bot commented on BEAM-1925:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2519


> Make DoFn invocation logic of Python SDK more extensible
> 
>
> Key: BEAM-1925
> URL: https://issues.apache.org/jira/browse/BEAM-1925
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> DoFn invocation logic of Python SDK is currently in DoFnRunner class.
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/common.py#L54
> At initialization of this, we parse a DoFn and create local state. We use 
> this state when invoking DoFn methods process, start_bundle, and 
> finish_bundle. For example, we store a list of  ArgPlaceholder objects within 
> the state of DoFnRunner to facilitate invocation of process method.
> We will need to extend this functionality when adding new features to DoFn 
> class (for example to support Splittable DoFn [1]). So I think it's good to 
> refactor this code to be more extensible. 
> I think a good approach for this is to add DoFnInvoker and DoFnSignature 
> classes similar to Java SDK [2].
> In this approach:
> A DoFnSignature captures the signature of a DoFn including methods and 
> arguments.
> A DoFnInvoker implements a particular way DoFn methods will be executed 
> (initially we'll have simple and per-window invokers [3]).
> A runner uses DoFnRunner to execute methods of a given DoFn. At 
> initialization, DoFnRunner crates a DoFnSignature and a DoFnInvoker for the 
> given DoFn.
> DoFnSignature and DoFnInvoker methods will be used by SplittableDoFn 
> implementation as well. 
> [1] 
> https://docs.google.com/document/d/1h_zprJrOilivK2xfvl4L42vaX4DMYGfH1YDmi-s_ozM/edit#heading=h.e6patunrpiql
> [2]https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
> [3] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/common.py#L200



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2519: [BEAM-1925] Updates DoFn invocation logic to be mor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2519


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Updates DoFn invocation logic to be more extensible.

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master afb96d72a -> 009469972


Updates DoFn invocation logic to be more extensible.

Adds following abstractions.

DoFnSignature: describes the signature of a given DoFn object.
DoFnInvoker: defines a particular way for invoking DoFn methods.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7db375d4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7db375d4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7db375d4

Branch: refs/heads/master
Commit: 7db375d40f9b230d989c087ccfc08844c29afdac
Parents: afb96d7
Author: chamik...@google.com 
Authored: Fri Apr 7 13:41:28 2017 -0700
Committer: chamik...@google.com 
Committed: Tue Apr 25 23:30:01 2017 -0700

--
 sdks/python/apache_beam/runners/common.pxd |  62 +++-
 sdks/python/apache_beam/runners/common.py  | 441 
 2 files changed, 344 insertions(+), 159 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7db375d4/sdks/python/apache_beam/runners/common.pxd
--
diff --git a/sdks/python/apache_beam/runners/common.pxd 
b/sdks/python/apache_beam/runners/common.pxd
index 5952942..f3395c1 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -28,32 +28,62 @@ cdef class Receiver(object):
   cpdef receive(self, WindowedValue windowed_value)
 
 
-cdef class DoFnRunner(Receiver):
+cdef class DoFnMethodWrapper(object):
+  cdef public object args
+  cdef public object defaults
+  cdef public object method_value
 
-  cdef object dofn
-  cdef object dofn_process
-  cdef object window_fn
+
+cdef class DoFnSignature(object):
+  cdef public DoFnMethodWrapper process_method
+  cdef public DoFnMethodWrapper start_bundle_method
+  cdef public DoFnMethodWrapper finish_bundle_method
+  cdef public object do_fn
+
+
+cdef class DoFnInvoker(object):
+  cdef public DoFnSignature signature
+  cdef OutputProcessor output_processor
+
+  cpdef invoke_process(self, WindowedValue windowed_value)
+  cpdef invoke_start_bundle(self)
+  cpdef invoke_finish_bundle(self)
+
+  # TODO(chamikara) define static method create_invoker() here.
+
+
+cdef class SimpleInvoker(DoFnInvoker):
+  cdef object process_method
+
+
+cdef class PerWindowInvoker(DoFnInvoker):
+  cdef list side_inputs
+  cdef DoFnContext context
+  cdef list args_for_process
+  cdef dict kwargs_for_process
+  cdef list placeholders
+  cdef bint has_windowed_inputs
+  cdef object process_method
+
+
+cdef class DoFnRunner(Receiver):
   cdef DoFnContext context
-  cdef object tagged_receivers
   cdef LoggingContext logging_context
   cdef object step_name
-  cdef list args
-  cdef dict kwargs
   cdef ScopedMetricsContainer scoped_metrics_container
   cdef list side_inputs
-  cdef bint has_windowed_inputs
-  cdef list placeholders
-  cdef bint use_simple_invoker
+  cdef DoFnInvoker do_fn_invoker
+
+  cpdef process(self, WindowedValue windowed_value)
 
-  cdef Receiver main_receivers
 
-  cpdef process(self, WindowedValue element)
-  cdef _dofn_invoker(self, WindowedValue element)
-  cdef _dofn_simple_invoker(self, WindowedValue element)
-  cdef _dofn_per_window_invoker(self, WindowedValue element)
+cdef class OutputProcessor(object):
+  cdef object window_fn
+  cdef Receiver main_receivers
+  cdef object tagged_receivers
 
   @cython.locals(windowed_value=WindowedValue)
-  cpdef _process_outputs(self, WindowedValue element, results)
+  cpdef process_outputs(self, WindowedValue element, results)
 
 
 cdef class DoFnContext(object):

http://git-wip-us.apache.org/repos/asf/beam/blob/7db375d4/sdks/python/apache_beam/runners/common.py
--
diff --git a/sdks/python/apache_beam/runners/common.py 
b/sdks/python/apache_beam/runners/common.py
index 64d6d00..08071a6 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -51,6 +51,262 @@ class Receiver(object):
 raise NotImplementedError
 
 
+class DoFnMethodWrapper(object):
+  """Represents a method of a DoFn object."""
+
+  def __init__(self, do_fn, method_name):
+"""
+Initiates a ``DoFnMethodWrapper``.
+
+Args:
+  do_fn: A DoFn object that contains the method.
+  method_name: name of the method as a string.
+"""
+
+args, _, _, defaults = do_fn.get_function_arguments(method_name)
+defaults = defaults if defaults else []
+method_value = getattr(do_fn, method_name)
+self.method_value = method_value
+self.args = args
+self.defaults = defaults
+
+
+class DoFnSignature(object):
+  """Represents the signature of a given ``DoFn`` object.
+
+  Signature of a ``DoFn`` provides a view of the properties of a given 
``DoFn``.
+  Amo

[2/2] beam git commit: This closes #2519

[chamikara]

This closes #2519


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/00946997
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/00946997
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/00946997

Branch: refs/heads/master
Commit: 00946997229bf8a764ce0a135b2112bacd7a3993
Parents: afb96d7 7db375d
Author: chamik...@google.com 
Authored: Tue Apr 25 23:31:35 2017 -0700
Committer: chamik...@google.com 
Committed: Tue Apr 25 23:31:35 2017 -0700

--
 sdks/python/apache_beam/runners/common.pxd |  62 +++-
 sdks/python/apache_beam/runners/common.py  | 441 
 2 files changed, 344 insertions(+), 159 deletions(-)
--

[jira] [Commented] (BEAM-2072) MicrobatchSource.reader stops reading after reaching maxNumRecords for the first time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984241#comment-15984241
 ] 

ASF GitHub Bot commented on BEAM-2072:
--

GitHub user staslev opened a pull request:

https://github.com/apache/beam/pull/2698

[BEAM-2072] Fixed MicrobatchSource.reader stops reading unexpectedly.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/staslev/beam 
BEAM-2072-MicrobatchSource-reader-stops-reading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2698.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2698


commit 5726e38b86cb3b7ef343463b3b9697bc583dcd80
Author: Stas Levin 
Date:   2017-04-26T06:28:08Z

[BEAM-2072] Fixed MicrobatchSource.reader stops reading after reaching 
maxNumRecords for the first time.




> MicrobatchSource.reader stops reading after reaching maxNumRecords for the 
> first time
> -
>
> Key: BEAM-2072
> URL: https://issues.apache.org/jira/browse/BEAM-2072
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{MicrobatchSource.Reader}} stops reading further data after the first batch 
> in which it has reached {{maxNumRecords}}. Subsequent read request will check 
> the {{recordsRead}} and conclude that no data can be read.
> The root cause being that {{recordsRead}} is maintained throughout the 
> reader's lifecycle, and can span multiple (spark) batches. 
> We should reset {{recordsRead}} every time the reader starts reading a fresh 
> batch.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2698: [BEAM-2072] Fixed MicrobatchSource.reader stops rea...

[staslev]

GitHub user staslev opened a pull request:

https://github.com/apache/beam/pull/2698

[BEAM-2072] Fixed MicrobatchSource.reader stops reading unexpectedly.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/staslev/beam 
BEAM-2072-MicrobatchSource-reader-stops-reading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2698.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2698


commit 5726e38b86cb3b7ef343463b3b9697bc583dcd80
Author: Stas Levin 
Date:   2017-04-26T06:28:08Z

[BEAM-2072] Fixed MicrobatchSource.reader stops reading after reaching 
maxNumRecords for the first time.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-59) Switch from IOChannelFactory to FileSystems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984163#comment-15984163
 ] 

ASF GitHub Bot commented on BEAM-59:


GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2697

[BEAM-59] Some low hanging fruit in 
FileSystems/LocalFileSystem/LocalResource

Pulling out from ongoing work to convert `FileBasedSink`.

R: @ssisk or @peihe or @reuvenlax as available

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam low-hanging-fruit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2697.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2697


commit b7b38379f34b50496d962b8ef6badb4181073a53
Author: Dan Halperin 
Date:   2017-04-26T03:48:52Z

LocalResourceId: make toString end in '/' for directories

commit aea01295a5b71b159870126c17d0a8878d8d16bc
Author: Dan Halperin 
Date:   2017-04-26T03:48:23Z

FileSystems: make tolerant of and more efficient for empty lists

commit 4e6c31d99ba87682431e6ab658aae8a85e968877
Author: Dan Halperin 
Date:   2017-04-26T03:48:02Z

LocalFileSystem: create parent directories if needed

And improve testing to confirm.




> Switch from IOChannelFactory to FileSystems
> ---
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: First stable release
>
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2697: [BEAM-59] Some low hanging fruit in FileSystems/Loc...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2697

[BEAM-59] Some low hanging fruit in 
FileSystems/LocalFileSystem/LocalResource

Pulling out from ongoing work to convert `FileBasedSink`.

R: @ssisk or @peihe or @reuvenlax as available

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam low-hanging-fruit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2697.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2697


commit b7b38379f34b50496d962b8ef6badb4181073a53
Author: Dan Halperin 
Date:   2017-04-26T03:48:52Z

LocalResourceId: make toString end in '/' for directories

commit aea01295a5b71b159870126c17d0a8878d8d16bc
Author: Dan Halperin 
Date:   2017-04-26T03:48:23Z

FileSystems: make tolerant of and more efficient for empty lists

commit 4e6c31d99ba87682431e6ab658aae8a85e968877
Author: Dan Halperin 
Date:   2017-04-26T03:48:02Z

LocalFileSystem: create parent directories if needed

And improve testing to confirm.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1283) DoFn finishBundle should be required to specify the window for output

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984139#comment-15984139
 ] 

Kenneth Knowles commented on BEAM-1283:
---

I believe the best course is to require value, timestamp, and window to all be 
specified in such cases.

If someone is going to output in finishBundle, they are responsible for 
managing windows, so they must already have the data stored to explicitly 
output to the window and timestamp, or they probably wrote an incorrect DoFn.

> DoFn finishBundle should be required to specify the window for output
> -
>
> Key: BEAM-1283
> URL: https://issues.apache.org/jira/browse/BEAM-1283
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The spec is here in Javadoc: 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to 
> use the {{WindowFn}} of the input {{PCollection}} to determine what windows 
> the element should be in, throwing an exception if the {{WindowFn}} attempts 
> to access any information about the input element. The output element will 
> have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically 
> wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is 
> allowed to see the element type. The spec just avoids this by limiting which 
> {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The 
> spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged 
> then you'll just have a bogus proto window regardless of explicit timestamp 
> or not.
> The use cases for these methods are best addressed by state plus window 
> expiry callback, so we should revisit this spec and probably just wipe it.
> There are some rare case where you might need to output from {{FinishBundle}} 
> in a way that is not _actually_ sensitive to bundling (perhaps modulo some 
> downstream notion of equivalence) in which case you had better know what 
> window you are outputting to. Often it should be the global window.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #2932

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1989) clean SyntaxWarning

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984118#comment-15984118
 ] 

Sourabh Bajaj commented on BEAM-1989:
-

This can be closed now.

> clean SyntaxWarning
> ---
>
> Key: BEAM-1989
> URL: https://issues.apache.org/jira/browse/BEAM-1989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> apache_beam/io/gcp/bigquery.py:326: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table=None, dataset=None, project=None, query=None,
> apache_beam/io/gcp/bigquery.py:431: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table, dataset=None, project=None, schema=None,
> cc: [~sb2nov][~chamikara]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1749) Upgrade pep8 to pycodestyle

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984116#comment-15984116
 ] 

Sourabh Bajaj commented on BEAM-1749:
-

This can be closed now.

> Upgrade pep8 to pycodestyle
> ---
>
> Key: BEAM-1749
> URL: https://issues.apache.org/jira/browse/BEAM-1749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: newbie, starter
>
> pep8 was deprecated and replaced with pycodestyle
> We should upgrade our linter to this module, and while doing that re-evaluate 
> our linter strategy and see if we can enable more rules. This is important 
> for keeping the code healthy as the community grows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2690: Datastore: fix use of deprecated function

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2690


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2690

[dhalperi]

This closes #2690


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/afb96d72
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/afb96d72
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/afb96d72

Branch: refs/heads/master
Commit: afb96d72a8d0ce47069d33ea3318eb419aba2aa9
Parents: 1675f03 6abc21e
Author: Dan Halperin 
Authored: Tue Apr 25 19:48:01 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 25 19:48:01 2017 -0700

--
 .../org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java| 4 ++--
 .../java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java| 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
--

[1/2] beam git commit: Datastore: fix use of deprecated function

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 1675f03f7 -> afb96d72a


Datastore: fix use of deprecated function

getMutableProperties().put() has been deprecated in favor of putProperties()


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6abc21e4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6abc21e4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6abc21e4

Branch: refs/heads/master
Commit: 6abc21e493183deb8d446b53fee91c1b3d6e38f7
Parents: 1675f03
Author: Dan Halperin 
Authored: Tue Apr 25 17:34:07 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 25 19:47:58 2017 -0700

--
 .../org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java| 4 ++--
 .../java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java| 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6abc21e4/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
index c09a1fa..ba8ac84 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
@@ -943,7 +943,7 @@ public class DatastoreV1Test {
 RunQueryResponse.Builder statKindResponse = RunQueryResponse.newBuilder();
 Entity.Builder entity = Entity.newBuilder();
 entity.setKey(makeKey("dummyKind", "dummyId"));
-entity.getMutableProperties().put("entity_bytes", 
makeValue(entitySizeInBytes).build());
+entity.putProperties("entity_bytes", makeValue(entitySizeInBytes).build());
 EntityResult.Builder entityResult = EntityResult.newBuilder();
 entityResult.setEntity(entity);
 QueryResultBatch.Builder batch = QueryResultBatch.newBuilder();
@@ -957,7 +957,7 @@ public class DatastoreV1Test {
 RunQueryResponse.Builder timestampResponse = RunQueryResponse.newBuilder();
 Entity.Builder entity = Entity.newBuilder();
 entity.setKey(makeKey("dummyKind", "dummyId"));
-entity.getMutableProperties().put("timestamp", makeValue(new 
Date(timestamp * 1000)).build());
+entity.putProperties("timestamp", makeValue(new Date(timestamp * 
1000)).build());
 EntityResult.Builder entityResult = EntityResult.newBuilder();
 entityResult.setEntity(entity);
 QueryResultBatch.Builder batch = QueryResultBatch.newBuilder();

http://git-wip-us.apache.org/repos/asf/beam/blob/6abc21e4/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java
index 526f035..25558c0 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java
@@ -104,7 +104,7 @@ class V1TestUtil {
 }
 
 entityBuilder.setKey(keyBuilder.build());
-entityBuilder.getMutableProperties().put("value", 
makeValue(value).build());
+entityBuilder.putProperties("value", makeValue(value).build());
 return entityBuilder.build();
   }
 

[GitHub] beam pull request #2696: Do not depend on message id in DataflowRunner

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2696

Do not depend on message id in DataflowRunner

This field is deprecated and causing messages to be repeated. Use of hash 
of messages instead of id.

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam numop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2696.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2696


commit c99b076e945b191d9f13e0acdd7e5496ee96f4a5
Author: Ahmet Altay 
Date:   2017-04-26T01:42:03Z

Do not depend on message id in DataflowRunner

This field is deprecated and causing messages to be repeated. Hash
message to avoid printing duplicate messages.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1749) Upgrade pep8 to pycodestyle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984005#comment-15984005
 ] 

ASF GitHub Bot commented on BEAM-1749:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2686


> Upgrade pep8 to pycodestyle
> ---
>
> Key: BEAM-1749
> URL: https://issues.apache.org/jira/browse/BEAM-1749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: newbie, starter
>
> pep8 was deprecated and replaced with pycodestyle
> We should upgrade our linter to this module, and while doing that re-evaluate 
> our linter strategy and see if we can enable more rules. This is important 
> for keeping the code healthy as the community grows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2686: [BEAM-1749] Upgrade to pycodestyle from pep8

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2686


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2686

[altay]

This closes #2686


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1675f03f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1675f03f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1675f03f

Branch: refs/heads/master
Commit: 1675f03f7fb70bcdbbc36dbe094bd7ece4b82fac
Parents: 9213f34 6571bae
Author: Ahmet Altay 
Authored: Tue Apr 25 19:02:50 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Apr 25 19:02:50 2017 -0700

--
 sdks/python/apache_beam/coders/coders_test.py   | 1 +
 sdks/python/apache_beam/coders/fast_coders_test.py  | 1 +
 sdks/python/apache_beam/coders/slow_coders_test.py  | 1 +
 sdks/python/apache_beam/examples/complete/autocomplete_test.py  | 1 +
 .../apache_beam/examples/complete/game/hourly_team_score.py | 1 +
 .../examples/complete/game/hourly_team_score_test.py| 1 +
 sdks/python/apache_beam/examples/complete/game/user_score.py| 1 +
 .../apache_beam/examples/complete/game/user_score_test.py   | 1 +
 .../examples/complete/juliaset/juliaset/juliaset_test.py| 1 +
 .../apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py | 1 +
 sdks/python/apache_beam/examples/wordcount.py   | 1 +
 sdks/python/apache_beam/examples/wordcount_debugging_test.py| 1 +
 sdks/python/apache_beam/internal/module_test.py | 1 +
 sdks/python/apache_beam/internal/pickler.py | 3 +++
 sdks/python/apache_beam/internal/pickler_test.py| 1 +
 sdks/python/apache_beam/io/concat_source_test.py| 1 +
 sdks/python/apache_beam/io/filebasedsource_test.py  | 1 +
 sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio_test.py | 1 +
 sdks/python/apache_beam/io/gcp/pubsub_test.py   | 1 +
 sdks/python/apache_beam/io/source_test_utils.py | 1 +
 sdks/python/apache_beam/io/source_test_utils_test.py| 1 +
 sdks/python/apache_beam/io/tfrecordio.py| 2 ++
 sdks/python/apache_beam/io/tfrecordio_test.py   | 1 +
 .../apache_beam/runners/dataflow/dataflow_metrics_test.py   | 1 +
 sdks/python/apache_beam/runners/dataflow/native_io/iobase.py| 1 +
 sdks/python/apache_beam/transforms/window.py| 4 
 sdks/python/apache_beam/transforms/window_test.py   | 2 ++
 sdks/python/apache_beam/typehints/decorators.py | 1 +
 sdks/python/apache_beam/typehints/opcodes.py| 5 -
 sdks/python/apache_beam/typehints/typehints_test.py | 1 +
 sdks/python/apache_beam/utils/annotations.py| 1 +
 .../python/apache_beam/utils/pipeline_options_validator_test.py | 1 +
 sdks/python/apache_beam/utils/test_stream_test.py   | 1 +
 sdks/python/run_pylint.sh   | 4 ++--
 sdks/python/tox.ini | 4 ++--
 35 files changed, 47 insertions(+), 5 deletions(-)
--

[1/2] beam git commit: [BEAM-1749] Upgrade to pycodestyle

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 9213f34d3 -> 1675f03f7


[BEAM-1749] Upgrade to pycodestyle


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6571bae3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6571bae3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6571bae3

Branch: refs/heads/master
Commit: 6571bae30acff45575f7397b92aef8c3d2d228fe
Parents: 9213f34
Author: Sourabh Bajaj 
Authored: Tue Apr 25 16:00:18 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Apr 25 19:02:32 2017 -0700

--
 sdks/python/apache_beam/coders/coders_test.py   | 1 +
 sdks/python/apache_beam/coders/fast_coders_test.py  | 1 +
 sdks/python/apache_beam/coders/slow_coders_test.py  | 1 +
 sdks/python/apache_beam/examples/complete/autocomplete_test.py  | 1 +
 .../apache_beam/examples/complete/game/hourly_team_score.py | 1 +
 .../examples/complete/game/hourly_team_score_test.py| 1 +
 sdks/python/apache_beam/examples/complete/game/user_score.py| 1 +
 .../apache_beam/examples/complete/game/user_score_test.py   | 1 +
 .../examples/complete/juliaset/juliaset/juliaset_test.py| 1 +
 .../apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py | 1 +
 sdks/python/apache_beam/examples/wordcount.py   | 1 +
 sdks/python/apache_beam/examples/wordcount_debugging_test.py| 1 +
 sdks/python/apache_beam/internal/module_test.py | 1 +
 sdks/python/apache_beam/internal/pickler.py | 3 +++
 sdks/python/apache_beam/internal/pickler_test.py| 1 +
 sdks/python/apache_beam/io/concat_source_test.py| 1 +
 sdks/python/apache_beam/io/filebasedsource_test.py  | 1 +
 sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio_test.py | 1 +
 sdks/python/apache_beam/io/gcp/pubsub_test.py   | 1 +
 sdks/python/apache_beam/io/source_test_utils.py | 1 +
 sdks/python/apache_beam/io/source_test_utils_test.py| 1 +
 sdks/python/apache_beam/io/tfrecordio.py| 2 ++
 sdks/python/apache_beam/io/tfrecordio_test.py   | 1 +
 .../apache_beam/runners/dataflow/dataflow_metrics_test.py   | 1 +
 sdks/python/apache_beam/runners/dataflow/native_io/iobase.py| 1 +
 sdks/python/apache_beam/transforms/window.py| 4 
 sdks/python/apache_beam/transforms/window_test.py   | 2 ++
 sdks/python/apache_beam/typehints/decorators.py | 1 +
 sdks/python/apache_beam/typehints/opcodes.py| 5 -
 sdks/python/apache_beam/typehints/typehints_test.py | 1 +
 sdks/python/apache_beam/utils/annotations.py| 1 +
 .../python/apache_beam/utils/pipeline_options_validator_test.py | 1 +
 sdks/python/apache_beam/utils/test_stream_test.py   | 1 +
 sdks/python/run_pylint.sh   | 4 ++--
 sdks/python/tox.ini | 4 ++--
 35 files changed, 47 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6571bae3/sdks/python/apache_beam/coders/coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/coders_test.py 
b/sdks/python/apache_beam/coders/coders_test.py
index ba505db..575503b 100644
--- a/sdks/python/apache_beam/coders/coders_test.py
+++ b/sdks/python/apache_beam/coders/coders_test.py
@@ -110,6 +110,7 @@ class FallbackCoderTest(unittest.TestCase):
 self.assertEqual(coder, coders.FastPrimitivesCoder())
 self.assertEqual(DummyClass(), coder.decode(coder.encode(DummyClass(
 
+
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)
   unittest.main()

http://git-wip-us.apache.org/repos/asf/beam/blob/6571bae3/sdks/python/apache_beam/coders/fast_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/fast_coders_test.py 
b/sdks/python/apache_beam/coders/fast_coders_test.py
index 55cf16c..a13334a 100644
--- a/sdks/python/apache_beam/coders/fast_coders_test.py
+++ b/sdks/python/apache_beam/coders/fast_coders_test.py
@@ -32,6 +32,7 @@ class FastCoders(unittest.TestCase):
 # pylint: disable=unused-variable
 import apache_beam.coders.stream
 
+
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)
   unittest.main()

http://git-wip-us.apache.org/repos/asf/beam/blob/6571bae3/sdks/python/apache_beam/coders/slow_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/slow_coders_test.py 
b/sdks/python/apache_beam/coders/slow_coders_test.py
index 1f22e3d..97aa3

[jira] [Commented] (BEAM-1989) clean SyntaxWarning

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983989#comment-15983989
 ] 

ASF GitHub Bot commented on BEAM-1989:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2687


> clean SyntaxWarning
> ---
>
> Key: BEAM-1989
> URL: https://issues.apache.org/jira/browse/BEAM-1989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> apache_beam/io/gcp/bigquery.py:326: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table=None, dataset=None, project=None, query=None,
> apache_beam/io/gcp/bigquery.py:431: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table, dataset=None, project=None, schema=None,
> cc: [~sb2nov][~chamikara]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2687: [BEAM-1989] Fix the syntax warning from import star

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2687


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1989] Fix the syntax warning from import star

[altay]

Repository: beam
Updated Branches:
  refs/heads/master b8c568f29 -> 9213f34d3


[BEAM-1989] Fix the syntax warning from import star


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/13177b81
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/13177b81
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/13177b81

Branch: refs/heads/master
Commit: 13177b81f8c27e1ace270c5b1a4c2f5e46ba47f7
Parents: b8c568f
Author: Sourabh Bajaj 
Authored: Tue Apr 25 16:04:12 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Apr 25 18:47:27 2017 -0700

--
 sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/13177b81/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
index a0ccbbb..c606133 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
@@ -96,7 +96,7 @@ class ReadFromDatastore(PTransform):
 # Import here to avoid adding the dependency for local running scenarios.
 try:
   # pylint: disable=wrong-import-order, wrong-import-position
-  from apitools.base.py import *
+  from apitools.base import py  # pylint: disable=unused-variable
 except ImportError:
   raise ImportError(
   'Google Cloud IO not available, '
@@ -382,7 +382,7 @@ class WriteToDatastore(_Mutate):
 # Import here to avoid adding the dependency for local running scenarios.
 try:
   # pylint: disable=wrong-import-order, wrong-import-position
-  from apitools.base.py import *
+  from apitools.base import py  # pylint: disable=unused-variable
 except ImportError:
   raise ImportError(
   'Google Cloud IO not available, '

[2/2] beam git commit: This closes #2687

[altay]

This closes #2687


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9213f34d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9213f34d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9213f34d

Branch: refs/heads/master
Commit: 9213f34d3fec8b7c31ef729dbd838c5b55ec9363
Parents: b8c568f 13177b8
Author: Ahmet Altay 
Authored: Tue Apr 25 18:47:30 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Apr 25 18:47:30 2017 -0700

--
 sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-2078) add BeamSQL feature branch in site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983971#comment-15983971
 ] 

ASF GitHub Bot commented on BEAM-2078:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/224


> add BeamSQL feature branch in site
> --
>
> Key: BEAM-2078
> URL: https://issues.apache.org/jira/browse/BEAM-2078
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> Add {{dsl_sql}} feature branch to page 
> 'https://beam.apache.org/contribute/work-in-progress/', to track the status.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #224: [BEAM-2078] add BeamSQL feature branch in site

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/224


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam-site git commit: add Beam SQL DSL to page 'work-in-progress'

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 973853241 -> 71dc72b80


add Beam SQL DSL to page 'work-in-progress'


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/676f6753
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/676f6753
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/676f6753

Branch: refs/heads/asf-site
Commit: 676f67533790a25f8c5734b28614bd8b73e51c50
Parents: 9738532
Author: mingmxu 
Authored: Tue Apr 25 14:25:00 2017 -0700
Committer: mingmxu 
Committed: Tue Apr 25 14:25:00 2017 -0700

--
 content/contribute/work-in-progress/index.html | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/676f6753/content/contribute/work-in-progress/index.html
--
diff --git a/content/contribute/work-in-progress/index.html 
b/content/contribute/work-in-progress/index.html
index 90eff24..f2eeab1 100644
--- a/content/contribute/work-in-progress/index.html
+++ b/content/contribute/work-in-progress/index.html
@@ -192,6 +192,12 @@
   -
   https://lists.apache.org/thread.html/e38ac4e4914a6cb1b865b1f32a6ca06c2be28ea4aa0f6b18393de66f@%3Cdev.beam.apache.org%3E";>thread
 
+
+  Beam SQL DSL
+  https://github.com/apache/beam/tree/DSL_SQL";>DSL_SQL
+  https://issues.apache.org/jira/browse/BEAM/component/12332480";>dsl-sql
+  https://issues.apache.org/jira/browse/BEAM-301";>BEAM-301
+
   
 
 

[3/3] beam-site git commit: This closes #224

[davor]

This closes #224


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/71dc72b8
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/71dc72b8
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/71dc72b8

Branch: refs/heads/asf-site
Commit: 71dc72b80f8914db258a686ba08afd118ea79530
Parents: 9738532 daa8738
Author: Davor Bonaci 
Authored: Tue Apr 25 18:29:14 2017 -0700
Committer: Davor Bonaci 
Committed: Tue Apr 25 18:29:14 2017 -0700

--
 content/documentation/sdks/python-custom-io/index.html | 2 ++
 1 file changed, 2 insertions(+)
--

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/daa87382
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/daa87382
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/daa87382

Branch: refs/heads/asf-site
Commit: daa873826c26aeaa839dd67c46d9ba6d792886c2
Parents: 676f675
Author: Davor Bonaci 
Authored: Tue Apr 25 18:29:14 2017 -0700
Committer: Davor Bonaci 
Committed: Tue Apr 25 18:29:14 2017 -0700

--
 content/contribute/work-in-progress/index.html | 6 --
 content/documentation/sdks/python-custom-io/index.html | 2 ++
 2 files changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/daa87382/content/contribute/work-in-progress/index.html
--
diff --git a/content/contribute/work-in-progress/index.html 
b/content/contribute/work-in-progress/index.html
index f2eeab1..90eff24 100644
--- a/content/contribute/work-in-progress/index.html
+++ b/content/contribute/work-in-progress/index.html
@@ -192,12 +192,6 @@
   -
   https://lists.apache.org/thread.html/e38ac4e4914a6cb1b865b1f32a6ca06c2be28ea4aa0f6b18393de66f@%3Cdev.beam.apache.org%3E";>thread
 
-
-  Beam SQL DSL
-  https://github.com/apache/beam/tree/DSL_SQL";>DSL_SQL
-  https://issues.apache.org/jira/browse/BEAM/component/12332480";>dsl-sql
-  https://issues.apache.org/jira/browse/BEAM-301";>BEAM-301
-
   
 
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/daa87382/content/documentation/sdks/python-custom-io/index.html
--
diff --git a/content/documentation/sdks/python-custom-io/index.html 
b/content/documentation/sdks/python-custom-io/index.html
index d078e33..629ef0f 100644
--- a/content/documentation/sdks/python-custom-io/index.html
+++ b/content/documentation/sdks/python-custom-io/index.html
@@ -342,6 +342,7 @@
 class 
CountingSource(iobase.BoundedSource):
 
   def __init__(self, count):
+self.records_read = Metrics.counter(self.__class__, 'recordsRead')
 self._count = count
 
   def estimate_size(self):
@@ -359,6 +360,7 @@
 for i in range(self._count):
   if not range_tracker.try_claim(i):
 return
+  self.records_read.inc()
   yield i
 
   def split(self, desired_bundle_size, start_position=None,

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983967#comment-15983967
 ] 

Stephen Sisk commented on BEAM-2031:


[~lcwik] also mentioned that it's definitely possible to serialize/deserialize 
Configurations from PipelineOptions, that roughly looks like:
* Use a mixin to override Configuration's jackson annotations
* Use that to add @JsonSerialize(using = Serializer.class) 
@JsonDeserialize(using = Deserializer.class) to override serialize/deserialize

To support this, we would need to add a serviceloader (probably in 
ProxyInvocationHandler.java) so that users (including the HadoopFileSystem 
code) can register arbitrary jackson modules.

For now, I'd like to use a simple Map as the type in 
PipelineOptions (and not use the above method) so we can get it working this 
week, but it should be very do-able to implement the "Serialize Configuration" 
option in the near future and have minimal change from a user's perspective 
(the command line would be the same, there'd be a programmatic change since 
they'd no longer be passing a string, but a configuration)

> Hadoop FileSystem needs to receive Hadoop Configuration
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #2931

[Apache Jenkins Server]

See 


Changes:

[tgroh] Make Most StandardCoders CustomCoders

[chamikara] [BEAM-1988] Add FileSystems Interface for accessing underlying FS

--
[...truncated 5.70 MB...]
[INFO] 2017-04-26T01:19:39.930Z: (634e6dfeb8f444e5): Fusing consumer 
PAssert$287/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into 
PAssert$287/GroupGlobally/GatherAllOutputs/GroupByKey/Read
[INFO] 2017-04-26T01:19:39.932Z: (634e6dfeb8f4445b): Fusing consumer 
PAssert$287/GroupGlobally/WindowIntoDummy/Window.Assign into 
PAssert$287/GroupGlobally/Create.Values/Read(CreateSource)
[INFO] 2017-04-26T01:19:40.012Z: (634e6dfeb8f44291): Adding StepResource setup 
and teardown to workflow graph.
[INFO] 2017-04-26T01:19:40.064Z: (939064c2efda911e): Executing operation 
PAssert$287/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2017-04-26T01:19:40.270Z: (16bfbc93a3545d9): Starting 1 workers...
[INFO] 2017-04-26T01:19:40.315Z: (939064c2efda9751): Executing operation 
Create.Values/Read(CreateSource)+PAssert$287/GroupGlobally/Window.Into()/Window.Assign+PAssert$287/GroupGlobally/GatherAllOutputs/ParDo(ReifyTimestampsAndWindows)+PAssert$287/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$287/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$287/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$287/GroupGlobally/GatherAllOutputs/GroupByKey/Write
[INFO] Staging files complete: 0 files cached, 110 files newly uploaded
[INFO] Staging files complete: 0 files cached, 110 files newly uploaded
[INFO] To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-04-25_18_19_42-18049386336424926276
[INFO] To cancel the job using the 'gcloud' tool, run:
> gcloud beta dataflow jobs --project=apache-beam-testing cancel 
> 2017-04-25_18_19_42-18049386336424926276
[INFO] Running Dataflow job 2017-04-25_18_19_42-18049386336424926276 with 1 
expected assertions.
[INFO] To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-04-25_18_19_43-2645401558949173805
[INFO] To cancel the job using the 'gcloud' tool, run:
> gcloud beta dataflow jobs --project=apache-beam-testing cancel 
> 2017-04-25_18_19_43-2645401558949173805
[INFO] Running Dataflow job 2017-04-25_18_19_43-2645401558949173805 with 1 
expected assertions.
[INFO] 2017-04-26T01:19:42.750Z: (fa7c4d3e44659bf0): Autoscaling is enabled for 
job 2017-04-25_18_19_42-18049386336424926276. The number of workers will be 
between 1 and 15.
[INFO] 2017-04-26T01:19:42.750Z: (fa7c4d3e446593c6): Autoscaling was 
automatically enabled for job 2017-04-25_18_19_42-18049386336424926276.
[INFO] 2017-04-26T01:19:43.826Z: (1dafcdd59c6b21f9): Checking required Cloud 
APIs are enabled.
[INFO] 2017-04-26T01:19:43.110Z: (24b65ab3b16e15d9): Autoscaling is enabled for 
job 2017-04-25_18_19_43-2645401558949173805. The number of workers will be 
between 1 and 15.
[INFO] 2017-04-26T01:19:43.111Z: (24b65ab3b16e13af): Autoscaling was 
automatically enabled for job 2017-04-25_18_19_43-2645401558949173805.
[INFO] 2017-04-26T01:19:44.051Z: (8858aa8dbd0a0356): Checking required Cloud 
APIs are enabled.
[INFO] 2017-04-26T01:19:42.618Z: (cd21556f4c54b8f1): Cleaning up.
[INFO] 2017-04-26T01:19:42.622Z: (cd21556f4c54b40d): Stopping worker pool...
[INFO] 2017-04-26T01:19:47.484Z: (1dafcdd59c6b27ab): Expanding GroupByKey 
operations into optimizable parts.
[INFO] 2017-04-26T01:19:47.488Z: (1dafcdd59c6b2579): Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
[INFO] 2017-04-26T01:19:47.506Z: (1dafcdd59c6b2a7f): Fusing adjacent ParDo, 
Read, Write, and Flatten operations
[INFO] 2017-04-26T01:19:47.509Z: (1dafcdd59c6b284d): Elided trivial flatten 
[INFO] 2017-04-26T01:19:47.511Z: (1dafcdd59c6b261b): Elided trivial flatten 
[INFO] 2017-04-26T01:19:47.513Z: (1dafcdd59c6b23e9): Elided trivial flatten 
[INFO] 2017-04-26T01:19:47.519Z: (1dafcdd59c6b2d53): Unzipping flatten s14 for 
input s12.12
[INFO] 2017-04-26T01:19:47.521Z: (1dafcdd59c6b2b21): Fusing unzipped copy of 
PAssert$289/GroupGlobally/GroupDummyAndContents/Reify, through flatten , into 
producer PAssert$289/GroupGlobally/WindowIntoDummy/Window.Assign
[INFO] 2017-04-26T01:19:47.524Z: (1dafcdd59c6b28ef): Fusing consumer 
PAssert$289/GroupGlobally/GroupDummyAndContents/Reify into 
PAssert$289/GroupGlobally/KeyForDummy/AddKeys/Map
[INFO] 2017-04-26T01:19:47.578Z: (1dafcdd59c6b2671): Fusing consumer 
PAssert$289/GroupGlobally/WindowIntoDummy/Window.Assign into 
PAssert$289/GroupGlobally/Create.Values/Read(CreateSource)
[INFO] 2017-04-26T01:19:47.704Z: (75b50c6949b07489): Executing operation 
PAssert$289/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2017-04-26T01:19:47.910Z: (71d700846d982c46): Starting 1 

[GitHub] beam pull request #2695: [BEAM-1316] Forbid output in StartBundle

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2695

[BEAM-1316] Forbid output in StartBundle

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
TODO: Add comments, audit any other context creation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam no_start_bundle_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2695.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2695


commit 5d369da267405911e534e49ad74fdb64bc2c3150
Author: Thomas Groh 
Date:   2017-04-26T01:20:21Z

Forbid output in StartBundle




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1316) DoFn#startBundle should not be able to output

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983961#comment-15983961
 ] 

ASF GitHub Bot commented on BEAM-1316:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2695

[BEAM-1316] Forbid output in StartBundle

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
TODO: Add comments, audit any other context creation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam no_start_bundle_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2695.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2695


commit 5d369da267405911e534e49ad74fdb64bc2c3150
Author: Thomas Groh 
Date:   2017-04-26T01:20:21Z

Forbid output in StartBundle




> DoFn#startBundle should not be able to output
> -
>
> Key: BEAM-1316
> URL: https://issues.apache.org/jira/browse/BEAM-1316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> While within startBundle and finishBundle, the window in which elements are 
> output is not generally defined. Elements must always be output from within a 
> windowed context, or the {{WindowFn}} used by the {{PCollection}} may not 
> operate appropriately.
> startBundle and finishBundle are suitable for operational duties, similarly 
> to {{setup}} and {{teardown}}, but within the scope of some collection of 
> input elements. This includes actions such as clearing field state within a 
> DoFn and ensuring all live RPCs complete successfully before committing 
> inputs.
> Sometimes it might be reasonable to output from {{@FinishBundle}} but it is 
> hard to imagine a situation where output from {{@StartBundle}} is useful in a 
> way that doesn't seriously abuse things.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983958#comment-15983958
 ] 

Stephen Sisk commented on BEAM-2031:


yeah - in that doc, I think that "2. Construct FileSystemConfig (conceptually a 
serializable map)" is the world I'm hoping to live in :)

Luke and I were talking, we think that there's a possible way to make multiple 
hadoopfilesystem configurations work - if the below assumptions are true.

Assumptions:
* fs.default.name is always set on Hadoop Configurations used to connect to 
filesystems
* fs.default.name always represents a unique prefix for different 
servers/useful configurations for user's purposes
* the user always uses prefixes that match to fs.default.name
(I'm not sure if those assumptions are true or not given my naivete in the 
hadoop ecosystem)

Given those, we could:
* Allow the user to provide a list of configurations (via pipelineoptions)
* Register for the unique set of schemes present in the configurations (might 
require some small changes to allow this to work)
* Inside of HadoopFileSystem, maintain a map of fs.default.name -> configuration
* When hadoop file system is given a uri, it would just look up the 
configuration based on the prefix, and then use that configuration.

This is aspirational for first stable release, but if anyone has insights into 
whether or not those assumptions are true, that'd be useful.

This may be moot if we use option 2 (Construct FileSystemConfig) in davor's doc.

> Hadoop FileSystem needs to receive Hadoop Configuration
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983956#comment-15983956
 ] 

ASF GitHub Bot commented on BEAM-2021:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2693

[BEAM-2021] Remove AtomicCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam rm_atomic_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2693


commit 91a060934456a915d9d7771a547e03455b905d61
Author: Thomas Groh 
Date:   2017-04-20T17:57:56Z

Remove AtomicCoder




> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2692: Dummy change to trigger jenkins

[jasonkuster]

Github user jasonkuster closed the pull request at:

https://github.com/apache/beam/pull/2692


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2694: Dummy change to trigger jenkins

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/2694

Dummy change to trigger jenkins

Signed-off-by: Jason Kuster 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam dummy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2694.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2694


commit 7a636618f4b66380024953fea81db077f01ad5a8
Author: Jason Kuster 
Date:   2017-04-26T00:42:10Z

Dummy change to trigger jenkins

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2693: [BEAM-2021] Remove AtomicCoder

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2693

[BEAM-2021] Remove AtomicCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam rm_atomic_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2693


commit 91a060934456a915d9d7771a547e03455b905d61
Author: Thomas Groh 
Date:   2017-04-20T17:57:56Z

Remove AtomicCoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-673) Data locality for Read.Bounded

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983950#comment-15983950
 ] 

Eugene Kirpichov commented on BEAM-673:
---

On second thought, this might be related to SDF: processing different 
restrictions of the same element may have different requirements.

Or more like: a design for DoFn's giving hints to runners about their resource 
requirements would need to include some data dependence. I don't have a good 
idea about how to express it in a way that will be modular and will combine 
well with the rest of the Beam model and various tricks runners are allowed to 
do (such as fusion or materialization).

> Data locality for Read.Bounded
> --
>
> Key: BEAM-673
> URL: https://issues.apache.org/jira/browse/BEAM-673
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Ismaël Mejía
> Fix For: First stable release
>
>
> In some distributed filesystems, such as HDFS, we should be able to hint to 
> Spark the preferred locations of splits.
> Here is an example of how Spark does that for Hadoop RDDs:
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-2031:
---
Summary: Hadoop FileSystem needs to receive Hadoop Configuration  (was: 
Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions)

> Hadoop FileSystem needs to receive Hadoop Configuration
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2692: Dummy change to trigger jenkins

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/2692

Dummy change to trigger jenkins

Signed-off-by: Jason Kuster 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam dummy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2692.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2692


commit 7a636618f4b66380024953fea81db077f01ad5a8
Author: Jason Kuster 
Date:   2017-04-26T00:42:10Z

Dummy change to trigger jenkins

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2691: [BEAM-2021] Add getElementCoders to UnionCoder

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2691

[BEAM-2021] Add getElementCoders to UnionCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This is an explicit method on UnionCoder that is completely unrelated to
the serialization and deserialization of the Coder.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam element_coders_union_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2691.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2691


commit abd5dd91ec8dc3a5f093638b67c74952a242f786
Author: Thomas Groh 
Date:   2017-04-26T00:39:38Z

Add getElementCoders to UnionCoder

This is an explicit method on UnionCoder that is completely unrelated to
the serialization and deserialization of the Coder.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983904#comment-15983904
 ] 

ASF GitHub Bot commented on BEAM-2021:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2691

[BEAM-2021] Add getElementCoders to UnionCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This is an explicit method on UnionCoder that is completely unrelated to
the serialization and deserialization of the Coder.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam element_coders_union_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2691.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2691


commit abd5dd91ec8dc3a5f093638b67c74952a242f786
Author: Thomas Groh 
Date:   2017-04-26T00:39:38Z

Add getElementCoders to UnionCoder

This is an explicit method on UnionCoder that is completely unrelated to
the serialization and deserialization of the Coder.




> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2690: Datastore: fix use of deprecated function

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2690

Datastore: fix use of deprecated function

getMutableProperties().put() has been deprecated in favor of putProperties()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam tiny-fixup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2690.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2690


commit e47e2905ee88d89b00d9a0b63776996def5918d4
Author: Dan Halperin 
Date:   2017-04-26T00:34:07Z

Datastore: fix use of deprecated function

getMutableProperties().put() has been deprecated in favor of putProperties()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983883#comment-15983883
 ] 

ASF GitHub Bot commented on BEAM-2021:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2689

[BEAM-2021] Add A CoderTranslator Interface

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This will enable the removal of StandardCoder.getComponents

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam coder_translators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2689


commit 492b1b0824625c2355849f6b191cbc02257dcc84
Author: Thomas Groh 
Date:   2017-04-26T00:24:27Z

Add A CoderTranslator Interface

This will enable the removal of StandardCoder.getComponents




> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2689: [BEAM-2021] Add A CoderTranslator Interface

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2689

[BEAM-2021] Add A CoderTranslator Interface

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This will enable the removal of StandardCoder.getComponents

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam coder_translators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2689


commit 492b1b0824625c2355849f6b191cbc02257dcc84
Author: Thomas Groh 
Date:   2017-04-26T00:24:27Z

Add A CoderTranslator Interface

This will enable the removal of StandardCoder.getComponents




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3456

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #2930

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3453

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #3454

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983833#comment-15983833
 ] 

Davor Bonaci edited comment on BEAM-2031 at 4/25/17 11:37 PM:
--

Ack. (I think the complexity of each of options is roughly the same.)

The proposal is described here in more details: 
https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit


was (Author: davor):
Ack. (I think the complexity of each of option is roughly the same.)

The proposal is described here in more details: 
https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit

> Hadoop FileSystem needs to receive Hadoop Configuration through 
> PipelineOptions
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983833#comment-15983833
 ] 

Davor Bonaci commented on BEAM-2031:


Ack. (I think the complexity of each of option is roughly the same.)

The proposal is described here in more details: 
https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit

> Hadoop FileSystem needs to receive Hadoop Configuration through 
> PipelineOptions
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2080) Add custom maven enforcer rules to catch banned classes and dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983824#comment-15983824
 ] 

ASF GitHub Bot commented on BEAM-2080:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2688

[BEAM-2080]: Add a custom enforcer rule to check for banned classes.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
- Add a custom rule to catch banned classes which can be specified using 
full path like `org/apache/beam/Foo.class` or using wildcards like 
`org/apache/maven/**` 
- Disable enforcer plugin in parent pom to avoid circular dependency issues 
with the custom rule
- Move enforcer executions to `pluginManagement` and explicitly include 
them in `sdks`, `examples` and `runners`.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam mvn_enforcer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2688.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2688


commit 48034b365ab89b4e4f37c64345269cb534a87212
Author: Vikas Kedigehalli 
Date:   2017-04-21T22:10:21Z

Add a custom rule to check for banned classes.




> Add custom maven enforcer rules to catch banned classes and dependencies
> 
>
> Key: BEAM-2080
> URL: https://issues.apache.org/jira/browse/BEAM-2080
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>Priority: Minor
>
> The maven enforcer plugin standard rules aren't sufficient to catch certain 
> issues like:
> * An artifact built as an uber/bundled jar (usually with shade plugin) 
> including banned classes. 
> * An artifact pom that depends on banned dependencies. (bannedDependencies 
> rule provided by enforcer plugin doesn't work always because it doesn't look 
> at the dependency-reduced-pom generated by shade plugin)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2688: [BEAM-2080]: Add a custom enforcer rule to check fo...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2688

[BEAM-2080]: Add a custom enforcer rule to check for banned classes.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
- Add a custom rule to catch banned classes which can be specified using 
full path like `org/apache/beam/Foo.class` or using wildcards like 
`org/apache/maven/**` 
- Disable enforcer plugin in parent pom to avoid circular dependency issues 
with the custom rule
- Move enforcer executions to `pluginManagement` and explicitly include 
them in `sdks`, `examples` and `runners`.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam mvn_enforcer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2688.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2688


commit 48034b365ab89b4e4f37c64345269cb534a87212
Author: Vikas Kedigehalli 
Date:   2017-04-21T22:10:21Z

Add a custom rule to check for banned classes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983814#comment-15983814
 ] 

Stephen Sisk edited comment on BEAM-2031 at 4/25/17 11:27 PM:
--

where/how would we do that? There's no way right now for a user to specify any 
config for a FileSystem except via PipelineOptions - they're instantiated 
behind the scenes via the registrars.

In my previous discussion on BEAM-2005 
(https://issues.apache.org/jira/browse/BEAM-2005?focusedCommentId=15977497&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15977497)
 I brought up that it'd be good to allow multiple configs and that it'd be nice 
to specify configuration information on the transform that gets passed to the 
FileSystem when doing a read/write, but it's not currently supported as far as 
I know.  (cc [~dhalp...@google.com] since this impacts the future of FBS/the 
sources reading. )

I assume we won't be attempting to do that for the first stable release, and 
thus PipelineOptions is the only way.


was (Author: sisk):
where/how would we do that? There's no way right now for a user to specify any 
config for a FileSystem except via PipelineOptions - they're instantiated 
behind the scenes via the registrars.

In my previous discussion on BEAM-2005 I brought up that it'd be good to allow 
multiple configs and that it'd be nice to specify configuration information on 
the transform that gets passed to the FileSystem when doing a read/write, but 
it's not currently supported as far as I know.  (cc [~dhalp...@google.com] 
since this impacts the future of FBS/the sources reading. )

I assume we won't be attempting to do that for the first stable release, and 
thus PipelineOptions is the only way.

> Hadoop FileSystem needs to receive Hadoop Configuration through 
> PipelineOptions
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983814#comment-15983814
 ] 

Stephen Sisk commented on BEAM-2031:


where/how would we do that? There's no way right now for a user to specify any 
config for a FileSystem except via PipelineOptions - they're instantiated 
behind the scenes via the registrars.

In my previous discussion on BEAM-2005 I brought up that it'd be good to allow 
multiple configs and that it'd be nice to specify configuration information on 
the transform that gets passed to the FileSystem when doing a read/write, but 
it's not currently supported as far as I know.  (cc [~dhalp...@google.com] 
since this impacts the future of FBS/the sources reading. )

I assume we won't be attempting to do that for the first stable release, and 
thus PipelineOptions is the only way.

> Hadoop FileSystem needs to receive Hadoop Configuration through 
> PipelineOptions
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1988) utils.path.join does not correctly handle GCS bucket roots

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983810#comment-15983810
 ] 

ASF GitHub Bot commented on BEAM-1988:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2665


> utils.path.join does not correctly handle GCS bucket roots
> --
>
> Key: BEAM-1988
> URL: https://issues.apache.org/jira/browse/BEAM-1988
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Here:
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/path.py#L22
> Joining a bucket root with a filename e.g. (gs://mybucket/ , myfile) results 
> in invalid 'gs://mybucket//myfile', notice the double // between mybucket and 
> myfile. (It actually does not handle anything that already ends with {{/}} 
> correctly)
> [~sb2nov] could you take this one? Also, should the `join` operation move to 
> a BeamFileSystem level code.
> (cc: [~chamikara])



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2665

[chamikara]

This closes #2665


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b8c568f2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b8c568f2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b8c568f2

Branch: refs/heads/master
Commit: b8c568f2950d333f44b72b57c24f59464a0c9836
Parents: 0d69611 ad6dcf4
Author: Chamikara Jayalath 
Authored: Tue Apr 25 16:22:26 2017 -0700
Committer: Chamikara Jayalath 
Committed: Tue Apr 25 16:22:26 2017 -0700

--
 sdks/python/.pylintrc   |   1 +
 sdks/python/apache_beam/io/filebasedsource.py   |  20 +-
 sdks/python/apache_beam/io/fileio.py|  25 +--
 sdks/python/apache_beam/io/filesystem.py|   6 +-
 sdks/python/apache_beam/io/filesystems.py   | 186 +++
 sdks/python/apache_beam/io/filesystems_test.py  | 224 +++
 .../apache_beam/io/localfilesystem_test.py  |   4 +-
 .../runners/dataflow/internal/apiclient.py  |   7 +-
 .../runners/dataflow/internal/dependency.py |  32 ++-
 .../dataflow/internal/dependency_test.py|   7 +-
 .../apache_beam/tests/pipeline_verifiers.py |   7 +-
 11 files changed, 452 insertions(+), 67 deletions(-)
--

[GitHub] beam pull request #2665: [BEAM-1988] Add FileSystems Interface for accessing...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2665


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1988] Add FileSystems Interface for accessing underlying FS correctly

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master 0d69611e2 -> b8c568f29


[BEAM-1988] Add FileSystems Interface for accessing underlying FS correctly


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ad6dcf4d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ad6dcf4d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ad6dcf4d

Branch: refs/heads/master
Commit: ad6dcf4d1d22b7e6e349db9027ef639a5410b494
Parents: 0d69611
Author: Sourabh Bajaj 
Authored: Tue Apr 25 12:01:21 2017 -0700
Committer: Chamikara Jayalath 
Committed: Tue Apr 25 16:21:30 2017 -0700

--
 sdks/python/.pylintrc   |   1 +
 sdks/python/apache_beam/io/filebasedsource.py   |  20 +-
 sdks/python/apache_beam/io/fileio.py|  25 +--
 sdks/python/apache_beam/io/filesystem.py|   6 +-
 sdks/python/apache_beam/io/filesystems.py   | 186 +++
 sdks/python/apache_beam/io/filesystems_test.py  | 224 +++
 .../apache_beam/io/localfilesystem_test.py  |   4 +-
 .../runners/dataflow/internal/apiclient.py  |   7 +-
 .../runners/dataflow/internal/dependency.py |  32 ++-
 .../dataflow/internal/dependency_test.py|   7 +-
 .../apache_beam/tests/pipeline_verifiers.py |   7 +-
 11 files changed, 452 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ad6dcf4d/sdks/python/.pylintrc
--
diff --git a/sdks/python/.pylintrc b/sdks/python/.pylintrc
index 429ebdb..6418249 100644
--- a/sdks/python/.pylintrc
+++ b/sdks/python/.pylintrc
@@ -95,6 +95,7 @@ disable =
   import-self,
   invalid-name,
   invalid-unary-operand-type,
+  len-as-condition,
   locally-disabled,
   locally-enabled,
   misplaced-bare-raise,

http://git-wip-us.apache.org/repos/asf/beam/blob/ad6dcf4d/sdks/python/apache_beam/io/filebasedsource.py
--
diff --git a/sdks/python/apache_beam/io/filebasedsource.py 
b/sdks/python/apache_beam/io/filebasedsource.py
index ef44b3e..e25f92e 100644
--- a/sdks/python/apache_beam/io/filebasedsource.py
+++ b/sdks/python/apache_beam/io/filebasedsource.py
@@ -30,7 +30,7 @@ from apache_beam.io import concat_source
 from apache_beam.io import iobase
 from apache_beam.io import range_trackers
 from apache_beam.io.filesystem import CompressionTypes
-from apache_beam.io.filesystems_util import get_filesystem
+from apache_beam.io.filesystems import FileSystems
 from apache_beam.transforms.display import DisplayDataItem
 from apache_beam.utils.value_provider import ValueProvider
 from apache_beam.utils.value_provider import StaticValueProvider
@@ -86,10 +86,6 @@ class FileBasedSource(iobase.BoundedSource):
 if isinstance(file_pattern, basestring):
   file_pattern = StaticValueProvider(str, file_pattern)
 self._pattern = file_pattern
-if file_pattern.is_accessible():
-  self._file_system = get_filesystem(file_pattern.get())
-else:
-  self._file_system = None
 
 self._concat_source = None
 self._min_bundle_size = min_bundle_size
@@ -118,9 +114,7 @@ class FileBasedSource(iobase.BoundedSource):
   pattern = self._pattern.get()
 
   single_file_sources = []
-  if self._file_system is None:
-self._file_system = get_filesystem(pattern)
-  match_result = self._file_system.match([pattern])[0]
+  match_result = FileSystems.match([pattern])[0]
   files_metadata = match_result.metadata_list
 
   # We create a reference for FileBasedSource that will be serialized along
@@ -155,7 +149,7 @@ class FileBasedSource(iobase.BoundedSource):
 return self._concat_source
 
   def open_file(self, file_name):
-return get_filesystem(file_name).open(
+return FileSystems.open(
 file_name, 'application/octet-stream',
 compression_type=self._compression_type)
 
@@ -164,11 +158,9 @@ class FileBasedSource(iobase.BoundedSource):
 """Validate if there are actual files in the specified glob pattern
 """
 pattern = self._pattern.get()
-if self._file_system is None:
-  self._file_system = get_filesystem(pattern)
 
 # Limit the responses as we only want to check if something exists
-match_result = self._file_system.match([pattern], limits=[1])[0]
+match_result = FileSystems.match([pattern], limits=[1])[0]
 if len(match_result.metadata_list) <= 0:
   raise IOError(
   'No files found based on the file pattern %s' % pattern)
@@ -183,9 +175,7 @@ class FileBasedSource(iobase.BoundedSource):
   @check_accessible(['_pattern'])
   def estimate_size(self):
 pattern = self._pattern.get()
-if self._file_system is None:
-  self._file_system = get_filesystem(pattern)
-match_result = self._file_sys

[jira] [Created] (BEAM-2080) Add custom maven enforcer rules to catch banned classes and dependencies

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-2080:
---

 Summary: Add custom maven enforcer rules to catch banned classes 
and dependencies
 Key: BEAM-2080
 URL: https://issues.apache.org/jira/browse/BEAM-2080
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Affects Versions: Not applicable
Reporter: Vikas Kedigehalli
Assignee: Vikas Kedigehalli
Priority: Minor


The maven enforcer plugin standard rules aren't sufficient to catch certain 
issues like:
* An artifact built as an uber/bundled jar (usually with shade plugin) 
including banned classes. 
* An artifact pom that depends on banned dependencies. (bannedDependencies rule 
provided by enforcer plugin doesn't work always because it doesn't look at the 
dependency-reduced-pom generated by shade plugin)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983802#comment-15983802
 ] 

Davor Bonaci commented on BEAM-2031:


[~lcwik] is likely to have an opinion on this.

I could see a world where we don't offer `hdfsConfiguration` option. Users can 
construct the configuration object themselves and give it to us as a Java 
object.

> Hadoop FileSystem needs to receive Hadoop Configuration through 
> PipelineOptions
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3452

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2687: [BEAM-1989] Fix the syntax warning from import star

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2687

[BEAM-1989] Fix the syntax warning from import star

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1989-fix-syntax-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2687


commit 678c05a7c82b59f1721a0504cc89ecc2946793e1
Author: Sourabh Bajaj 
Date:   2017-04-25T23:04:12Z

[BEAM-1989] Fix the syntax warning from import star




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1989) clean SyntaxWarning

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983790#comment-15983790
 ] 

ASF GitHub Bot commented on BEAM-1989:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2687

[BEAM-1989] Fix the syntax warning from import star

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1989-fix-syntax-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2687


commit 678c05a7c82b59f1721a0504cc89ecc2946793e1
Author: Sourabh Bajaj 
Date:   2017-04-25T23:04:12Z

[BEAM-1989] Fix the syntax warning from import star




> clean SyntaxWarning
> ---
>
> Key: BEAM-1989
> URL: https://issues.apache.org/jira/browse/BEAM-1989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>
> apache_beam/io/gcp/bigquery.py:326: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table=None, dataset=None, project=None, query=None,
> apache_beam/io/gcp/bigquery.py:431: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table, dataset=None, project=None, schema=None,
> cc: [~sb2nov][~chamikara]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1989) clean SyntaxWarning

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj reassigned BEAM-1989:
---

Assignee: Sourabh Bajaj

> clean SyntaxWarning
> ---
>
> Key: BEAM-1989
> URL: https://issues.apache.org/jira/browse/BEAM-1989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> apache_beam/io/gcp/bigquery.py:326: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table=None, dataset=None, project=None, query=None,
> apache_beam/io/gcp/bigquery.py:431: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table, dataset=None, project=None, schema=None,
> cc: [~sb2nov][~chamikara]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1749) Upgrade pep8 to pycodestyle

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj reassigned BEAM-1749:
---

Assignee: Sourabh Bajaj

> Upgrade pep8 to pycodestyle
> ---
>
> Key: BEAM-1749
> URL: https://issues.apache.org/jira/browse/BEAM-1749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: newbie, starter
>
> pep8 was deprecated and replaced with pycodestyle
> We should upgrade our linter to this module, and while doing that re-evaluate 
> our linter strategy and see if we can enable more rules. This is important 
> for keeping the code healthy as the community grows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-539) Error when writing to the root of a GCS location

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983788#comment-15983788
 ] 

Chamikara Jayalath commented on BEAM-539:
-

That PR will not automatically fix this but we can use that. I'll send out a PR 
for this after PR-2665 is in.

> Error when writing to the root of a GCS location
> 
>
> Key: BEAM-539
> URL: https://issues.apache.org/jira/browse/BEAM-539
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Labels: newbie, starter
> Fix For: First stable release
>
>
> User issue: 
> http://stackoverflow.com/questions/38811152/google-dataflow-python-pipeline-write-failure
> Reproduction: use a TextFileSink and set output locations as gs://mybucket 
> and it fails. Change it to gs://mybucket/ and it works.
> The final output path is generated here:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L495
> And this seemingly works in the Java SDK.
> Stack:
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/iobase.py", 
> line 1058, in finish_bundle
> yield window.TimestampedValue(self.writer.close(), window.MAX_TIMESTAMP)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/fileio.py", 
> line 601, in close
> self.sink.close(self.temp_handle)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/fileio.py", 
> line 687, in close
> file_handle.close()
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcsio.py", line 
> 617, in close
> self._flush_write_buffer()
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcsio.py", line 
> 647, in _flush_write_buffer
> raise self.upload_thread.last_error  # pylint: disable=raising-bad-type
> HttpError: HttpError accessing 
> :
>  response: <{'status': '404', 'alternate-protocol': '443:quic', 
> 'content-length': '165', 'vary': 'Origin, X-Origin', 'server': 
> 'UploadServer', 'x-guploader-uploadid': 
> 'AEnB2Uq6ZGb_CsrMVxozv6aL48k4OMMiRgYVeVGmJrM-sMQWRGeGMkesOQg5F0W7HZuaqTBog_d4ml-DlIars_ZvJTejdfcbAUr4gswZWVieq82ufc3WR2g',
>  'date': 'Mon, 08 Aug 2016 21:29:46 GMT', 'alt-svc': 'quic=":443"; 
> ma=2592000; v="36,35,34,33,32,31,30"', 'content-type': 'application/json; 
> charset=UTF-8'}>, content <{
>  "error": {
>   "errors": [
>{
> "domain": "global",
> "reason": "notFound",
> "message": "Not Found"
>}
>   ],
>   "code": 404,
>   "message": "Not Found"
>  }
> }



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1749) Upgrade pep8 to pycodestyle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983787#comment-15983787
 ] 

ASF GitHub Bot commented on BEAM-1749:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2686

[BEAM-1749] Upgrade to pycodestyle from pep8

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1749-pycodestyle

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2686.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2686


commit 1641287cfa80c386dd5fb449061293afedb3a264
Author: Sourabh Bajaj 
Date:   2017-04-25T23:00:18Z

[BEAM-1749] Upgrade to pycodestyle




> Upgrade pep8 to pycodestyle
> ---
>
> Key: BEAM-1749
> URL: https://issues.apache.org/jira/browse/BEAM-1749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>  Labels: newbie, starter
>
> pep8 was deprecated and replaced with pycodestyle
> We should upgrade our linter to this module, and while doing that re-evaluate 
> our linter strategy and see if we can enable more rules. This is important 
> for keeping the code healthy as the community grows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2686: [BEAM-1749] Upgrade to pycodestyle from pep8

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2686

[BEAM-1749] Upgrade to pycodestyle from pep8

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1749-pycodestyle

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2686.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2686


commit 1641287cfa80c386dd5fb449061293afedb3a264
Author: Sourabh Bajaj 
Date:   2017-04-25T23:00:18Z

[BEAM-1749] Upgrade to pycodestyle




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983783#comment-15983783
 ] 

ASF GitHub Bot commented on BEAM-2021:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2668


> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2668: [BEAM-2021] Make Most StandardCoders CustomCoders

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2668


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam git commit: This closes #2668

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 2e3c17a35 -> 0d69611e2


This closes #2668


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0d69611e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0d69611e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0d69611e

Branch: refs/heads/master
Commit: 0d69611e23122f0ec3951bf0da1e497fa23d2997
Parents: 2e3c17a cbcec7c
Author: Thomas Groh 
Authored: Tue Apr 25 15:31:33 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 25 15:31:33 2017 -0700

--
 .../apex/translation/utils/ApexStreamTuple.java |   6 +-
 runners/core-construction-java/pom.xml  |   5 -
 .../beam/runners/core/construction/Coders.java  |   6 +
 .../UnboundedReadFromBoundedSource.java |  16 +--
 .../runners/core/construction/CodersTest.java   |   2 +
 runners/core-java/pom.xml   |   5 -
 .../beam/runners/core/KeyedWorkItemCoder.java   |  22 +---
 .../beam/runners/core/TimerInternals.java   |  18 +--
 .../streaming/SingletonKeyedWorkItemCoder.java  |  22 +---
 .../runners/dataflow/BatchViewOverrides.java|  21 +---
 .../runners/dataflow/internal/IsmFormat.java|  64 ++
 .../org/apache/beam/sdk/coders/AvroCoder.java   |  46 +++-
 .../org/apache/beam/sdk/coders/CustomCoder.java |   3 +-
 .../org/apache/beam/sdk/coders/MapCoder.java|  26 ++---
 .../apache/beam/sdk/coders/NullableCoder.java   |  27 +++--
 .../beam/sdk/coders/SerializableCoder.java  |  22 
 .../org/apache/beam/sdk/testing/TestStream.java | 116 ---
 .../org/apache/beam/sdk/transforms/Combine.java |  14 +--
 .../apache/beam/sdk/transforms/CombineFns.java  |  15 +--
 .../beam/sdk/transforms/join/CoGbkResult.java   |  27 +
 .../beam/sdk/transforms/join/UnionCoder.java|  14 +--
 .../apache/beam/sdk/util/ValueWithRecordId.java |  16 +--
 .../beam/sdk/values/TimestampedValue.java   |  22 +---
 .../beam/sdk/values/ValueInSingleWindow.java|  23 +---
 .../apache/beam/sdk/coders/AvroCoderTest.java   |  15 ++-
 .../beam/sdk/coders/CoderRegistryTest.java  |   2 +-
 .../beam/sdk/coders/DelegateCoderTest.java  |   2 +-
 .../beam/sdk/coders/LengthPrefixCoderTest.java  |   7 --
 .../beam/sdk/coders/NullableCoderTest.java  |   2 +-
 .../beam/sdk/coders/SerializableCoderTest.java  |  11 +-
 .../apache/beam/sdk/testing/TestStreamTest.java |  24 
 .../beam/sdk/transforms/CombineFnsTest.java |   4 +-
 .../beam/sdk/util/SerializableUtilsTest.java|  56 +
 .../sdk/extensions/protobuf/ProtoCoder.java |  14 ---
 sdks/java/io/google-cloud-platform/pom.xml  |   5 -
 .../sdk/io/gcp/bigquery/ShardedKeyCoder.java|  17 +--
 sdks/java/io/hadoop-common/pom.xml  |   5 -
 .../beam/sdk/io/hadoop/WritableCoder.java   |  29 +
 .../apache/beam/sdk/io/xml/JAXBCoderTest.java   |   4 +-
 39 files changed, 116 insertions(+), 639 deletions(-)
--

[3/3] beam git commit: Make Most StandardCoders CustomCoders

[tgroh]

Make Most StandardCoders CustomCoders

Standard Coders have a defined serialization format and are understood
within the Runner API, Custom Coders are not. Move existing
"StandardCoders" to extend CustomCoder, and remove custom cloud object
related serialization logic where possible.

Still remaining: Splitting the CustomCoder side of the class hierarchy
from the StandardCoder side of the hierarchy, moving IterableLikeCoder
to be a CustomCoder, and have IterableCoder forward to an internal
implementation (to ensure it remains a StandardCoder).


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cbcec7c0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cbcec7c0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cbcec7c0

Branch: refs/heads/master
Commit: cbcec7c0520f151f348604621cc251b6ed9b1616
Parents: 2e3c17a
Author: Thomas Groh 
Authored: Fri Apr 21 17:58:51 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 25 15:31:33 2017 -0700

--
 .../apex/translation/utils/ApexStreamTuple.java |   6 +-
 runners/core-construction-java/pom.xml  |   5 -
 .../beam/runners/core/construction/Coders.java  |   6 +
 .../UnboundedReadFromBoundedSource.java |  16 +--
 .../runners/core/construction/CodersTest.java   |   2 +
 runners/core-java/pom.xml   |   5 -
 .../beam/runners/core/KeyedWorkItemCoder.java   |  22 +---
 .../beam/runners/core/TimerInternals.java   |  18 +--
 .../streaming/SingletonKeyedWorkItemCoder.java  |  22 +---
 .../runners/dataflow/BatchViewOverrides.java|  21 +---
 .../runners/dataflow/internal/IsmFormat.java|  64 ++
 .../org/apache/beam/sdk/coders/AvroCoder.java   |  46 +++-
 .../org/apache/beam/sdk/coders/CustomCoder.java |   3 +-
 .../org/apache/beam/sdk/coders/MapCoder.java|  26 ++---
 .../apache/beam/sdk/coders/NullableCoder.java   |  27 +++--
 .../beam/sdk/coders/SerializableCoder.java  |  22 
 .../org/apache/beam/sdk/testing/TestStream.java | 116 ---
 .../org/apache/beam/sdk/transforms/Combine.java |  14 +--
 .../apache/beam/sdk/transforms/CombineFns.java  |  15 +--
 .../beam/sdk/transforms/join/CoGbkResult.java   |  27 +
 .../beam/sdk/transforms/join/UnionCoder.java|  14 +--
 .../apache/beam/sdk/util/ValueWithRecordId.java |  16 +--
 .../beam/sdk/values/TimestampedValue.java   |  22 +---
 .../beam/sdk/values/ValueInSingleWindow.java|  23 +---
 .../apache/beam/sdk/coders/AvroCoderTest.java   |  15 ++-
 .../beam/sdk/coders/CoderRegistryTest.java  |   2 +-
 .../beam/sdk/coders/DelegateCoderTest.java  |   2 +-
 .../beam/sdk/coders/LengthPrefixCoderTest.java  |   7 --
 .../beam/sdk/coders/NullableCoderTest.java  |   2 +-
 .../beam/sdk/coders/SerializableCoderTest.java  |  11 +-
 .../apache/beam/sdk/testing/TestStreamTest.java |  24 
 .../beam/sdk/transforms/CombineFnsTest.java |   4 +-
 .../beam/sdk/util/SerializableUtilsTest.java|  56 +
 .../sdk/extensions/protobuf/ProtoCoder.java |  14 ---
 sdks/java/io/google-cloud-platform/pom.xml  |   5 -
 .../sdk/io/gcp/bigquery/ShardedKeyCoder.java|  17 +--
 sdks/java/io/hadoop-common/pom.xml  |   5 -
 .../beam/sdk/io/hadoop/WritableCoder.java   |  29 +
 .../apache/beam/sdk/io/xml/JAXBCoderTest.java   |   4 +-
 39 files changed, 116 insertions(+), 639 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cbcec7c0/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
index 79a4f1b..4ce351b 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
@@ -20,7 +20,6 @@ package org.apache.beam.runners.apex.translation.utils;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.datatorrent.api.Operator;
-
 import java.io.DataInputStream;
 import java.io.DataOutputStream;
 import java.io.IOException;
@@ -29,11 +28,10 @@ import java.io.OutputStream;
 import java.util.Arrays;
 import java.util.List;
 import java.util.Objects;
-
 import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.StandardCoder;
+import org.apache.beam.sdk.coders.CustomCoder;
 
 /**
  * The common interface for all objects transmitted through streams.
@@ -151,7 +149,7 @@ public interface ApexStre

[2/3] beam git commit: Make Most StandardCoders CustomCoders

[tgroh]

http://git-wip-us.apache.org/repos/asf/beam/blob/cbcec7c0/sdks/java/io/hadoop-common/pom.xml
--
diff --git a/sdks/java/io/hadoop-common/pom.xml 
b/sdks/java/io/hadoop-common/pom.xml
index ad31ded..ebb4b39 100644
--- a/sdks/java/io/hadoop-common/pom.xml
+++ b/sdks/java/io/hadoop-common/pom.xml
@@ -37,11 +37,6 @@
 
 
 
-  com.fasterxml.jackson.core
-  jackson-annotations
-
-
-
   org.apache.hadoop
   hadoop-client
   provided

http://git-wip-us.apache.org/repos/asf/beam/blob/cbcec7c0/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java
--
diff --git 
a/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java
 
b/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java
index 0ba367d..9589fb1 100644
--- 
a/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java
+++ 
b/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java
@@ -17,18 +17,16 @@
  */
 package org.apache.beam.sdk.io.hadoop;
 
-import com.fasterxml.jackson.annotation.JsonCreator;
-import com.fasterxml.jackson.annotation.JsonProperty;
 import java.io.DataInputStream;
 import java.io.DataOutputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.util.Collections;
 import java.util.List;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.StandardCoder;
-import org.apache.beam.sdk.util.CloudObject;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.hadoop.io.NullWritable;
 import org.apache.hadoop.io.Writable;
 
@@ -45,7 +43,7 @@ import org.apache.hadoop.io.Writable;
  *
  * @param  the type of elements handled by this coder.
  */
-public class WritableCoder extends StandardCoder {
+public class WritableCoder extends CustomCoder {
   private static final long serialVersionUID = 0L;
 
   /**
@@ -56,18 +54,6 @@ public class WritableCoder extends 
StandardCoder {
 return new WritableCoder<>(clazz);
   }
 
-  @JsonCreator
-  @SuppressWarnings("unchecked")
-  public static WritableCoder of(@JsonProperty("type") String classType)
-  throws ClassNotFoundException {
-Class clazz = Class.forName(classType);
-if (!Writable.class.isAssignableFrom(clazz)) {
-  throw new ClassNotFoundException(
-  "Class " + classType + " does not implement Writable");
-}
-return of((Class) clazz);
-  }
-
   private final Class type;
 
   public WritableCoder(Class type) {
@@ -97,14 +83,7 @@ public class WritableCoder extends 
StandardCoder {
 
   @Override
   public List> getCoderArguments() {
-return null;
-  }
-
-  @Override
-  public CloudObject initializeCloudObject() {
-CloudObject result = CloudObject.forClass(getClass());
-result.put("type", type.getName());
-return result;
+return Collections.emptyList();
   }
 
   @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/cbcec7c0/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
--
diff --git 
a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java 
b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
index 5f1330d..276c231 100644
--- 
a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
+++ 
b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
@@ -33,7 +33,7 @@ import java.util.concurrent.atomic.AtomicReference;
 import javax.xml.bind.annotation.XmlRootElement;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.StandardCoder;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.coders.VarLongCoder;
 import org.apache.beam.sdk.testing.CoderProperties;
@@ -171,7 +171,7 @@ public class JAXBCoderTest {
   /**
* A coder that surrounds the value with two values, to demonstrate nesting.
*/
-  private static class TestCoder extends StandardCoder {
+  private static class TestCoder extends CustomCoder {
 private final JAXBCoder jaxbCoder;
 public TestCoder(JAXBCoder jaxbCoder) {
   this.jaxbCoder = jaxbCoder;

[GitHub] beam pull request #2685: Make WindowedValueCoder an Interface

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2685

Make WindowedValueCoder an Interface

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Implement it in both FullWindowedValueCoder and
ValueOnlyWindowedValueCoder

Make ValueOnlyWindowedValueCoder a CustomCoder

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam windowed_value_only_custom_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2685


commit dacb978fd01367b838db3917c385e92e6e56be74
Author: Thomas Groh 
Date:   2017-04-25T22:55:47Z

Make WindowedValueCoder an Interface

Implement it in both FullWindowedValueCoder and
ValueOnlyWindowedValueCoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2031) Hadoop FileSystem needs to receive Hadoop Configuration through PipelineOptions

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983776#comment-15983776
 ] 

Stephen Sisk commented on BEAM-2031:


We need to allow users to pass in Configuration for hadoop - it's the way to 
set up connection info/etc.. (Configuration is the hadoop class that manages 
this - kinda sorta similar to PipelineOptions, but it is more like a generic 
bag than ours)

>From a reading of the Configuration source code, Configuration objects can 
>store int/float/bool/etc..., but those are ultimately stored as strings.

In both scenarios it seems useful to handle the configuration as a simple flat 
json map, so the user would pass in the configuration like so:

--hdfsConfiguration={
  'fs.default.name': 'hdfs://my-host:8020/,
  'another-key': 'value',
  'a-third-key': 'another-value'} 

in line in a larger commandline:
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
 -Dexec.args="--runner=DataflowRunner  \
  --inputFile=hdfs://my-host/shakespeare/* 
--output=hdfs://my-host/counts \
  --hdfsConfiguration={'fs.default.name': 'hdfs://my-host:8020/, 
'another-key': 'value', 'a-third-key': 'another-value'} \
 -Pdataflow-runner

I propose handling the PipelineOption type as Map and then we 
call set for each key/value pair.

I'm going to test this out to see if there are any holes in my understanding of 
how this all works.

cc [~davor]

> Hadoop FileSystem needs to receive Hadoop Configuration through 
> PipelineOptions
> ---
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to 
> pass a Hadoop Configuration through PipelineOptions. I think that's very 
> solvable, but it does seem semi-complicated.
> cc [~pei...@gmail.com] I believe you mentioned in the past that you had an 
> answer to this - is that written down anywhere?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3451

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983737#comment-15983737
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2678


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2678: [BEAM-1871] Move Bigquery/Pubsub options to sdks/ja...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2678


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam git commit: [BEAM-1871] Move Bigquery/Pubsub options to sdks/java/io/google-cloud-platform

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 652a919ed -> 2e3c17a35


http://git-wip-us.apache.org/repos/asf/beam/blob/6f9f7bcd/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
new file mode 100644
index 000..2bdfffa
--- /dev/null
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.testing;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.BigqueryScopes;
+import com.google.api.services.bigquery.model.QueryRequest;
+import com.google.api.services.bigquery.model.QueryResponse;
+import com.google.api.services.bigquery.model.TableCell;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.auth.Credentials;
+import com.google.auth.http.HttpCredentialsAdapter;
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
+import com.google.common.hash.HashCode;
+import com.google.common.hash.Hashing;
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.NotThreadSafe;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.testing.SerializableMatcher;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Transport;
+import org.hamcrest.Description;
+import org.hamcrest.TypeSafeMatcher;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A matcher to verify data in BigQuery by processing given query
+ * and comparing with content's checksum.
+ *
+ * Example:
+ * {@code [
+ *   assertThat(job, new BigqueryMatcher(appName, projectId, queryString, 
expectedChecksum));
+ * ]}
+ */
+@NotThreadSafe
+public class BigqueryMatcher extends TypeSafeMatcher
+implements SerializableMatcher {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BigqueryMatcher.class);
+
+  // The maximum number of retries to execute a BigQuery RPC
+  static final int MAX_QUERY_RETRIES = 4;
+
+  // The initial backoff for executing a BigQuery RPC
+  private static final Duration INITIAL_BACKOFF = Duration.standardSeconds(1L);
+
+  // The total number of rows in query response to be formatted for debugging 
purpose
+  private static final int TOTAL_FORMATTED_ROWS = 20;
+
+  // The backoff factory with initial configs
+  static final FluentBackoff BACKOFF_FACTORY =
+  FluentBackoff.DEFAULT
+  .withMaxRetries(MAX_QUERY_RETRIES)
+  .withInitialBackoff(INITIAL_BACKOFF);
+
+  private final String applicationName;
+  private final String projectId;
+  private final String query;
+  private final String expectedChecksum;
+  private String actualChecksum;
+  private transient QueryResponse response;
+
+  public BigqueryMatcher(
+  String applicationName, String projectId, String query, String 
expectedChecksum) {
+validateArgument("applicationName", applicationName);
+validateArgument("projectId", projectId);
+validateArgument("query", query);
+validateArgument("expectedChecksum", expectedChecksum);
+
+this.applicationName = applicationName;
+this.projectId = projectId;
+this.query = query;
+   

[2/3] beam git commit: [BEAM-1871] Move Bigquery/Pubsub options to sdks/java/io/google-cloud-platform

[lcwik]

[BEAM-1871] Move Bigquery/Pubsub options to sdks/java/io/google-cloud-platform


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6f9f7bcd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6f9f7bcd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6f9f7bcd

Branch: refs/heads/master
Commit: 6f9f7bcd914d18eee4e00bd16d69c4d53636e244
Parents: 652a919
Author: Luke Cwik 
Authored: Mon Apr 24 15:58:47 2017 -0700
Committer: Luke Cwik 
Committed: Tue Apr 25 15:19:44 2017 -0700

--
 examples/java/pom.xml   |  22 ++
 .../beam/examples/common/ExampleUtils.java  |  60 -
 .../examples/cookbook/BigQueryTornadoesIT.java  |   4 +-
 pom.xml |   7 +
 .../options/DataflowPipelineOptions.java|   4 +-
 sdks/java/extensions/gcp-core/pom.xml   |  29 +--
 .../beam/sdk/options/BigQueryOptions.java   |  32 ---
 .../options/GcpPipelineOptionsRegistrar.java|   2 -
 .../apache/beam/sdk/options/PubsubOptions.java  |  36 ---
 .../beam/sdk/testing/BigqueryMatcher.java   | 256 --
 .../apache/beam/sdk/testing/package-info.java   |  21 --
 .../org/apache/beam/sdk/util/Transport.java |  36 ---
 .../org/apache/beam/GcpCoreApiSurfaceTest.java  |   2 -
 .../sdk/options/GoogleApiDebugOptionsTest.java  |   7 +-
 .../beam/sdk/testing/BigqueryMatcherTest.java   | 176 -
 sdks/java/extensions/protobuf/pom.xml   |  19 +-
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java|   1 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|   1 -
 .../sdk/io/gcp/bigquery/BigQueryOptions.java|  39 +++
 .../io/gcp/bigquery/BigQueryQuerySource.java|   1 -
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   1 -
 .../io/gcp/bigquery/BigQueryServicesImpl.java   |  41 ++-
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java |   1 -
 .../io/gcp/bigquery/BigQueryTableSource.java|   1 -
 .../beam/sdk/io/gcp/bigquery/CreateTables.java  |   1 -
 .../beam/sdk/io/gcp/bigquery/PrepareWrite.java  |   1 -
 .../sdk/io/gcp/bigquery/StreamingWriteFn.java   |   1 -
 .../beam/sdk/io/gcp/bigquery/WriteRename.java   |   1 -
 .../beam/sdk/io/gcp/bigquery/WriteTables.java   |   1 -
 .../common/GcpIoPipelineOptionsRegistrar.java   |  39 +++
 .../beam/sdk/io/gcp/common/package-info.java|  20 ++
 .../beam/sdk/io/gcp/pubsub/PubsubClient.java|   1 -
 .../sdk/io/gcp/pubsub/PubsubGrpcClient.java |   1 -
 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java |   1 -
 .../sdk/io/gcp/pubsub/PubsubJsonClient.java |   1 -
 .../beam/sdk/io/gcp/pubsub/PubsubOptions.java   |  44 
 .../sdk/io/gcp/pubsub/PubsubTestClient.java |   1 -
 .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java  |   1 -
 .../io/gcp/pubsub/PubsubUnboundedSource.java|   1 -
 .../beam/sdk/io/gcp/testing/package-info.java   |  21 ++
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |   2 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   1 -
 .../io/gcp/bigquery/FakeBigQueryServices.java   |   1 -
 .../sdk/io/gcp/testing/BigqueryMatcher.java | 257 +++
 .../sdk/io/gcp/testing/BigqueryMatcherTest.java | 177 +
 45 files changed, 737 insertions(+), 636 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6f9f7bcd/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ae3d63d..9317136 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -485,6 +485,21 @@
 
 
 
+  com.google.cloud.bigdataoss
+  util
+
+
+
+  com.google.auth
+  google-auth-library-oauth2-http
+
+
+
+  com.google.auth
+  google-auth-library-credentials
+
+
+
   org.apache.avro
   avro
 
@@ -552,6 +567,13 @@
 
 
 
+  org.apache.beam
+  beam-sdks-java-io-google-cloud-platform
+  tests
+  test
+
+
+
   org.mockito
   mockito-all
   test

http://git-wip-us.apache.org/repos/asf/beam/blob/6f9f7bcd/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
index 6962571..2650f8e 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
@@ -19,6 +19,7 @@ package org.apache.beam.examples.common;
 
 import com.google.api.client.googleapis.json.GoogleJsonResponseException;
 import com.google.api.client.googleapis.services.AbstractGoogleClientRequest;
+import com.google.

[jira] [Resolved] (BEAM-2068) Upgrade Google-Apitools to latest version

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-2068.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Upgrade Google-Apitools to latest version
> -
>
> Key: BEAM-2068
> URL: https://issues.apache.org/jira/browse/BEAM-2068
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: First stable release
>
>
> In 0.5.9 apitools is pinned to setuptools 18.5 which is really old as the 
> current release is 35.0.1 at the time of creating the issue. Updating to 
> 0.5.9 causes issues for other dependencies so we're going to try to address 
> this upstream first and then upgrade to the latest version in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: [BEAM-1871] Move Bigquery/Pubsub options to sdks/java/io/google-cloud-platform

[lcwik]

[BEAM-1871] Move Bigquery/Pubsub options to sdks/java/io/google-cloud-platform

This closes #2678


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2e3c17a3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2e3c17a3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2e3c17a3

Branch: refs/heads/master
Commit: 2e3c17a3560ac32da8fc5c944c836107b2b76312
Parents: 652a919 6f9f7bc
Author: Luke Cwik 
Authored: Tue Apr 25 15:19:56 2017 -0700
Committer: Luke Cwik 
Committed: Tue Apr 25 15:19:56 2017 -0700

--
 examples/java/pom.xml   |  22 ++
 .../beam/examples/common/ExampleUtils.java  |  60 -
 .../examples/cookbook/BigQueryTornadoesIT.java  |   4 +-
 pom.xml |   7 +
 .../options/DataflowPipelineOptions.java|   4 +-
 sdks/java/extensions/gcp-core/pom.xml   |  29 +--
 .../beam/sdk/options/BigQueryOptions.java   |  32 ---
 .../options/GcpPipelineOptionsRegistrar.java|   2 -
 .../apache/beam/sdk/options/PubsubOptions.java  |  36 ---
 .../beam/sdk/testing/BigqueryMatcher.java   | 256 --
 .../apache/beam/sdk/testing/package-info.java   |  21 --
 .../org/apache/beam/sdk/util/Transport.java |  36 ---
 .../org/apache/beam/GcpCoreApiSurfaceTest.java  |   2 -
 .../sdk/options/GoogleApiDebugOptionsTest.java  |   7 +-
 .../beam/sdk/testing/BigqueryMatcherTest.java   | 176 -
 sdks/java/extensions/protobuf/pom.xml   |  19 +-
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java|   1 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|   1 -
 .../sdk/io/gcp/bigquery/BigQueryOptions.java|  39 +++
 .../io/gcp/bigquery/BigQueryQuerySource.java|   1 -
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   1 -
 .../io/gcp/bigquery/BigQueryServicesImpl.java   |  41 ++-
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java |   1 -
 .../io/gcp/bigquery/BigQueryTableSource.java|   1 -
 .../beam/sdk/io/gcp/bigquery/CreateTables.java  |   1 -
 .../beam/sdk/io/gcp/bigquery/PrepareWrite.java  |   1 -
 .../sdk/io/gcp/bigquery/StreamingWriteFn.java   |   1 -
 .../beam/sdk/io/gcp/bigquery/WriteRename.java   |   1 -
 .../beam/sdk/io/gcp/bigquery/WriteTables.java   |   1 -
 .../common/GcpIoPipelineOptionsRegistrar.java   |  39 +++
 .../beam/sdk/io/gcp/common/package-info.java|  20 ++
 .../beam/sdk/io/gcp/pubsub/PubsubClient.java|   1 -
 .../sdk/io/gcp/pubsub/PubsubGrpcClient.java |   1 -
 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java |   1 -
 .../sdk/io/gcp/pubsub/PubsubJsonClient.java |   1 -
 .../beam/sdk/io/gcp/pubsub/PubsubOptions.java   |  44 
 .../sdk/io/gcp/pubsub/PubsubTestClient.java |   1 -
 .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java  |   1 -
 .../io/gcp/pubsub/PubsubUnboundedSource.java|   1 -
 .../beam/sdk/io/gcp/testing/package-info.java   |  21 ++
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |   2 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   1 -
 .../io/gcp/bigquery/FakeBigQueryServices.java   |   1 -
 .../sdk/io/gcp/testing/BigqueryMatcher.java | 257 +++
 .../sdk/io/gcp/testing/BigqueryMatcherTest.java | 177 +
 45 files changed, 737 insertions(+), 636 deletions(-)
--

[1/2] beam git commit: This closes #2681

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/hdfs [created] 162369aa8


This closes #2681


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/162369aa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/162369aa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/162369aa

Branch: refs/heads/hdfs
Commit: 162369aa8f11961b6c2c5af0c89be8ed08cb4b45
Parents: 4fabaef fac7b83
Author: Thomas Groh 
Authored: Tue Apr 25 14:51:39 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 25 14:51:39 2017 -0700

--
 .../beam/sdk/io/hdfs/HadoopResourceId.java  | 48 ++-
 .../beam/sdk/io/hdfs/HadoopResourceIdTest.java  | 83 
 2 files changed, 129 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-1283) DoFn finishBundle should be required to specify the window for output

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983733#comment-15983733
 ] 

Sourabh Bajaj commented on BEAM-1283:
-

[~kenn] have we reached a resolution on this?

Currently in Python if the user doesn't return a timestamped value we assign it 
to -1 for the output[1]. As we don't have state and timers implemented 
completely yet should we just assert and that we're getting timestampedValues 
only.

[1] 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/common.py#L310

> DoFn finishBundle should be required to specify the window for output
> -
>
> Key: BEAM-1283
> URL: https://issues.apache.org/jira/browse/BEAM-1283
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The spec is here in Javadoc: 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to 
> use the {{WindowFn}} of the input {{PCollection}} to determine what windows 
> the element should be in, throwing an exception if the {{WindowFn}} attempts 
> to access any information about the input element. The output element will 
> have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically 
> wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is 
> allowed to see the element type. The spec just avoids this by limiting which 
> {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The 
> spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged 
> then you'll just have a bogus proto window regardless of explicit timestamp 
> or not.
> The use cases for these methods are best addressed by state plus window 
> expiry callback, so we should revisit this spec and probably just wipe it.
> There are some rare case where you might need to output from {{FinishBundle}} 
> in a way that is not _actually_ sensitive to bundling (perhaps modulo some 
> downstream notion of equivalence) in which case you had better know what 
> window you are outputting to. Often it should be the global window.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2068) Upgrade Google-Apitools to latest version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983731#comment-15983731
 ] 

ASF GitHub Bot commented on BEAM-2068:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2684


> Upgrade Google-Apitools to latest version
> -
>
> Key: BEAM-2068
> URL: https://issues.apache.org/jira/browse/BEAM-2068
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Ahmet Altay
>Priority: Minor
>
> In 0.5.9 apitools is pinned to setuptools 18.5 which is really old as the 
> current release is 35.0.1 at the time of creating the issue. Updating to 
> 0.5.9 causes issues for other dependencies so we're going to try to address 
> this upstream first and then upgrade to the latest version in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Add HadoopResourceId

[tgroh]

Add HadoopResourceId


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fac7b838
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fac7b838
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fac7b838

Branch: refs/heads/hdfs
Commit: fac7b838bb2642aeb34d12d359737fd641d52aec
Parents: 4fabaef
Author: Stephen Sisk 
Authored: Fri Apr 21 17:23:55 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 25 14:51:39 2017 -0700

--
 .../beam/sdk/io/hdfs/HadoopResourceId.java  | 48 ++-
 .../beam/sdk/io/hdfs/HadoopResourceIdTest.java  | 83 
 2 files changed, 129 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fac7b838/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
--
diff --git 
a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
 
b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
index 2a29bb9..5524cac 100644
--- 
a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
+++ 
b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
@@ -17,26 +17,70 @@
  */
 package org.apache.beam.sdk.io.hdfs;
 
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import java.net.URI;
 import org.apache.beam.sdk.io.fs.ResolveOptions;
 import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.hadoop.fs.Path;
 
 /**
  * {@link ResourceId} implementation for the {@link HadoopFileSystem}.
  */
 public class HadoopResourceId implements ResourceId {
 
+  private final URI uri;
+
+  /**
+   * Constructs a HadoopResourceId from the provided absolute path. If only a 
relative path is
+   * available, you can create a {@link HadoopResourceId} from the absolute 
path of the root of the
+   * server, and then use resolve to add the relative path to the root.
+   */
+  public static HadoopResourceId fromPath(Path path) {
+checkNotNull(path, "path must not be null");
+checkArgument(path.isAbsolute(), "path must be absolute");
+return new HadoopResourceId(path);
+  }
+
+  private HadoopResourceId(Path path) {
+this.uri = path.toUri();
+  }
+
   @Override
   public ResourceId resolve(String other, ResolveOptions resolveOptions) {
-throw new UnsupportedOperationException();
+checkArgument(
+
resolveOptions.equals(ResolveOptions.StandardResolveOptions.RESOLVE_FILE)
+|| 
resolveOptions.equals(ResolveOptions.StandardResolveOptions.RESOLVE_DIRECTORY),
+String.format("ResolveOptions: [%s] is not supported. "
++ "Supported ResolveOptions are RESOLVE_FILE and 
RESOLVE_DIRECTORY.", resolveOptions));
+if 
(resolveOptions.equals(ResolveOptions.StandardResolveOptions.RESOLVE_FILE)) {
+  checkArgument(
+  !other.endsWith("/"),
+  "ResolveOptions: [%s] ends with '/', which is not supported for 
RESOLVE_FILE.",
+  other);
+}
+return new HadoopResourceId(new Path(new Path(uri), other));
   }
 
   @Override
   public ResourceId getCurrentDirectory() {
+// See BEAM-2069. Possible workaround: inject FileSystem into this class, 
and call
+// org.apache.hadoop.fs.FileSystem#isDirectory.
 throw new UnsupportedOperationException();
   }
 
   @Override
   public String getScheme() {
-throw new UnsupportedOperationException();
+return uri.getScheme();
+  }
+
+  public Path getPath() {
+return new Path(uri);
+  }
+
+  @Override
+  public String toString() {
+return uri.toString();
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/fac7b838/sdks/java/io/hdfs/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopResourceIdTest.java
--
diff --git 
a/sdks/java/io/hdfs/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopResourceIdTest.java
 
b/sdks/java/io/hdfs/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopResourceIdTest.java
new file mode 100644
index 000..e4eadfa
--- /dev/null
+++ 
b/sdks/java/io/hdfs/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopResourceIdTest.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable

[GitHub] beam pull request #2684: [BEAM-2068] Update to latest version of apitools

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2684


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2068] Update to latest version of apitools

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 4fabaef80 -> 652a919ed


[BEAM-2068] Update to latest version of apitools


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c0fa7473
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c0fa7473
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c0fa7473

Branch: refs/heads/master
Commit: c0fa7473ccdb979ad2b52031d6af1406ddd4f567
Parents: 4fabaef
Author: Sourabh Bajaj 
Authored: Tue Apr 25 14:29:44 2017 -0700
Committer: Sourabh Bajaj 
Committed: Tue Apr 25 14:29:44 2017 -0700

--
 sdks/python/setup.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c0fa7473/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 182c6b2..615931b 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -103,8 +103,7 @@ REQUIRED_TEST_PACKAGES = [
 ]
 
 GCP_REQUIREMENTS = [
-  # TODO(BEAM-2068): Upgrade google-apitools once the issue is fixed.
-  'google-apitools==0.5.8',
+  'google-apitools==0.5.10',
   'proto-google-cloud-datastore-v1==0.90.0',
   'googledatastore==7.0.1',
   # GCP packages required by tests

[2/2] beam git commit: This closes #2684

[altay]

This closes #2684


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/652a919e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/652a919e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/652a919e

Branch: refs/heads/master
Commit: 652a919ed2da5d7a0487c97fa5e7bfeb92a4802c
Parents: 4fabaef c0fa747
Author: Ahmet Altay 
Authored: Tue Apr 25 15:17:53 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Apr 25 15:17:53 2017 -0700

--
 sdks/python/setup.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-2068) Upgrade Google-Apitools to latest version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983727#comment-15983727
 ] 

ASF GitHub Bot commented on BEAM-2068:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2684

[BEAM-2068] Update to latest version of apitools

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2068-upgrade-apitols

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2684


commit c0fa7473ccdb979ad2b52031d6af1406ddd4f567
Author: Sourabh Bajaj 
Date:   2017-04-25T21:29:44Z

[BEAM-2068] Update to latest version of apitools




> Upgrade Google-Apitools to latest version
> -
>
> Key: BEAM-2068
> URL: https://issues.apache.org/jira/browse/BEAM-2068
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Ahmet Altay
>Priority: Minor
>
> In 0.5.9 apitools is pinned to setuptools 18.5 which is really old as the 
> current release is 35.0.1 at the time of creating the issue. Updating to 
> 0.5.9 causes issues for other dependencies so we're going to try to address 
> this upstream first and then upgrade to the latest version in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2684: [BEAM-2068] Update to latest version of apitools

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2684

[BEAM-2068] Update to latest version of apitools

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2068-upgrade-apitols

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2684


commit c0fa7473ccdb979ad2b52031d6af1406ddd4f567
Author: Sourabh Bajaj 
Date:   2017-04-25T21:29:44Z

[BEAM-2068] Update to latest version of apitools




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-539) Error when writing to the root of a GCS location

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983679#comment-15983679
 ] 

Ahmet Altay commented on BEAM-539:
--

[~chamikara], is this fixed with {{join}} in BFS?

> Error when writing to the root of a GCS location
> 
>
> Key: BEAM-539
> URL: https://issues.apache.org/jira/browse/BEAM-539
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Labels: newbie, starter
> Fix For: First stable release
>
>
> User issue: 
> http://stackoverflow.com/questions/38811152/google-dataflow-python-pipeline-write-failure
> Reproduction: use a TextFileSink and set output locations as gs://mybucket 
> and it fails. Change it to gs://mybucket/ and it works.
> The final output path is generated here:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L495
> And this seemingly works in the Java SDK.
> Stack:
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/iobase.py", 
> line 1058, in finish_bundle
> yield window.TimestampedValue(self.writer.close(), window.MAX_TIMESTAMP)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/fileio.py", 
> line 601, in close
> self.sink.close(self.temp_handle)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/fileio.py", 
> line 687, in close
> file_handle.close()
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcsio.py", line 
> 617, in close
> self._flush_write_buffer()
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcsio.py", line 
> 647, in _flush_write_buffer
> raise self.upload_thread.last_error  # pylint: disable=raising-bad-type
> HttpError: HttpError accessing 
> :
>  response: <{'status': '404', 'alternate-protocol': '443:quic', 
> 'content-length': '165', 'vary': 'Origin, X-Origin', 'server': 
> 'UploadServer', 'x-guploader-uploadid': 
> 'AEnB2Uq6ZGb_CsrMVxozv6aL48k4OMMiRgYVeVGmJrM-sMQWRGeGMkesOQg5F0W7HZuaqTBog_d4ml-DlIars_ZvJTejdfcbAUr4gswZWVieq82ufc3WR2g',
>  'date': 'Mon, 08 Aug 2016 21:29:46 GMT', 'alt-svc': 'quic=":443"; 
> ma=2592000; v="36,35,34,33,32,31,30"', 'content-type': 'application/json; 
> charset=UTF-8'}>, content <{
>  "error": {
>   "errors": [
>{
> "domain": "global",
> "reason": "notFound",
> "message": "Not Found"
>}
>   ],
>   "code": 404,
>   "message": "Not Found"
>  }
> }



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2078) add BeamSQL feature branch in site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983678#comment-15983678
 ] 

ASF GitHub Bot commented on BEAM-2078:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam-site/pull/224

[BEAM-2078] add BeamSQL feature branch in site



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam-site BEAM-2078

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/224.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #224


commit 676f67533790a25f8c5734b28614bd8b73e51c50
Author: mingmxu 
Date:   2017-04-25T21:25:00Z

add Beam SQL DSL to page 'work-in-progress'




> add BeamSQL feature branch in site
> --
>
> Key: BEAM-2078
> URL: https://issues.apache.org/jira/browse/BEAM-2078
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> Add {{dsl_sql}} feature branch to page 
> 'https://beam.apache.org/contribute/work-in-progress/', to track the status.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #224: [BEAM-2078] add BeamSQL feature branch in site

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam-site/pull/224

[BEAM-2078] add BeamSQL feature branch in site



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam-site BEAM-2078

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/224.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #224


commit 676f67533790a25f8c5734b28614bd8b73e51c50
Author: mingmxu 
Date:   2017-04-25T21:25:00Z

add Beam SQL DSL to page 'work-in-progress'




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2078) add BeamSQL feature branch in site

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2078:
-
Component/s: dsl-sql

> add BeamSQL feature branch in site
> --
>
> Key: BEAM-2078
> URL: https://issues.apache.org/jira/browse/BEAM-2078
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> Add {{dsl_sql}} feature branch to page 
> 'https://beam.apache.org/contribute/work-in-progress/', to track the status.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1327) Replace OutputTimeFn with enum

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983676#comment-15983676
 ] 

ASF GitHub Bot commented on BEAM-1327:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2683

[BEAM-1327] Replace OutputTimeFn UDF with TimestampCombiner enum

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam deprecate-OutputTimeFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2683.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2683


commit 1bcb8333e16102ad57a4524e0dea52aa6d4c92c2
Author: Kenneth Knowles 
Date:   2017-01-27T03:56:06Z

Replace OutputTimeFn UDF with TimestampCombiner enum




> Replace OutputTimeFn with enum
> --
>
> Key: BEAM-1327
> URL: https://issues.apache.org/jira/browse/BEAM-1327
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The class {{OutputTimeFn}} is overkill for a Fn API crossing. There are only 
> three sensible values known: MIN, MAX, EOW. The interface is right for 
> implementing these, but the full class is left over from the days when there 
> was little cost to shipping new kinds of fns. An enum is concise.
> This can be done "mostly" backwards compatibly with legacy adapters in place, 
> but might be less confusing without them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2683: [BEAM-1327] Replace OutputTimeFn UDF with Timestamp...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2683

[BEAM-1327] Replace OutputTimeFn UDF with TimestampCombiner enum

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam deprecate-OutputTimeFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2683.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2683


commit 1bcb8333e16102ad57a4524e0dea52aa6d4c92c2
Author: Kenneth Knowles 
Date:   2017-01-27T03:56:06Z

Replace OutputTimeFn UDF with TimestampCombiner enum




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983674#comment-15983674
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2682

[BEAM-1871] Move ByteStringCoder to sdks/java/extensions/protobuf

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2682


commit 8798d79bc6213ee6690091409148eab1cbfb44cf
Author: Luke Cwik 
Date:   2017-04-25T21:23:11Z

[BEAM-1871] Move ByteStringCoder to sdks/java/extensions/protobuf




> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)