[jira] [Created] (BEAM-2071) AttributeError in dataflow_metrics

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2071:
-

 Summary: AttributeError in dataflow_metrics
 Key: BEAM-2071
 URL: https://issues.apache.org/jira/browse/BEAM-2071
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Pablo Estrada


https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1977/consoleFull

(dataflow job: 
https://pantheon.corp.google.com/dataflow/job/2017-04-24_22_35_11-7339696457089964477?project=apache-beam-testing)

Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount_it_test.py",
 line 61, in test_wordcount_it
wordcount.run(test_pipeline.get_full_options_as_args(**extra_opts))
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/examples/wordcount.py",
 line 116, in run
query_result = result.metrics().query(empty_lines_filter)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py",
 line 152, in query
counters = self._populate_metric_results(response)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py",
 line 124, in _populate_metric_results
attempted = metric['tentative'].scalar.integer_value
AttributeError: 'NoneType' object has no attribute 'integer_value'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #2926

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1977

[Apache Jenkins Server]

See 


Changes:

[altay] Pin apitools to 0.5.8

--
[...truncated 781.39 KB...]
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/SideInput-s16.output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": "write/Write/WriteImpl/FinalizeWrite/SideInput-s16"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "SideInput-s15": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s15"
  }, 
  "SideInput-s16": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s16"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/FinalizeWrite.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/Do"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-04-24_22_35_11-7339696457089964477]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-04-24_22_35_11-7339696457089964477
root: INFO: Job 2017-04-24_22_35_11-7339696457089964477 is in state 
JOB_STATE_RUNNING
root: INFO: None: 2017-04-25T05:35:11.469Z: JOB_MESSAGE_WARNING: 
(65dbd70a540a52c1): Setting the number of workers (1) disables autoscaling for 
this job. If you are trying to cap autoscaling, consider only setting 
max_num_workers. If you want to disable autoscaling altogether, the documented 
way is to explicitly use autoscalingAlgorithm=NONE.
root: INFO: None: 2017-04-25T05:35:12.513Z: JOB_MESSAGE_DETAILED: 
(c01cc15245b434e8): Checking required Cloud APIs are enabled.
root: INFO: None: 2017-04-25T05:35:13.407Z: JOB_MESSAGE_DEBUG: 
(c01cc15245b43b3d): Combiner lifting skipped for step 
write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
root: INFO: None: 2017-04-25T05:35:13.409Z: JOB_MESSAGE_DEBUG: 
(c01cc15245b439f3): Combiner lifting skipped for step group: GroupByKey not 
followed by a combiner.
root: INFO: None: 2017-04-25T05:35:13.411Z: JOB_MESSAGE_DETAILED: 
(c01cc15245b438a9): Expanding GroupByKey operations into optimizable parts.
root: INFO: None: 2017-04-25T05:35:13.414Z: JOB_MESSAGE_DETAILED: 
(c01cc15245b4375f): Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: None: 2017-04-25T05:35:13.421Z: JOB_MESSAGE_DEBUG: 
(c01cc15245b43381): Annotating grap

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3443

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3442

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2068) Upgrade Google-Apitools to latest version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982360#comment-15982360
 ] 

ASF GitHub Bot commented on BEAM-2068:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2667


> Upgrade Google-Apitools to latest version
> -
>
> Key: BEAM-2068
> URL: https://issues.apache.org/jira/browse/BEAM-2068
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Ahmet Altay
>Priority: Minor
>
> In 0.5.9 apitools is pinned to setuptools 18.5 which is really old as the 
> current release is 35.0.1 at the time of creating the issue. Updating to 
> 0.5.9 causes issues for other dependencies so we're going to try to address 
> this upstream first and then upgrade to the latest version in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2667: [BEAM-2068] Pin apitools to 0.5.8

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2667


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Pin apitools to 0.5.8

[altay]

Repository: beam
Updated Branches:
  refs/heads/master e5e1b0894 -> 6f6817283


Pin apitools to 0.5.8


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/664ccb0d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/664ccb0d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/664ccb0d

Branch: refs/heads/master
Commit: 664ccb0d58e608caf07e1fb640b72dcbe02abdc6
Parents: e5e1b08
Author: Sourabh Bajaj 
Authored: Mon Apr 24 15:56:30 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 24 21:48:37 2017 -0700

--
 sdks/python/setup.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/664ccb0d/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index a6efe21..182c6b2 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -103,7 +103,8 @@ REQUIRED_TEST_PACKAGES = [
 ]
 
 GCP_REQUIREMENTS = [
-  'google-apitools>=0.5.8,<1.0.0',
+  # TODO(BEAM-2068): Upgrade google-apitools once the issue is fixed.
+  'google-apitools==0.5.8',
   'proto-google-cloud-datastore-v1==0.90.0',
   'googledatastore==7.0.1',
   # GCP packages required by tests

[2/2] beam git commit: This closes #2667

[altay]

This closes #2667


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6f681728
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6f681728
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6f681728

Branch: refs/heads/master
Commit: 6f6817283a3490a307d8638393314dadb05ab8b6
Parents: e5e1b08 664ccb0
Author: Ahmet Altay 
Authored: Mon Apr 24 21:48:40 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 24 21:48:40 2017 -0700

--
 sdks/python/setup.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982336#comment-15982336
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/2670


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2670: [BEAM-1871] Move Bigquery/Pubsub options to sdks/ja...

[lukecwik]

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/2670


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2663: [BEAM-2066] TextIO & AvroIO no longer validate sche...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2663


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2066) Strip IOChannel validation calls from file based IOs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982334#comment-15982334
 ] 

ASF GitHub Bot commented on BEAM-2066:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2663


> Strip IOChannel validation calls from file based IOs
> 
>
> Key: BEAM-2066
> URL: https://issues.apache.org/jira/browse/BEAM-2066
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>Priority: Minor
>  Labels: backward-incompatible
>
> IOs like TextIO are still calling IOChannelUtils to validate the particular 
> spec they're given. This results in errors since in the new Beam FileSystem 
> world, IOChannelFactory does not know about all the registered IO schemas.
> per [~dhalp...@google.com], we can just strip out this validation code - if 
> there is an issue, we will catch it later in the read process.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: TextIO & AvroIO no longer validate schemas against IOChannelFactory

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 7e6f1b78b -> e5e1b0894


TextIO & AvroIO no longer validate schemas against IOChannelFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cf96ac4e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cf96ac4e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cf96ac4e

Branch: refs/heads/master
Commit: cf96ac4e01ee1294550a3800d2f67f259e82f403
Parents: 7a2fe68
Author: Stephen Sisk 
Authored: Mon Apr 24 12:48:41 2017 -0700
Committer: Stephen Sisk 
Committed: Mon Apr 24 13:23:27 2017 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIO.java | 85 ++-
 .../java/org/apache/beam/sdk/io/TextIO.java | 86 +---
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 24 +-
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 36 ++--
 4 files changed, 34 insertions(+), 197 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cf96ac4e/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index 5cba3f3..6a06972 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -18,14 +18,12 @@
 package org.apache.beam.sdk.io;
 
 import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkState;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Maps;
 import com.google.common.io.BaseEncoding;
 
-import java.io.IOException;
 import java.nio.channels.Channels;
 import java.nio.channels.WritableByteChannel;
 import java.util.Map;
@@ -46,7 +44,6 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
-import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.util.MimeTypes;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
@@ -185,18 +182,6 @@ public class AvroIO {
 }
 
 /**
- * Returns a {@link PTransform} that reads Avro file(s)
- * that has GCS path validation on pipeline creation disabled.
- *
- * This can be useful in the case where the GCS input location does
- * not exist at the pipeline creation time, but is expected to be available
- * at execution time.
- */
-public static Bound withoutValidation() {
-  return new Bound<>(GenericRecord.class).withoutValidation();
-}
-
-/**
  * A {@link PTransform} that reads from an Avro file (or multiple Avro
  * files matching a pattern) and returns a bounded {@link PCollection} 
containing
  * the decoding of each record.
@@ -213,19 +198,16 @@ public class AvroIO {
   /** The schema of the input file. */
   @Nullable
   final Schema schema;
-  /** An option to indicate if input validation is desired. Default is 
true. */
-  final boolean validate;
 
   Bound(Class type) {
-this(null, null, type, null, true);
+this(null, null, type, null);
   }
 
-  Bound(String name, String filepattern, Class type, Schema schema, 
boolean validate) {
+  Bound(String name, String filepattern, Class type, Schema schema) {
 super(name);
 this.filepattern = filepattern;
 this.type = type;
 this.schema = schema;
-this.validate = validate;
   }
 
   /**
@@ -237,7 +219,7 @@ public class AvroIO {
* Does not modify this object.
*/
   public Bound from(String filepattern) {
-return new Bound<>(name, filepattern, type, schema, validate);
+return new Bound<>(name, filepattern, type, schema);
   }
 
   /**
@@ -251,7 +233,7 @@ public class AvroIO {
* the resulting PCollection
*/
   public  Bound withSchema(Class type) {
-return new Bound<>(name, filepattern, type, 
ReflectData.get().getSchema(type), validate);
+return new Bound<>(name, filepattern, type, 
ReflectData.get().getSchema(type));
   }
 
   /**
@@ -261,7 +243,7 @@ public class AvroIO {
* Does not modify this object.
*/
   public Bound withSchema(Schema schema) {
-return new Bound<>(name, filepattern, GenericRecord.class, schema, 
validate);
+return new Bound<>(name, filepattern, GenericRecord.class, schema);
   }
 
   /**
@@ -275,20 +257,6 @@ public class AvroIO {

[2/2] beam git commit: This closes #2663: TextIO & AvroIO no longer validate schemas against IOChannelFactory

[kenn]

This closes #2663: TextIO & AvroIO no longer validate schemas against 
IOChannelFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e5e1b089
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e5e1b089
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e5e1b089

Branch: refs/heads/master
Commit: e5e1b089428e3c88c0058135383e4dae2b768d3a
Parents: 7e6f1b7 cf96ac4
Author: Kenneth Knowles 
Authored: Mon Apr 24 19:47:18 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Apr 24 19:47:18 2017 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIO.java | 85 ++-
 .../java/org/apache/beam/sdk/io/TextIO.java | 86 +---
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 24 +-
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 36 ++--
 4 files changed, 34 insertions(+), 197 deletions(-)
--

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982303#comment-15982303
 ] 

Jean-Baptiste Onofré commented on BEAM-2005:


+1 (sorry I'm late on this, I was busy with CassandraIO and Spark 2 support). 
Gonna help you in the coming days.

> Add a Hadoop FileSystem implementation of Beam's FileSystem
> ---
>
> Key: BEAM-2005
> URL: https://issues.apache.org/jira/browse/BEAM-2005
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Beam's FileSystem creates an abstraction for reading from files in many 
> different places. 
> We should add a Hadoop FileSystem implementation 
> (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html)
>  - that would enable us to read from any file system that implements 
> FileSystem (including HDFS, azure, s3, etc..)
> I'm investigating this now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982276#comment-15982276
 ] 

Ahmet Altay commented on BEAM-2025:
---

We may eventually break Python code including examples across the repository. 
However this is not happening now due to the differences in the native build 
systems used for Java nad Python.

Using symlinks is also a not good solution because they do not work in all 
platforms. We could a a README file in the root examples directory.

(I am not sure how this is related to the website issue though.)

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Hadar Hod
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3441

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982270#comment-15982270
 ] 

Melissa Pashniak edited comment on BEAM-2025 at 4/25/17 2:00 AM:
-

good point. [~altay] any thoughts on moving the python examples directory to 
the main examples directory, so they are alongside the java ones?


was (Author: melap):
good point. [~altay] any thoughts on moving the python examples directory to 
the main examples directory, so they were alongside the java ones?

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Hadar Hod
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982270#comment-15982270
 ] 

Melissa Pashniak commented on BEAM-2025:


good point. [~altay] any thoughts on moving the python examples directory to 
the main examples directory, so they were alongside the java ones?

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Hadar Hod
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #2924

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Mitar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982249#comment-15982249
 ] 

Mitar commented on BEAM-2025:
-

Ah, this is again an example that the Python code is all under one directory, 
but Java code is all around the repository. Maybe putting a README or symlnks 
into https://github.com/apache/beam/tree/master/examples to point to examples 
in other languages could be helpful?

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Hadar Hod
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3440

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn

[Reuven Lax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuven Lax updated BEAM-1831:
-

This is not double checked locking - the extra check is only there as a
performance optimization. The data structure itself is thread safe.




> Checking of containment in createdTables may have race condition in 
> StreamingWriteFn
> 
>
> Key: BEAM-1831
> URL: https://issues.apache.org/jira/browse/BEAM-1831
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
>   public TableReference getOrCreateTable(BigQueryOptions options, String 
> tableSpec)
>   throws InterruptedException, IOException {
> TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
> if (createDisposition != createDisposition.CREATE_NEVER
> && !createdTables.contains(tableSpec)) {
>   synchronized (createdTables) {
> // Another thread may have succeeded in creating the table in the 
> meanwhile, so
> // check again. This check isn't needed for correctness, but we add 
> it to prevent
> // every thread from attempting a create and overwhelming our 
> BigQuery quota.
> DatasetService datasetService = bqServices.getDatasetService(options);
> if (!createdTables.contains(tableSpec)) {
> {code}
> The first createdTables.contains() check is outside synchronized block.
> At least createdTables should be declared volatile for the double checked 
> locking to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982215#comment-15982215
 ] 

ASF GitHub Bot commented on BEAM-2021:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2612


> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2612: [BEAM-2021] Stop Extending AtomicCoder

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2612


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Stop Extending AtomicCoder

[tgroh]

http://git-wip-us.apache.org/repos/asf/beam/blob/7ad9efc7/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseResultCoder.java
--
diff --git 
a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseResultCoder.java
 
b/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseResultCoder.java
index 94f324a..0004d03 100644
--- 
a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseResultCoder.java
+++ 
b/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseResultCoder.java
@@ -21,9 +21,8 @@ import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
 import java.io.Serializable;
-
-import org.apache.beam.sdk.coders.AtomicCoder;
 import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
 import org.apache.hadoop.hbase.protobuf.generated.ClientProtos;
@@ -32,7 +31,7 @@ import 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos;
  * A {@link Coder} that serializes and deserializes the {@link Result} objects 
using {@link
  * ProtobufUtil}.
  */
-class HBaseResultCoder extends AtomicCoder implements Serializable {
+class HBaseResultCoder extends CustomCoder implements Serializable {
   private static final HBaseResultCoder INSTANCE = new HBaseResultCoder();
 
   private HBaseResultCoder() {}

http://git-wip-us.apache.org/repos/asf/beam/blob/7ad9efc7/sdks/java/io/kafka/pom.xml
--
diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index 785699c..2ae940d 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -83,11 +83,6 @@
 
 
 
-  com.fasterxml.jackson.core
-  jackson-annotations
-
-
-
   com.google.guava
   guava
 

http://git-wip-us.apache.org/repos/asf/beam/blob/7ad9efc7/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index fbd96eb..68efb9a 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -53,11 +53,11 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.coders.AtomicCoder;
 import org.apache.beam.sdk.coders.AvroCoder;
 import org.apache.beam.sdk.coders.ByteArrayCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.io.Read.Unbounded;
 import org.apache.beam.sdk.io.UnboundedSource;
@@ -1321,7 +1321,7 @@ public class KafkaIO {
 }
   }
 
-  private static class NullOnlyCoder extends AtomicCoder {
+  private static class NullOnlyCoder extends CustomCoder {
 @Override
 public void encode(T value, OutputStream outStream, Context context) {
   checkArgument(value == null, "Can only encode nulls");

http://git-wip-us.apache.org/repos/asf/beam/blob/7ad9efc7/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
index 25ef7df..160e8ce 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
@@ -17,28 +17,23 @@
  */
 package org.apache.beam.sdk.io.kafka;
 
-import com.fasterxml.jackson.annotation.JsonCreator;
-import com.fasterxml.jackson.annotation.JsonProperty;
-
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
 import java.util.List;
-
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.StandardCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.util.PropertyNames;
 import org.apache.beam.sdk.values.KV;
 
 /**
  * {@link Coder} for {@link KafkaRecord}.
  */
-public class KafkaRecordCoder extends StandardCoder> {
+public class Kaf

[1/3] beam git commit: This closes #2612

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 02e60a5c5 -> 7e6f1b78b


This closes #2612


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7e6f1b78
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7e6f1b78
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7e6f1b78

Branch: refs/heads/master
Commit: 7e6f1b78bb7962d117a847fa75556710ad075799
Parents: 02e60a5 7ad9efc
Author: Thomas Groh 
Authored: Mon Apr 24 16:47:29 2017 -0700
Committer: Thomas Groh 
Committed: Mon Apr 24 16:47:29 2017 -0700

--
 .../direct/CloningBundleFactoryTest.java| 10 ++---
 .../beam/runners/direct/DirectRunnerTest.java   |  5 +--
 .../UnboundedReadEvaluatorFactoryTest.java  |  4 +-
 .../runners/dataflow/internal/IsmFormat.java| 41 +++-
 .../runners/dataflow/util/RandomAccessData.java | 14 +--
 .../apache/beam/sdk/coders/BigDecimalCoder.java | 31 +--
 .../beam/sdk/coders/BigEndianIntegerCoder.java  | 12 --
 .../beam/sdk/coders/BigEndianLongCoder.java | 10 -
 .../apache/beam/sdk/coders/BigIntegerCoder.java | 23 +++
 .../apache/beam/sdk/coders/ByteArrayCoder.java  | 18 -
 .../org/apache/beam/sdk/coders/ByteCoder.java   |  9 +++--
 .../apache/beam/sdk/coders/ByteStringCoder.java |  7 ++--
 .../org/apache/beam/sdk/coders/CustomCoder.java | 22 ++-
 .../org/apache/beam/sdk/coders/DoubleCoder.java |  9 +++--
 .../apache/beam/sdk/coders/DurationCoder.java   | 19 +
 .../apache/beam/sdk/coders/InstantCoder.java| 25 +++-
 .../beam/sdk/coders/SerializableCoder.java  |  2 +-
 .../apache/beam/sdk/coders/StringUtf8Coder.java |  7 ++--
 .../beam/sdk/coders/TextualIntegerCoder.java| 14 +--
 .../org/apache/beam/sdk/coders/VarIntCoder.java | 12 --
 .../apache/beam/sdk/coders/VarLongCoder.java| 22 +--
 .../org/apache/beam/sdk/coders/VoidCoder.java   |  5 ++-
 .../org/apache/beam/sdk/io/FileBasedSink.java   |  9 +
 .../org/apache/beam/sdk/transforms/Mean.java| 10 -
 .../sdk/transforms/windowing/GlobalWindow.java  | 17 +++-
 .../transforms/windowing/IntervalWindow.java| 20 --
 .../beam/sdk/transforms/windowing/PaneInfo.java |  7 +++-
 .../org/apache/beam/sdk/util/BitSetCoder.java   | 13 +++
 .../apache/beam/sdk/testing/PAssertTest.java|  4 +-
 .../sdk/testing/SerializableMatchersTest.java   |  4 +-
 .../beam/sdk/testing/WindowSupplierTest.java|  4 +-
 .../apache/beam/sdk/transforms/CreateTest.java  |  3 +-
 .../beam/sdk/transforms/GroupByKeyTest.java |  9 +++--
 .../apache/beam/sdk/transforms/ParDoTest.java   |  6 ++-
 .../apache/beam/sdk/util/CoderUtilsTest.java|  4 +-
 .../sdk/extensions/protobuf/ProtoCoder.java |  4 +-
 .../io/gcp/bigquery/TableDestinationCoder.java  |  6 +--
 .../sdk/io/gcp/bigquery/TableRowInfoCoder.java  |  6 +--
 .../sdk/io/gcp/bigquery/TableRowJsonCoder.java  | 12 --
 .../io/gcp/bigquery/WriteBundlesToFiles.java|  7 ++--
 .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java  |  4 +-
 .../io/gcp/pubsub/PubsubUnboundedSource.java|  4 +-
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 10 +++--
 .../beam/sdk/io/hbase/HBaseMutationCoder.java   |  5 +--
 .../beam/sdk/io/hbase/HBaseResultCoder.java |  5 +--
 sdks/java/io/kafka/pom.xml  |  5 ---
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  4 +-
 .../beam/sdk/io/kafka/KafkaRecordCoder.java | 16 +---
 .../beam/sdk/io/kinesis/KinesisRecordCoder.java |  4 +-
 sdks/java/io/xml/pom.xml|  5 ---
 .../org/apache/beam/sdk/io/xml/JAXBCoder.java   | 39 ++-
 51 files changed, 342 insertions(+), 225 deletions(-)
--

[jira] [Assigned] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak reassigned BEAM-2025:
--

Assignee: Hadar Hod  (was: Melissa Pashniak)

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Hadar Hod
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: Stop Extending AtomicCoder

[tgroh]

Stop Extending AtomicCoder


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7ad9efc7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7ad9efc7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7ad9efc7

Branch: refs/heads/master
Commit: 7ad9efc7a678f042d67cd90536f21aca4736e4c3
Parents: 02e60a5
Author: Thomas Groh 
Authored: Thu Apr 20 09:39:12 2017 -0700
Committer: Thomas Groh 
Committed: Mon Apr 24 16:47:29 2017 -0700

--
 .../direct/CloningBundleFactoryTest.java| 10 ++---
 .../beam/runners/direct/DirectRunnerTest.java   |  5 +--
 .../UnboundedReadEvaluatorFactoryTest.java  |  4 +-
 .../runners/dataflow/internal/IsmFormat.java| 41 +++-
 .../runners/dataflow/util/RandomAccessData.java | 14 +--
 .../apache/beam/sdk/coders/BigDecimalCoder.java | 31 +--
 .../beam/sdk/coders/BigEndianIntegerCoder.java  | 12 --
 .../beam/sdk/coders/BigEndianLongCoder.java | 10 -
 .../apache/beam/sdk/coders/BigIntegerCoder.java | 23 +++
 .../apache/beam/sdk/coders/ByteArrayCoder.java  | 18 -
 .../org/apache/beam/sdk/coders/ByteCoder.java   |  9 +++--
 .../apache/beam/sdk/coders/ByteStringCoder.java |  7 ++--
 .../org/apache/beam/sdk/coders/CustomCoder.java | 22 ++-
 .../org/apache/beam/sdk/coders/DoubleCoder.java |  9 +++--
 .../apache/beam/sdk/coders/DurationCoder.java   | 19 +
 .../apache/beam/sdk/coders/InstantCoder.java| 25 +++-
 .../beam/sdk/coders/SerializableCoder.java  |  2 +-
 .../apache/beam/sdk/coders/StringUtf8Coder.java |  7 ++--
 .../beam/sdk/coders/TextualIntegerCoder.java| 14 +--
 .../org/apache/beam/sdk/coders/VarIntCoder.java | 12 --
 .../apache/beam/sdk/coders/VarLongCoder.java| 22 +--
 .../org/apache/beam/sdk/coders/VoidCoder.java   |  5 ++-
 .../org/apache/beam/sdk/io/FileBasedSink.java   |  9 +
 .../org/apache/beam/sdk/transforms/Mean.java| 10 -
 .../sdk/transforms/windowing/GlobalWindow.java  | 17 +++-
 .../transforms/windowing/IntervalWindow.java| 20 --
 .../beam/sdk/transforms/windowing/PaneInfo.java |  7 +++-
 .../org/apache/beam/sdk/util/BitSetCoder.java   | 13 +++
 .../apache/beam/sdk/testing/PAssertTest.java|  4 +-
 .../sdk/testing/SerializableMatchersTest.java   |  4 +-
 .../beam/sdk/testing/WindowSupplierTest.java|  4 +-
 .../apache/beam/sdk/transforms/CreateTest.java  |  3 +-
 .../beam/sdk/transforms/GroupByKeyTest.java |  9 +++--
 .../apache/beam/sdk/transforms/ParDoTest.java   |  6 ++-
 .../apache/beam/sdk/util/CoderUtilsTest.java|  4 +-
 .../sdk/extensions/protobuf/ProtoCoder.java |  4 +-
 .../io/gcp/bigquery/TableDestinationCoder.java  |  6 +--
 .../sdk/io/gcp/bigquery/TableRowInfoCoder.java  |  6 +--
 .../sdk/io/gcp/bigquery/TableRowJsonCoder.java  | 12 --
 .../io/gcp/bigquery/WriteBundlesToFiles.java|  7 ++--
 .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java  |  4 +-
 .../io/gcp/pubsub/PubsubUnboundedSource.java|  4 +-
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 10 +++--
 .../beam/sdk/io/hbase/HBaseMutationCoder.java   |  5 +--
 .../beam/sdk/io/hbase/HBaseResultCoder.java |  5 +--
 sdks/java/io/kafka/pom.xml  |  5 ---
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  4 +-
 .../beam/sdk/io/kafka/KafkaRecordCoder.java | 16 +---
 .../beam/sdk/io/kinesis/KinesisRecordCoder.java |  4 +-
 sdks/java/io/xml/pom.xml|  5 ---
 .../org/apache/beam/sdk/io/xml/JAXBCoder.java   | 39 ++-
 51 files changed, 342 insertions(+), 225 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7ad9efc7/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
index 3d14a12..c6054b6 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
@@ -33,8 +33,8 @@ import java.io.InputStream;
 import java.io.OutputStream;
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
-import org.apache.beam.sdk.coders.AtomicCoder;
 import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.coders.VarIntCoder;
@@ -175,7 +175,7 @@ public c

[jira] [Assigned] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1831:
--

Assignee: Reuven Lax  (was: Davor Bonaci)

> Checking of containment in createdTables may have race condition in 
> StreamingWriteFn
> 
>
> Key: BEAM-1831
> URL: https://issues.apache.org/jira/browse/BEAM-1831
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
>   public TableReference getOrCreateTable(BigQueryOptions options, String 
> tableSpec)
>   throws InterruptedException, IOException {
> TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
> if (createDisposition != createDisposition.CREATE_NEVER
> && !createdTables.contains(tableSpec)) {
>   synchronized (createdTables) {
> // Another thread may have succeeded in creating the table in the 
> meanwhile, so
> // check again. This check isn't needed for correctness, but we add 
> it to prevent
> // every thread from attempting a create and overwhelming our 
> BigQuery quota.
> DatasetService datasetService = bqServices.getDatasetService(options);
> if (!createdTables.contains(tableSpec)) {
> {code}
> The first createdTables.contains() check is outside synchronized block.
> At least createdTables should be declared volatile for the double checked 
> locking to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1897) Remove Sink

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1897:
--

Assignee: Reuven Lax  (was: Davor Bonaci)

> Remove Sink
> ---
>
> Key: BEAM-1897
> URL: https://issues.apache.org/jira/browse/BEAM-1897
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>
> Sink is really only used for file sinks. We should remove the abstract base 
> class, and just keep around FileBasedSink for file sinks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2039) number the chapters in page 'Beam Programming Guide'

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2039:
--

Assignee: Melissa Pashniak  (was: Davor Bonaci)

> number the chapters in page 'Beam Programming Guide'
> 
>
> Key: BEAM-2039
> URL: https://issues.apache.org/jira/browse/BEAM-2039
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Xu Mingmin
>Assignee: Melissa Pashniak
>Priority: Minor
>
> Now more and more content are added to page 'Beam Programming 
> Guide'(https://beam.apache.org/documentation/programming-guide/). Not easy to 
> read to me, a well numbered table content could give some help.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1897) Remove Sink

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982211#comment-15982211
 ] 

Davor Bonaci commented on BEAM-1897:


Done?

> Remove Sink
> ---
>
> Key: BEAM-1897
> URL: https://issues.apache.org/jira/browse/BEAM-1897
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>
> Sink is really only used for file sinks. We should remove the abstract base 
> class, and just keep around FileBasedSink for file sinks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982212#comment-15982212
 ] 

Melissa Pashniak commented on BEAM-2025:


I poked around and It looks like the links are being fixed as part of the 
changes in [apache/beam-site#222|https://github.com/apache/beam-site/pull/222]


> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Melissa Pashniak
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2037) PTransform identifier(name) documentation in Java SDK.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2037:
--

Assignee: Ben Chambers  (was: Davor Bonaci)

> PTransform identifier(name) documentation in Java SDK.
> --
>
> Key: BEAM-2037
> URL: https://issues.apache.org/jira/browse/BEAM-2037
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Ben Chambers
>
> Add/improve PTransform identifier(name) documentation in Java SDK.
> Specifically address this in Javadoc of {{Pipeline#apply}}, {{PBegin#apply}}, 
> {{PCollection#apply}}, {{Metrics}}, {{MetricsResult}}, {{MetricsFilter}}.
> Documentation around Metrics {{step}} (Which corresponds to the PTransform 
> application identifier) in the classes mentioned above is a bit confusing 
> currently, this should be reworded in correlation with the changes 
> {{Pipeline#apply}}, {{PBegin#apply}}, {{PCollection#apply}}.
> See parent issue BEAM-2035 for more information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2036) PTransform identifier(name) documentation in website.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2036:
--

Assignee: Melissa Pashniak  (was: Davor Bonaci)

> PTransform identifier(name) documentation in website.
> -
>
> Key: BEAM-2036
> URL: https://issues.apache.org/jira/browse/BEAM-2036
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Aviem Zur
>Assignee: Melissa Pashniak
>
> Add PTransform identifier(name) documentation to the website.
> Section 'Applying transforms' - 
> https://beam.apache.org/documentation/programming-guide/#transforms
> See parent issue BEAM-2035 for more information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2046) Better API for querying metrics

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2046:
--

Assignee: Ben Chambers  (was: Davor Bonaci)

> Better API for querying metrics
> ---
>
> Key: BEAM-2046
> URL: https://issues.apache.org/jira/browse/BEAM-2046
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: First stable release
>Reporter: Malo Denielou
>Assignee: Ben Chambers
>
> I just want to read a metric :-).
> Can we have a better API than this:
> Iterable> seenMetrics = job.metrics()
> .queryMetrics(
> MetricsFilter.builder()
> .addNameFilter(MetricNameFilter.named("XX", "YY"))
> .build())
> .counters();
> long seenSentinels = Iterables.isEmpty(seenMetrics) ? 0
> : Iterables.getFirst(seenMetrics, null).committed();
> This is very clunky :-P.
> Ideally I'd like to read a metric with a name, and provide a default value if 
> the metric is not there.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2052) Windowed file sinks should support dynamic sharding

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2052:
--

Assignee: Reuven Lax  (was: Davor Bonaci)

> Windowed file sinks should support dynamic sharding
> ---
>
> Key: BEAM-2052
> URL: https://issues.apache.org/jira/browse/BEAM-2052
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>
> Currently windowed file sinks (WriteFiles and FileBasedSink) require 
> withNumShards to be set explicitly. We should remove this requirement, and 
> allow dynamic output.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982184#comment-15982184
 ] 

Davor Bonaci commented on BEAM-2025:


Thanks [~mitar] for this report. [~melap], would you mind taking a look?

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Melissa Pashniak
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2025) WordCount Example lacks a link to Python full source code

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2025:
--

Assignee: Melissa Pashniak  (was: Davor Bonaci)

> WordCount Example lacks a link to Python full source code
> -
>
> Key: BEAM-2025
> URL: https://issues.apache.org/jira/browse/BEAM-2025
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Mitar
>Assignee: Melissa Pashniak
>
> I am reading WorldCount Example 
> (https://beam.apache.org/get-started/wordcount-example/) but despite me 
> switching to Python SDK, the "To run this example, follow the instructions in 
> the Beam Examples README. To view the full code, see MinimalWordCount." text 
> points only to Java example. I could not find whole code in Python for this 
> example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2062) EventHandler jaxb unmarshaller should be optional

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982181#comment-15982181
 ] 

Davor Bonaci commented on BEAM-2062:


Indeed; thanks for the report [~dgouyette].

If you'd be interested in proposing a pull request to improve this, it would be 
much appreciated! This is a great issue for someone to get started contributing.

> EventHandler jaxb unmarshaller should be optional
> -
>
> Key: BEAM-2062
> URL: https://issues.apache.org/jira/browse/BEAM-2062
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>  Labels: newbie, starter
>
> Sometimes you don't want a runtime exception when validation fails (by 
> example additionals fields).
> That would be great if this behaviour could be disabled
> https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlSource.java#L146



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2062) EventHandler jaxb unmarshaller should be optional

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-2062:
---
Labels: newbie starter  (was: )

> EventHandler jaxb unmarshaller should be optional
> -
>
> Key: BEAM-2062
> URL: https://issues.apache.org/jira/browse/BEAM-2062
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>  Labels: newbie, starter
>
> Sometimes you don't want a runtime exception when validation fails (by 
> example additionals fields).
> That would be great if this behaviour could be disabled
> https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlSource.java#L146



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2062) EventHandler jaxb unmarshaller should be optional

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2062:
--

Assignee: (was: Davor Bonaci)

> EventHandler jaxb unmarshaller should be optional
> -
>
> Key: BEAM-2062
> URL: https://issues.apache.org/jira/browse/BEAM-2062
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>
> Sometimes you don't want a runtime exception when validation fails (by 
> example additionals fields).
> That would be great if this behaviour could be disabled
> https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlSource.java#L146



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2062) EventHandler jaxb unmarshaller should be optional

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-2062:
---
Component/s: (was: sdk-java-core)
 sdk-java-extensions

> EventHandler jaxb unmarshaller should be optional
> -
>
> Key: BEAM-2062
> URL: https://issues.apache.org/jira/browse/BEAM-2062
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>
> Sometimes you don't want a runtime exception when validation fails (by 
> example additionals fields).
> That would be great if this behaviour could be disabled
> https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlSource.java#L146



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982174#comment-15982174
 ] 

Stephen Sisk commented on BEAM-2005:


also - all the sub-tasks on this are assigned to me since we don't want FSR 
items unassigned, but I'm only currently working on one of them. Let me know if 
you're interested on working on this with me.

> Add a Hadoop FileSystem implementation of Beam's FileSystem
> ---
>
> Key: BEAM-2005
> URL: https://issues.apache.org/jira/browse/BEAM-2005
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Beam's FileSystem creates an abstraction for reading from files in many 
> different places. 
> We should add a Hadoop FileSystem implementation 
> (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html)
>  - that would enable us to read from any file system that implements 
> FileSystem (including HDFS, azure, s3, etc..)
> I'm investigating this now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2033) Implement ResourceIds for HadoopFileSystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982169#comment-15982169
 ] 

ASF GitHub Bot commented on BEAM-2033:
--

GitHub user ssisk opened a pull request:

https://github.com/apache/beam/pull/2671

[BEAM-2033] Add HadoopResourceId

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
merging person: I believe we'll want to do the implementation of this in a 
feature branch - this'd be a good time to create that feature branch. 

R: @tgroh 

It's worth noting that GcsResourceId spends a lot of time on worrying about 
slashes - that's not a thing we'll need for this since Hadoop's Path does a 
good job of cleaning it up, so all that's handled at a lower level.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssisk/beam hfs-pre-req

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2671.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2671


commit 405cc1471c4fb0668c30fe6ffa232bd21c31c4ee
Author: Stephen Sisk 
Date:   2017-04-22T00:23:55Z

Add HadoopResourceId




> Implement ResourceIds for HadoopFileSystem
> --
>
> Key: BEAM-2033
> URL: https://issues.apache.org/jira/browse/BEAM-2033
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> There's an empty class for HadoopResourceId already - we will implement it.
> This blocks most of the other implementation since we can't do much without 
> ResourceIds



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2671: [BEAM-2033] Add HadoopResourceId

[ssisk]

GitHub user ssisk opened a pull request:

https://github.com/apache/beam/pull/2671

[BEAM-2033] Add HadoopResourceId

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
merging person: I believe we'll want to do the implementation of this in a 
feature branch - this'd be a good time to create that feature branch. 

R: @tgroh 

It's worth noting that GcsResourceId spends a lot of time on worrying about 
slashes - that's not a thing we'll need for this since Hadoop's Path does a 
good job of cleaning it up, so all that's handled at a lower level.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssisk/beam hfs-pre-req

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2671.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2671


commit 405cc1471c4fb0668c30fe6ffa232bd21c31c4ee
Author: Stephen Sisk 
Date:   2017-04-22T00:23:55Z

Add HadoopResourceId




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1767

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2070) Implement match for HadoopFileSystem

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-2070:
--

 Summary: Implement match for HadoopFileSystem
 Key: BEAM-2070
 URL: https://issues.apache.org/jira/browse/BEAM-2070
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-extensions
Reporter: Stephen Sisk
Assignee: Stephen Sisk
 Fix For: First stable release






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982158#comment-15982158
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2670

[BEAM-1871] Move Bigquery/Pubsub options to 
sdks/java/io/google-cloud-platform

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2670.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2670


commit dc29ba500e3631991a74ee627a44c1adc59bbba7
Author: Luke Cwik 
Date:   2017-04-24T22:58:47Z

[BEAM-1871] Move Bigquery/Pubsub options to 
sdks/java/io/google-cloud-platform

commit 161a1009e005c4770951e413ce07810bb3e23c1b
Author: Luke Cwik 
Date:   2017-04-25T00:10:56Z

fixup! Fix classpath ordering.




> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2670: [BEAM-1871] Move Bigquery/Pubsub options to sdks/ja...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2670

[BEAM-1871] Move Bigquery/Pubsub options to 
sdks/java/io/google-cloud-platform

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2670.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2670


commit dc29ba500e3631991a74ee627a44c1adc59bbba7
Author: Luke Cwik 
Date:   2017-04-24T22:58:47Z

[BEAM-1871] Move Bigquery/Pubsub options to 
sdks/java/io/google-cloud-platform

commit 161a1009e005c4770951e413ce07810bb3e23c1b
Author: Luke Cwik 
Date:   2017-04-25T00:10:56Z

fixup! Fix classpath ordering.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1411) Unable to downgrade to lower guava version after upgrade to beam-0.5

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982146#comment-15982146
 ] 

Davor Bonaci commented on BEAM-1411:


Thanks [~pawelszc...@gmail.com]; much appreciated. Unfortunately, many users 
will be impacted by this change... not sure what we can do.

> Unable to downgrade to lower guava version after upgrade to beam-0.5
> 
>
> Key: BEAM-1411
> URL: https://issues.apache.org/jira/browse/BEAM-1411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Michael Luckey
>Assignee: Davor Bonaci
> Fix For: First stable release
>
>
> While testing upgrading to 0.5 version of beam we ran into NoSuchMethodErrors
> {noformat}
> java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>  at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$SerializableSplit.(HDFSFileSource.java:473)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:81)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:78)
>  at 
> com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451)
>  at java.util.AbstractList$Itr.next(AbstractList.java:358)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:202)
>  at 
> org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:65)
>  at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:168)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:329)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:71)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:178)
>  at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:258)
> {noformat}
> This seems to be caused by the [HDFS 
> IO|https://github.com/apache/beam/tree/master/sdks/java/io/hdfs] components 
> guava dependency not being shaded - in contrast to the core components - and 
> revealed by the [recent update to 
> guava-20.0|https://github.com/apache/beam/commit/0b4b2becb45b9f637ba31f599ebe8be0331bd633]
>  and the therefore incorporated api changes made by [overloading 
> methods|https://github.com/google/guava/commit/892e323fca32945cdfb25395ca6e346dd0fffa5b#diff-fe7358934fa6eba23c2791eb40cec030]
> Did not check, whether more components are affected to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1981) Serialization error with TimerInternals in ApexGroupByKeyOperator

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982135#comment-15982135
 ] 

Thomas Weise commented on BEAM-1981:


Definitely. I also want to move the Apex dependency to 3.6.0 if that release is 
out in time.

> Serialization error with TimerInternals in ApexGroupByKeyOperator
> -
>
> Key: BEAM-1981
> URL: https://issues.apache.org/jira/browse/BEAM-1981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Daniel Halperin
>Assignee: Thomas Weise
> Fix For: First stable release
>
>
> Logs below. We tried switching to Java serialization, but that didn't work. 
> We made the field transient (which is broken but let us make progress) and 
> that did.
> Stack trace
> {code}
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.beam.runners.core.AutoValue_TimerInternals_TimerData
> Serialization trace:
> activeTimers 
> (org.apache.beam.runners.apex.translation.operators.ApexGroupByKeyOperator)
>   at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>   at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>   at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
>   at 
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:192)
>   at 
> com.datatorrent.common.util.FSStorageAgent.load(FSStorageAgent.java:137)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.deployNodes(StreamingContainer.java:914)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:862)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:820)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
>  context: 
> PTContainer[id=6(container_1492195730173_0001_01_12),state=ACTIVE,operators=[
> {code}
> Larger logs with more scope:
> {code}
> 2017-04-14 18:56:49,961 INFO com.datatorrent.stram.StreamingAppMaster: Master 
> starting with classpath: 
> ./portability-demo-bundled-apex.jar:/etc/hadoop/conf:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-common-2.7.3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.3.jar:/usr/lib/hadoop/hadoop-nfs-2.7.3.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-common-2.7.3.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/snappy-java-1.0.5.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/bigquery-connector-0.10.1-hadoop2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/avro-1.7.7.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/gcs-connector-1.

[jira] [Resolved] (BEAM-1830) add 'withTopic()' api to KafkaIO Reader

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi resolved BEAM-1830.

Resolution: Fixed

> add 'withTopic()' api to KafkaIO Reader
> ---
>
> Key: BEAM-1830
> URL: https://issues.apache.org/jira/browse/BEAM-1830
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: First stable release
>
>
> Most of instances of KafkaIO readers consume just one topic. The existing 
> method {{withTopics(List topics)}} forces the users to make a list 
> containing single topic.
> It would be simpler to add {{withTopic(String topic)}} method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Properly Declare Protobuf extension dependencies

[lcwik]

Properly Declare Protobuf extension dependencies

This closes #2669


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/02e60a5c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/02e60a5c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/02e60a5c

Branch: refs/heads/master
Commit: 02e60a5c5b05604b6c94198b1338cad09228c92d
Parents: 57535f3 c443986
Author: Luke Cwik 
Authored: Mon Apr 24 16:46:16 2017 -0700
Committer: Luke Cwik 
Committed: Mon Apr 24 16:46:16 2017 -0700

--
 pom.xml| 6 ++
 sdks/java/io/google-cloud-platform/pom.xml | 5 +
 .../java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java   | 2 +-
 .../org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java| 2 +-
 4 files changed, 13 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Properly Declare Protobuf extension dependencies

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 57535f35e -> 02e60a5c5


Properly Declare Protobuf extension dependencies

Import the correct ProtoCoder in BigtableIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c4439862
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c4439862
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c4439862

Branch: refs/heads/master
Commit: c4439862efc723237c7005952f8b8bdb9075d8e7
Parents: 57535f3
Author: Thomas Groh 
Authored: Mon Apr 24 16:05:28 2017 -0700
Committer: Thomas Groh 
Committed: Mon Apr 24 16:10:38 2017 -0700

--
 pom.xml| 6 ++
 sdks/java/io/google-cloud-platform/pom.xml | 5 +
 .../java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java   | 2 +-
 .../org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java| 2 +-
 4 files changed, 13 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c4439862/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8723ed9..aec434e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -358,6 +358,12 @@
 
   
 org.apache.beam
+beam-sdks-java-extensions-protobuf
+${project.version}
+  
+
+  
+org.apache.beam
 beam-sdks-java-extensions-gcp-core
 ${project.version}
   

http://git-wip-us.apache.org/repos/asf/beam/blob/c4439862/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 3778a63..7fa3842 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -72,6 +72,11 @@
 
 
 
+  org.apache.beam
+  beam-sdks-java-extensions-protobuf
+
+
+
   com.fasterxml.jackson.core
   jackson-annotations
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c4439862/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 28f8878..7bba1a6 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -44,7 +44,7 @@ import java.util.concurrent.ConcurrentLinkedQueue;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.protobuf.ProtoCoder;
+import org.apache.beam.sdk.extensions.protobuf.ProtoCoder;
 import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
 import org.apache.beam.sdk.io.range.ByteKey;

http://git-wip-us.apache.org/repos/asf/beam/blob/c4439862/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
index 713a9a9..cf96b65 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
@@ -78,7 +78,7 @@ import org.apache.beam.sdk.coders.ByteStringCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.IterableCoder;
 import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.protobuf.ProtoCoder;
+import org.apache.beam.sdk.extensions.protobuf.ProtoCoder;
 import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
 import org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.BigtableSource;
 import org.apache.beam.sdk.io.range.ByteKey;

[GitHub] beam pull request #2669: Properly Declare Protobuf extension dependencies

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2669


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2069) Remove ResourceId.getCurrentDirectory()?

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982123#comment-15982123
 ] 

Stephen Sisk commented on BEAM-2069:


It may be worth considering moving directory over onto the FileSystem 
implementation - it's not clear that a wrapper around a string (which is what 
resourceId is) is ever going to be able to answer this question, and it's how 
hadoop implements this. (org.apache.hadoop...FileSystem has an isDirectory() 
method)

It was pointed out to me that I could inject hadoop's FileSystem into the 
ResourceId and use isDirectory there to solve this problem for hadoop, so there 
is likely a solution for hadoop.

> Remove ResourceId.getCurrentDirectory()?
> 
>
> Key: BEAM-2069
> URL: https://issues.apache.org/jira/browse/BEAM-2069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Beam ResourceId currently has a getCurrentDirectory method that returns the 
> current resource id if it's a directory, or the parent directory if it's a 
> directory.
> To implement this you need to know whether or not a particular path is a 
> directory or not.
> I'm trying to implement the Hadoop ResourceId implementation, and it's not 
> clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
> (they are stripped), nor do hadoop paths tell you if something is a 
> directory, so it's not possible to determine if a given path is a file that 
> does not have a suffix, or a directory.
> It's not clear to me that all file systems can determine whether a path is a 
> directory and thus I don't believe it can be implemented reliably.
> The only usages of getCurrentDirectory that I could find are in tests so it's 
> not clear we actually need this.
> I propose that we remove this method.
> cc [~davor]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2069) Remove ResourceId.getCurrentDirectory()?

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-2069:
---
Description: 
Beam ResourceId currently has a getCurrentDirectory method that returns the 
current resource id if it's a directory, or the parent directory if it's a 
directory.

To implement this you need to know whether or not a particular path is a 
directory or not.

I'm trying to implement the Hadoop ResourceId implementation, and it's not 
clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
(they are stripped), nor do hadoop paths tell you if something is a directory, 
so it's not possible to determine if a given path is a file that does not have 
a suffix, or a directory.

It's not clear to me that all file systems can determine whether a path is a 
directory and thus I don't believe it can be implemented reliably.

The only usages of getCurrentDirectory that I could find are in tests so it's 
not clear we actually need this.

I propose that we remove this method.

cc [~davor]

  was:
Beam FileSystem currently has a getCurrentDirectory method that returns the 
current resource id if it's a directory, or the parent directory if it's a 
directory.

To implement this you need to know whether or not a particular path is a 
directory or not.

I'm trying to implement the Hadoop FileSystem implementation, and it's not 
clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
(they are stripped), nor do hadoop paths tell you if something is a directory, 
so it's not possible to determine if a given path is a file that does not have 
a suffix, or a directory.

It's not clear to me that all file systems can determine whether a path is a 
directory and thus I don't believe it can be implemented reliably.

The only usages of getCurrentDirectory that I could find are in tests so it's 
not clear we actually need this.

I propose that we remove this method.

cc [~davor]


> Remove ResourceId.getCurrentDirectory()?
> 
>
> Key: BEAM-2069
> URL: https://issues.apache.org/jira/browse/BEAM-2069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Beam ResourceId currently has a getCurrentDirectory method that returns the 
> current resource id if it's a directory, or the parent directory if it's a 
> directory.
> To implement this you need to know whether or not a particular path is a 
> directory or not.
> I'm trying to implement the Hadoop ResourceId implementation, and it's not 
> clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
> (they are stripped), nor do hadoop paths tell you if something is a 
> directory, so it's not possible to determine if a given path is a file that 
> does not have a suffix, or a directory.
> It's not clear to me that all file systems can determine whether a path is a 
> directory and thus I don't believe it can be implemented reliably.
> The only usages of getCurrentDirectory that I could find are in tests so it's 
> not clear we actually need this.
> I propose that we remove this method.
> cc [~davor]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2069) Remove ResourceId.getCurrentDirectory()?

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-2069:
---
Summary: Remove ResourceId.getCurrentDirectory()?  (was: Remove 
FileSystem.getCurrentDirectory()?)

> Remove ResourceId.getCurrentDirectory()?
> 
>
> Key: BEAM-2069
> URL: https://issues.apache.org/jira/browse/BEAM-2069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Beam FileSystem currently has a getCurrentDirectory method that returns the 
> current resource id if it's a directory, or the parent directory if it's a 
> directory.
> To implement this you need to know whether or not a particular path is a 
> directory or not.
> I'm trying to implement the Hadoop FileSystem implementation, and it's not 
> clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
> (they are stripped), nor do hadoop paths tell you if something is a 
> directory, so it's not possible to determine if a given path is a file that 
> does not have a suffix, or a directory.
> It's not clear to me that all file systems can determine whether a path is a 
> directory and thus I don't believe it can be implemented reliably.
> The only usages of getCurrentDirectory that I could find are in tests so it's 
> not clear we actually need this.
> I propose that we remove this method.
> cc [~davor]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2069) Remove FileSystem.getCurrentDirectory()?

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-2069:
---
Affects Version/s: First stable release

> Remove FileSystem.getCurrentDirectory()?
> 
>
> Key: BEAM-2069
> URL: https://issues.apache.org/jira/browse/BEAM-2069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Beam FileSystem currently has a getCurrentDirectory method that returns the 
> current resource id if it's a directory, or the parent directory if it's a 
> directory.
> To implement this you need to know whether or not a particular path is a 
> directory or not.
> I'm trying to implement the Hadoop FileSystem implementation, and it's not 
> clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
> (they are stripped), nor do hadoop paths tell you if something is a 
> directory, so it's not possible to determine if a given path is a file that 
> does not have a suffix, or a directory.
> It's not clear to me that all file systems can determine whether a path is a 
> directory and thus I don't believe it can be implemented reliably.
> The only usages of getCurrentDirectory that I could find are in tests so it's 
> not clear we actually need this.
> I propose that we remove this method.
> cc [~davor]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2069) Remove FileSystem.getCurrentDirectory()?

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982105#comment-15982105
 ] 

Stephen Sisk commented on BEAM-2069:


marking FSR until we know whether or not we want to do this, since it breaks 
back-compat

> Remove FileSystem.getCurrentDirectory()?
> 
>
> Key: BEAM-2069
> URL: https://issues.apache.org/jira/browse/BEAM-2069
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Beam FileSystem currently has a getCurrentDirectory method that returns the 
> current resource id if it's a directory, or the parent directory if it's a 
> directory.
> To implement this you need to know whether or not a particular path is a 
> directory or not.
> I'm trying to implement the Hadoop FileSystem implementation, and it's not 
> clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
> (they are stripped), nor do hadoop paths tell you if something is a 
> directory, so it's not possible to determine if a given path is a file that 
> does not have a suffix, or a directory.
> It's not clear to me that all file systems can determine whether a path is a 
> directory and thus I don't believe it can be implemented reliably.
> The only usages of getCurrentDirectory that I could find are in tests so it's 
> not clear we actually need this.
> I propose that we remove this method.
> cc [~davor]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982100#comment-15982100
 ] 

Stephen Sisk commented on BEAM-2005:


Also, I filed https://issues.apache.org/jira/browse/BEAM-2069 (Remove 
FileSystem.getCurrentDirectory()?)  based on an issue I ran into while 
implementing HadoopResourceId.

For now, I'm having getCurrentDirectory() throw NotImplementedException()

> Add a Hadoop FileSystem implementation of Beam's FileSystem
> ---
>
> Key: BEAM-2005
> URL: https://issues.apache.org/jira/browse/BEAM-2005
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Beam's FileSystem creates an abstraction for reading from files in many 
> different places. 
> We should add a Hadoop FileSystem implementation 
> (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html)
>  - that would enable us to read from any file system that implements 
> FileSystem (including HDFS, azure, s3, etc..)
> I'm investigating this now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2069) Remove FileSystem.getCurrentDirectory()?

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-2069:
--

 Summary: Remove FileSystem.getCurrentDirectory()?
 Key: BEAM-2069
 URL: https://issues.apache.org/jira/browse/BEAM-2069
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Stephen Sisk
Assignee: Davor Bonaci


Beam FileSystem currently has a getCurrentDirectory method that returns the 
current resource id if it's a directory, or the parent directory if it's a 
directory.

To implement this you need to know whether or not a particular path is a 
directory or not.

I'm trying to implement the Hadoop FileSystem implementation, and it's not 
clear if it's possible. Hadoop's Paths do not end a / if they are a directory 
(they are stripped), nor do hadoop paths tell you if something is a directory, 
so it's not possible to determine if a given path is a file that does not have 
a suffix, or a directory.

It's not clear to me that all file systems can determine whether a path is a 
directory and thus I don't believe it can be implemented reliably.

The only usages of getCurrentDirectory that I could find are in tests so it's 
not clear we actually need this.

I propose that we remove this method.

cc [~davor]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1411) Unable to downgrade to lower guava version after upgrade to beam-0.5

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982073#comment-15982073
 ] 

Pawel Szczur edited comment on BEAM-1411 at 4/24/17 11:07 PM:
--

Just for a notice. I've encountered this error (with different stacktrace) 
after upgrading from Beam 0.4 to 0.6. I had Guava 19 in dependencies, upgraded 
to 21, problem went away.

(leaving for future people googling stacktrace)


was (Author: pawelszc...@gmail.com):
Just for a notice. I've encountered this error (with different stacktrace) 
after upgrading from Beam 0.4 to 0.6. I had Guava 19 in dependencies, upgraded 
to 21, problem went away.

> Unable to downgrade to lower guava version after upgrade to beam-0.5
> 
>
> Key: BEAM-1411
> URL: https://issues.apache.org/jira/browse/BEAM-1411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Michael Luckey
>Assignee: Davor Bonaci
> Fix For: First stable release
>
>
> While testing upgrading to 0.5 version of beam we ran into NoSuchMethodErrors
> {noformat}
> java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>  at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$SerializableSplit.(HDFSFileSource.java:473)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:81)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:78)
>  at 
> com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451)
>  at java.util.AbstractList$Itr.next(AbstractList.java:358)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:202)
>  at 
> org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:65)
>  at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:168)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:329)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:71)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:178)
>  at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:258)
> {noformat}
> This seems to be caused by the [HDFS 
> IO|https://github.com/apache/beam/tree/master/sdks/java/io/hdfs] components 
> guava dependency not being shaded - in contrast to the core components - and 
> revealed by the [recent update to 
> guava-20.0|https://github.com/apache/beam/commit/0b4b2becb45b9f637ba31f599ebe8be0331bd633]
>  and the therefore incorporated api changes made by [overloading 
> methods|https://github.com/google/guava/commit/892e323fca32945cdfb25395ca6e346dd0fffa5b#diff-fe7358934fa6eba23c2791eb40cec030]
> Did not check, whether more components are affected to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2669: Properly Declare Protobuf extension dependencies

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2669

Properly Declare Protobuf extension dependencies

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Import the correct ProtoCoder in BigtableIO



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam proto_deps

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2669.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2669


commit 7a038d9cd7c1eca4e216f912a564c64b2ef1b1ba
Author: Thomas Groh 
Date:   2017-04-24T23:05:28Z

Properly Declare Protobuf extension dependencies

Import the correct ProtoCoder in BigtableIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (BEAM-1411) Unable to downgrade to lower guava version after upgrade to beam-0.5

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982073#comment-15982073
 ] 

Pawel Szczur edited comment on BEAM-1411 at 4/24/17 11:06 PM:
--

Just for a notice. I've encountered this error (with different stacktrace) 
after upgrading from Beam 0.4 to 0.6. I had Guava 19 in dependencies, upgraded 
to 21, problem went away.


was (Author: pawelszc...@gmail.com):
Just for a notice. I've encountered this error after upgrading from Beam 0.4 to 
0.6. I had Guava 19 in dependencies, upgraded to 21, problem went away.

> Unable to downgrade to lower guava version after upgrade to beam-0.5
> 
>
> Key: BEAM-1411
> URL: https://issues.apache.org/jira/browse/BEAM-1411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Michael Luckey
>Assignee: Davor Bonaci
> Fix For: First stable release
>
>
> While testing upgrading to 0.5 version of beam we ran into NoSuchMethodErrors
> {noformat}
> java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>  at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$SerializableSplit.(HDFSFileSource.java:473)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:81)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:78)
>  at 
> com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451)
>  at java.util.AbstractList$Itr.next(AbstractList.java:358)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:202)
>  at 
> org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:65)
>  at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:168)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:329)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:71)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:178)
>  at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:258)
> {noformat}
> This seems to be caused by the [HDFS 
> IO|https://github.com/apache/beam/tree/master/sdks/java/io/hdfs] components 
> guava dependency not being shaded - in contrast to the core components - and 
> revealed by the [recent update to 
> guava-20.0|https://github.com/apache/beam/commit/0b4b2becb45b9f637ba31f599ebe8be0331bd633]
>  and the therefore incorporated api changes made by [overloading 
> methods|https://github.com/google/guava/commit/892e323fca32945cdfb25395ca6e346dd0fffa5b#diff-fe7358934fa6eba23c2791eb40cec030]
> Did not check, whether more components are affected to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1411) Unable to downgrade to lower guava version after upgrade to beam-0.5

[Pawel Szczur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982073#comment-15982073
 ] 

Pawel Szczur commented on BEAM-1411:


Just for a notice. I've encountered this error after upgrading from Beam 0.4 to 
0.6. I had Guava 19 in dependencies, upgraded to 21, problem went away.

> Unable to downgrade to lower guava version after upgrade to beam-0.5
> 
>
> Key: BEAM-1411
> URL: https://issues.apache.org/jira/browse/BEAM-1411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Michael Luckey
>Assignee: Davor Bonaci
> Fix For: First stable release
>
>
> While testing upgrading to 0.5 version of beam we ran into NoSuchMethodErrors
> {noformat}
> java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>  at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$SerializableSplit.(HDFSFileSource.java:473)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:81)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:78)
>  at 
> com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451)
>  at java.util.AbstractList$Itr.next(AbstractList.java:358)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:202)
>  at 
> org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:65)
>  at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:168)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:329)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:71)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:178)
>  at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:258)
> {noformat}
> This seems to be caused by the [HDFS 
> IO|https://github.com/apache/beam/tree/master/sdks/java/io/hdfs] components 
> guava dependency not being shaded - in contrast to the core components - and 
> revealed by the [recent update to 
> guava-20.0|https://github.com/apache/beam/commit/0b4b2becb45b9f637ba31f599ebe8be0331bd633]
>  and the therefore incorporated api changes made by [overloading 
> methods|https://github.com/google/guava/commit/892e323fca32945cdfb25395ca6e346dd0fffa5b#diff-fe7358934fa6eba23c2791eb40cec030]
> Did not check, whether more components are affected to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2068) Upgrade Google-Apitools to latest version

[Sourabh Bajaj (JIRA)]

Sourabh Bajaj created BEAM-2068:
---

 Summary: Upgrade Google-Apitools to latest version
 Key: BEAM-2068
 URL: https://issues.apache.org/jira/browse/BEAM-2068
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Sourabh Bajaj
Assignee: Ahmet Altay
Priority: Minor


In 0.5.9 apitools is pinned to setuptools 18.5 which is really old as the 
current release is 35.0.1 at the time of creating the issue. Updating to 0.5.9 
causes issues for other dependencies so we're going to try to address this 
upstream first and then upgrade to the latest version in the future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1766

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982052#comment-15982052
 ] 

ASF GitHub Bot commented on BEAM-2021:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2668

[BEAM-2021] Make Most StandardCoders CustomCoders

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Standard Coders have a defined serialization format and are understood
within the Runner API, Custom Coders are not. Move existing
"StandardCoders" to extend CustomCoder, and remove custom cloud object
related serialization logic where possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fewer_standard_coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2668


commit 77f416ec28233fd4cae676c604334295ae30a082
Author: Thomas Groh 
Date:   2017-04-20T16:39:12Z

Stop Extending AtomicCoder

commit 6fc50b2bb7e0416bb04b93595248175020185107
Author: Thomas Groh 
Date:   2017-04-22T00:58:51Z

Make Most StandardCoders CustomCoders

Standard Coders have a defined serialization format and are understood
within the Runner API, Custom Coders are not. Move existing
"StandardCoders" to extend CustomCoder, and remove custom cloud object
related serialization logic where possible.




> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2668: [BEAM-2021] Make Most StandardCoders CustomCoders

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2668

[BEAM-2021] Make Most StandardCoders CustomCoders

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Standard Coders have a defined serialization format and are understood
within the Runner API, Custom Coders are not. Move existing
"StandardCoders" to extend CustomCoder, and remove custom cloud object
related serialization logic where possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fewer_standard_coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2668


commit 77f416ec28233fd4cae676c604334295ae30a082
Author: Thomas Groh 
Date:   2017-04-20T16:39:12Z

Stop Extending AtomicCoder

commit 6fc50b2bb7e0416bb04b93595248175020185107
Author: Thomas Groh 
Date:   2017-04-22T00:58:51Z

Make Most StandardCoders CustomCoders

Standard Coders have a defined serialization format and are understood
within the Runner API, Custom Coders are not. Move existing
"StandardCoders" to extend CustomCoder, and remove custom cloud object
related serialization logic where possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2667: Pin apitools to 0.5.8

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2667

Pin apitools to 0.5.8

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-fix-apitools-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2667.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2667


commit f5be041d96e2ad870d7de5eb21a00abf7ca8
Author: Sourabh Bajaj 
Date:   2017-04-24T22:49:13Z

Pin apitools to 0.5.8




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982030#comment-15982030
 ] 

Stephen Sisk commented on BEAM-2005:


note that adding support for other Hadoop FileSystem should be simple (I'm not 
doing anything to block them) and might be as straightforward as adding another 
FileSystemRegistrar that returns the HadoopFileSystem for different schemes.

> Add a Hadoop FileSystem implementation of Beam's FileSystem
> ---
>
> Key: BEAM-2005
> URL: https://issues.apache.org/jira/browse/BEAM-2005
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Beam's FileSystem creates an abstraction for reading from files in many 
> different places. 
> We should add a Hadoop FileSystem implementation 
> (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html)
>  - that would enable us to read from any file system that implements 
> FileSystem (including HDFS, azure, s3, etc..)
> I'm investigating this now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #2923

[Apache Jenkins Server]

See 


Changes:

[robertwb] Add instructions to regenerate Python proto wrappers.

[robertwb] Generate python proto wrappers for runner and fn API.

[robertwb] Add apache licence to generated files.

[robertwb] Adapt to PR #2505 changes to protos.

[lcwik] [BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf

--
[...truncated 1.35 MB...]
2017-04-24T22:37:27.952 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.8.Final/netty-buffer-4.1.8.Final.pom
2017-04-24T22:37:27.979 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.8.Final/netty-buffer-4.1.8.Final.pom
 (2 KB at 49.8 KB/sec)
2017-04-24T22:37:27.994 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.8.Final/netty-common-4.1.8.Final.pom
2017-04-24T22:37:28.024 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.8.Final/netty-common-4.1.8.Final.pom
 (10 KB at 327.9 KB/sec)
2017-04-24T22:37:28.032 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.pom
2017-04-24T22:37:28.060 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.pom
 (2 KB at 48.2 KB/sec)
2017-04-24T22:37:28.068 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.8.Final/netty-handler-4.1.8.Final.pom
2017-04-24T22:37:28.100 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.8.Final/netty-handler-4.1.8.Final.pom
 (3 KB at 73.2 KB/sec)
2017-04-24T22:37:28.107 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.pom
2017-04-24T22:37:28.135 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.pom
 (2 KB at 67.3 KB/sec)
2017-04-24T22:37:28.146 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.pom
2017-04-24T22:37:28.177 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.pom
 (2 KB at 42.3 KB/sec)
2017-04-24T22:37:28.184 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-okhttp/1.2.0/grpc-okhttp-1.2.0.pom
2017-04-24T22:37:28.213 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-okhttp/1.2.0/grpc-okhttp-1.2.0.pom
 (3 KB at 82.0 KB/sec)
2017-04-24T22:37:28.215 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.pom
2017-04-24T22:37:28.254 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.pom
 (2 KB at 35.4 KB/sec)
2017-04-24T22:37:28.256 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/squareup/okhttp/parent/2.5.0/parent-2.5.0.pom
2017-04-24T22:37:28.286 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/squareup/okhttp/parent/2.5.0/parent-2.5.0.pom
 (9 KB at 298.6 KB/sec)
2017-04-24T22:37:28.289 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/squareup/okio/okio/1.6.0/okio-1.6.0.pom
2017-04-24T22:37:28.321 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/squareup/okio/okio/1.6.0/okio-1.6.0.pom
 (2 KB at 45.1 KB/sec)
2017-04-24T22:37:28.323 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/squareup/okio/okio-parent/1.6.0/okio-parent-1.6.0.pom
2017-04-24T22:37:28.351 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/squareup/okio/okio-parent/1.6.0/okio-parent-1.6.0.pom
 (4 KB at 111.4 KB/sec)
2017-04-24T22:37:28.354 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-nano/1.2.0/grpc-protobuf-nano-1.2.0.pom
2017-04-24T22:37:28.382 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-nano/1.2.0/grpc-protobuf-nano-1.2.0.pom
 (3 KB at 74.7 KB/sec)
2017-04-24T22:37:28.385 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/nano/protobuf-javanano/3.0.0-alpha-5/protobuf-javanano-3.0.0-alpha-5.pom
2017-04-24T22:37:28.413 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/nano/protobuf-javanano/3.0.0-alpha-5/protobuf-javanano-3.0.0-alpha-5.pom
 (11 KB at 386.3 KB/sec)
2017-04-24T22:37:28.416 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-protos/0.9.6/bigtable-protos-0.9.6.pom
2017-04-24T22:37:28.443 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-protos/0.9.6/bigtable-protos-0.9.6.pom
 (8 KB at 275.0 KB/sec)
2017-04-24T22:37:28.445 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bi

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3439

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf

--
[...truncated 927.19 KB...]
2017-04-24T22:28:07.580 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http2/4.1.8.Final/netty-codec-http2-4.1.8.Final.jar
2017-04-24T22:28:07.595 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api/grpc/grpc-google-pubsub-v1/0.1.0/grpc-google-pubsub-v1-0.1.0.jar
 (262 KB at 3636.7 KB/sec)
2017-04-24T22:28:07.595 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http/4.1.8.Final/netty-codec-http-4.1.8.Final.jar
2017-04-24T22:28:07.606 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api/grpc/grpc-google-iam-v1/0.1.0/grpc-google-iam-v1-0.1.0.jar
 (78 KB at 930.7 KB/sec)
2017-04-24T22:28:07.606 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.jar
2017-04-24T22:28:07.636 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.jar
 (20 KB at 179.9 KB/sec)
2017-04-24T22:28:07.636 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.jar
2017-04-24T22:28:07.660 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/datastore/datastore-v1-protos/1.3.0/datastore-v1-protos-1.3.0.jar
 (437 KB at 3212.6 KB/sec)
2017-04-24T22:28:07.660 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.8.Final/netty-handler-4.1.8.Final.jar
2017-04-24T22:28:07.687 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.jar
 (118 KB at 724.2 KB/sec)
2017-04-24T22:28:07.687 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.8.Final/netty-buffer-4.1.8.Final.jar
2017-04-24T22:28:07.699 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-netty/1.2.0/grpc-netty-1.2.0.jar
 (162 KB at 929.6 KB/sec)
2017-04-24T22:28:07.699 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.8.Final/netty-common-4.1.8.Final.jar
2017-04-24T22:28:07.720 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http/4.1.8.Final/netty-codec-http-4.1.8.Final.jar
 (533 KB at 2728.8 KB/sec)
2017-04-24T22:28:07.721 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-transport/4.1.8.Final/netty-transport-4.1.8.Final.jar
2017-04-24T22:28:07.760 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http2/4.1.8.Final/netty-codec-http2-4.1.8.Final.jar
 (355 KB at 1510.6 KB/sec)
2017-04-24T22:28:07.760 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.jar
2017-04-24T22:28:07.772 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.8.Final/netty-handler-4.1.8.Final.jar
 (325 KB at 1312.3 KB/sec)
2017-04-24T22:28:07.772 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec/4.1.8.Final/netty-codec-4.1.8.Final.jar
2017-04-24T22:28:07.797 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.8.Final/netty-buffer-4.1.8.Final.jar
 (254 KB at 930.2 KB/sec)
2017-04-24T22:28:07.797 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-all/1.2.0/grpc-all-1.2.0.jar
2017-04-24T22:28:07.806 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.jar
 (29 KB at 103.6 KB/sec)
2017-04-24T22:28:07.806 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-okhttp/1.2.0/grpc-okhttp-1.2.0.jar
2017-04-24T22:28:07.807 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-transport/4.1.8.Final/netty-transport-4.1.8.Final.jar
 (418 KB at 1480.2 KB/sec)
2017-04-24T22:28:07.807 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.jar
2017-04-24T22:28:07.823 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-all/1.2.0/grpc-all-1.2.0.jar 
(359 B at 1.2 KB/sec)
2017-04-24T22:28:07.823 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/squareup/okio/okio/1.6.0/okio-1.6.0.jar
2017-04-24T22:28:07.858 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec/4.1.8.Final/netty-codec-4.1.8.Final.jar
 (302 KB at 905.6 KB/sec)
2017-04-24T22:28:07.858 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-nano/1.2.0/grpc-protobuf-nano-1.2.0.jar
2017-04-24T22:28:07.859 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/squareup/okio/okio/1.6.0/okio-1.6.0.jar
 (65 KB at 1

[jira] [Commented] (BEAM-818) Remove Pipeline.getPipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982018#comment-15982018
 ] 

ASF GitHub Bot commented on BEAM-818:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2666

[BEAM-818,BEAM-827,BEAM-828] Remove some construction-time uses of 
PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam PipelineOptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2666.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2666


commit a94c216ccbd2b2dd19617f2a6410d2200d15d828
Author: Kenneth Knowles 
Date:   2017-04-24T19:49:39Z

Move stable name validation to Pipeline.run()

The action taken when a pipeline does not have unique stable
names depends on the PipelineOptions, which will not available
during construction. Moving this later removes one blocker
from the refactor to PipelineOptions availability.

commit 93755eb3722eadd1bce5d99aaee4625d632f3625
Author: Kenneth Knowles 
Date:   2017-04-24T20:57:23Z

Remove PipelineOptions from createWriteOperation()

commit f28f3e67776d7203d3635b7d0732bd5adf83366f
Author: Kenneth Knowles 
Date:   2017-04-24T21:37:30Z

Move PTransform.validate to post-construction

commit 7d575eea183863441a7f8646efdd8876fa0cdc3d
Author: Kenneth Knowles 
Date:   2017-04-24T19:50:57Z

Supply PipelineOptions at Pipeline.run()




> Remove Pipeline.getPipelineOptions
> --
>
> Key: BEAM-818
> URL: https://issues.apache.org/jira/browse/BEAM-818
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This stops transforms from changing their operation based on 
> construction-time options, and instead requires that configuration to be 
> explicit, or to obtain the configuration at runtime.
> https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk--XmGMGgDwwNwWZaFbxLKvPqEQ/edit#



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2666: [BEAM-818,BEAM-827,BEAM-828] Remove some constructi...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2666

[BEAM-818,BEAM-827,BEAM-828] Remove some construction-time uses of 
PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam PipelineOptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2666.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2666


commit a94c216ccbd2b2dd19617f2a6410d2200d15d828
Author: Kenneth Knowles 
Date:   2017-04-24T19:49:39Z

Move stable name validation to Pipeline.run()

The action taken when a pipeline does not have unique stable
names depends on the PipelineOptions, which will not available
during construction. Moving this later removes one blocker
from the refactor to PipelineOptions availability.

commit 93755eb3722eadd1bce5d99aaee4625d632f3625
Author: Kenneth Knowles 
Date:   2017-04-24T20:57:23Z

Remove PipelineOptions from createWriteOperation()

commit f28f3e67776d7203d3635b7d0732bd5adf83366f
Author: Kenneth Knowles 
Date:   2017-04-24T21:37:30Z

Move PTransform.validate to post-construction

commit 7d575eea183863441a7f8646efdd8876fa0cdc3d
Author: Kenneth Knowles 
Date:   2017-04-24T19:50:57Z

Supply PipelineOptions at Pipeline.run()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #2922

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2664: [BEAM-1871] Move ProtoCoder to new sdks/java/extens...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2664


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981991#comment-15981991
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2664


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam git commit: [BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.

[lcwik]

[BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ff1fe7fa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ff1fe7fa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ff1fe7fa

Branch: refs/heads/master
Commit: ff1fe7fa53816fe4327c7572c13a616fc4243dc9
Parents: d7e7af8
Author: Luke Cwik 
Authored: Mon Apr 24 14:32:21 2017 -0700
Committer: Luke Cwik 
Committed: Mon Apr 24 15:04:13 2017 -0700

--
 sdks/java/core/pom.xml  |   6 -
 .../apache/beam/sdk/coders/CoderRegistry.java   |  10 +-
 .../beam/sdk/coders/StringDelegateCoder.java|   4 +-
 .../beam/sdk/coders/protobuf/ProtoCoder.java| 405 ---
 .../beam/sdk/coders/protobuf/ProtobufUtil.java  | 171 
 .../beam/sdk/coders/protobuf/package-info.java  |  24 --
 sdks/java/core/src/main/proto/README.md |  45 ---
 .../main/proto/proto2_coder_test_messages.proto |  53 ---
 .../beam/sdk/coders/CoderRegistryTest.java  |  14 -
 .../sdk/coders/protobuf/ProtoCoderTest.java | 182 -
 .../sdk/coders/protobuf/ProtobufUtilTest.java   | 192 -
 sdks/java/extensions/pom.xml|   1 +
 sdks/java/extensions/protobuf/pom.xml   | 142 +++
 .../sdk/extensions/protobuf/ProtoCoder.java | 405 +++
 .../sdk/extensions/protobuf/ProtobufUtil.java   | 171 
 .../sdk/extensions/protobuf/package-info.java   |  24 ++
 .../sdk/extensions/protobuf/ProtoCoderTest.java | 181 +
 .../extensions/protobuf/ProtobufUtilTest.java   | 191 +
 .../test/proto/proto2_coder_test_messages.proto |  53 +++
 19 files changed, 1173 insertions(+), 1101 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ff1fe7fa/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index ac7a3bb..6c46453 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -294,12 +294,6 @@
 
 
 
-  com.google.cloud.dataflow
-  google-cloud-dataflow-java-proto-library-all
-  test
-
-
-
   com.esotericsoftware.kryo
   kryo
   2.21

http://git-wip-us.apache.org/repos/asf/beam/blob/ff1fe7fa/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
index 4238293..e0b2b3a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
@@ -44,7 +44,6 @@ import java.util.Set;
 import javax.annotation.Nonnull;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.coders.CannotProvideCoderException.ReasonCode;
-import org.apache.beam.sdk.coders.protobuf.ProtoCoder;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.util.CoderUtils;
@@ -77,9 +76,7 @@ import org.slf4j.LoggerFactory;
  *   the default {@code Coder} type. The {@link Coder} class must satisfy 
the requirements
  *   of {@link CoderProviders#fromStaticMethods}.
  *   Fallback: A fallback {@link CoderProvider} is used to attempt to 
provide a {@link Coder}
- *   for any type. By default, there are two chained fallback coders:
- *   {@link ProtoCoder#coderProvider}, which can provide a coder to 
efficiently serialize any
- *   Protocol Buffers message, and then {@link 
SerializableCoder#PROVIDER}, which can provide a
+ *   for any type. By default, there is {@link 
SerializableCoder#PROVIDER}, which can provide a
  *   {@link Coder} for any type that is serializable via Java 
serialization. The fallback
  *   {@link CoderProvider} can be get and set respectively using
  *   {@link #getFallbackCoderProvider()} and {@link 
#setFallbackCoderProvider}. Multiple
@@ -165,7 +162,7 @@ public class CoderRegistry implements CoderProvider {
   private CoderRegistry() {
 coderFactoryMap = new HashMap<>(REGISTERED_CODER_FACTORIES_PER_CLASS);
 setFallbackCoderProvider(
-CoderProviders.firstOf(ProtoCoder.coderProvider(), 
SerializableCoder.PROVIDER));
+CoderProviders.firstOf(SerializableCoder.PROVIDER));
   }
 
   /**
@@ -423,8 +420,7 @@ public class CoderRegistry implements CoderProvider {
* providing a {@code Coder} for a type {@code T}, then the registry will 
attempt to create
* a {@link Coder} using this {@link CoderProvider}.
*
-   * By default, this is set to the chain 

[3/3] beam git commit: [BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.

[lcwik]

[BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.

This closes #2664


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/57535f35
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/57535f35
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/57535f35

Branch: refs/heads/master
Commit: 57535f35e04beeff022d450b5e737a05314f3c6c
Parents: d7e7af8 ff1fe7f
Author: Luke Cwik 
Authored: Mon Apr 24 15:04:34 2017 -0700
Committer: Luke Cwik 
Committed: Mon Apr 24 15:04:34 2017 -0700

--
 sdks/java/core/pom.xml  |   6 -
 .../apache/beam/sdk/coders/CoderRegistry.java   |  10 +-
 .../beam/sdk/coders/StringDelegateCoder.java|   4 +-
 .../beam/sdk/coders/protobuf/ProtoCoder.java| 405 ---
 .../beam/sdk/coders/protobuf/ProtobufUtil.java  | 171 
 .../beam/sdk/coders/protobuf/package-info.java  |  24 --
 sdks/java/core/src/main/proto/README.md |  45 ---
 .../main/proto/proto2_coder_test_messages.proto |  53 ---
 .../beam/sdk/coders/CoderRegistryTest.java  |  14 -
 .../sdk/coders/protobuf/ProtoCoderTest.java | 182 -
 .../sdk/coders/protobuf/ProtobufUtilTest.java   | 192 -
 sdks/java/extensions/pom.xml|   1 +
 sdks/java/extensions/protobuf/pom.xml   | 142 +++
 .../sdk/extensions/protobuf/ProtoCoder.java | 405 +++
 .../sdk/extensions/protobuf/ProtobufUtil.java   | 171 
 .../sdk/extensions/protobuf/package-info.java   |  24 ++
 .../sdk/extensions/protobuf/ProtoCoderTest.java | 181 +
 .../extensions/protobuf/ProtobufUtilTest.java   | 191 +
 .../test/proto/proto2_coder_test_messages.proto |  53 +++
 19 files changed, 1173 insertions(+), 1101 deletions(-)
--

[1/3] beam git commit: [BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master d7e7af8eb -> 57535f35e


http://git-wip-us.apache.org/repos/asf/beam/blob/ff1fe7fa/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
--
diff --git 
a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
 
b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
new file mode 100644
index 000..a8c6a81
--- /dev/null
+++ 
b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.ListCoder;
+import 
org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageA;
+import 
org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageB;
+import 
org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageC;
+import 
org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageWithMap;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link ProtoCoder}.
+ */
+@RunWith(JUnit4.class)
+public class ProtoCoderTest {
+
+  @Rule public ExpectedException thrown = ExpectedException.none();
+
+  @Test
+  public void testFactoryMethodAgreement() throws Exception {
+assertEquals(ProtoCoder.of(new TypeDescriptor() {}), 
ProtoCoder.of(MessageA.class));
+
+assertEquals(
+ProtoCoder.of(new TypeDescriptor() {}),
+ProtoCoder.coderProvider().getCoder(new TypeDescriptor() 
{}));
+  }
+
+  @Test
+  public void testProviderCannotProvideCoder() throws Exception {
+thrown.expect(CannotProvideCoderException.class);
+thrown.expectMessage("java.lang.Integer is not a subclass of 
com.google.protobuf.Message");
+
+ProtoCoder.coderProvider().getCoder(new TypeDescriptor() {});
+  }
+
+  @Test
+  public void testCoderEncodeDecodeEqual() throws Exception {
+MessageA value =
+MessageA.newBuilder()
+.setField1("hello")
+.addField2(MessageB.newBuilder().setField1(true).build())
+.addField2(MessageB.newBuilder().setField1(false).build())
+.build();
+CoderProperties.coderDecodeEncodeEqual(ProtoCoder.of(MessageA.class), 
value);
+  }
+
+  @Test
+  public void testCoderEncodeDecodeEqualNestedContext() throws Exception {
+MessageA value1 =
+MessageA.newBuilder()
+.setField1("hello")
+.addField2(MessageB.newBuilder().setField1(true).build())
+.addField2(MessageB.newBuilder().setField1(false).build())
+.build();
+MessageA value2 =
+MessageA.newBuilder()
+.setField1("world")
+.addField2(MessageB.newBuilder().setField1(false).build())
+.addField2(MessageB.newBuilder().setField1(true).build())
+.build();
+CoderProperties.coderDecodeEncodeEqual(
+ListCoder.of(ProtoCoder.of(MessageA.class)), ImmutableList.of(value1, 
value2));
+  }
+
+  @Test
+  public void testCoderEncodeDecodeExtensionsEqual() throws Exception {
+MessageC value =
+MessageC.newBuilder()
+.setExtension(
+Proto2CoderTestMessages.field1,
+MessageA.newBuilder()
+.setField1("hello")
+.addField2(MessageB.newBuilder().setField1(true).build())
+.bu

[jira] [Created] (BEAM-2067) Add support for generic CoderProvider -> CoderFactory mapping with CoderRegistrar

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-2067:
---

 Summary: Add support for generic CoderProvider -> CoderFactory 
mapping with CoderRegistrar
 Key: BEAM-2067
 URL: https://issues.apache.org/jira/browse/BEAM-2067
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Affects Versions: First stable release
Reporter: Luke Cwik
Assignee: Luke Cwik


Instead of relying on Class to CoderFactory in the CoderRegistrar, we should 
really ask if CoderFactory can handle a specific type or allow for 
CoderProvider be able answer on its behalf.

This includes re-adding support for ProtoCoder as a "fallback" coder provider.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1988) utils.path.join does not correctly handle GCS bucket roots

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981981#comment-15981981
 ] 

ASF GitHub Bot commented on BEAM-1988:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2665

[BEAM-1988] Add FileSystems Interface for accessing underlying FS

…rectly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @chamikaramj @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1988-filesystems-interface

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2665.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2665


commit ec3b30feb58914aec03ef5d1d10e373c96dc7b4e
Author: Sourabh Bajaj 
Date:   2017-04-24T21:57:09Z

[BEAM-1988] Add FileSystems Interface for accessing underlying FS correctly




> utils.path.join does not correctly handle GCS bucket roots
> --
>
> Key: BEAM-1988
> URL: https://issues.apache.org/jira/browse/BEAM-1988
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Here:
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/path.py#L22
> Joining a bucket root with a filename e.g. (gs://mybucket/ , myfile) results 
> in invalid 'gs://mybucket//myfile', notice the double // between mybucket and 
> myfile. (It actually does not handle anything that already ends with {{/}} 
> correctly)
> [~sb2nov] could you take this one? Also, should the `join` operation move to 
> a BeamFileSystem level code.
> (cc: [~chamikara])



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2665: [BEAM-1988] Add FileSystems Interface for accessing...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2665

[BEAM-1988] Add FileSystems Interface for accessing underlying FS

…rectly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @chamikaramj @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1988-filesystems-interface

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2665.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2665


commit ec3b30feb58914aec03ef5d1d10e373c96dc7b4e
Author: Sourabh Bajaj 
Date:   2017-04-24T21:57:09Z

[BEAM-1988] Add FileSystems Interface for accessing underlying FS correctly




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/7] beam git commit: Generate python proto wrappers for runner and fn API.

[robertwb]

Generate python proto wrappers for runner and fn API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2a59b057
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2a59b057
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2a59b057

Branch: refs/heads/master
Commit: 2a59b0574c988661a17b57c2380a16227f5e6529
Parents: 11103af
Author: Robert Bradshaw 
Authored: Fri Apr 21 10:00:20 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Apr 24 14:52:27 2017 -0700

--
 .../apache_beam/runners/api/beam_fn_api_pb2.py  | 2723 ++
 .../runners/api/beam_fn_api_pb2_grpc.py |  186 ++
 .../runners/api/beam_runner_api_pb2.py  |  315 +-
 3 files changed, 3107 insertions(+), 117 deletions(-)
--

[2/7] beam git commit: Generate python proto wrappers for runner and fn API.

[robertwb]

http://git-wip-us.apache.org/repos/asf/beam/blob/2a59b057/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
new file mode 100644
index 000..dea47e3
--- /dev/null
+++ b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
@@ -0,0 +1,2723 @@
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# source: beam_fn_api.proto
+
+import sys
+_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf import descriptor_pb2
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+from google.protobuf import timestamp_pb2 as 
google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+  name='beam_fn_api.proto',
+  package='org.apache.beam.fn.v1',
+  syntax='proto3',
+  
serialized_pb=_b('\n\x11\x62\x65\x61m_fn_api.proto\x12\x15org.apache.beam.fn.v1\x1a\x19google/protobuf/any.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"t\n\x06Target\x12%\n\x1dprimitive_transform_reference\x18\x01
 \x01(\t\x12\x0c\n\x04name\x18\x02 
\x01(\t\x1a\x35\n\x04List\x12-\n\x06target\x18\x01 
\x03(\x0b\x32\x1d.org.apache.beam.fn.v1.Target\"&\n\x0bPCollection\x12\x17\n\x0f\x63oder_reference\x18\x01
 \x01(\t\"\xcb\x04\n\x12PrimitiveTransform\x12\n\n\x02id\x18\x01 
\x01(\t\x12:\n\rfunction_spec\x18\x02 
\x01(\x0b\x32#.org.apache.beam.fn.v1.FunctionSpec\x12\x45\n\x06inputs\x18\x03 
\x03(\x0b\x32\x35.org.apache.beam.fn.v1.PrimitiveTransform.InputsEntry\x12G\n\x07outputs\x18\x04
 
\x03(\x0b\x32\x36.org.apache.beam.fn.v1.PrimitiveTransform.OutputsEntry\x12N\n\x0bside_inputs\x18\x05
 
\x03(\x0b\x32\x39.org.apache.beam.fn.v1.PrimitiveTransform.SideInputsEntry\x12\x11\n\tstep_name\x18\x06
 \x01(\t\x1aQ\n\x0bInputsEntry\x12\x0b\n\x03key\x18\x01 
\x01(\t\x12\x31\n\x05value\x18\x02 \x01(\x0b\x32\"
 
.org.apache.beam.fn.v1.Target.List:\x02\x38\x01\x1aR\n\x0cOutputsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x31\n\x05value\x18\x02 
\x01(\x0b\x32\".org.apache.beam.fn.v1.PCollection:\x02\x38\x01\x1aS\n\x0fSideInputsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12/\n\x05value\x18\x02 \x01(\x0b\x32 
.org.apache.beam.fn.v1.SideInput:\x02\x38\x01\"j\n\x0c\x46unctionSpec\x12\n\n\x02id\x18\x01
 \x01(\t\x12\x0b\n\x03urn\x18\x02 
\x01(\t\x12\x1d\n\x15\x65nvironment_reference\x18\x03 
\x01(\t\x12\"\n\x04\x64\x61ta\x18\x04 
\x01(\x0b\x32\x14.google.protobuf.Any\"o\n\tSideInput\x12,\n\x05input\x18\x01 
\x01(\x0b\x32\x1d.org.apache.beam.fn.v1.Target\x12\x34\n\x07view_fn\x18\x02 
\x01(\x0b\x32#.org.apache.beam.fn.v1.FunctionSpec\"f\n\x05\x43oder\x12:\n\rfunction_spec\x18\x01
 
\x01(\x0b\x32#.org.apache.beam.fn.v1.FunctionSpec\x12!\n\x19\x63omponent_coder_reference\x18\x02
 \x03(\t\"]\n\x0eRemoteGrpcPort\x12K\n\x16\x61pi_service_descriptor\x18\x01 
\x01(\x0b\x32+.org.apache.beam.fn.v1.ApiServiceDescriptor\"\xe8\x0
 2\n\x12InstructionRequest\x12\x16\n\x0einstruction_id\x18\x01 
\x01(\t\x12;\n\x08register\x18\xe8\x07 
\x01(\x0b\x32&.org.apache.beam.fn.v1.RegisterRequestH\x00\x12\x46\n\x0eprocess_bundle\x18\xe9\x07
 
\x01(\x0b\x32+.org.apache.beam.fn.v1.ProcessBundleRequestH\x00\x12W\n\x17process_bundle_progress\x18\xea\x07
 
\x01(\x0b\x32\x33.org.apache.beam.fn.v1.ProcessBundleProgressRequestH\x00\x12Q\n\x14process_bundle_split\x18\xeb\x07
 
\x01(\x0b\x32\x30.org.apache.beam.fn.v1.ProcessBundleSplitRequestH\x00\x42\t\n\x07request\"\xfd\x02\n\x13InstructionResponse\x12\x16\n\x0einstruction_id\x18\x01
 \x01(\t\x12\r\n\x05\x65rror\x18\x02 \x01(\t\x12<\n\x08register\x18\xe8\x07 
\x01(\x0b\x32\'.org.apache.beam.fn.v1.RegisterResponseH\x00\x12G\n\x0eprocess_bundle\x18\xe9\x07
 
\x01(\x0b\x32,.org.apache.beam.fn.v1.ProcessBundleResponseH\x00\x12X\n\x17process_bundle_progress\x18\xea\x07
 
\x01(\x0b\x32\x34.org.apache.beam.fn.v1.ProcessBundleProgressResponseH\x00\x12R\n\x14process_bundle_split\x18\xeb\x07
 \x01(\x0b\x
 
32\x31.org.apache.beam.fn.v1.ProcessBundleSplitResponseH\x00\x42\n\n\x08response\"d\n\x0fRegisterRequest\x12Q\n\x19process_bundle_descriptor\x18\x01
 
\x03(\x0b\x32..org.apache.beam.fn.v1.ProcessBundleDescriptor\"\x12\n\x10RegisterResponse\"\x9b\x01\n\x17ProcessBundleDescriptor\x12\n\n\x02id\x18\x01
 \x01(\t\x12\x46\n\x13primitive_transform\x18\x02 
\x03(\x0b\x32).org.apache.beam.fn.v1.PrimitiveTransform\x12,\n\x06\x63oders\x18\x04
 
\x03(\x0b\x32\x1c.org.apache.beam.fn.v1.Coder\"|\n\x14ProcessBundleRequest\x12+\n#process_bundle_descriptor_reference\x18\x01
 \x01(\t\x12\x37\n\x0c\x63\x61\x63he_tokens\x18\x02 
\x03(\x0b\x32!.org.apache.beam.fn.v1.CacheToken\"\x17\n\x15ProcessBundleR

[1/7] beam git commit: Generate python proto wrappers for runner and fn API.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 0191e042b -> d7e7af8eb


http://git-wip-us.apache.org/repos/asf/beam/blob/2a59b057/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py 
b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
new file mode 100644
index 000..f86861c
--- /dev/null
+++ b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
@@ -0,0 +1,186 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+import grpc
+from grpc.framework.common import cardinality
+from grpc.framework.interfaces.face import utilities as face_utilities
+
+import beam_fn_api_pb2 as beam__fn__api__pb2
+
+
+class BeamFnControlStub(object):
+  """
+  Control Plane API
+
+  Progress reporting and splitting still need further vetting. Also, this may 
change
+  with the addition of new types of instructions/responses related to metrics.
+
+  An API that describes the work that a SDK harness is meant to do.
+  Stable
+  """
+
+  def __init__(self, channel):
+"""Constructor.
+
+Args:
+  channel: A grpc.Channel.
+"""
+self.Control = channel.stream_stream(
+'/org.apache.beam.fn.v1.BeamFnControl/Control',
+
request_serializer=beam__fn__api__pb2.InstructionResponse.SerializeToString,
+response_deserializer=beam__fn__api__pb2.InstructionRequest.FromString,
+)
+
+
+class BeamFnControlServicer(object):
+  """
+  Control Plane API
+
+  Progress reporting and splitting still need further vetting. Also, this may 
change
+  with the addition of new types of instructions/responses related to metrics.
+
+  An API that describes the work that a SDK harness is meant to do.
+  Stable
+  """
+
+  def Control(self, request_iterator, context):
+"""Instructions sent by the runner to the SDK requesting different types
+of work.
+"""
+context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+context.set_details('Method not implemented!')
+raise NotImplementedError('Method not implemented!')
+
+
+def add_BeamFnControlServicer_to_server(servicer, server):
+  rpc_method_handlers = {
+  'Control': grpc.stream_stream_rpc_method_handler(
+  servicer.Control,
+  
request_deserializer=beam__fn__api__pb2.InstructionResponse.FromString,
+  
response_serializer=beam__fn__api__pb2.InstructionRequest.SerializeToString,
+  ),
+  }
+  generic_handler = grpc.method_handlers_generic_handler(
+  'org.apache.beam.fn.v1.BeamFnControl', rpc_method_handlers)
+  server.add_generic_rpc_handlers((generic_handler,))
+
+
+class BeamFnDataStub(object):
+  """Stable
+  """
+
+  def __init__(self, channel):
+"""Constructor.
+
+Args:
+  channel: A grpc.Channel.
+"""
+self.Data = channel.stream_stream(
+'/org.apache.beam.fn.v1.BeamFnData/Data',
+request_serializer=beam__fn__api__pb2.Elements.SerializeToString,
+response_deserializer=beam__fn__api__pb2.Elements.FromString,
+)
+
+
+class BeamFnDataServicer(object):
+  """Stable
+  """
+
+  def Data(self, request_iterator, context):
+"""Used to send data between harnesses.
+"""
+context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+context.set_details('Method not implemented!')
+raise NotImplementedError('Method not implemented!')
+
+
+def add_BeamFnDataServicer_to_server(servicer, server):
+  rpc_method_handlers = {
+  'Data': grpc.stream_stream_rpc_method_handler(
+  servicer.Data,
+  request_deserializer=beam__fn__api__pb2.Elements.FromString,
+  response_serializer=beam__fn__api__pb2.Elements.SerializeToString,
+  ),
+  }
+  generic_handler = grpc.method_handlers_generic_handler(
+  'org.apache.beam.fn.v1.BeamFnData', rpc_method_handlers)
+  server.add_generic_rpc_handlers((generic_handler,))
+
+
+class BeamFnStateStub(object):
+
+  def __init__(self, channel):
+"""Constructor.
+
+Args:
+  channel: A grpc.Channel.
+"""
+self.State = channel.stream_stream(
+'/org.apache.beam.fn.v1.BeamFnState/State',
+request_serializer=beam__fn__api__pb2.StateRequest.SerializeToString,
+response_deserializer=beam__fn__api__pb2.StateResponse.FromString,
+)
+
+
+class BeamFnStateServicer(object):
+
+  def State(self, request_iterator, context):
+"""Used to get/append/clear state stored by the runner on behalf of the 
SDK.
+"""
+context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+context.set_details('Method not implemented!')
+raise NotImplementedError('Method not implemented!')
+
+
+def add_BeamFnStateServicer_to_server(servicer, server):
+  rpc_method_handlers = {
+  'State': grpc.stream_stream_rpc_method_handler(
+  servicer.State,
+  request_deserializer=beam__fn__api__pb2.StateRequest.FromString,
+  
response_serializer=beam__fn__api__pb2.StateRes

[6/7] beam git commit: Adapt to PR #2505 changes to protos.

[robertwb]

Adapt to PR #2505 changes to protos.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/488f3c67
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/488f3c67
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/488f3c67

Branch: refs/heads/master
Commit: 488f3c6730edd4a22005132f5d8c819fa2deb3a1
Parents: fbeb7e1
Author: Robert Bradshaw 
Authored: Fri Apr 21 14:22:07 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Apr 24 14:52:28 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/488f3c67/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 8e811bc..0f4c8db 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -336,7 +336,7 @@ class Pipeline(object):
 # argument evaluation order.
 root_transform_id = context.transforms.get_id(self._root_transform())
 proto = beam_runner_api_pb2.Pipeline(
-root_transform_id=root_transform_id,
+root_transform_ids=[root_transform_id],
 components=context.to_runner_api())
 return proto
 
@@ -345,8 +345,9 @@ class Pipeline(object):
 p = Pipeline(runner=runner, options=options)
 from apache_beam.runners import pipeline_context
 context = pipeline_context.PipelineContext(proto.components)
+root_transform_id, = proto.root_transform_ids
 p.transforms_stack = [
-context.transforms.get_by_id(proto.root_transform_id)]
+context.transforms.get_by_id(root_transform_id)]
 # TODO(robertwb): These are only needed to continue construction. Omit?
 p.applied_labels = set([
 t.unique_name for t in proto.components.transforms.values()])

[GitHub] beam pull request #2632: Generate Runner/Fn API protos needed for Fn harness

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2632


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[5/7] beam git commit: Add apache licence to generated files.

[robertwb]

Add apache licence to generated files.

Also add needed dependency (for docs) and fix lint.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fbeb7e13
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fbeb7e13
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fbeb7e13

Branch: refs/heads/master
Commit: fbeb7e13a2d678ddb2852c2e270eb972280a40f3
Parents: 2a59b05
Author: Robert Bradshaw 
Authored: Fri Apr 21 12:15:02 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Apr 24 14:52:28 2017 -0700

--
 pom.xml|  3 +++
 .../apache_beam/runners/api/beam_fn_api_pb2.py | 17 +
 .../runners/api/beam_fn_api_pb2_grpc.py| 17 +
 .../apache_beam/runners/api/beam_runner_api_pb2.py | 17 +
 sdks/python/run_pylint.sh  |  5 -
 sdks/python/setup.py   |  1 +
 6 files changed, 59 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fbeb7e13/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 5764204..8723ed9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1193,6 +1193,9 @@
   **/.factorypath
   **/.project
   **/.settings/**/*
+
+  
+  **/sdks/python/apache_beam/runners/api/*.py
 
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/fbeb7e13/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
index dea47e3..769b13c 100644
--- a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
+++ b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 # Generated by the protocol buffer compiler.  DO NOT EDIT!
 # source: beam_fn_api.proto
 

http://git-wip-us.apache.org/repos/asf/beam/blob/fbeb7e13/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py 
b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
index f86861c..7fbe4c1 100644
--- a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
+++ b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 # Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
 import grpc
 from grpc.framework.common import cardinality

http://git-wip-us.apache.org/repos/asf/beam/blob/fbeb7e13/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
index 3c612a9..d2006d7 100644
--- a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
+++ b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) u

[4/7] beam git commit: Add instructions to regenerate Python proto wrappers.

[robertwb]

Add instructions to regenerate Python proto wrappers.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/11103afa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/11103afa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/11103afa

Branch: refs/heads/master
Commit: 11103afa4a46cc1835f1479651bcd7c808d6a33c
Parents: 0191e04
Author: Robert Bradshaw 
Authored: Fri Apr 21 09:59:52 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Apr 24 14:52:27 2017 -0700

--
 sdks/python/apache_beam/runners/api/__init__.py | 14 ++
 1 file changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/11103afa/sdks/python/apache_beam/runners/api/__init__.py
--
diff --git a/sdks/python/apache_beam/runners/api/__init__.py 
b/sdks/python/apache_beam/runners/api/__init__.py
index cce3aca..e94673c 100644
--- a/sdks/python/apache_beam/runners/api/__init__.py
+++ b/sdks/python/apache_beam/runners/api/__init__.py
@@ -14,3 +14,17 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
+"""Checked in to avoid protoc dependency for Python development.
+
+Regenerate files with::
+
+protoc -I../common/runner-api/src/main/proto/ \
+--python_out=apache_beam/runners/api/ \
+../common/runner-api/src/main/proto/*.proto
+
+protoc -I../common/{fn,runner}-api/src/main/proto/ \
+--python_out=apache_beam/runners/api/ \
+--grpc_python_out=apache_beam/runners/api/ \
+../common/fn-api/src/main/proto/*.proto
+"""

[7/7] beam git commit: Closes #2632

[robertwb]

Closes #2632


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d7e7af8e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d7e7af8e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d7e7af8e

Branch: refs/heads/master
Commit: d7e7af8eb8aac09e189d4dfb6f103783d0c7ca05
Parents: 0191e04 488f3c6
Author: Robert Bradshaw 
Authored: Mon Apr 24 14:52:29 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Apr 24 14:52:29 2017 -0700

--
 pom.xml |3 +
 sdks/python/apache_beam/pipeline.py |5 +-
 sdks/python/apache_beam/runners/api/__init__.py |   14 +
 .../apache_beam/runners/api/beam_fn_api_pb2.py  | 2740 ++
 .../runners/api/beam_fn_api_pb2_grpc.py |  203 ++
 .../runners/api/beam_runner_api_pb2.py  |  298 +-
 sdks/python/run_pylint.sh   |5 +-
 sdks/python/setup.py|1 +
 8 files changed, 3166 insertions(+), 103 deletions(-)
--

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981966#comment-15981966
 ] 

Stephen Sisk commented on BEAM-2005:


Since I haven't heard anything and we are trying to finish soon for FSR, I'm 
doing as follows:
1. Implement a simple version of FileSystem that supports only HDFS, and only 
one configuration and gets that configuration via pipelineOptions that looks 
like a json blob
2. It will be checked in at the same location as the existing 
HadoopFileSystem.java (and thus, this will live in extensions)

I like the idea of the scheme -> module mapping, but that probably belongs in 
another JIRA if we want it.

> Add a Hadoop FileSystem implementation of Beam's FileSystem
> ---
>
> Key: BEAM-2005
> URL: https://issues.apache.org/jira/browse/BEAM-2005
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Beam's FileSystem creates an abstraction for reading from files in many 
> different places. 
> We should add a Hadoop FileSystem implementation 
> (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html)
>  - that would enable us to read from any file system that implements 
> FileSystem (including HDFS, azure, s3, etc..)
> I'm investigating this now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981958#comment-15981958
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2664

[BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Note that I got rid of ProtoCoder as a fallback coder after discussing with 
@kennknowles with a plan to replace the mechanism with a different override 
mechanism which is better with an automatic service registration scheme.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2664.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2664


commit 52609ce96ffee9c96f4130831a9a9626757f1c0a
Author: Luke Cwik 
Date:   2017-04-24T21:32:21Z

[BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.




> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2664: [BEAM-1871] Move ProtoCoder to new sdks/java/extens...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2664

[BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Note that I got rid of ProtoCoder as a fallback coder after discussing with 
@kennknowles with a plan to replace the mechanism with a different override 
mechanism which is better with an automatic service registration scheme.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2664.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2664


commit 52609ce96ffee9c96f4130831a9a9626757f1c0a
Author: Luke Cwik 
Date:   2017-04-24T21:32:21Z

[BEAM-1871] Move ProtoCoder to new sdks/java/extensions/protobuf package.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2633: Make it possible to test runners that don't support...

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/2633


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---