[beam] branch master updated (de6cdb0 -> 7681f8a)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from de6cdb0 Merge pull request #10248: [BEAM-7274] Add type conversions factory add b07415a change definition of has_unbounded_sources in PIN to a pre-determined list of sources add 83fd08f typo add a4d881e lint add c042886 remove BigQueryReader from list add c2cb314 lint add d8403e8 remove external add 0e0e0c95 remove external add 7681f8a [BEAM-8335] Change has_unbounded_sources to predetermined list of sources No new revisions were added by this update. Summary of changes: .../runners/interactive/pipeline_instrument.py | 33 ++ 1 file changed, 15 insertions(+), 18 deletions(-)
[beam] branch master updated (eb49b69 -> 9fc641a)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from eb49b69 [BEAM-8645] A test case for TimestampCombiner. (#10081) new 8e71e59 [BEAM-2929] Ensure that the Beam Java SDK sends the property "use_indexed_format" to Dataflow for side inputs which use a multimap materialization. new 31acee9 [BEAM-2929] Ensure that the Beam Go SDK sends the property "use_indexed_format" to Dataflow for side inputs which use a multimap materialization. new 9fc641a [BEAM-2929] Ensure that the Beam Python SDK sends the property "use_indexed_format" to Dataflow for side inputs which use a multimap materialization. The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dataflow/DataflowPipelineTranslator.java | 8 ++- .../dataflow/DataflowPipelineTranslatorTest.java | 62 +- .../beam/runners/dataflow/dataflowlib/messages.go | 7 +-- .../beam/runners/dataflow/dataflowlib/translate.go | 20 --- .../runners/dataflow/dataflow_runner.py| 22 +--- .../apache_beam/runners/dataflow/internal/names.py | 1 + 6 files changed, 100 insertions(+), 20 deletions(-)
[beam] 03/03: [BEAM-2929] Ensure that the Beam Python SDK sends the property "use_indexed_format" to Dataflow for side inputs which use a multimap materialization.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 9fc641ac8f7c1385c82e71e35d55c1d4d77a1147
Author: Luke Cwik
AuthorDate: Wed Nov 27 12:56:59 2019 -0800
[BEAM-2929] Ensure that the Beam Python SDK sends the property
"use_indexed_format" to Dataflow for side inputs which use a multimap
materialization.
---
.../runners/dataflow/dataflow_runner.py| 22 ++
.../apache_beam/runners/dataflow/internal/names.py | 1 +
2 files changed, 15 insertions(+), 8 deletions(-)
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index d07abc4..718ab61 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -607,7 +607,8 @@ class DataflowRunner(PipelineRunner):
return step
def _add_singleton_step(
- self, label, full_label, tag, input_step, windowing_strategy):
+ self, label, full_label, tag, input_step, windowing_strategy,
+ access_pattern):
"""Creates a CollectionToSingleton step used to handle ParDo side
inputs."""
# Import here to avoid adding the dependency for local running scenarios.
from apache_beam.runners.dataflow.internal import apiclient
@@ -620,12 +621,16 @@ class DataflowRunner(PipelineRunner):
PropertyNames.STEP_NAME: input_step.proto.name,
PropertyNames.OUTPUT_NAME: input_step.get_output(tag)})
step.encoding = self._get_side_input_encoding(input_step.encoding)
-step.add_property(
-PropertyNames.OUTPUT_INFO,
-[{PropertyNames.USER_NAME: (
-'%s.%s' % (full_label, PropertyNames.OUTPUT)),
- PropertyNames.ENCODING: step.encoding,
- PropertyNames.OUTPUT_NAME: PropertyNames.OUT}])
+
+output_info = {
+PropertyNames.USER_NAME: '%s.%s' % (full_label, PropertyNames.OUTPUT),
+PropertyNames.ENCODING: step.encoding,
+PropertyNames.OUTPUT_NAME: PropertyNames.OUT
+}
+if common_urns.side_inputs.MULTIMAP.urn == access_pattern:
+ output_info[PropertyNames.USE_INDEXED_FORMAT] = True
+step.add_property(PropertyNames.OUTPUT_INFO, [output_info])
+
step.add_property(
PropertyNames.WINDOWING_STRATEGY,
self.serialize_windowing_strategy(windowing_strategy))
@@ -820,7 +825,8 @@ class DataflowRunner(PipelineRunner):
self._add_singleton_step(
step_name, si_full_label, side_pval.pvalue.tag,
self._cache.get_pvalue(side_pval.pvalue),
- side_pval.pvalue.windowing)
+ side_pval.pvalue.windowing,
+ side_pval._side_input_data().access_pattern)
si_dict[si_label] = {
'@type': 'OutputReference',
PropertyNames.STEP_NAME: step_name,
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py
b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index 5b2dd89..fdce49b 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -117,6 +117,7 @@ class PropertyNames(object):
SOURCE_STEP_INPUT = 'custom_source_step_input'
SERIALIZED_TEST_STREAM = 'serialized_test_stream'
STEP_NAME = 'step_name'
+ USE_INDEXED_FORMAT = 'use_indexed_format'
USER_FN = 'user_fn'
USER_NAME = 'user_name'
VALIDATE_SINK = 'validate_sink'
[beam] 02/03: [BEAM-2929] Ensure that the Beam Go SDK sends the property "use_indexed_format" to Dataflow for side inputs which use a multimap materialization.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 31acee91c6fbbd8d99491d7df3fd51e68c887945
Author: Luke Cwik
AuthorDate: Wed Nov 27 11:11:31 2019 -0800
[BEAM-2929] Ensure that the Beam Go SDK sends the property
"use_indexed_format" to Dataflow for side inputs which use a multimap
materialization.
---
.../beam/runners/dataflow/dataflowlib/messages.go| 7 ---
.../beam/runners/dataflow/dataflowlib/translate.go | 20 ++--
2 files changed, 18 insertions(+), 9 deletions(-)
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/messages.go
b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/messages.go
index dc0c36b..6cf39fd 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/messages.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/messages.go
@@ -97,9 +97,10 @@ type propertiesWithPubSubMessage struct {
}
type output struct {
- UserName string `json:"user_name,omitempty"`
- OutputName string `json:"output_name,omitempty"`
- Encoding *graphx.CoderRef `json:"encoding,omitempty"`
+ UserName string `json:"user_name,omitempty"`
+ OutputName string `json:"output_name,omitempty"`
+ Encoding *graphx.CoderRef `json:"encoding,omitempty"`
+ UseIndexedFormat bool `json:"use_indexed_format,omitempty"`
}
type integer struct {
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
index 047d818..3a67015 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
@@ -137,7 +137,7 @@ func (x *translator) translateTransform(trunk string, id
string) ([]*df.Step, er
rem := reflectx.ShallowClone(t.Inputs).(map[string]string)
prop.NonParallelInputs = make(map[string]*outputReference)
- for key := range payload.SideInputs {
+ for key, side_input := range payload.SideInputs {
// Side input require an additional conversion step,
which must
// be before the present one.
delete(rem, key)
@@ -146,16 +146,24 @@ func (x *translator) translateTransform(trunk string, id
string) ([]*df.Step, er
ref := x.pcollections[t.Inputs[key]]
c := x.translateCoder(pcol, pcol.CoderId)
+ var output_info output
+ output_info = output{
+ UserName: "i0",
+ OutputName: "i0",
+ Encoding: graphx.WrapIterable(c),
+ }
+ if graphx.URNMultimapSideInput ==
side_input.GetAccessPattern().GetUrn() {
+ output_info.UseIndexedFormat = true
+ }
+
side := &df.Step{
Name: fmt.Sprintf("view%v_%v", id, key),
Kind: sideInputKind,
Properties: newMsg(properties{
ParallelInput: ref,
- OutputInfo: []output{{
- UserName: "i0",
- OutputName: "i0",
- Encoding:
graphx.WrapIterable(c),
- }},
+ OutputInfo: []output{
+ output_info,
+ },
UserName: userName(trunk,
fmt.Sprintf("AsView%v_%v", id, key)),
}),
}
[beam] 01/03: [BEAM-2929] Ensure that the Beam Java SDK sends the property "use_indexed_format" to Dataflow for side inputs which use a multimap materialization.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8e71e59eae495ab74c5c20f50456a5c9c5416905
Author: Luke Cwik
AuthorDate: Wed Nov 27 10:39:12 2019 -0800
[BEAM-2929] Ensure that the Beam Java SDK sends the property
"use_indexed_format" to Dataflow for side inputs which use a multimap
materialization.
---
.../dataflow/DataflowPipelineTranslator.java | 8 ++-
.../dataflow/DataflowPipelineTranslatorTest.java | 62 +-
2 files changed, 67 insertions(+), 3 deletions(-)
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 41e5cbb..45e0c33 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -83,6 +83,7 @@ import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.Materializations;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.View;
@@ -710,8 +711,11 @@ public class DataflowPipelineTranslator {
String generatedName = String.format("%s.out%d", stepName,
outputInfoList.size());
addString(outputInfo, PropertyNames.USER_NAME, generatedName);
- if (value instanceof PCollection
- &&
translator.runner.doesPCollectionRequireIndexedFormat((PCollection) value)) {
+ if ((value instanceof PCollection
+ &&
translator.runner.doesPCollectionRequireIndexedFormat((PCollection) value))
+ || ((value instanceof PCollectionView)
+ && (Materializations.MULTIMAP_MATERIALIZATION_URN.equals(
+ ((PCollectionView)
value).getViewFn().getMaterialization().getUrn() {
addBoolean(outputInfo, PropertyNames.USE_INDEXED_FORMAT, true);
}
if (valueCoder != null) {
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index 85b1e22..c5c3ddc 100644
---
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -736,8 +736,8 @@ public class DataflowPipelineTranslatorTest implements
Serializable {
@Test
public void testSplittableParDoTranslationFnApi() throws Exception {
DataflowPipelineOptions options = buildPipelineOptions();
-DataflowRunner runner = DataflowRunner.fromOptions(options);
options.setExperiments(Arrays.asList("beam_fn_api"));
+DataflowRunner runner = DataflowRunner.fromOptions(options);
DataflowPipelineTranslator translator =
DataflowPipelineTranslator.fromOptions(options);
Pipeline pipeline = Pipeline.create(options);
@@ -852,6 +852,66 @@ public class DataflowPipelineTranslatorTest implements
Serializable {
}
@Test
+ public void testToSingletonTranslationWithFnApiSideInput() throws Exception {
+// A "change detector" test that makes sure the translation
+// of getting a PCollectionView does not change
+// in bad ways during refactor
+
+DataflowPipelineOptions options = buildPipelineOptions();
+options.setExperiments(Arrays.asList("beam_fn_api"));
+DataflowPipelineTranslator translator =
DataflowPipelineTranslator.fromOptions(options);
+
+Pipeline pipeline = Pipeline.create(options);
+pipeline.apply(Create.of(1)).apply(View.asSingleton());
+DataflowRunner runner = DataflowRunner.fromOptions(options);
+runner.replaceTransforms(pipeline);
+Job job = translator.translate(pipeline, runner,
Collections.emptyList()).getJob();
+assertAllStepOutputsHaveUniqueIds(job);
+
+List steps = job.getSteps();
+assertEquals(14, steps.size());
+
+Step collectionToSingletonStep = steps.get(steps.size() - 1);
+assertEquals("CollectionToSingleton", collectionToSingletonStep.getKind());
+
+@SuppressWarnings("unchecked")
+List> ctsOutputs =
+(List>)
+steps.get(steps.size() -
1).getProperties().get(PropertyNames.OUTPUT_INFO);
+assertTrue(Structs.getBoolean(Iterables.getOnlyElement(ctsOutputs),
"u
[beam] branch master updated (1f64ba3 -> 5e9739e)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1f64ba3 Merge pull request #10105: [BEAM-4776] Add metrics support to Java PortableRunner add 26596c8 [BEAM-8733] Handle the registration request synchronously in the Python SDK harness. add 5e9739e [BEAM-8733] Handle the registration request synchronously in the Python&Java SDK harness No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 14 ++ 1 file changed, 6 insertions(+), 8 deletions(-)
[beam] branch master updated (5bd9d1d -> 3a79080)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5bd9d1d Merge pull request #10234 from apache/aaltay-patch-3 add a320d84 [BEAM-8832] Allow GCS staging upload chunk size to be increased >1M when setting GcsUploadBufferSizeBytes pipeline option, to improve performance add 3a79080 [BEAM-8832] Allow GCS staging upload chunk size to be increased >1M when setting GcsUploadBufferSizeBytes pipeline option, to improve performance No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (dce302d -> a8f3195)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dce302d Merge pull request #10126 from sunjincheng121/BEAM-8619-PR add dac10c9 [BEAM-8805] Remove obsolete worker_threads experiment in tests add a8f3195 [BEAM-8805] Remove obsolete worker_threads experiment in tests No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 - sdks/python/test-suites/portable/py2/build.gradle | 2 -- 2 files changed, 3 deletions(-)
[beam] branch master updated (031a8c3 -> dce302d)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 031a8c3 Merge pull request #10191 [BEAM-8802] Don't clear watermark hold when adding elements. new f5c4e8b [BEAM-8619] Move reusable information to BundleProcessor. new 10b097a [BEAM-8619] Extract tearDown functions into BundleProcessor. new 209d0de [BEAM-8619] Reuse the BundleProcessor between bundles for the same bundle descriptor. new 5773420 [BEAM-8619] Teardown the DoFns when upon control service termination for the Java SDK harness. new dce302d Merge pull request #10126 from sunjincheng121/BEAM-8619-PR The 24383 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/core/metrics/CounterCell.java | 6 + .../runners/core/metrics/DistributionCell.java | 6 + .../core/metrics/ExecutionStateSampler.java| 5 + .../core/metrics/ExecutionStateTracker.java| 21 ++ .../beam/runners/core/metrics/GaugeCell.java | 6 + .../beam/runners/core/metrics/MetricCell.java | 3 + .../runners/core/metrics/MetricsContainerImpl.java | 13 + .../core/metrics/MetricsContainerStepMap.java | 8 + .../runners/core/metrics/SimpleExecutionState.java | 5 + .../runners/core/metrics/SimpleStateRegistry.java | 7 + .../beam/runners/core/metrics/CounterCellTest.java | 11 + .../runners/core/metrics/DistributionCellTest.java | 11 + .../core/metrics/ExecutionStateSamplerTest.java| 7 + .../core/metrics/ExecutionStateTrackerTest.java| 84 + .../beam/runners/core/metrics/GaugeCellTest.java | 11 + .../core/metrics/MetricsContainerStepMapTest.java | 44 +++ .../core/metrics/SimpleExecutionStateTest.java | 3 + .../core/metrics/SimpleStateRegistryTest.java | 17 + .../runners/dataflow/worker/DeltaCounterCell.java | 5 + .../dataflow/worker/DeltaDistributionCell.java | 5 + .../beam/fn/harness/BeamFnDataReadRunner.java | 3 + .../beam/fn/harness/BeamFnDataWriteRunner.java | 3 + .../beam/fn/harness/BoundedSourceRunner.java | 3 + .../org/apache/beam/fn/harness/CombineRunners.java | 3 + .../fn/harness/DoFnPTransformRunnerFactory.java| 6 + .../org/apache/beam/fn/harness/FlattenRunner.java | 3 + .../apache/beam/fn/harness/FnApiDoFnRunner.java| 5 + .../java/org/apache/beam/fn/harness/FnHarness.java | 1 + .../org/apache/beam/fn/harness/MapFnRunners.java | 3 + .../beam/fn/harness/PTransformRunnerFactory.java | 4 + .../harness/SplittableProcessElementsRunner.java | 5 + .../fn/harness/control/ProcessBundleHandler.java | 344 +- .../harness/data/PCollectionConsumerRegistry.java | 5 + .../harness/data/PTransformFunctionRegistry.java | 5 + .../beam/fn/harness/AssignWindowsRunnerTest.java | 1 + .../beam/fn/harness/BeamFnDataReadRunnerTest.java | 6 + .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 6 + .../beam/fn/harness/BoundedSourceRunnerTest.java | 4 + .../apache/beam/fn/harness/CombineRunnersTest.java | 4 + .../apache/beam/fn/harness/FlattenRunnerTest.java | 2 + .../beam/fn/harness/FnApiDoFnRunnerTest.java | 34 ++ .../apache/beam/fn/harness/MapFnRunnersTest.java | 10 + .../harness/control/ProcessBundleHandlerTest.java | 404 - 43 files changed, 1049 insertions(+), 93 deletions(-) create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/ExecutionStateTrackerTest.java
[beam] branch master updated (543abed -> 4c18cb4)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 543abed Merge pull request #10163 from ibzib/artifact-port
add 4c18cb4 [BEAM-7948] Add time-based cache threshold support in the
Java data s… (#9949)
No new revisions were added by this update.
Summary of changes:
.../worker/fn/data/BeamFnDataGrpcService.java | 35 +
.../worker/fn/control/TimerReceiverTest.java | 5 +-
.../control/DefaultJobBundleFactory.java | 3 +-
.../runners/fnexecution/data/GrpcDataService.java | 19 ++-
.../runners/fnexecution/EmbeddedSdkHarness.java| 3 +-
.../fnexecution/control/RemoteExecutionTest.java | 5 +-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 4 +-
.../fnexecution/data/GrpcDataServiceTest.java | 9 +-
.../beam/runners/samza/SamzaExecutionContext.java | 3 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 146 +++--
...mFnDataSizeBasedBufferingOutboundObserver.java} | 44 ++-
...amFnDataTimeBasedBufferingOutboundObserver.java | 116
...ataSizeBasedBufferingOutboundObserverTest.java} | 36 +++--
...DataTimeBasedBufferingOutboundObserverTest.java | 138 +++
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 27 +---
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 2 +-
16 files changed, 382 insertions(+), 213 deletions(-)
copy
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/{BeamFnDataBufferingOutboundObserver.java
=> BeamFnDataSizeBasedBufferingOutboundObserver.java} (69%)
create mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataTimeBasedBufferingOutboundObserver.java
rename
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/{BeamFnDataBufferingOutboundObserverTest.java
=> BeamFnDataSizeBasedBufferingOutboundObserverTest.java} (80%)
create mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataTimeBasedBufferingOutboundObserverTest.java
[beam] branch master updated (7d73314 -> 7a8a26b)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7d73314 [BEAM-8016] Pipeline Graph (#10132) add 7a8a26b [BEAM-7278, BEAM-2530] Add support for using a Java linkage testing tool to aid upgrading dependencies. (#10184) No new revisions were added by this update. Summary of changes: build.gradle | 38 ++ 1 file changed, 38 insertions(+)
[beam] branch master updated (d1b513d -> 9407578)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d1b513d [BEAM-8575] Add a Python test to test windowing in DoFn finish_bundle() (#10145) add 9407578 [BEAM-3419] Flesh out iterable side inputs and key enumeration for multimaps in shared libraries (#10147) No new revisions were added by this update. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 60 +- .../core/InMemoryMultimapSideInputView.java| 62 +- .../core/InMemoryMultimapSideInputViewTest.java| 30 + .../functions/FlinkExecutableStageFunction.java| 2 + .../functions/FlinkSideInputReader.java| 7 +- .../FlinkStreamingSideInputHandlerFactory.java | 94 ++- .../streaming/ExecutableStageDoFnOperator.java | 2 + .../dataflow/worker/IsmSideInputReader.java| 8 +- .../control/DataflowSideInputHandlerFactory.java | 54 +- .../fn/control/ProcessRemoteBundleOperation.java | 2 + .../DataflowSideInputHandlerFactoryTest.java | 40 +- .../control/ProcessBundleDescriptors.java | 6 +- .../fnexecution/state/StateRequestHandlers.java| 279 ++-- .../translation/BatchSideInputHandlerFactory.java | 159 ++--- .../fnexecution/control/RemoteExecutionTest.java | 85 ++- .../BatchSideInputHandlerFactoryTest.java | 95 ++- .../translation/SparkExecutableStageFunction.java | 2 + .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 779 - .../beam/model/pipeline_v1/beam_runner_api.pb.go | 187 ++--- .../org/apache/beam/sdk/transforms/DoFnTester.java | 14 +- .../beam/sdk/transforms/Materializations.java | 53 +- .../beam/fn/harness/state/MultimapSideInput.java | 18 + .../runners/portability/fn_api_runner.py | 35 +- .../apache_beam/runners/worker/bundle_processor.py | 17 +- 24 files changed, 1259 insertions(+), 831 deletions(-)
[beam] branch master updated (600cb26 -> 3685a6e)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 600cb26 Allow metrics update to be tolerant to uninitalized metric containers. (#10148) add 3685a6e [BEAM-3493] Prevent users from "implementing" PipelineOptions. (#10005) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/options/PipelineOptions.java | 3 +++ .../core/src/test/java/org/apache/beam/sdk/PipelineTest.java | 11 +++ 2 files changed, 14 insertions(+)
[beam] branch master updated (eb81905 -> 600cb26)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from eb81905 [BEAM-8575] Test flatten a single PC and test flatten a flattened PC (#10102) add 600cb26 Allow metrics update to be tolerant to uninitalized metric containers. (#10148) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/statesampler_fast.pyx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (7da5071 -> 0a9410e)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7da5071 [BEAM-8729] Gracefully skip irrelevant http/https lines from gradle-version-plugin (#10154) add 0a9410e [BEAM-8335] Add timestamp and duration to/from protos to Python SDK (#10146) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/utils/timestamp.py | 54 + sdks/python/apache_beam/utils/timestamp_test.py | 36 + 2 files changed, 90 insertions(+)
[beam] branch master updated (860e46f -> 7da5071)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 860e46f Merge pull request #10128: [BEAM-8672] Keep Python process alive when using LOOPBACK execution mode add 7da5071 [BEAM-8729] Gracefully skip irrelevant http/https lines from gradle-version-plugin (#10154) No new revisions were added by this update. Summary of changes: .../jenkins/dependency_check/dependency_check_report_generator.py | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated (97093f4 -> 76b9bdd)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 97093f4 Merge pull request #10131 from Ardagan/LowerExamplesTimeout add bda8f62 [BEAM-8575] Test a customized window fn work as expected add b41fe1a fixup add 3358b37 fixup add c6c8fbd Merge branch 'master' into addTest add 76b9bdd [BEAM-8575] Test a customized window fn work as expected No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/window_test.py | 33 +++ 1 file changed, 33 insertions(+)
[beam] branch master updated (d16f2a9 -> 5de27d5)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d16f2a9 Bump python precommit timeout add 33cc30e [BEAM-8151] Swap to create SdkWorkers on demand when processing jobs add 5de27d5 [BEAM-8151] Swap to create SdkWorkers on demand when processing jobs No new revisions were added by this update. Summary of changes: .../runners/portability/fn_api_runner.py | 8 +- .../runners/portability/fn_api_runner_test.py | 12 +-- .../runners/portability/portable_runner.py | 1 - .../apache_beam/runners/worker/sdk_worker.py | 103 ++--- .../apache_beam/runners/worker/sdk_worker_main.py | 30 -- .../runners/worker/sdk_worker_main_test.py | 47 +- .../apache_beam/runners/worker/sdk_worker_test.py | 7 +- .../apache_beam/runners/worker/worker_pool_main.py | 16 +--- sdks/python/apache_beam/transforms/environments.py | 24 ++--- .../apache_beam/transforms/environments_test.py| 9 +- .../apache_beam/utils/thread_pool_executor_test.py | 8 ++ 11 files changed, 55 insertions(+), 210 deletions(-)
[beam] branch master updated (85248f3 -> d16f2a9)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 85248f3 [BEAM-8557] Add log for the dropped unknown response add de9cd0b Bump python precommit timeout add d16f2a9 Bump python precommit timeout No new revisions were added by this update. Summary of changes: .test-infra/jenkins/job_PreCommit_Python.groovy | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (8f54d07 -> 85248f3)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8f54d07 Merge pull request #10052 from robertwb/no-gauge add 4013214 [BEAM-8557] Add log for the dropped unknown response add 85248f3 [BEAM-8557] Add log for the dropped unknown response No new revisions were added by this update. Summary of changes: .../fnexecution/control/FnApiControlClient.java| 22 -- .../harness/state/BeamFnStateGrpcClientCache.java | 14 -- 2 files changed, 20 insertions(+), 16 deletions(-)
[beam] branch master updated (db11e06 -> e35e6b8)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from db11e06 [BEAM-8661] Moving other modules to have per-module loggers add 308f4ca [BEAM-8667] Bound the number of element bundles buffered off the data channel. add e35e6b8 [BEAM-8667] Bound elements buffered off the data channel. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/data_plane.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[beam] branch master updated (8bf9468 -> db11e06)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8bf9468 [BEAM-8661] Moving io module to have per-module logger add fab9e3e [BEAM-8661] Moving other modules to have per-module loggers add d4818ec Fix lint add ccb6af2 Removing extra line between constants add db11e06 [BEAM-8661] Moving other modules to have per-module loggers No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/internal/gcp/auth.py | 10 +++--- sdks/python/apache_beam/options/pipeline_options.py | 11 +++ .../apache_beam/testing/benchmarks/nexmark/nexmark_util.py | 4 +++- .../testing/load_tests/load_test_metrics_utils.py| 10 ++ sdks/python/apache_beam/testing/pipeline_verifiers.py| 8 +--- sdks/python/apache_beam/transforms/core.py | 8 +--- sdks/python/apache_beam/transforms/trigger.py| 7 +-- sdks/python/apache_beam/typehints/typehints.py | 6 -- sdks/python/apache_beam/utils/profiler.py| 12 +++- sdks/python/apache_beam/utils/retry.py | 7 +-- sdks/python/apache_beam/utils/subprocess_server.py | 12 +++- 11 files changed, 61 insertions(+), 34 deletions(-)
[beam] branch master updated (7588387 -> 8bf9468)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7588387 Merge pull request #10108 from ibzib/artifact-endpoint add a1e0daa [BEAM-8661] Moving io module to have per-module logger add 38d0f5d Fix lint add c9e2699 Removing extra space between constants add 8bf9468 [BEAM-8661] Moving io module to have per-module logger No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/filebasedsink.py| 19 +++-- sdks/python/apache_beam/io/filebasedsink_test.py | 4 ++- sdks/python/apache_beam/io/fileio.py | 17 ++- sdks/python/apache_beam/io/filesystemio_test.py| 4 ++- .../io/gcp/big_query_query_to_table_it_test.py | 5 +++- sdks/python/apache_beam/io/gcp/bigquery.py | 13 + .../apache_beam/io/gcp/bigquery_file_loads.py | 14 + .../apache_beam/io/gcp/bigquery_file_loads_test.py | 8 -- .../apache_beam/io/gcp/bigquery_read_it_test.py| 9 -- sdks/python/apache_beam/io/gcp/bigquery_test.py| 9 -- sdks/python/apache_beam/io/gcp/bigquery_tools.py | 33 +++--- .../apache_beam/io/gcp/bigquery_write_it_test.py | 9 -- .../apache_beam/io/gcp/datastore/v1/datastoreio.py | 21 -- .../apache_beam/io/gcp/datastore/v1/helper.py | 5 +++- .../datastore/v1new/datastore_write_it_pipeline.py | 12 .../io/gcp/datastore/v1new/datastoreio.py | 17 ++- .../io/gcp/datastore/v1new/types_test.py | 7 +++-- .../io/gcp/datastore_write_it_pipeline.py | 13 + sdks/python/apache_beam/io/gcp/gcsio.py| 21 -- sdks/python/apache_beam/io/gcp/gcsio_overrides.py | 6 ++-- .../apache_beam/io/gcp/tests/bigquery_matcher.py | 18 ++-- .../apache_beam/io/gcp/tests/pubsub_matcher.py | 4 ++- sdks/python/apache_beam/io/gcp/tests/utils.py | 5 +++- sdks/python/apache_beam/io/hadoopfilesystem.py | 4 ++- sdks/python/apache_beam/io/iobase.py | 5 +++- sdks/python/apache_beam/io/mongodbio.py| 7 +++-- sdks/python/apache_beam/io/mongodbio_it_test.py| 10 --- sdks/python/apache_beam/io/range_trackers.py | 13 + sdks/python/apache_beam/io/source_test_utils.py| 11 +--- sdks/python/apache_beam/io/textio.py | 5 +++- sdks/python/apache_beam/io/tfrecordio.py | 5 +++- sdks/python/apache_beam/io/vcfio.py| 5 +++- 32 files changed, 211 insertions(+), 127 deletions(-)
[beam] branch master updated (0b78d91 -> e265826)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0b78d91 [BEAM-8612] Convert []beam.T to the underlying type []T when passed to a universal type. (#10066) new e9f766f [BEAM-8151, BEAM-7848] Add an UnboundedThreadPoolExecutor implementation. new 1b62310 [BEAM-8151, BEAM-7848] Swap to using a thread pool which is unbounded and shrinks when threads are idle. new e265826 Merge pull request #9477 from lukecwik/python The 23981 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../runners/portability/artifact_service_test.py | 6 +- .../runners/portability/expansion_service_test.py | 4 +- .../runners/portability/fn_api_runner.py | 30 +--- .../runners/portability/local_job_service.py | 6 +- .../runners/portability/portable_stager_test.py| 4 +- .../apache_beam/runners/worker/data_plane_test.py | 4 +- .../apache_beam/runners/worker/log_handler_test.py | 4 +- .../apache_beam/runners/worker/sdk_worker.py | 14 +- .../apache_beam/runners/worker/sdk_worker_main.py | 2 +- .../apache_beam/runners/worker/sdk_worker_test.py | 4 +- .../apache_beam/runners/worker/worker_pool_main.py | 4 +- .../apache_beam/utils/thread_pool_executor.py | 170 + .../apache_beam/utils/thread_pool_executor_test.py | 106 + sdks/python/scripts/generate_pydoc.sh | 1 + 14 files changed, 308 insertions(+), 51 deletions(-) create mode 100644 sdks/python/apache_beam/utils/thread_pool_executor.py create mode 100644 sdks/python/apache_beam/utils/thread_pool_executor_test.py
[beam] branch master updated (bff9a1a -> ee3f951)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bff9a1a Merge pull request #10086: [BEAM-7078] Update amazon-kinesis-client to 1.13.0 add ee3f951 [BEAM-3419] Explicitly model iterable side inputs as part of the Beam portability APIs. (#10092) No new revisions were added by this update. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 34 + .../pipeline/src/main/proto/beam_runner_api.proto | 9 + .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 634 -- .../beam/model/jobmanagement_v1/beam_job_api.pb.go | 284 +++--- .../beam/model/pipeline_v1/beam_runner_api.pb.go | 949 +++-- 5 files changed, 1077 insertions(+), 833 deletions(-)
[beam] branch master updated (a27acf3 -> 7b482e4)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a27acf3 Merge pull request #9892: [BEAM-8427] [SQL] buildIOWrite for MongoDb Table add 7b482e4 [BEAM-7233, BEAM-6868] Create an implementation for a bundle finalizer that stores bundle finalization requests in memory. (#10074) No new revisions were added by this update. Summary of changes: .../control/BundleFinalizationHandler.java | 11 +++- .../control/BundleFinalizationHandlers.java| 65 ++ .../fnexecution/control/SdkHarnessClient.java | 4 +- .../control/BundleFinalizationHandlersTest.java| 56 +++ .../fnexecution/control/SdkHarnessClientTest.java | 12 +--- 5 files changed, 134 insertions(+), 14 deletions(-) create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlers.java create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlersTest.java
[beam] branch master updated (de2cfae -> c5e3253)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from de2cfae Beam changes to bypass recording timing for first N times seeing a new batch size. (#10063) add c5e3253 [BEAM-8575] Windows idempotency: Applying the same window fn (or wind… (#10071) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/window_test.py | 31 +++ 1 file changed, 31 insertions(+)
[beam] branch beam7233 created (now b834e53)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch beam7233 in repository https://gitbox.apache.org/repos/asf/beam.git. at b834e53 [BEAM-8594] Remove unnecessary error check in DataFlow Runner No new revisions were added by this update.
[beam] branch master updated (c2e58c5 -> b834e53)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c2e58c5 Merge pull request #10047: [BEAM-8028] Add spark_runner.py add 5d1 [BEAM-8594] Remove unnecessary error check in DataFlow Runner add f038020 fixup add 3265605 fix codestyle add b834e53 [BEAM-8594] Remove unnecessary error check in DataFlow Runner No new revisions were added by this update. Summary of changes: .../dataflow/worker/FnApiWindowMappingFn.java | 19 +- .../control/RegisterAndProcessBundleOperation.java | 75 +++--- .../control/InstructionRequestHandler.java | 5 +- 3 files changed, 31 insertions(+), 68 deletions(-)
[beam] branch master updated (0d4f3de -> 212fb09)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 0d4f3de Merge pull request #10033 [BEAM-8575] Discarding mode with
early data trigger test.
add ef7ec62 [BEAM-2939, BEAM-6868, BEAM-7233] Expose residual roots and
bundle finalization callbacks to portable runners.
add 212fb09 [BEAM-2939, BEAM-5600, BEAM-6868, BEAM-7233] Expose residual
roots and bundle finalization callbacks to portable runners.
No new revisions were added by this update.
Summary of changes:
...stHandler.java => BundleCheckpointHandler.java} | 15 ++-
...Factory.java => BundleFinalizationHandler.java} | 15 +--
.../fnexecution/control/SdkHarnessClient.java | 68 +++--
.../fnexecution/control/SdkHarnessClientTest.java | 106 +
4 files changed, 188 insertions(+), 16 deletions(-)
copy
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{InstructionRequestHandler.java
=> BundleCheckpointHandler.java} (67%)
copy
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{JobBundleFactory.java
=> BundleFinalizationHandler.java} (63%)
[beam] branch master updated (c7be064 -> e852ac8)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c7be064 Merge pull request #10036 [BEAM-8589] Print pipeline proto and pipeline options in DEBUG loglevel. add c32842b [BEAM-5600, BEAM-2939] Add SplittableParDo expansion logic to runner's core. add 45152b8 fixup! Lint add 91fd0df fixup! Fix checkstyle & spotbugs add e852ac8 [BEAM-5600, BEAM-2939] Add SplittableParDo expansion logic to runner's core. No new revisions were added by this update. Summary of changes: .../graph/GreedyPCollectionFusers.java | 11 +- .../core/construction/graph/ProtoOverrides.java| 26 +- .../core/construction/graph/QueryablePipeline.java | 4 + .../graph/SplittableParDoExpander.java | 273 + .../graph/SplittableParDoExpanderTest.java | 122 + .../beam/runners/flink/FlinkPipelineRunner.java| 13 +- .../beam/runners/samza/SamzaPipelineRunner.java| 13 +- .../beam/runners/spark/SparkPipelineRunner.java| 13 +- .../runners/portability/flink_runner_test.py | 3 - .../runners/portability/portable_runner.py | 5 +- 10 files changed, 463 insertions(+), 20 deletions(-) create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpander.java create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpanderTest.java
[beam] branch master updated (b951993 -> f95de04)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b951993 [BEAM-8587] TestStream for Dataflow runner add 42da6ae BEAM-8579: Strip UTF-8 BOM from TextSource output. add f95de04 [BEAM-8579] Strip UTF-8 BOM from TextSource output. No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/TextSource.java| 6 + .../org/apache/beam/sdk/io/TextSourceTest.java | 158 + 2 files changed, 164 insertions(+) create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java
[beam] branch master updated (36d6534 -> b951993)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 36d6534 Merge pull request #10031: [BEAM-8427] [SQL] Create a table and a table provider for MongoDB add 3535958 Adds TestStream support for Dataflow runner. add f6ce376 Marks run_TestStream as not a unittest, since it was trying to be 'run' when running other tests. Moves bad copy of internal/names.py to dataflow/internal/names.py, where it was supposed to go. add be08efd Fixes line-to-long lint error. add f31a0df Replaces unittest annotation with setting __test__ = False to prevent nosetest from trying to run the method. add b951993 [BEAM-8587] TestStream for Dataflow runner No new revisions were added by this update. Summary of changes: .../runners/dataflow/dataflow_runner.py| 46 ++ .../apache_beam/runners/dataflow/internal/names.py | 11 -- sdks/python/apache_beam/testing/test_stream.py | 2 +- .../python/apache_beam/testing/test_stream_test.py | 23 +++ 4 files changed, 70 insertions(+), 12 deletions(-)
[beam] branch master updated (7124651 -> 60845e5)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7124651 [BEAM-8605] Function display_graph() in example do not exist add ad84e55 Still use fake pubsub signals when using windmill appliance with dataflow streaming. add 60845e5 [BEAM-8524] Still use fake pubsub signals when using windmill appliance No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/dataflow/DataflowRunner.java | 3 ++- .../org/apache/beam/runners/dataflow/TransformTranslator.java | 8 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 8 +++- 3 files changed, 17 insertions(+), 2 deletions(-)
[beam] branch master updated (47d5ee9 -> 7124651)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 47d5ee9 Merge pull request #9778 from robinyqiu/hll add a1d8d21 Update pipeline_graph.py add 7124651 [BEAM-8605] Function display_graph() in example do not exist No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (0a75cb6 -> c7ffbde)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0a75cb6 Merge pull request #10015 from andrefaraujo/master Option to skip display operations in beam's InteractiveRunner. add 15b3db2 [BEAM-8472] test Java default GCP region add c7ffbde [BEAM-8472] test Java default GCP region No new revisions were added by this update. Summary of changes: .../dataflow/options/DataflowPipelineOptions.java | 57 ++ .../options/DataflowPipelineOptionsTest.java | 48 -- 2 files changed, 80 insertions(+), 25 deletions(-)
[beam] branch master updated (1d81898 -> f481cd4)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1d81898 Merge pull request #9963 from apache/dependabot/pip/sdks/python/container/pyyaml-5.1 add 4af507f Always use shadow configuration for direct runner dep add f481cd4 [BEAM-8602] Always use shadow configuration for direct runner dependencies No new revisions were added by this update. Summary of changes: sdks/java/extensions/euphoria/build.gradle | 2 +- sdks/java/extensions/jackson/build.gradle | 2 +- sdks/java/extensions/join-library/build.gradle | 2 +- sdks/java/extensions/kryo/build.gradle | 2 +- sdks/java/extensions/sketching/build.gradle | 2 +- sdks/java/extensions/sorter/build.gradle| 2 +- sdks/java/extensions/sql/build.gradle | 2 +- sdks/java/extensions/zetasketch/build.gradle| 2 +- sdks/java/io/amazon-web-services/build.gradle | 2 +- sdks/java/io/amazon-web-services2/build.gradle | 2 +- sdks/java/io/amqp/build.gradle | 2 +- sdks/java/io/cassandra/build.gradle | 2 +- sdks/java/io/clickhouse/build.gradle| 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle | 2 +- .../java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle | 2 +- sdks/java/io/hadoop-file-system/build.gradle| 2 +- sdks/java/io/hadoop-format/build.gradle | 2 +- sdks/java/io/hbase/build.gradle | 2 +- sdks/java/io/hcatalog/build.gradle | 2 +- sdks/java/io/jdbc/build.gradle | 2 +- sdks/java/io/jms/build.gradle | 2 +- sdks/java/io/kafka/build.gradle | 2 +- sdks/java/io/kinesis/build.gradle | 2 +- sdks/java/io/kudu/build.gradle | 2 +- sdks/java/io/mongodb/build.gradle | 2 +- sdks/java/io/mqtt/build.gradle | 2 +- sdks/java/io/parquet/build.gradle | 2 +- sdks/java/io/rabbitmq/build.gradle | 2 +- sdks/java/io/redis/build.gradle | 2 +- sdks/java/io/solr/build.gradle | 2 +- sdks/java/io/synthetic/build.gradle | 2 +- sdks/java/io/tika/build.gradle | 2 +- sdks/java/io/xml/build.gradle | 2 +- 35 files changed, 35 insertions(+), 35 deletions(-)
[beam] branch master updated (df0e7e8 -> 4364a21)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from df0e7e8 Merge pull request #9974 from ibzib/java-default-region add 2e5f89a Javadoc that the sdk.util package is internal add 4364a21 Javadoc that the sdk.util package is internal No new revisions were added by this update. Summary of changes: .../core/src/main/java/org/apache/beam/sdk/util/package-info.java | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-)
[beam] branch master updated (fd06b7e -> ec4fd27)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fd06b7e Fix generic comparisons on protobuf messages add 8327160 Update DoFn.SplitRestriction signature add df34941 Clarify that SplitRestriction outputs Restrictions add ec4fd27 [BEAM-8573] Updates documented signature for @SplitRestriction No new revisions were added by this update. Summary of changes: .../core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated: Removing some trailing whitespace.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7f90f2b Removing some trailing whitespace.
new 1bff5ae Merge pull request #10001 from youngoli/patch-9
7f90f2b is described below
commit 7f90f2bc1f3b5e57b45727b8b36548dde087863d
Author: Daniel Oliveira
AuthorDate: Tue Nov 5 14:41:08 2019 -0800
Removing some trailing whitespace.
Not a big change, but a linter was complaining about this and it got
annoying.
The whitespace seems like an accident, but just in case it's intentional
I'll have the original author review.
---
.../apache_beam/runners/interactive/display/pcoll_visualization.py| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
index 3709deb..1f0e925 100644
--- a/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
+++ b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
@@ -73,8 +73,8 @@ _OVERVIEW_HTML_TEMPLATE = """
document.querySelector("#{display_id}").protoInput =
"{protostr}";
"""
_DATAFRAME_PAGINATION_TEMPLATE = """
-https://ajax.googleapis.com/ajax/libs/jquery/2.2.2/jquery.min.js"</a>;>
-https://cdn.datatables.net/1.10.16/js/jquery.dataTables.js"</a>;>
+https://ajax.googleapis.com/ajax/libs/jquery/2.2.2/jquery.min.js"</a>;>
+https://cdn.datatables.net/1.10.16/js/jquery.dataTables.js"</a>;>
https://cdn.datatables.net/1.10.16/css/jquery.dataTables.css";>
{dataframe_html}
[beam] branch master updated (16fbe89 -> ddd115a)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 16fbe89 Merge pull request #9836 [BEAM-8435] Implement PaneInfo computation for Python. add b09b1b6 [BEAM-8539] Provide an initial definition of all job states and the state transition diagram add ddd115a Merge pull request #9969 from apache/beam8539 No new revisions were added by this update. Summary of changes: model/job-management/src/main/proto/beam_job_api.proto | 11 +++ 1 file changed, 11 insertions(+)
[beam] branch beam8539 created (now b09b1b6)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch beam8539 in repository https://gitbox.apache.org/repos/asf/beam.git. at b09b1b6 [BEAM-8539] Provide an initial definition of all job states and the state transition diagram This branch includes the following new commits: new b09b1b6 [BEAM-8539] Provide an initial definition of all job states and the state transition diagram The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: [BEAM-8539] Provide an initial definition of all job states and the state transition diagram
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch beam8539
in repository https://gitbox.apache.org/repos/asf/beam.git
commit b09b1b643e9413f7ff7a1ccf5f6ac3f6700dcb0b
Author: Lukasz Cwik
AuthorDate: Fri Nov 1 11:46:51 2019 -0700
[BEAM-8539] Provide an initial definition of all job states and the state
transition diagram
---
model/job-management/src/main/proto/beam_job_api.proto | 11 +++
1 file changed, 11 insertions(+)
diff --git a/model/job-management/src/main/proto/beam_job_api.proto
b/model/job-management/src/main/proto/beam_job_api.proto
index 2ebc1de..e9f0eb9 100644
--- a/model/job-management/src/main/proto/beam_job_api.proto
+++ b/model/job-management/src/main/proto/beam_job_api.proto
@@ -201,6 +201,16 @@ message JobMessagesResponse {
}
// Enumeration of all JobStates
+//
+// The state transition diagram is:
+// STOPPED -> STARTING -> RUNNING -> DONE
+// \> FAILED
+// \> CANCELLING -> CANCELLED
+// \> UPDATING -> UPDATED
+// \> DRAINING -> DRAINED
+//
+// Transitions are optional such that a job may go from STOPPED to RUNNING
+// without needing to pass through STARTING.
message JobState {
enum Enum {
UNSPECIFIED = 0;
@@ -214,6 +224,7 @@ message JobState {
DRAINED = 8;
STARTING = 9;
CANCELLING = 10;
+UPDATING = 11;
}
}
[beam] branch master updated (5fa2c7f -> f196d8e)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5fa2c7f [Go SDK] Delete spammy log in datamgr.go new dc6b7a3 Avoid polluting Stackdriver logs with noise during autoscaling events. Add message directing users to ignore them when appropriate. new db87ebc Spotless Java applied new f196d8e Merge pull request #9821 from nahuellofeudo/logs-cleanup The 23705 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java| 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (7a02887 -> 5fa2c7f)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7a02887 Merge pull request #9843 from kamilwu/portable-runners-metrics add cb5a467 [Go SDK] Delete spammy log in datamgr.go add 5fa2c7f [Go SDK] Delete spammy log in datamgr.go No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (1deab4c -> d664592)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1deab4c Merge pull request #9931: [BEAM-8518] Fail when pipeline options conversion fails due to incompatible jackson-core version add 174b218 [BEAM-8398] Upgrade Google Cloud Java Client APIs add ceb9a99 fix PubsubJsonClientTest NPE add d664592 [BEAM-8398] Upgrade Dataflow Java Client API No new revisions were added by this update. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 12 ++-- .../org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-)
[beam] branch master updated (aa47745 -> b02bc8a)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aa47745 [BEAM-8517] Fix timezone conversion issue due to usage of different calls on DateTime for getting values. (#9929) new b26e821 [BEAM-8515, BEAM-2699] Add equals/hashCode to ValueProvider classes. new 6a900d2 [BEAM-7230] Cache DataSource instances based upon equivalent DataSourceConfiguration instances. new b02bc8a Merge pull request #9927 from lukecwik/beam7230 The 23682 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/options/ValueProvider.java | 39 + .../apache/beam/sdk/options/ValueProviderTest.java | 6 +- .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 94 +++--- .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 19 + 4 files changed, 109 insertions(+), 49 deletions(-)
[beam] branch master updated (c8bed1c -> aa47745)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c8bed1c Merge pull request #9863: [BEAM-8468] Predicate push down for in memory table add aa47745 [BEAM-8517] Fix timezone conversion issue due to usage of different calls on DateTime for getting values. (#9929) No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 9 + 1 file changed, 5 insertions(+), 4 deletions(-)
[beam] branch master updated (eb7edf3 -> d96c6b2)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from eb7edf3 Update the Beam Katas Java to use the new course metadata yaml format (#9901) add d96c6b2 BEAM-3658 Port SpannerIOReadTest off DoFnTester (#9898) No new revisions were added by this update. Summary of changes: .../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 173 +++-- 1 file changed, 93 insertions(+), 80 deletions(-)
[beam] branch master updated (6007c8c -> 0fcdbe1)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6007c8c Merge pull request #9902 from henryken/katas-python-course-yaml add 0fcdbe1 [BEAM-8432] Move javaVersion to gradle.properties (#9830) No new revisions were added by this update. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 9 +++-- gradle.properties| 2 ++ 2 files changed, 5 insertions(+), 6 deletions(-)
[beam] branch master updated (aa8acb0 -> 5e94175)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aa8acb0 Merge pull request #9809 [BEAM-8355] Add standard bool coder to Go SDK add 5e94175 Small improvement. (#9818) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (aa8acb0 -> 5e94175)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aa8acb0 Merge pull request #9809 [BEAM-8355] Add standard bool coder to Go SDK add 5e94175 Small improvement. (#9818) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (1458891 -> 0ff53f1)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1458891 Fixing a logging bug (#9795) add 0ff53f1 Fixing a logging bug in auth (#9796) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/internal/gcp/auth.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (fa74467 -> 1458891)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fa74467 Merge pull request #9775 from robertwb/easy-flink add 1458891 Fixing a logging bug (#9795) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/portability/stager.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (b8bef24 -> 2a367c3)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b8bef24 Merge pull request #9773 from RyanSkraba/BEAM-7073-avro-sql-unit-test add 2a367c3 [BEAM-4046] Remove old project name mappings. (#8915) No new revisions were added by this update. Summary of changes: build.gradle| 4 - deprecation-warning.txt | 18 - gradlew | 184 -- gradlew.bat | 93 +++- gradlew_orig| 190 gradlew_orig.bat| 102 -- project-mappings| 115 - 7 files changed, 236 insertions(+), 470 deletions(-) delete mode 100644 deprecation-warning.txt delete mode 100755 gradlew_orig delete mode 100644 gradlew_orig.bat delete mode 100644 project-mappings
[beam] branch master updated (71a4690 -> f3ce866)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 71a4690 [BEAM-8287] Python 3 docs updates for 2.16.0 (#9700) add f3ce866 Change the default status port of BatchDataflowWorker to 8081 (#9763) No new revisions were added by this update. Summary of changes: .../org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (0cb56a2 -> 025e514)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0cb56a2 Merge pull request #9731: [BEAM-8343] Added nessesary methods to BeamSqlTable to enable support for predicate/project push-down add 025e514 [BEAM-8378] Downgrade build-scan plugin to 2.3 (#9762) No new revisions were added by this update. Summary of changes: build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (e20b9e1 -> 9bb7dd5)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e20b9e1 [Go-SDK] Check correct cache for wrapped structs. add 9bb7dd5 Handle newlines in dataflow side input labels. (#9712) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/bundle_processor.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (3125a83 -> 487c9dfc)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3125a83 Merge pull request #9617: BigQueryIO: add clarification about authorized views (doc only pr) add 487c9dfc [BEAM-6923] limit gcs buffer size to 1MB for artifact upload (#9647) No new revisions were added by this update. Summary of changes: runners/flink/flink_runner.gradle | 1 + .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java | 7 ++- runners/spark/build.gradle| 1 + .../java/org/apache/beam/runners/spark/SparkJobServerDriver.java | 8 +++- 4 files changed, 15 insertions(+), 2 deletions(-)
[beam] branch master updated (cb414aa -> 74ea749)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cb414aa Merge pull request #9666 from lukecwik/beam8315 add 74ea749 [BEAM-8313] Follow up on PR comments for #9663 (#9668) No new revisions were added by this update. Summary of changes: .../worker/fn/data/BeamFnDataGrpcServiceTest.java | 12 ++--- .../runners/fnexecution/data/GrpcDataService.java | 4 +- .../fnexecution/data/GrpcDataServiceTest.java | 10 ++-- .../data/BeamFnDataBufferingOutboundObserver.java | 6 +-- .../apache/beam/sdk/fn/data/LogicalEndpoint.java | 2 +- .../BeamFnDataBufferingOutboundObserverTest.java | 4 +- .../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 4 +- .../beam/fn/harness/data/BeamFnDataGrpcClient.java | 4 +- .../fn/harness/data/QueueingBeamFnDataClient.java | 4 +- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 63 ++ .../fn/harness/data/BeamFnDataGrpcClientTest.java | 10 ++-- .../harness/data/QueueingBeamFnDataClientTest.java | 10 ++-- 12 files changed, 62 insertions(+), 71 deletions(-)
[beam] branch master updated (ba9a7ff -> 17805bc)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ba9a7ff Merge pull request #9648 from angoenka/flink_options add 17805bc [BEAM-8313] Rename certain proto fields to be consistent across fn/runner APIs No new revisions were added by this update. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 40 +- .../streaming/ExecutableStageDoFnOperatorTest.java |2 +- .../dataflow/worker/FnApiWindowMappingFn.java |2 +- .../control/RegisterAndProcessBundleOperation.java | 26 +- .../logging/DataflowWorkerLoggingHandler.java |6 +- .../dataflow/worker/FnApiWindowMappingFnTest.java |3 +- .../RegisterAndProcessBundleOperationTest.java | 13 +- .../worker/fn/data/BeamFnDataGrpcServiceTest.java | 12 +- .../fn/logging/BeamFnLoggingServiceTest.java | 18 +- .../logging/DataflowWorkerLoggingHandlerTest.java |2 +- .../fnexecution/control/SdkHarnessClient.java |2 +- .../fnexecution/state/GrpcStateService.java|5 +- .../fnexecution/state/StateRequestHandlers.java|6 +- .../runners/fnexecution/ServerFactoryTest.java |4 +- .../fnexecution/data/GrpcDataServiceTest.java | 12 +- .../logging/GrpcLoggingServiceTest.java| 18 +- .../fnexecution/state/GrpcStateServiceTest.java|4 +- sdks/go/pkg/beam/core/runtime/exec/translate.go|2 +- sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 12 +- .../pkg/beam/core/runtime/harness/datamgr_test.go |6 +- sdks/go/pkg/beam/core/runtime/harness/harness.go |6 +- sdks/go/pkg/beam/core/runtime/harness/logging.go |4 +- sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 12 +- sdks/go/pkg/beam/doc_test.go |4 +- .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 599 + .../model/jobmanagement_v1/beam_artifact_api.pb.go | 125 +- .../beam/model/pipeline_v1/beam_runner_api.pb.go | 1364 ++-- .../data/BeamFnDataBufferingOutboundObserver.java |8 +- .../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 11 +- .../sdk/fn/data/BeamFnDataInboundObserver.java |4 +- .../BeamFnDataBufferingOutboundObserverTest.java |8 +- .../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java |8 +- .../harness/SplittableProcessElementsRunner.java |4 +- .../fn/harness/control/ProcessBundleHandler.java |6 +- .../apache/beam/fn/harness/state/BagUserState.java |4 +- .../beam/fn/harness/state/FnApiStateAccessor.java |4 +- .../beam/fn/harness/state/MultimapSideInput.java |4 +- .../beam/fn/harness/FnApiDoFnRunnerTest.java |4 +- .../harness/control/ProcessBundleHandlerTest.java | 28 +- .../fn/harness/data/BeamFnDataGrpcClientTest.java | 20 +- .../data/BeamFnDataInboundObserverTest.java|4 +- .../harness/data/QueueingBeamFnDataClientTest.java | 20 +- .../beam/fn/harness/state/BagUserStateTest.java|2 +- .../state/BeamFnStateGrpcClientCacheTest.java | 14 +- .../fn/harness/state/MultimapSideInputTest.java|2 +- .../runners/portability/fn_api_runner.py | 40 +- .../apache_beam/runners/worker/bundle_processor.py | 14 +- .../apache_beam/runners/worker/data_plane.py | 22 +- .../apache_beam/runners/worker/data_plane_test.py | 16 +- .../apache_beam/runners/worker/sdk_worker.py | 36 +- 50 files changed, 1001 insertions(+), 1591 deletions(-)
[beam] branch master updated (f07bd7a -> 2acbfbd)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f07bd7a Merge pull request #9646 from ibzib/retrieve-fail add b32e332 [BEAM-8021] Stop publishing unneeded artifacts add 5ea2878 [BEAM-8021] Add Automatic-Module-Name headers to beam's artifacts. new 2acbfbd Merge branch 'lgajowy-BEAM-8021' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 33 ++ examples/java/build.gradle | 2 +- examples/kotlin/build.gradle | 2 +- model/fn-execution/build.gradle| 5 +++- model/job-management/build.gradle | 10 --- model/pipeline/build.gradle| 5 +++- runners/apex/build.gradle | 2 +- runners/core-construction-java/build.gradle| 2 +- runners/core-java/build.gradle | 2 +- runners/direct-java/build.gradle | 24 runners/extensions-java/metrics/build.gradle | 2 +- runners/flink/flink_runner.gradle | 3 +- runners/flink/job-server/flink_job_server.gradle | 1 + runners/gearpump/build.gradle | 2 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../worker/windmill/build.gradle | 1 + runners/java-fn-execution/build.gradle | 2 +- runners/jet/build.gradle | 2 +- runners/local-java/build.gradle| 1 + runners/reference/java/build.gradle| 2 +- runners/samza/build.gradle | 2 +- runners/samza/job-server/build.gradle | 1 + runners/spark/build.gradle | 2 +- runners/spark/job-server/build.gradle | 1 + sdks/java/build-tools/build.gradle | 2 +- sdks/java/core/build.gradle| 23 --- sdks/java/extensions/euphoria/build.gradle | 2 +- .../google-cloud-platform-core/build.gradle| 2 +- sdks/java/extensions/jackson/build.gradle | 1 + sdks/java/extensions/join-library/build.gradle | 2 +- sdks/java/extensions/kryo/build.gradle | 1 + sdks/java/extensions/protobuf/build.gradle | 2 +- sdks/java/extensions/sketching/build.gradle| 2 +- sdks/java/extensions/sorter/build.gradle | 2 +- sdks/java/extensions/sql/build.gradle | 1 + sdks/java/extensions/sql/datacatalog/build.gradle | 2 +- sdks/java/extensions/sql/hcatalog/build.gradle | 2 +- sdks/java/extensions/sql/jdbc/build.gradle | 1 + sdks/java/extensions/zetasketch/build.gradle | 2 +- sdks/java/fn-execution/build.gradle| 2 +- sdks/java/harness/build.gradle | 1 + sdks/java/io/amazon-web-services/build.gradle | 2 +- sdks/java/io/amazon-web-services2/build.gradle | 2 +- sdks/java/io/amqp/build.gradle | 2 +- sdks/java/io/bigquery-io-perf-tests/build.gradle | 2 +- sdks/java/io/cassandra/build.gradle| 2 +- sdks/java/io/clickhouse/build.gradle | 1 + sdks/java/io/common/build.gradle | 2 +- .../elasticsearch-tests-2/build.gradle | 1 + .../elasticsearch-tests-5/build.gradle | 1 + .../elasticsearch-tests-6/build.gradle | 1 + .../elasticsearch-tests-common/build.gradle| 1 + sdks/java/io/elasticsearch/build.gradle| 2 +- sdks/java/io/file-based-io-tests/build.gradle | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 1 + sdks/java/io/hadoop-common/build.gradle| 2 +- sdks/java/io/hadoop-file-system/build.gradle | 2 +- sdks/java/io/hadoop-format/build.gradle| 2 +- sdks/java/io/hbase/build.gradle| 2 +- sdks/java/io/hcatalog/build.gradle | 2 +- sdks/java/io/jdbc/build.gradle | 2 +- sdks/java/io/jms/build.gradle | 2 +- sdks/java/io/kafka/build.gradle| 2 +- sdks/java/io/kinesis/build.gradle | 2 +- sdks/java/io/kudu/build.gradle | 2 +- sdks/java/io/mongodb/build.gradle | 2 +- sdks/java/io/mqtt/build.gradle | 2 +- sdks/java/io/parquet/build.gradle | 2 +- sdks/java/io/rabbitmq/build.gradle | 2 +- sdks/java/io/redis/build.gradle| 2 +- sdks/java/io/solr/build.gradl
[beam] 01/01: Merge branch 'lgajowy-BEAM-8021'
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2acbfbd23a23ee6eed77cbe74690f66f985c9815 Merge: f07bd7a 5ea2878 Author: Luke Cwik AuthorDate: Tue Sep 24 14:02:44 2019 -0700 Merge branch 'lgajowy-BEAM-8021' .../org/apache/beam/gradle/BeamModulePlugin.groovy | 33 ++ examples/java/build.gradle | 2 +- examples/kotlin/build.gradle | 2 +- model/fn-execution/build.gradle| 5 +++- model/job-management/build.gradle | 10 --- model/pipeline/build.gradle| 5 +++- runners/apex/build.gradle | 2 +- runners/core-construction-java/build.gradle| 2 +- runners/core-java/build.gradle | 2 +- runners/direct-java/build.gradle | 24 runners/extensions-java/metrics/build.gradle | 2 +- runners/flink/flink_runner.gradle | 3 +- runners/flink/job-server/flink_job_server.gradle | 1 + runners/gearpump/build.gradle | 2 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../worker/windmill/build.gradle | 1 + runners/java-fn-execution/build.gradle | 2 +- runners/jet/build.gradle | 2 +- runners/local-java/build.gradle| 1 + runners/reference/java/build.gradle| 2 +- runners/samza/build.gradle | 2 +- runners/samza/job-server/build.gradle | 1 + runners/spark/build.gradle | 2 +- runners/spark/job-server/build.gradle | 1 + sdks/java/build-tools/build.gradle | 2 +- sdks/java/core/build.gradle| 23 --- sdks/java/extensions/euphoria/build.gradle | 2 +- .../google-cloud-platform-core/build.gradle| 2 +- sdks/java/extensions/jackson/build.gradle | 1 + sdks/java/extensions/join-library/build.gradle | 2 +- sdks/java/extensions/kryo/build.gradle | 1 + sdks/java/extensions/protobuf/build.gradle | 2 +- sdks/java/extensions/sketching/build.gradle| 2 +- sdks/java/extensions/sorter/build.gradle | 2 +- sdks/java/extensions/sql/build.gradle | 1 + sdks/java/extensions/sql/datacatalog/build.gradle | 2 +- sdks/java/extensions/sql/hcatalog/build.gradle | 2 +- sdks/java/extensions/sql/jdbc/build.gradle | 1 + sdks/java/extensions/zetasketch/build.gradle | 2 +- sdks/java/fn-execution/build.gradle| 2 +- sdks/java/harness/build.gradle | 1 + sdks/java/io/amazon-web-services/build.gradle | 2 +- sdks/java/io/amazon-web-services2/build.gradle | 2 +- sdks/java/io/amqp/build.gradle | 2 +- sdks/java/io/bigquery-io-perf-tests/build.gradle | 2 +- sdks/java/io/cassandra/build.gradle| 2 +- sdks/java/io/clickhouse/build.gradle | 1 + sdks/java/io/common/build.gradle | 2 +- .../elasticsearch-tests-2/build.gradle | 1 + .../elasticsearch-tests-5/build.gradle | 1 + .../elasticsearch-tests-6/build.gradle | 1 + .../elasticsearch-tests-common/build.gradle| 1 + sdks/java/io/elasticsearch/build.gradle| 2 +- sdks/java/io/file-based-io-tests/build.gradle | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 1 + sdks/java/io/hadoop-common/build.gradle| 2 +- sdks/java/io/hadoop-file-system/build.gradle | 2 +- sdks/java/io/hadoop-format/build.gradle| 2 +- sdks/java/io/hbase/build.gradle| 2 +- sdks/java/io/hcatalog/build.gradle | 2 +- sdks/java/io/jdbc/build.gradle | 2 +- sdks/java/io/jms/build.gradle | 2 +- sdks/java/io/kafka/build.gradle| 2 +- sdks/java/io/kinesis/build.gradle | 2 +- sdks/java/io/kudu/build.gradle | 2 +- sdks/java/io/mongodb/build.gradle | 2 +- sdks/java/io/mqtt/build.gradle | 2 +- sdks/java/io/parquet/build.gradle | 2 +- sdks/java/io/rabbitmq/build.gradle | 2 +- sdks/java/io/redis/build.gradle| 2 +- sdks/java/io/solr/build.gradle | 2 +- sdks/java/io/synthetic/build.gradle| 2 +- sdks/java/io/tika/build.gradle | 2 +- sdks/java/io/xml/build.gradle | 2 +- sdks/java/javadoc/build.gradle | 2 +- sdks/java/maven-archetypes/examples/build.gradle | 2 +- sdks/java/maven-archetypes/starter/build.grad
[beam] branch master updated (031b378 -> ac45af9)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 031b378 [BEAM-5820] Use vendored calcite
add 77bfad4 Remove the 'experimental' word from Jet Runner
modules/directories
add ac45af9 [BEAM-7305] Remove the 'experimental' word from Jet Runner
modules/directories
No new revisions were added by this update.
Summary of changes:
runners/{jet-experimental => jet}/build.gradle | 0
.../src/main/java/org/apache/beam/runners/jet/DAGBuilder.java | 0
.../org/apache/beam/runners/jet/FailedRunningPipelineResults.java | 0
.../src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java | 0
.../main/java/org/apache/beam/runners/jet/JetPipelineOptions.java | 0
.../main/java/org/apache/beam/runners/jet/JetPipelineResult.java| 0
.../src/main/java/org/apache/beam/runners/jet/JetRunner.java| 0
.../main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java | 0
.../java/org/apache/beam/runners/jet/JetTransformTranslator.java| 0
.../java/org/apache/beam/runners/jet/JetTransformTranslators.java | 0
.../java/org/apache/beam/runners/jet/JetTranslationContext.java | 0
.../src/main/java/org/apache/beam/runners/jet/Utils.java| 0
.../java/org/apache/beam/runners/jet/metrics/AbstractMetric.java| 0
.../main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java | 0
.../java/org/apache/beam/runners/jet/metrics/DistributionImpl.java | 0
.../main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java| 0
.../java/org/apache/beam/runners/jet/metrics/JetMetricResults.java | 0
.../org/apache/beam/runners/jet/metrics/JetMetricsContainer.java| 0
.../main/java/org/apache/beam/runners/jet/metrics/package-info.java | 0
.../src/main/java/org/apache/beam/runners/jet/package-info.java | 0
.../java/org/apache/beam/runners/jet/processors/AbstractParDoP.java | 0
.../java/org/apache/beam/runners/jet/processors/AssignWindowP.java | 0
.../java/org/apache/beam/runners/jet/processors/BoundedSourceP.java | 0
.../main/java/org/apache/beam/runners/jet/processors/FlattenP.java | 0
.../main/java/org/apache/beam/runners/jet/processors/ImpulseP.java | 0
.../main/java/org/apache/beam/runners/jet/processors/ParDoP.java| 0
.../java/org/apache/beam/runners/jet/processors/StatefulParDoP.java | 0
.../org/apache/beam/runners/jet/processors/UnboundedSourceP.java| 0
.../src/main/java/org/apache/beam/runners/jet/processors/ViewP.java | 0
.../java/org/apache/beam/runners/jet/processors/WindowGroupP.java | 0
.../java/org/apache/beam/runners/jet/processors/package-info.java | 0
.../java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java| 0
.../src/test/java/org/apache/beam/runners/jet/TestJetRunner.java| 0
.../src/test/java/org/apache/beam/runners/jet/TestStreamP.java | 0
.../examples/src/main/resources/archetype-resources/pom.xml | 6 +++---
settings.gradle | 2 +-
36 files changed, 4 insertions(+), 4 deletions(-)
rename runners/{jet-experimental => jet}/build.gradle (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetRunner.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/Utils.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runn
[beam] branch master updated (6eb8605 -> 031b378)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6eb8605 Merge pull request #9645 from robertwb/no-grpc add 899f45c [BEAM-5820] Switch Beam SQL to vendored Calcite add 0d9972f not include shadow direct runner dependency in nexmark add 031b378 [BEAM-5820] Use vendored calcite No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 1 + .../src/main/resources/beam/checkstyle.xml | 8 ++ sdks/java/extensions/sql/build.gradle | 119 + sdks/java/extensions/sql/datacatalog/build.gradle | 3 +- .../sql/example/BeamSqlDataCatalogExample.java | 2 +- .../datacatalog/DataCatalogTableProvider.java | 4 +- .../sql/meta/provider/datacatalog/SchemaUtils.java | 4 +- .../sql/meta/provider/datacatalog/TableUtils.java | 2 +- sdks/java/extensions/sql/hcatalog/build.gradle | 2 +- sdks/java/extensions/sql/jdbc/build.gradle | 2 +- sdks/java/extensions/sql/shell/build.gradle| 4 +- .../extensions/sql/src/main/codegen/config.fmpp| 10 +- .../beam/sdk/extensions/sql/SqlTransform.java | 4 +- .../extensions/sql/TableNameExtractionUtils.java | 14 +-- .../sdk/extensions/sql/impl/BeamCalciteSchema.java | 17 +-- .../sql/impl/BeamCalciteSchemaFactory.java | 20 ++-- .../sdk/extensions/sql/impl/BeamCalciteTable.java | 30 +++--- .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 14 +-- .../sql/impl/BeamSqlPipelineOptionsRegistrar.java | 2 +- .../extensions/sql/impl/BeamTableStatistics.java | 14 +-- .../sql/impl/CalciteConnectionWrapper.java | 16 +-- .../extensions/sql/impl/CalciteFactoryWrapper.java | 24 ++--- .../extensions/sql/impl/CalciteQueryPlanner.java | 72 ++--- .../sdk/extensions/sql/impl/JdbcConnection.java| 8 +- .../beam/sdk/extensions/sql/impl/JdbcDriver.java | 30 +++--- .../beam/sdk/extensions/sql/impl/JdbcFactory.java | 36 --- .../beam/sdk/extensions/sql/impl/QueryPlanner.java | 2 +- .../extensions/sql/impl/ScalarFunctionImpl.java| 55 +- .../extensions/sql/impl/TableResolutionUtils.java | 6 +- .../beam/sdk/extensions/sql/impl/UdafImpl.java | 12 +-- .../beam/sdk/extensions/sql/impl/UdfImpl.java | 12 ++- .../sql/impl/UdfImplReflectiveFunctionBase.java| 19 ++-- .../sql/impl/parser/SqlCheckConstraint.java| 18 ++-- .../sql/impl/parser/SqlColumnDeclaration.java | 20 ++-- .../sql/impl/parser/SqlCreateExternalTable.java| 30 +++--- .../extensions/sql/impl/parser/SqlDdlNodes.java| 20 ++-- .../extensions/sql/impl/parser/SqlDropObject.java | 24 ++--- .../extensions/sql/impl/parser/SqlDropTable.java | 10 +- .../sql/impl/parser/SqlSetOptionBeam.java | 20 ++-- .../extensions/sql/impl/planner/BeamCostModel.java | 11 +- .../sql/impl/planner/BeamJavaTypeFactory.java | 12 +-- .../sql/impl/planner/BeamRelDataTypeSystem.java| 4 +- .../extensions/sql/impl/planner/BeamRuleSets.java | 56 +- .../sql/impl/planner/NodeStatsMetadata.java| 12 +-- .../sql/impl/planner/RelMdNodeStats.java | 12 +-- .../sql/impl/rel/BeamAggregationRel.java | 20 ++-- .../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 68 ++-- .../extensions/sql/impl/rel/BeamCoGBKJoinRel.java | 14 +-- .../sql/impl/rel/BeamEnumerableConverter.java | 40 +++ .../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java | 22 ++-- .../extensions/sql/impl/rel/BeamIOSourceRel.java | 14 +-- .../extensions/sql/impl/rel/BeamIntersectRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 34 +++--- .../sql/impl/rel/BeamLogicalConvention.java| 12 +-- .../sdk/extensions/sql/impl/rel/BeamMinusRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamRelNode.java | 12 ++- .../sql/impl/rel/BeamSetOperatorRelBase.java | 2 +- .../sql/impl/rel/BeamSideInputJoinRel.java | 14 +-- .../sql/impl/rel/BeamSideInputLookupJoinRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamSortRel.java | 30 +++--- .../extensions/sql/impl/rel/BeamSqlRelUtils.java | 6 +- .../extensions/sql/impl/rel/BeamUncollectRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamUnionRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 24 ++--- .../sdk/extensions/sql/impl/rel/BeamValuesRel.java | 20 ++-- .../sdk/extensions/sql/impl/rel/package-info.java | 5 +- .../sql/impl/rule/BeamAggregationRule.java | 24 ++--- .../sql/impl/rule/BeamBasicAggregationRule.java| 14 +-- .../sdk/extensions/sql/impl/rule/BeamCalcRule.java | 14 +-- .../sql/impl/rule/BeamCoGBKJoinRule.java | 12 +-- .../sql/impl/rule/BeamEnumerableConverterRule.java | 8 +- .../extensions/sql/impl/rule
[beam] branch master updated: More guarding against missing grpc.
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 32f03f9 More guarding against missing grpc. new 6eb8605 Merge pull request #9645 from robertwb/no-grpc 32f03f9 is described below commit 32f03f969aa82da2ebecd9edb5ca4a630f32aa47 Author: Robert Bradshaw AuthorDate: Mon Sep 23 17:15:19 2019 -0700 More guarding against missing grpc. --- sdks/python/apache_beam/transforms/external.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/transforms/external.py b/sdks/python/apache_beam/transforms/external.py index 0e241b7..75fe766 100644 --- a/sdks/python/apache_beam/transforms/external.py +++ b/sdks/python/apache_beam/transforms/external.py @@ -39,13 +39,13 @@ from apache_beam.typehints.native_type_compatibility import convert_to_beam_type from apache_beam.typehints.trivial_inference import instance_to_type from apache_beam.typehints.typehints import Union from apache_beam.typehints.typehints import UnionConstraint -from apache_beam.utils import subprocess_server # Protect against environments where grpc is not available. # pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports try: import grpc from apache_beam.portability.api import beam_expansion_api_pb2_grpc + from apache_beam.utils import subprocess_server except ImportError: grpc = None # pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
[beam] branch master updated (28525b6 -> 511caa3)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 28525b6 [BEAM-8093] Fail tox environments that don't set command add 113d9c1 Fix typo for procesing --> processing in programming-guide.md add 511caa3 Fix typo for procesing --> processing in programming-guide.md No new revisions were added by this update. Summary of changes: website/src/documentation/programming-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (06b91a7 -> 28525b6)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 06b91a7 [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override add 433f0e7 [BEAM-8093] Fail tox environments the don't set command add 28525b6 [BEAM-8093] Fail tox environments that don't set command No new revisions were added by this update. Summary of changes: sdks/python/tox.ini | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (2e59c27 -> 06b91a7)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2e59c27 [BEAM-8093] Commit change to ensure py36-gcp runs tests add 1e00170 [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override add 06b91a7 [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override No new revisions were added by this update. Summary of changes: .../apache_beam/runners/dataflow/dataflow_runner.py | 5 ++--- .../runners/dataflow/dataflow_runner_test.py | 18 ++ .../runners/dataflow/internal/apiclient.py| 19 ++- 3 files changed, 30 insertions(+), 12 deletions(-)
[beam] branch master updated (af00fc4 -> 2e59c27)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from af00fc4 [BEAM-8224] Fix bug in _fn_takes_side_inputs (#9563) add 451dd9c [BEAM-8093] Commit change to ensure py36-gcp runs tests add 2e59c27 [BEAM-8093] Commit change to ensure py36-gcp runs tests No new revisions were added by this update. Summary of changes: sdks/python/tox.ini | 7 +++ 1 file changed, 7 insertions(+)
[beam] branch master updated (d3ffdce -> d93e4e0)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d3ffdce Merge pull request #9521: [BEAM-8178] Fix broken Load Tests running on Flink add f39da9d Update execution-model.md add d93e4e0 [BEAM-7595]Update execution-model.md No new revisions were added by this update. Summary of changes: website/src/documentation/execution-model.md | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam] branch master updated (cb10019 -> 6b3f7d6)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cb10019 [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions add 7e91a7f [BEAM-7947] Improves the interfaces of classes such as FnDataService, BundleProcessor, ActiveBundle, etc to change the parameter type from WindowedValue to T add be2cfec fix code style add 6b3f7d6 [BEAM-7947] Improves the interfaces of classes such as FnDataService,… No new revisions were added by this update. Summary of changes: .../FlinkExecutableStageFunctionTest.java | 2 +- .../streaming/ExecutableStageDoFnOperatorTest.java | 2 +- .../worker/fn/data/BeamFnDataGrpcService.java | 12 +++ .../dataflow/worker/FnApiWindowMappingFnTest.java | 15 - .../worker/fn/control/TimerReceiverTest.java | 2 +- .../control/DefaultJobBundleFactory.java | 10 +++--- .../control/ProcessBundleDescriptors.java | 18 +-- .../runners/fnexecution/control/RemoteBundle.java | 3 +- .../fnexecution/control/SdkHarnessClient.java | 27 +++- .../SingleEnvironmentInstanceJobBundleFactory.java | 8 ++--- .../runners/fnexecution/data/FnDataService.java| 8 ++--- .../runners/fnexecution/data/GrpcDataService.java | 8 ++--- .../fnexecution/control/RemoteExecutionTest.java | 36 +++-- .../fnexecution/control/SdkHarnessClientTest.java | 4 +-- .../SparkExecutableStageFunctionTest.java | 4 +-- .../data/BeamFnDataBufferingOutboundObserver.java | 14 .../sdk/fn/data/BeamFnDataInboundObserver.java | 13 +++- .../beam/fn/harness/data/BeamFnDataClient.java | 9 +++--- .../beam/fn/harness/data/BeamFnDataGrpcClient.java | 9 +++--- .../fn/harness/data/QueueingBeamFnDataClient.java | 37 +++--- .../data/BeamFnDataInboundObserverTest.java| 4 +-- 21 files changed, 108 insertions(+), 137 deletions(-)
[beam] branch master updated (da2fec3 -> cb10019)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from da2fec3 Merge pull request #9506: [BEAM-8105] Docker images release scripts add 804a295 [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions add 6df5f74 fixup! [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions add 86daba6 fixup! fixup! [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions add cb10019 [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions No new revisions were added by this update. Summary of changes: .../sql/example/BeamSqlDataCatalogExample.java | 5 +- .../datacatalog/DataCatalogClientAdapter.java | 94 -- .../datacatalog/DataCatalogTableProvider.java | 38 + .../datacatalog/DataCatalogBigQueryIT.java | 4 +- 4 files changed, 29 insertions(+), 112 deletions(-) delete mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogClientAdapter.java
[beam] branch master updated (fcf1698 -> 6a63c06)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fcf1698 Merge pull request #9546 [BEAM-8188] Allow PyArrow imports to fail on systems other than Win+Py27 add d0fd5b3 Downgrade warning log to debug log. add 6a63c06 Make JRH logs less noisy No new revisions were added by this update. Summary of changes: .../worker/fn/control/BeamFnMapTaskExecutor.java | 29 -- .../control/RegisterAndProcessBundleOperation.java | 4 +++ 2 files changed, 20 insertions(+), 13 deletions(-)
svn commit: r35740 - /dev/beam/KEYS
Author: lcwik Date: Tue Sep 10 23:40:15 2019 New Revision: 35740 Log: Add release manager's public key Modified: dev/beam/KEYS Modified: dev/beam/KEYS == --- dev/beam/KEYS (original) +++ dev/beam/KEYS Tue Sep 10 23:40:15 2019 @@ -1223,3 +1223,85 @@ PF+gdmGg8CXa9qwpUTo2Z+kGCuWOXlVwu1vU5e53 I0tCrK6brUzqJwPi8Vk= =++rr -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2019-09-10 [SC] + C110B1C82074883A4241D977599D6305FF3ABB32 +uid [ultimate] Mark Liu (GPG key) +sub rsa4096 2019-09-10 [E] + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF14IZ0BEADTYB26xLbK8SVRX8ObawprHKUaT6Obo6HIRysM3CDqTUSlIHc7 +yx5hByDRKbpD2Y/4LHQAusPRERrg1bsPfRKiTj0QGo9Wllk5b5bT/DqgqtYFCjvb +ZaOb8krtKunPedCZqk6Zfrl2VjZoQI/Rm0mYRM8q97JLclTT3EVbmz+AAAi+veLA +rIiqz7wDlGILnWQ5VrK6hF8W4MMJIAsG830D1MHNDawraTt0LcJLB5iGZ0S9F2od +UOhV2jLktmbE99lRCsMLGfmMY4YdddcEKYEcIVNB21/WJ7w5gw5WCbEUdgt7w1al +LvuXdAadX2D1SFpN+nUy2vU5sv2YrQJmN/WccD4gyniFRa7KCkLm3MFuurx3G0lR +Xt3Gn7xAQ5s1Gvj8D731yhZKUeU1IfCHhLvKcpdUdb9/Fzi48RNdcitdSh7JGqGI +GD2yOabPNhjq4RGzpJgx8MDgwhrTChdnpUoQwbsxILEIdZ2BtCncIMEfnoAYTS1f +hmPRbsQCqJobVxGBqRuVoX0TPp1TbaU8M5fefYtsBzYNBxZNo5sBpZflpME8bvwj +zoCXmlIA0M1EIH7CIhW1aLvfX8MUgiGlxIsZu8dJtMKBfnGFKpwi+Vknvkr/26ZO +SLIq1TOcTQMkBaIsJPTZ5AKNCYWnhTbCQ7rZT8Ky4u/APhVmvqQxx2iwjQARAQAB +tCdNYXJrIExpdSAoR1BHIGtleSkgPG1hcmtsaXVAYXBhY2hlLm9yZz6JAk4EEwEK +ADgWIQTBELHIIHSIOkJB2XdZnWMF/zq7MgUCXXgltQIbAwULCQgHAgYVCgkICwIE +FgIDAQIeAQIXgAAKCRBZnWMF/zq7Mm4kD/0eZ6WZXtjnm44TSIPUvYQ1O1aSQoyz +cCv9eUk5dOCGbARqFYYcj0nXSisFbz3wKWyM1sAmmQHlgXWeReJZXe5STH7j3eI9 +l9Kz/JuU5+F731WfWzEXHTugDG2DjzESg4R9mEgL7E+2qckEXrBneHmmo4Lv5vit +irkmeRWSUNbE7NsddpyPyY96SQ4Y0L2a5fYhulTcyd8lhZqpJrSrSkyx1qiUSF6L +0D9D0f+CeNoI6j+o/0S84k04AegPN/oh8qGeCekzMtFi+xX1mUjbbSSSjoe8ZMGj +XnKaqdf/6XZhWjcNSxrg/xZrRNJZlF0uzfGzd0VsFImqJt31nXW9OsyE98tJoINc +tJWgset2gp9jDQHzCoDiCKRb8A9OkjLExughO783n1qO7iuT035W0s9dW2CtG1sG +4HWcfeiNqXGuxSYVK1UHhTdaQiowzEQsfFCTQeweJ0h64CPHcXczB0LlzAnTGZG2 +eCWucKAhBkwEj9ZmvBdnrjn5eGOMIIYOjwyDZ1973bGc8uVvDcxZ3yXfkUI5J1pu +VShLC9pwIU01fuP+txofqWly6doawKjY5N5boqlMOk+6+ckJ6h1ZD8fnVsJB5NwQ +653Uxr+K1UeGwg92f7x4wjYpQ1Dr4/T4fRxtpy5pygZjw23cZZWeEzhhTPzB6PXy +Yb43Mil+9QlfYrQnTWFyayBMaXUgKEdQRyBrZXkpIDxtYXJrbGl1QGdvb2dsZS5j +b20+iQI2BDABCgAgFiEEwRCxyCB0iDpCQdl3WZ1jBf86uzIFAl14J5QCHSAACgkQ +WZ1jBf86uzIBihAAiZQ0EDx7BVbWM8E5UwgkWlv1L6+mCBUivrTagIrrXAOfgp8o +JEJ+AcLz9P22kRe7f4VM6xVyYm+GGOBeUD+bUCr14eZhfs7Buu4q9cz1yimnWg8d +riH9A1vxIFmZ86n1uFyWn8sG4FpJF09edAyCOYmvmJ4fdZiU3QSILfq0L3UT3294 +qIwWQRG9H27NBkcR4ivtyQ4uvwrql7Ut0Ew9Kj4T53T4FADG9n2RCCHjCKxShqGa +jAoFLktxw/K75pgDV4XdkS+81XHws1KcozBVpR8eWJWLmfE5Fu9lhm03RVcwS3Uu +Xg4An4P1wKv/BcViaY/hqR0KBJuOMeeC9HMxqL1wkyZ5O3s+Vf5tfP2DWPKMyfz+ +bw+4c5rH4mzja7bC5DCS0XNM2Ga8VhH4cN8Y/NyKu3ldaauAiLtB9i2XBXFEMe4q +yoe/x3/tTBgIQUaXTTuC5811M15PwQBd+rFlDf2YT8qQYi0vZDqXJF6SKSl/youG +8mwoUlOB5cLoinFAleGmHHN9LDmmOZdMZgoCqryo/cEHlNkNtVYf5KpMMzoca6T0 +r1ihVPO2O78/R0fVit2+Fm5IeHTaJsMLtMS/5b9f79fydw2924sEv4egLwo4tPus +9legKEYCfHhdED7jqc2icqtk+MPImIw+P6WZt2/RlTOaTjDg4lAGNoihfpCJAk4E +EwEKADgWIQTBELHIIHSIOkJB2XdZnWMF/zq7MgUCXXghnQIbAwULCQgHAgYVCgkI +CwIEFgIDAQIeAQIXgAAKCRBZnWMF/zq7Mj4GEADG1JeAQw1bRjFQ6Qg32kRkOwds +ZkRhO9ssoqfDVKKu1YvfhFtIYjGYHkQnTuWnEp5q4l0B2iSRY46CtpXcqExTuFen +omElJ1R6Ufi/OtULzfhCBS8GVLrvEfdHC5v/Uv9BVlKzoT+MY1jPeNq+wNqsVwEm +CwLQRXVrvBaPq6BKFEWOUItEvurJ3YklMBh74OFOYkYz7SfWmZQL0Bq15vwzXPjP +JZw+014cOQiDdSaV1R0RO2upNoPfDbzcy4H/4JGSXbSeFrwRV4R0cZH7PkEiqFbe +bxEzSzFkM0TZzhI9RO6BOGxWTVJIpt0bA0z67SO+U8Kc4V9/u67BkOUn5w3PayG0 +hHKayL6tIrxcAUclSJhmMDRDoe1EUb7jjkoCtjikK0LQ8zYada+RZHAlKolI3ZXJ +52uhqrcpKznUbV+vA0EmrpFMc4hRNtQMo5RO4se6S1hZj5JngHv7H5xB1ow/FQlG +Vn6tWxzxnA4eq0sotN4bgYqfrktFsm3hmwmKDcJ8digkKtt8WKY4zhlYWugcAswB +Vuyn1I1ktc65kX4nVyF0anunJ9dJtTcAj3U/gyyB1nx2Z/5WnRGJbfJgMjoMUC/R +dD9OWvn90+BPle94C9hYNvHnJm7oObprLCrpsMrB1iNnGbW93O9lY1VZlRdn99Fk +VvMw7dCbjClAMZ8yorkCDQRdeCGdARAAwT2COsvNmaqxmnzGOCZIMlgiBNMijX8/ +mUA/UkZqNaP/LPRn2w1Br0XAW5wkaOKH+3R5Q7BdIvJ+AshnmuKubSRSgdulvIUj +55zNIHaQvMEw7d6ya4Km7Is9edrv3GMtdf8R1bSmH84dFGgwDbayLtRHn4b9nOSP +8iic+RVlfxCpDRQv4vJFRWO7jv9J10VHteNoZj+1XL2JoP+DESRtCz3xGE7eObz9 +b2sLF/vAU0ZR/pmQ61sksLeSs8G3K006sOVAzFIA4vCX11hRT9iztdWCnPliGXlS +WKK5tVXcx7sT/aF48509iuXb1NEpMVWxn85Db1Lqg+gmvfsdJpiNoUDqFpySmEHT +vRruHC/iUXanejuC4dI4HM6eLLVMxdt4qvfZfwzZPY/rIr447sDv6z0ab3kEHXuy +QG0jz1nHiwPByapLk4LtaxHko0I0/QyBcpThMCH8a56gsELZshW5LkWZDyKoDUu2 +wC+uGpbUSWlJqknYK7uVFed/1oQQz6GvR5Rjmfmqd54/bNIbVwjY6Rac8IOL1ah3 +bxq0QMKoR865ReLPNZCFZLtwmS0p1GeGSvJnc4BreLkgDSoVl5CFTxl0gWgmbCfN +b/Irbg3gXcVxgTb1AbBNPEZ/1iFa/+vHt5vOjADFIlEWwOX0BRXci6px5DTslrgh +Sd6KalTCnD8AEQEAAYkCNgQYAQoAIBYhBMEQscggdIg6QkHZd1mdYwX/OrsyBQJd +eCGdAhsMAAoJEFmdYwX/OrsytbgQAMNCtV9jTpln4UdWR/+9qwcAw1j4cNQgPpaX +7qEPM1n0PfltC8joH5LQzuk22r7g2srLtmu4R5VJ4iAFSfA3hg7qDNoe1WVhggvD +Iw8/enSR9Ea195VIUMKfzsY1gMnJ+QH4bkYNgfMrI5OfSQM/ITLSIdclTOXwu64a ++ZoLHKlBzW1mgiKEgH+OlRi6yPi87FyUlpKeXCGmjjCCBMCPfLSeYHs92cSXO0Tp +vAZ+5O8cvTsO+4jsQaVwhOixiQ5HobLvbRgg+q6T/7Loc2jEzsq+6xqGWoI
svn commit: r35741 - /release/beam/KEYS
Author: lcwik Date: Tue Sep 10 23:40:29 2019 New Revision: 35741 Log: Add release manager's key Modified: release/beam/KEYS Modified: release/beam/KEYS == --- release/beam/KEYS (original) +++ release/beam/KEYS Tue Sep 10 23:40:29 2019 @@ -1223,3 +1223,85 @@ PF+gdmGg8CXa9qwpUTo2Z+kGCuWOXlVwu1vU5e53 I0tCrK6brUzqJwPi8Vk= =++rr -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2019-09-10 [SC] + C110B1C82074883A4241D977599D6305FF3ABB32 +uid [ultimate] Mark Liu (GPG key) +sub rsa4096 2019-09-10 [E] + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF14IZ0BEADTYB26xLbK8SVRX8ObawprHKUaT6Obo6HIRysM3CDqTUSlIHc7 +yx5hByDRKbpD2Y/4LHQAusPRERrg1bsPfRKiTj0QGo9Wllk5b5bT/DqgqtYFCjvb +ZaOb8krtKunPedCZqk6Zfrl2VjZoQI/Rm0mYRM8q97JLclTT3EVbmz+AAAi+veLA +rIiqz7wDlGILnWQ5VrK6hF8W4MMJIAsG830D1MHNDawraTt0LcJLB5iGZ0S9F2od +UOhV2jLktmbE99lRCsMLGfmMY4YdddcEKYEcIVNB21/WJ7w5gw5WCbEUdgt7w1al +LvuXdAadX2D1SFpN+nUy2vU5sv2YrQJmN/WccD4gyniFRa7KCkLm3MFuurx3G0lR +Xt3Gn7xAQ5s1Gvj8D731yhZKUeU1IfCHhLvKcpdUdb9/Fzi48RNdcitdSh7JGqGI +GD2yOabPNhjq4RGzpJgx8MDgwhrTChdnpUoQwbsxILEIdZ2BtCncIMEfnoAYTS1f +hmPRbsQCqJobVxGBqRuVoX0TPp1TbaU8M5fefYtsBzYNBxZNo5sBpZflpME8bvwj +zoCXmlIA0M1EIH7CIhW1aLvfX8MUgiGlxIsZu8dJtMKBfnGFKpwi+Vknvkr/26ZO +SLIq1TOcTQMkBaIsJPTZ5AKNCYWnhTbCQ7rZT8Ky4u/APhVmvqQxx2iwjQARAQAB +tCdNYXJrIExpdSAoR1BHIGtleSkgPG1hcmtsaXVAYXBhY2hlLm9yZz6JAk4EEwEK +ADgWIQTBELHIIHSIOkJB2XdZnWMF/zq7MgUCXXgltQIbAwULCQgHAgYVCgkICwIE +FgIDAQIeAQIXgAAKCRBZnWMF/zq7Mm4kD/0eZ6WZXtjnm44TSIPUvYQ1O1aSQoyz +cCv9eUk5dOCGbARqFYYcj0nXSisFbz3wKWyM1sAmmQHlgXWeReJZXe5STH7j3eI9 +l9Kz/JuU5+F731WfWzEXHTugDG2DjzESg4R9mEgL7E+2qckEXrBneHmmo4Lv5vit +irkmeRWSUNbE7NsddpyPyY96SQ4Y0L2a5fYhulTcyd8lhZqpJrSrSkyx1qiUSF6L +0D9D0f+CeNoI6j+o/0S84k04AegPN/oh8qGeCekzMtFi+xX1mUjbbSSSjoe8ZMGj +XnKaqdf/6XZhWjcNSxrg/xZrRNJZlF0uzfGzd0VsFImqJt31nXW9OsyE98tJoINc +tJWgset2gp9jDQHzCoDiCKRb8A9OkjLExughO783n1qO7iuT035W0s9dW2CtG1sG +4HWcfeiNqXGuxSYVK1UHhTdaQiowzEQsfFCTQeweJ0h64CPHcXczB0LlzAnTGZG2 +eCWucKAhBkwEj9ZmvBdnrjn5eGOMIIYOjwyDZ1973bGc8uVvDcxZ3yXfkUI5J1pu +VShLC9pwIU01fuP+txofqWly6doawKjY5N5boqlMOk+6+ckJ6h1ZD8fnVsJB5NwQ +653Uxr+K1UeGwg92f7x4wjYpQ1Dr4/T4fRxtpy5pygZjw23cZZWeEzhhTPzB6PXy +Yb43Mil+9QlfYrQnTWFyayBMaXUgKEdQRyBrZXkpIDxtYXJrbGl1QGdvb2dsZS5j +b20+iQI2BDABCgAgFiEEwRCxyCB0iDpCQdl3WZ1jBf86uzIFAl14J5QCHSAACgkQ +WZ1jBf86uzIBihAAiZQ0EDx7BVbWM8E5UwgkWlv1L6+mCBUivrTagIrrXAOfgp8o +JEJ+AcLz9P22kRe7f4VM6xVyYm+GGOBeUD+bUCr14eZhfs7Buu4q9cz1yimnWg8d +riH9A1vxIFmZ86n1uFyWn8sG4FpJF09edAyCOYmvmJ4fdZiU3QSILfq0L3UT3294 +qIwWQRG9H27NBkcR4ivtyQ4uvwrql7Ut0Ew9Kj4T53T4FADG9n2RCCHjCKxShqGa +jAoFLktxw/K75pgDV4XdkS+81XHws1KcozBVpR8eWJWLmfE5Fu9lhm03RVcwS3Uu +Xg4An4P1wKv/BcViaY/hqR0KBJuOMeeC9HMxqL1wkyZ5O3s+Vf5tfP2DWPKMyfz+ +bw+4c5rH4mzja7bC5DCS0XNM2Ga8VhH4cN8Y/NyKu3ldaauAiLtB9i2XBXFEMe4q +yoe/x3/tTBgIQUaXTTuC5811M15PwQBd+rFlDf2YT8qQYi0vZDqXJF6SKSl/youG +8mwoUlOB5cLoinFAleGmHHN9LDmmOZdMZgoCqryo/cEHlNkNtVYf5KpMMzoca6T0 +r1ihVPO2O78/R0fVit2+Fm5IeHTaJsMLtMS/5b9f79fydw2924sEv4egLwo4tPus +9legKEYCfHhdED7jqc2icqtk+MPImIw+P6WZt2/RlTOaTjDg4lAGNoihfpCJAk4E +EwEKADgWIQTBELHIIHSIOkJB2XdZnWMF/zq7MgUCXXghnQIbAwULCQgHAgYVCgkI +CwIEFgIDAQIeAQIXgAAKCRBZnWMF/zq7Mj4GEADG1JeAQw1bRjFQ6Qg32kRkOwds +ZkRhO9ssoqfDVKKu1YvfhFtIYjGYHkQnTuWnEp5q4l0B2iSRY46CtpXcqExTuFen +omElJ1R6Ufi/OtULzfhCBS8GVLrvEfdHC5v/Uv9BVlKzoT+MY1jPeNq+wNqsVwEm +CwLQRXVrvBaPq6BKFEWOUItEvurJ3YklMBh74OFOYkYz7SfWmZQL0Bq15vwzXPjP +JZw+014cOQiDdSaV1R0RO2upNoPfDbzcy4H/4JGSXbSeFrwRV4R0cZH7PkEiqFbe +bxEzSzFkM0TZzhI9RO6BOGxWTVJIpt0bA0z67SO+U8Kc4V9/u67BkOUn5w3PayG0 +hHKayL6tIrxcAUclSJhmMDRDoe1EUb7jjkoCtjikK0LQ8zYada+RZHAlKolI3ZXJ +52uhqrcpKznUbV+vA0EmrpFMc4hRNtQMo5RO4se6S1hZj5JngHv7H5xB1ow/FQlG +Vn6tWxzxnA4eq0sotN4bgYqfrktFsm3hmwmKDcJ8digkKtt8WKY4zhlYWugcAswB +Vuyn1I1ktc65kX4nVyF0anunJ9dJtTcAj3U/gyyB1nx2Z/5WnRGJbfJgMjoMUC/R +dD9OWvn90+BPle94C9hYNvHnJm7oObprLCrpsMrB1iNnGbW93O9lY1VZlRdn99Fk +VvMw7dCbjClAMZ8yorkCDQRdeCGdARAAwT2COsvNmaqxmnzGOCZIMlgiBNMijX8/ +mUA/UkZqNaP/LPRn2w1Br0XAW5wkaOKH+3R5Q7BdIvJ+AshnmuKubSRSgdulvIUj +55zNIHaQvMEw7d6ya4Km7Is9edrv3GMtdf8R1bSmH84dFGgwDbayLtRHn4b9nOSP +8iic+RVlfxCpDRQv4vJFRWO7jv9J10VHteNoZj+1XL2JoP+DESRtCz3xGE7eObz9 +b2sLF/vAU0ZR/pmQ61sksLeSs8G3K006sOVAzFIA4vCX11hRT9iztdWCnPliGXlS +WKK5tVXcx7sT/aF48509iuXb1NEpMVWxn85Db1Lqg+gmvfsdJpiNoUDqFpySmEHT +vRruHC/iUXanejuC4dI4HM6eLLVMxdt4qvfZfwzZPY/rIr447sDv6z0ab3kEHXuy +QG0jz1nHiwPByapLk4LtaxHko0I0/QyBcpThMCH8a56gsELZshW5LkWZDyKoDUu2 +wC+uGpbUSWlJqknYK7uVFed/1oQQz6GvR5Rjmfmqd54/bNIbVwjY6Rac8IOL1ah3 +bxq0QMKoR865ReLPNZCFZLtwmS0p1GeGSvJnc4BreLkgDSoVl5CFTxl0gWgmbCfN +b/Irbg3gXcVxgTb1AbBNPEZ/1iFa/+vHt5vOjADFIlEWwOX0BRXci6px5DTslrgh +Sd6KalTCnD8AEQEAAYkCNgQYAQoAIBYhBMEQscggdIg6QkHZd1mdYwX/OrsyBQJd +eCGdAhsMAAoJEFmdYwX/OrsytbgQAMNCtV9jTpln4UdWR/+9qwcAw1j4cNQgPpaX +7qEPM1n0PfltC8joH5LQzuk22r7g2srLtmu4R5VJ4iAFSfA3hg7qDNoe1WVhggvD +Iw8/enSR9Ea195VIUMKfzsY1gMnJ+QH4bkYNgfMrI5OfSQM/ITLSIdclTOXwu64a ++ZoLHKlBzW1mgiKEgH+OlRi6yPi87FyUlpKeXCGmjjCCBMCPfLSeYHs92cSXO0Tp +vAZ+5O8cvTsO+4jsQaVwhOixiQ5HobLvbRgg+q6T/7Loc2
[beam] branch master updated (9043b3e -> 9390339)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9043b3e Merge pull request #9372: [Beam-6858] Validate that side-input parameters match the type of the PCollectionView add eefe965 Remove obsolete field from proto. new 9390339 Remove obsolete field from proto. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: model/job-management/src/main/proto/beam_artifact_api.proto | 4 1 file changed, 4 deletions(-)
[beam] 01/01: Remove obsolete field from proto.
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 93903397b172ee4a991b6bcb9ff2ef7a4c925758 Merge: 9043b3e eefe965 Author: Lukasz Cwik AuthorDate: Sat Sep 7 09:09:25 2019 -0700 Remove obsolete field from proto. model/job-management/src/main/proto/beam_artifact_api.proto | 4 1 file changed, 4 deletions(-)
[beam] branch master updated (58a12b6 -> 014bbee)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 58a12b6 Merge pull request #9454: [BEAM-8111] Add ValidatesRunner test to AvroSchemaTest add 36aaabc [BEAM-8161] Update joda time to 2.10.3 to get TZDB 2019b add 014bbee [BEAM-8161] Update joda time to 2.10.3 to get TZDB 2019b No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- learning/katas/java/build.gradle| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (9bbd89a -> a9e14ff)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 9bbd89a Merge pull request #9440: [BEAM-5428] Modify cache token
Proto design to only include tokens in ProcessBundleRequest
add f1d02e9 [BEAM-3489] add PubSub messageId to PubsubMessage
add 71f6ddf [BEAM-3489] add Coder test
add d0b9cac [BEAM-3489] add messageId coders to
PubsubUnboundedSource.getOutputCoder()
add b8313b0 [BEAM-3489] update docs
add a9e14ff [BEAM-3489] add PubSub messageId in PubsubMessage
No new revisions were added by this update.
Summary of changes:
.../beam/runners/dataflow/DataflowRunner.java | 10 +--
.../beam/runners/dataflow/worker/PubsubReader.java | 4 +-
.../gcp/pubsub/PubsubCoderProviderRegistrar.java | 7 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java| 47 -
.../beam/sdk/io/gcp/pubsub/PubsubMessage.java | 18 -
...subMessageWithAttributesAndMessageIdCoder.java} | 30 -
...r.java => PubsubMessageWithMessageIdCoder.java} | 32 +
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 78 +++---
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 2 +-
.../pubsub/PubsubMessagePayloadOnlyCoderTest.java} | 35 ++
...MessageWithAttributesAndMessageIdCoderTest.java | 62 +
.../PubsubMessageWithAttributesCoderTest.java | 60 +
.../PubsubMessageWithMessageIdCoderTest.java} | 36 ++
.../beam/sdk/io/gcp/pubsub/PubsubReadIT.java | 47 +
14 files changed, 388 insertions(+), 80 deletions(-)
copy
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/{PubsubMessageWithAttributesCoder.java
=> PubsubMessageWithAttributesAndMessageIdCoder.java} (73%)
copy
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/{PubsubMessagePayloadOnlyCoder.java
=> PubsubMessageWithMessageIdCoder.java} (64%)
copy
sdks/java/{core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java =>
io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoderTest.java}
(51%)
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoderTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesCoderTest.java
copy
sdks/java/{core/src/test/java/org/apache/beam/sdk/coders/VoidCoderTest.java =>
io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoderTest.java}
(50%)
[beam] branch master updated (c7f0b49 -> 7f09832)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c7f0b49 Merge pull request #9439 from iemejia/BEAM-8107-commons-compress-update new c9da964 Revert "Merge pull request #8943: Schema conversion cleanup" new dbcb14c Revert "Update portable schema representation and java SchemaTranslation (#8853)" new 7f09832 Merge pull request #9443 from TheNeuralBit/revert-portable-schema The 22910 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../pipeline/src/main/proto/beam_runner_api.proto | 55 ++ model/pipeline/src/main/proto/schema.proto | 85 - .../core/construction/SchemaTranslation.java | 194 - .../core/construction/SchemaTranslationTest.java | 89 -- .../util/SchemaCoderCloudObjectTranslator.java | 6 +- 5 files changed, 130 insertions(+), 299 deletions(-) delete mode 100644 model/pipeline/src/main/proto/schema.proto delete mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
[beam] branch master updated (8a3399b -> f0618c5)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8a3399b Housekeeping for GCP dependencies add a8d8235 Add mongodbio to beam website io links add f0618c5 Add python mongodbio to beam website IO links No new revisions were added by this update. Summary of changes: website/src/documentation/io/built-in.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (5bb21c8 -> 8a3399b)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5bb21c8 Document StateKey enums for state backed iterable use case. add 203a98f Use consistent names for auto-generated GCP client artifacts. add 6303d41 Alphabetize imports of GCP library entries add 26ecb3d Depend on proto- instead of grpc- package for google_cloud_bigtable add 74e9fd5 Merge branch 'master' into beam_java_dependencies add a1bfde3 Merge branch 'master' into beam_java_dependencies add 8a3399b Housekeeping for GCP dependencies No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 11 +++-- examples/java/build.gradle | 14 +++--- examples/kotlin/build.gradle | 14 +++--- runners/google-cloud-dataflow-java/build.gradle| 22 - sdks/java/io/google-cloud-platform/build.gradle| 53 +++--- 5 files changed, 58 insertions(+), 56 deletions(-)
[beam] branch master updated (cb6a86d -> 5bb21c8)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cb6a86d [BEAM-6127] upgrade junit dependency add 45b595b Update beam_fn_api.proto add 2e9f052 Update beam_fn_api.proto add 5bb21c8 Document StateKey enums for state backed iterable use case. No new revisions were added by this update. Summary of changes: model/fn-execution/src/main/proto/beam_fn_api.proto | 5 + 1 file changed, 5 insertions(+)
[beam] branch master updated: [BEAM-8006] Add retracting to windowing strategy translation.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ff7964c [BEAM-8006] Add retracting to windowing strategy translation.
ff7964c is described below
commit ff7964c7252c8a0c670f69bb4291230ca6136afd
Author: amaliujia
AuthorDate: Tue Aug 20 15:29:29 2019 -0700
[BEAM-8006] Add retracting to windowing strategy translation.
---
.../core/construction/WindowingStrategyTranslation.java | 4
.../core/construction/WindowingStrategyTranslationTest.java | 7 +++
.../java/org/apache/beam/sdk/values/WindowingStrategy.java | 12 ++--
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
index 5fd0f33..a57aa9b 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
@@ -57,6 +57,8 @@ public class WindowingStrategyTranslation implements
Serializable {
return AccumulationMode.DISCARDING_FIRED_PANES;
case ACCUMULATING:
return AccumulationMode.ACCUMULATING_FIRED_PANES;
+ case RETRACTING:
+return AccumulationMode.RETRACTING_FIRED_PANES;
case UNRECOGNIZED:
default:
// Whether or not it is proto that cannot recognize it (due to the
version of the
@@ -77,6 +79,8 @@ public class WindowingStrategyTranslation implements
Serializable {
return RunnerApi.AccumulationMode.Enum.DISCARDING;
case ACCUMULATING_FIRED_PANES:
return RunnerApi.AccumulationMode.Enum.ACCUMULATING;
+ case RETRACTING_FIRED_PANES:
+return RunnerApi.AccumulationMode.Enum.RETRACTING;
default:
throw new IllegalArgumentException(
String.format(
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
index eee9c3f..9f50c0b 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
@@ -85,6 +85,13 @@ public class WindowingStrategyTranslationTest {
.withMode(AccumulationMode.DISCARDING_FIRED_PANES)
.withTrigger(REPRESENTATIVE_TRIGGER)
.withAllowedLateness(Duration.millis(93))
+.withTimestampCombiner(TimestampCombiner.LATEST)),
+toProtoAndBackSpec(
+WindowingStrategy.of(REPRESENTATIVE_WINDOW_FN)
+.withClosingBehavior(ClosingBehavior.FIRE_IF_NON_EMPTY)
+.withMode(AccumulationMode.RETRACTING_FIRED_PANES)
+.withTrigger(REPRESENTATIVE_TRIGGER)
+.withAllowedLateness(Duration.millis(100))
.withTimestampCombiner(TimestampCombiner.LATEST)));
}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
index dfd9562..6b2c4d6 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
@@ -44,10 +44,18 @@ import org.joda.time.Duration;
*/
public class WindowingStrategy implements
Serializable {
- /** The accumulation modes that can be used with windowing. */
+ /**
+ * The accumulation modes that can be used with windowing.
+ *
+ * Experimental {@link AccumulationMode.RETRACTING_FIRED_PANES} for
enabling retractions in
+ * pipelines. There is no backwards-compatibility guarantees.
+ */
public enum AccumulationMode {
DISCARDING_FIRED_PANES,
-ACCUMULATING_FIRED_PANES
+ACCUMULATING_FIRED_PANES,
+// RETRACTING_FIRED_PANES is experimental. There is no
backwards-compatibility guarantees.
+@Experimental
+RETRACTING_FIRED_PANES,
}
private static final Duration DEFAULT_ALLOWED_LATENESS = Duration.ZERO;
[beam] branch master updated (797c27f -> eddaf92)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 797c27f [BEAM-7994] Fixing unsafe pointer usage for Go 1.13 add c170081 Fix the exception type of InMemoryJobService when job id not found add 86c2ebd Update to also catch StatusRuntimeException add eddaf92 [BEAM-7720] Fix the exception type of InMemoryJobService when job id not found No new revisions were added by this update. Summary of changes: .../fnexecution/jobsubmission/InMemoryJobService.java| 12 1 file changed, 12 insertions(+)
[beam] branch master updated (d1f99ce -> 797c27f)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from d1f99ce Merge pull request #9286: [BEAM-5980] Remove redundant
combine tests
add 873f776 [BEAM-7994] Fixing unsafe pointer usage for Go 1.13
add 797c27f [BEAM-7994] Fixing unsafe pointer usage for Go 1.13
No new revisions were added by this update.
Summary of changes:
sdks/go/pkg/beam/core/util/reflectx/functions.go | 4 ++-
.../beam/core/util/reflectx/functions_test.go} | 31 --
2 files changed, 20 insertions(+), 15 deletions(-)
copy sdks/go/{test/integration/primitives/pardo_test.go =>
pkg/beam/core/util/reflectx/functions_test.go} (64%)
[beam] branch master updated: [BEAM-7965] add retracting mode to model proto.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 530f1d7 [BEAM-7965] add retracting mode to model proto.
new 544604c Merge pull request #9329 from
amaliujia/add_retraction_in_java_sdk
530f1d7 is described below
commit 530f1d70a6b143a1f6c0e03c9a49808f2e3cedf8
Author: amaliujia
AuthorDate: Tue Aug 13 10:27:48 2019 -0700
[BEAM-7965] add retracting mode to model proto.
---
model/pipeline/src/main/proto/beam_runner_api.proto | 3 +++
1 file changed, 3 insertions(+)
diff --git a/model/pipeline/src/main/proto/beam_runner_api.proto
b/model/pipeline/src/main/proto/beam_runner_api.proto
index 204c408..736bcdc 100644
--- a/model/pipeline/src/main/proto/beam_runner_api.proto
+++ b/model/pipeline/src/main/proto/beam_runner_api.proto
@@ -734,6 +734,9 @@ message AccumulationMode {
// The aggregation is accumulated across outputs
ACCUMULATING = 2;
+
+// The aggregation emits retractions when it is output
+RETRACTING = 3;
}
}
[beam] branch master updated: release vendor calcite
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f97b04a release vendor calcite
new cde2c71 Merge pull request #9333 from vectorijk/release-vendor-calcite
f97b04a is described below
commit f97b04ab0a112a8fc59b9919cf06cf955c044d79
Author: Kai Jiang
AuthorDate: Tue Aug 13 15:29:37 2019 -0700
release vendor calcite
---
settings.gradle| 1 +
vendor/calcite-1_20_0/build.gradle | 63 ++
2 files changed, 64 insertions(+)
diff --git a/settings.gradle b/settings.gradle
index 8a2ec8e..6fc34d1 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -143,6 +143,7 @@ include ":sdks:python:test-suites:tox:py36"
include ":sdks:python:test-suites:tox:py37"
include ":vendor:grpc-1_21_0"
include ":vendor:bytebuddy-1_9_3"
+include ":vendor:calcite-1_20_0"
include ":vendor:sdks-java-extensions-protobuf"
include ":vendor:guava-26_0-jre"
include ":website"
diff --git a/vendor/calcite-1_20_0/build.gradle
b/vendor/calcite-1_20_0/build.gradle
new file mode 100644
index 000..09d9fa6
--- /dev/null
+++ b/vendor/calcite-1_20_0/build.gradle
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.vendor-java' }
+
+description = "Apache Beam :: Vendored Dependencies :: Calcite 1.20.0"
+
+group = "org.apache.beam"
+version = "0.1"
+
+def calcite_version = "1.20.0"
+def avatica_version = "1.15.0"
+def prefix = "org.apache.beam.vendor.calcite.v1_20_0"
+
+List packagesToRelocate = [
+"com.esri",
+"com.google.common",
+"com.google.thirdparty",
+"com.google.protobuf",
+"com.fasterxml",
+"com.jayway",
+"com.yahoo",
+"org.apache.calcite",
+"org.apache.commons",
+"org.apache.http",
+"org.codehaus",
+"org.pentaho",
+"org.yaml"
+]
+
+vendorJava(
+dependencies: [
+"org.apache.calcite:calcite-core:$calcite_version",
+"org.apache.calcite:calcite-linq4j:$calcite_version",
+"org.apache.calcite.avatica:avatica-core:$avatica_version",
+library.java.protobuf_java,
+library.java.slf4j_api
+],
+relocations: packagesToRelocate.collectEntries {
+[ (it): "${prefix}.${it}" ]
+},
+exclusions: [
+"org/slf4j/**"
+],
+groupId: group,
+artifactId: "beam-vendor-calcite-1_20_0",
+version: version,
+)
[beam] branch master updated (2785f14 -> 5d9bb45)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2785f14 Merge pull request #9354 from apache/aaltay-patch-2-1 add 5d9bb45 [BEAM-7987] Drop empty Windmill workitem in WindowingWindmillReader (#9336) No new revisions were added by this update. Summary of changes: .../dataflow/worker/WindowingWindmillReader.java | 62 +++--- 1 file changed, 43 insertions(+), 19 deletions(-)
[beam] branch master updated (22eff79 -> b645856)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 22eff79 [BEAM-7959] Simplify use of Gradle property and task in Python add ead80b4 [BEAM-7954] Validate beam modules to disable guava imports from vendored gRPC new b645856 [BEAM-7954] Validate beam modules to disable guava imports from vendored gRPC The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/runners/core/metrics/DefaultMetricResults.java| 2 +- .../beam/runners/flink/metrics/FlinkMetricContainerTest.java | 2 +- sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 8 .../main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java | 2 +- .../sdk/extensions/sql/meta/provider/kafka/KafkaCSVTableIT.java | 2 +- 5 files changed, 12 insertions(+), 4 deletions(-)
[beam] 01/01: [BEAM-7954] Validate beam modules to disable guava imports from vendored gRPC
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit b645856a6cbb82a6c119e1678ecb38ea4f8b770c Merge: 22eff79 ead80b4 Author: Lukasz Cwik AuthorDate: Tue Aug 13 10:19:08 2019 -0700 [BEAM-7954] Validate beam modules to disable guava imports from vendored gRPC .../apache/beam/runners/core/metrics/DefaultMetricResults.java| 2 +- .../beam/runners/flink/metrics/FlinkMetricContainerTest.java | 2 +- sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 8 .../main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java | 2 +- .../sdk/extensions/sql/meta/provider/kafka/KafkaCSVTableIT.java | 2 +- 5 files changed, 12 insertions(+), 4 deletions(-)
[beam] branch master updated (4ff1347 -> 22eff79)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4ff1347 Merge pull request #9314: [BEAM-7916] Add ElasticsearchIO query parameter to take a ValueProvider add 89498f2 [BEAM-7959] Simplify use of Gradle property and task in Python add 22eff79 [BEAM-7959] Simplify use of Gradle property and task in Python No new revisions were added by this update. Summary of changes: sdks/python/build.gradle | 32 ++- sdks/python/test-suites/dataflow/py2/build.gradle | 62 +- sdks/python/test-suites/dataflow/py35/build.gradle | 28 ++ sdks/python/test-suites/dataflow/py36/build.gradle | 26 + sdks/python/test-suites/dataflow/py37/build.gradle | 34 +++- sdks/python/test-suites/direct/py2/build.gradle| 23 sdks/python/test-suites/direct/py35/build.gradle | 10 ++-- sdks/python/test-suites/direct/py36/build.gradle | 10 ++-- sdks/python/test-suites/direct/py37/build.gradle | 15 +++--- sdks/python/test-suites/portable/py2/build.gradle | 30 +-- 10 files changed, 160 insertions(+), 110 deletions(-)
[beam] 01/01: [BEAM-7854] Resolve parent folder recursively in LocalFileSystem matc…
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 22d70f9af8658dc15458f777f2656dfa95febb8f Merge: d79c06d badef9c Author: Lukasz Cwik AuthorDate: Tue Aug 13 07:48:38 2019 -0700 [BEAM-7854] Resolve parent folder recursively in LocalFileSystem matc… .../org/apache/beam/sdk/io/LocalFileSystem.java| 19 - .../apache/beam/sdk/io/LocalFileSystemTest.java| 31 ++ 2 files changed, 49 insertions(+), 1 deletion(-)
[beam] branch master updated (d79c06d -> 22d70f9)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d79c06d Merge pull request #9295: [BEAM-7721] Add write test to BigQueryIOIT add 614916a [BEAM-7854] Resolve parent folder recursively in LocalFileSystem match + test add e3cd3b9 Use regex to detect non glob prefix of path add 770519a Change tests add 48955be Refactor + put test back add 21ddfd1 Change logic + fix tests add 66c3dea Spotless add badef9c Spotless new 22d70f9 [BEAM-7854] Resolve parent folder recursively in LocalFileSystem matc… The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/io/LocalFileSystem.java| 19 - .../apache/beam/sdk/io/LocalFileSystemTest.java| 31 ++ 2 files changed, 49 insertions(+), 1 deletion(-)
[beam] 01/01: Fix: Make sure experiments list if modifiable
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 7eca11756febe55354034791e9aabe008537ecee Merge: 12837e8 8b388a7 Author: Lukasz Cwik AuthorDate: Mon Aug 12 16:20:09 2019 -0700 Fix: Make sure experiments list if modifiable .../org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java| 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (12837e8 -> 7eca117)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 12837e8 [BEAM-3608] Remove remaining unused non-vendored guava dependencies add 8b388a7 Fix: Make sure experiments list if modifiable new 7eca117 Fix: Make sure experiments list if modifiable The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java| 2 ++ 1 file changed, 2 insertions(+)
[beam] 01/01: [BEAM-3608] Remove remaining unused non-vendored guava dependencies
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 12837e847155a901652d538e9d50cbd580e0 Merge: de18f4b 13b45e6 Author: Lukasz Cwik AuthorDate: Mon Aug 12 16:09:40 2019 -0700 [BEAM-3608] Remove remaining unused non-vendored guava dependencies runners/samza/job-server/build.gradle | 1 - runners/spark/build.gradle | 1 - sdks/java/extensions/euphoria/build.gradle | 1 - sdks/java/io/bigquery-io-perf-tests/build.gradle | 1 - sdks/java/io/file-based-io-tests/build.gradle | 1 - sdks/java/io/google-cloud-platform/build.gradle| 1 - sdks/java/io/hcatalog/build.gradle | 1 - sdks/java/io/kinesis/build.gradle | 1 - sdks/java/io/synthetic/build.gradle| 1 - sdks/java/maven-archetypes/examples/build.gradle | 1 - .../examples/src/main/resources/archetype-resources/pom.xml| 7 --- 11 files changed, 17 deletions(-)
[beam] branch master updated (de18f4b -> 12837e8)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from de18f4b Merge pull request #9322 from akedin/disable-kafkatableit add 13b45e6 [BEAM-3608] Remove remaining unused non-vendored guava dependencies new 12837e8 [BEAM-3608] Remove remaining unused non-vendored guava dependencies The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/samza/job-server/build.gradle | 1 - runners/spark/build.gradle | 1 - sdks/java/extensions/euphoria/build.gradle | 1 - sdks/java/io/bigquery-io-perf-tests/build.gradle | 1 - sdks/java/io/file-based-io-tests/build.gradle | 1 - sdks/java/io/google-cloud-platform/build.gradle| 1 - sdks/java/io/hcatalog/build.gradle | 1 - sdks/java/io/kinesis/build.gradle | 1 - sdks/java/io/synthetic/build.gradle| 1 - sdks/java/maven-archetypes/examples/build.gradle | 1 - .../examples/src/main/resources/archetype-resources/pom.xml| 7 --- 11 files changed, 17 deletions(-)
[beam] branch master updated (0704bb8 -> e5e9cab)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0704bb8 Merge pull request #9317: [BEAM-7940] Quick fix sdks:python:depSnapshot add d2d8bac Add JobService.GetJobs to the job API add e0a65ec Fix python code style new e5e9cab [BEAM-7927] Add JobService.GetJobs to the job API The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/proto/beam_job_api.proto | 19 .../jobsubmission/InMemoryJobService.java | 23 .../fnexecution/jobsubmission/JobInvocation.java | 10 + .../jobsubmission/InMemoryJobServiceTest.java | 25 ++ .../runners/portability/local_job_service.py | 11 ++ 5 files changed, 88 insertions(+)
