[beam] branch master updated (ded0325 -> 76c6129)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ded0325 [BEAM-11075] Go SDK SideInput load tests (#13436) add f1c1514 [BEAM-11267] Remove unecessary reshuffle for stateful ParDo after keyed operation. add 76c6129 Merge pull request #13353: [BEAM-11267] Remove unnecessary reshuffle for stateful ParDo after key… No new revisions were added by this update. Summary of changes: .../FlinkStreamingPortablePipelineTranslator.java | 4 +- .../flink/FlinkStreamingTransformTranslators.java | 32 +- .../flink/FlinkStreamingTranslationContext.java| 14 ++- .../wrappers/streaming/WorkItemKeySelector.java| 9 +- .../FlinkStreamingPipelineTranslatorTest.java | 109 + .../wrappers/streaming/WindowDoFnOperatorTest.java | 3 +- 6 files changed, 159 insertions(+), 12 deletions(-)
[beam] branch master updated (f87c984 -> 56d4ac9)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f87c984 Merge pull request #13470 from [BEAM-10114] Convert PubsubLiteIO read to use SplittableDoFn. add dee8d49 [BEAM-11443] Implement byte-based GBK for Flink Streaming Runner. add 56d4ac9 Merge pull request #13538: [BEAM-11443] Implement byte-based GBK for Flink Streaming Runner. No new revisions were added by this update. Summary of changes: .../FlinkStreamingPortablePipelineTranslator.java | 23 +-- .../flink/FlinkStreamingTransformTranslators.java | 189 ++--- .../flink/FlinkStreamingTranslationContext.java| 10 +- .../wrappers/streaming/DoFnOperator.java | 5 +- .../streaming/SingletonKeyedWorkItemCoder.java | 13 +- .../wrappers/streaming/WorkItemKeySelector.java| 4 +- .../wrappers/streaming/WindowDoFnOperatorTest.java | 2 +- 7 files changed, 152 insertions(+), 94 deletions(-)
[beam] branch master updated (c4af7f9 -> 0424485)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c4af7f9 Merge pull request #13456 from [BEAM-10480] Add a small announcement for Splittable DoFn. add 44fc9fe [BEAM-10402] Supress nullness and keyfor DoFnOperatorTest warnings. add 0424485 Merge pull request #13525: [BEAM-10402] Supress nullness and keyfor DoFnOperatorTest warnings. No new revisions were added by this update. Summary of changes: .../runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (d009fe9 -> 744637d)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d009fe9 Merge pull request #13503: [BEAM-11415] In-progress Go ValidatesRunner framework. add cd2c312 [BEAM-11435] Reuse already set timers. add 744637d Merge pull request #13522: [BEAM-11435] Reuse already set timers. No new revisions were added by this update. Summary of changes: .../wrappers/streaming/DoFnOperator.java | 31 +++--- 1 file changed, 22 insertions(+), 9 deletions(-)
[beam] branch master updated (b6ca2ab -> ccc2fa4)
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from b6ca2ab [BEAM-9723] Add DLP integration transforms (#11566)
add 0c45136 [BEAM-8850,BEAM-8849] FlinkBatchTranslationContext respects
input parameters during lookups. Add support for a multiple translation
variants for a single urn.
add 3c4536d [BEAM-8848] Flink Batch Runner: optimize GBK translation for
non-merging windows.
add 23a1fb7 [BEAM-8848] Code review.
add ccc2fa4 Merge pull request #11874: Flink Batch Runner: optimize GBK
translation for non-merging windows.
No new revisions were added by this update.
Summary of changes:
.../flink/FlinkBatchPipelineTranslator.java| 15 ++-
.../flink/FlinkBatchTransformTranslators.java | 124 +
.../flink/FlinkBatchTranslationContext.java| 61 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 7 ++
...ction.java => FlinkExplodeWindowsFunction.java} | 25 ++---
.../functions/FlinkNonMergingReduceFunction.java | 112 +++
...KeySelector.java => WindowedKvKeySelector.java} | 16 ++-
.../utils/CountingPipelineVisitor.java}| 32 +++---
.../translation/utils/LookupPipelineVisitor.java | 98
.../flink/batch/NonMergingGroupByKeyTest.java | 80 +
.../shortcodes/flink_java_pipeline_options.html| 5 +
.../shortcodes/flink_python_pipeline_options.html | 5 +
12 files changed, 493 insertions(+), 87 deletions(-)
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/{FlinkIdentityFunction.java
=> FlinkExplodeWindowsFunction.java} (59%)
create mode 100644
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkNonMergingReduceFunction.java
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/{KvKeySelector.java
=> WindowedKvKeySelector.java} (68%)
copy
runners/{google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/RecordingPipelineVisitor.java
=>
flink/src/main/java/org/apache/beam/runners/flink/translation/utils/CountingPipelineVisitor.java}
(59%)
create mode 100644
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/utils/LookupPipelineVisitor.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/batch/NonMergingGroupByKeyTest.java
[beam] branch master updated (1bd1508 -> 1a8c563)
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1bd1508 Merge pull request #11871: [website] Fix programming guide
typo
add 6dc08a4 [BEAM-8608] Chain DoFns in Flink batch runner when possible.
add 3689296 [BEAM-8608] Code review. Fix checkstyle.
new 1a8c563 Merge pull request #11858: [BEAM-8608] Chain DoFns in Flink
batch runner when possible.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../beam/runners/flink/FlinkCapabilities.java} | 17 ++-
.../beam/runners/flink/FlinkCapabilities.java} | 17 ++-
.../flink/FlinkBatchTransformTranslators.java | 21 ++-
.../translation/functions/FlinkDoFnFunction.java | 156 ++---
.../functions/FlinkStatefulDoFnFunction.java | 7 +-
.../src/main/resources/beam/suppressions.xml | 1 +
6 files changed, 152 insertions(+), 67 deletions(-)
copy
runners/flink/{src/main/java/org/apache/beam/runners/flink/FlinkStateBackendFactory.java
=> 1.10/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java}
(66%)
copy
runners/flink/{src/main/java/org/apache/beam/runners/flink/FlinkStateBackendFactory.java
=> 1.8/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java}
(66%)
[beam] 01/01: Merge pull request #11858: [BEAM-8608] Chain DoFns in Flink batch runner when possible.
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1a8c56380c924e25135a13dfdc0a5cb101cfddf1 Merge: 1bd1508 3689296 Author: David Moravek AuthorDate: Mon Jun 1 09:46:07 2020 +0200 Merge pull request #11858: [BEAM-8608] Chain DoFns in Flink batch runner when possible. [BEAM-8608] Chain DoFns in Flink batch runner when possible. .../beam/runners/flink/FlinkCapabilities.java | 34 + .../beam/runners/flink/FlinkCapabilities.java | 34 + .../flink/FlinkBatchTransformTranslators.java | 21 ++- .../translation/functions/FlinkDoFnFunction.java | 156 ++--- .../functions/FlinkStatefulDoFnFunction.java | 7 +- .../src/main/resources/beam/suppressions.xml | 1 + 6 files changed, 194 insertions(+), 59 deletions(-)
[beam] branch master updated (2967e3a -> 5cc46df)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2967e3a Katas - Convert task description from HTML to Markdown (#11736) add 3719e6a [BEAM-9900] Fix polling behavior in UnboundedSourceWrapper add 77ea885 [BEAM-9900] Add test case for polling idle readers add 5cc46df Merge pull request #11750: [BEAM-9900] Fix polling behavior in UnboundedSourceWrapper No new revisions were added by this update. Summary of changes: .../streaming/io/UnboundedSourceWrapper.java | 14 +- .../streaming/io/UnboundedSourceWrapperTest.java | 246 ++--- 2 files changed, 227 insertions(+), 33 deletions(-)
[beam] branch master updated (e5dfa5b -> 26f694d)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e5dfa5b Merge pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164) add 96804c9 [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput add 26f694d Merge pull request #11612: [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput No new revisions were added by this update. Summary of changes: .../beam/runners/core/StatefulDoFnRunner.java | 6 ++-- .../beam/runners/core/StatefulDoFnRunnerTest.java | 41 ++ 2 files changed, 44 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #11530: [BEAM-9824] Do not ignore chained Reshuffles on flink batch runner.
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 164bef38f6487e646a5a57e1db644dd882ee11ac Merge: 1890696 482bbc0 Author: David Moravek AuthorDate: Tue Apr 28 09:33:01 2020 +0200 Merge pull request #11530: [BEAM-9824] Do not ignore chained Reshuffles on flink batch runner. .../flink/FlinkBatchTransformTranslators.java | 12 ++- .../beam/runners/flink/batch/ReshuffleTest.java| 104 + 2 files changed, 113 insertions(+), 3 deletions(-)
[beam] branch master updated (1890696 -> 164bef3)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1890696 Merge pull request #11546 from [BEAM-9832] Ensuring tags are not added to ProcessBundleDescriptors o… add 482bbc0 [BEAM-9824] Do not ignore chained Reshuffles on flink batch runner. new 164bef3 Merge pull request #11530: [BEAM-9824] Do not ignore chained Reshuffles on flink batch runner. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../flink/FlinkBatchTransformTranslators.java | 12 ++- .../beam/runners/flink/batch/ReshuffleTest.java| 104 + 2 files changed, 113 insertions(+), 3 deletions(-) create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/batch/ReshuffleTest.java
[beam] branch master updated (b744df1 -> 0b9c65e)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b744df1 Merge pull request #10610: [BEAM-9083] Exclude testOutputTimestamp from Flink PVR tests add f59fa64 [BEAM-9123] HadoopResourceId returns wrong directoryName bugfix add 0b9c65e Merge pull request #10597: [BEAM-9123] HadoopResourceId returns wrong directoryName bugfix No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/io/hdfs/HadoopResourceId.java | 4 .../org/apache/beam/sdk/io/hdfs/HadoopResourceIdTest.java | 15 +++ 2 files changed, 19 insertions(+)
[beam] branch master updated (076a037 -> eb9dea3)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 076a037 Merge pull request #9811: [BEAM-8402] Create a class hierarchy to represent Python environments add 4caefa6 [BEAM-8568] Fixed problem that LocalFileSystem no longer supports wildcard relative paths. add eb9dea3 Merge pull request #10024: [BEAM-8568] Fixed problem that LocalFileSystem no longer supports wil… No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/LocalFileSystem.java | 2 +- .../java/org/apache/beam/sdk/io/LocalFileSystemTest.java | 16 2 files changed, 17 insertions(+), 1 deletion(-)
[beam] branch revert-9464-hotfix created (now be30557)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch revert-9464-hotfix in repository https://gitbox.apache.org/repos/asf/beam.git. at be30557 Revert "[hotfix] Encode keys as NESTED for flink keyselector" No new revisions were added by this update.
[beam] branch master updated (7e1eddf -> ce4cd3b)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7e1eddf [BEAM-7909] fix python container image name (#9478) add 4a0eaf3 [hotfix] Encode keys as NESTED for flink keyselector add ce4cd3b Merge pull request #9464: [hotfix] Encode keys as NESTED for flink keyselector No new revisions were added by this update. Summary of changes: .../runners/flink/streaming/FlinkStateInternalsTest.java | 12 .../runners/flink/streaming/FlinkStateInternalsTest.java | 12 .../flink/translation/wrappers/streaming/FlinkKeyUtils.java | 7 +-- .../wrappers/streaming/state/FlinkStateInternals.java| 2 +- .../translation/wrappers/streaming/DoFnOperatorTest.java | 12 ++-- .../translation/wrappers/streaming/FlinkKeyUtilsTest.java| 5 +++-- 6 files changed, 39 insertions(+), 11 deletions(-)
[beam] branch master updated (84d5c87 -> a1b1cfe)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 84d5c87 Merge pull request #9453 from rahul8383/handle-unsupported-lookupjoins add 562f928 [BEAM-8025] Temporarily disable CassandraIOTest add a1b1cfe Merge pull request #9470: [BEAM-8025] Temporarily disable CassandraIOTest No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (fb83467 -> c787b5e)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fb83467 Merge pull request #8946: [BEAM-7574] fix Combine performance for SparkRunner add bfdb169 Add arbitrary value matchers to MetricsResultMatchers. add d088904 [BEAM-6883] Improve long-running spark streaming test. add c787b5e Merge pull request #9178: [BEAM-6883] Improve long-running spark streaming test. No new revisions were added by this update. Summary of changes: .../streaming/StreamingSourceMetricsTest.java | 29 ++-- .../beam/sdk/metrics/MetricResultsMatchers.java| 158 ++--- 2 files changed, 129 insertions(+), 58 deletions(-)
[beam] branch master updated (4e40b53 -> 8f6247e)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4e40b53 Merge pull request #9170: [BEAM-7830] Make spark default duration consistent with PipelineResult add 3d087be [BEAM-7366] Restore Spotless check for Flink Runner new 8f6247e Merge pull request #9176: [BEAM-7366] Restore Spotless check for Flink Runner The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 8 +--- runners/flink/flink_runner.gradle | 11 +++ 2 files changed, 12 insertions(+), 7 deletions(-)
[beam] 01/01: Merge pull request #9176: [BEAM-7366] Restore Spotless check for Flink Runner
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8f6247ef2382c8fd321e23435bbaf401cdb05b50 Merge: 4e40b53 3d087be Author: David Moravek AuthorDate: Mon Jul 29 18:06:58 2019 +0200 Merge pull request #9176: [BEAM-7366] Restore Spotless check for Flink Runner .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 8 +--- runners/flink/flink_runner.gradle | 11 +++ 2 files changed, 12 insertions(+), 7 deletions(-)
[beam] branch master updated (5e5e7a5 -> eb5a8c2)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5e5e7a5 Merge pull request #8849: [BEAM-7561] Enable HdfsFileSystems.match to match a directory add b0bf009 [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction add 175a8d2 [BEAM-7543] deprecate old #combineBy(Sums.ofLongs(), TypeDescriptors.longs()) new eb5a8c2 Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../euphoria/core/client/operator/CountByKey.java | 2 +- .../euphoria/core/client/operator/ReduceByKey.java | 307 +++-- .../core/client/operator/ReduceWindow.java | 292 +--- .../extensions/euphoria/core/client/util/Sums.java | 58 +++- .../core/translate/ReduceByKeyTranslator.java | 83 +- .../core/client/operator/ReduceByKeyTest.java | 99 +-- .../euphoria/core/client/util/SumsTest.java| 13 +- .../euphoria/core/testkit/ReduceByKeyTest.java | 47 +++- 8 files changed, 799 insertions(+), 102 deletions(-)
[beam] 01/01: Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit eb5a8c2cbd973972491dedef23aff98e450a9f5a Merge: 5e5e7a5 175a8d2 Author: David Moravek AuthorDate: Wed Jun 19 16:40:32 2019 +0200 Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction .../euphoria/core/client/operator/CountByKey.java | 2 +- .../euphoria/core/client/operator/ReduceByKey.java | 307 +++-- .../core/client/operator/ReduceWindow.java | 292 +--- .../extensions/euphoria/core/client/util/Sums.java | 58 +++- .../core/translate/ReduceByKeyTranslator.java | 83 +- .../core/client/operator/ReduceByKeyTest.java | 99 +-- .../euphoria/core/client/util/SumsTest.java| 13 +- .../euphoria/core/testkit/ReduceByKeyTest.java | 47 +++- 8 files changed, 799 insertions(+), 102 deletions(-)
[beam] branch master updated (2798d62 -> d754094)
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 2798d62 [BEAM-7510] Fixing fileio tests checking JSON serialization
(flaky) (#8795)
add c603563 [BEAM-7529] Add Sums.ofFloats() and Sums.ofDoubles()
add d754094 Merge pull request #8820: [BEAM-7529] Add Sums.ofFloats() and
Sums.ofDoubles()
No new revisions were added by this update.
Summary of changes:
.../extensions/euphoria/core/client/util/Sums.java | 16 +--
.../euphoria/core/client/util/SumsTest.java} | 33 +-
2 files changed, 33 insertions(+), 16 deletions(-)
copy
sdks/java/{core/src/test/java/org/apache/beam/sdk/transforms/windowing/NeverTest.java
=>
extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/SumsTest.java}
(58%)
[beam] branch master updated (1eb2164 -> 504de16)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1eb2164 Merge pull request #8116: Update a guideline for commits history new b03de9e [BEAM-6875] expose timestampSkew to user in FlatMap#eventTimeBy new a3cc066 [BEAM-6875] add allowedTimestampSkew to AssignEventTime new ee7ce73 [BEAM-6875] require allowedTimestampSkew to be non null new 881dc9a [BEAM-6875] finalize implementation new 504de16 Merge pull request #8106: [BEAM-6875] expose timestampSkew to user in FlatMap#eventTimeBy The 20708 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/extensions/euphoria/build.gradle | 4 -- .../core/client/operator/AssignEventTime.java | 27 - .../euphoria/core/client/operator/FlatMap.java | 44 -- .../euphoria/core/translate/FlatMapTranslator.java | 15 +++- .../euphoria/core/client/operator/FlatMapTest.java | 25 5 files changed, 103 insertions(+), 12 deletions(-)
[beam] branch master updated: Create a custom hash paritioner that deals with arrays during combines when used in spark
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 18820bb Create a custom hash paritioner that deals with arrays during
combines when used in spark
new 32bc6da Merge pull request #8042: [BEAM-6812]: Convert keys to
ByteArray in Combine.perKey to make sure hashCode is consistent
18820bb is described below
commit 18820bb49d123030a6ba2712692c2b2bb51dac6a
Author: Ankit Jhalaria
AuthorDate: Tue Mar 12 14:16:45 2019 -0700
Create a custom hash paritioner that deals with arrays during combines when
used in spark
---
.../runners/spark/translation/GroupCombineFunctions.java | 12
.../beam/runners/spark/translation/TranslationUtils.java | 10 +++---
2 files changed, 15 insertions(+), 7 deletions(-)
diff --git
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
index 95ff95a..0ec217d 100644
---
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
+++
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
@@ -143,10 +143,10 @@ public class GroupCombineFunctions {
// Once Spark provides a way to include keys in the arguments of
combine/merge functions,
// we won't need to duplicate the keys anymore.
// Key has to bw windowed in order to group by window as well.
-JavaPairRDD>> inRddDuplicatedKeyPair =
-rdd.mapToPair(TranslationUtils.toPairByKeyInWindowedValue());
+JavaPairRDD>>
inRddDuplicatedKeyPair =
+rdd.mapToPair(TranslationUtils.toPairByKeyInWindowedValue(keyCoder));
-JavaPairRDD>> accumulatedResult =
+JavaPairRDD>>
accumulatedResult =
inRddDuplicatedKeyPair.combineByKey(
input ->
SerializableAccumulator.of(sparkCombineFn.createCombiner(input),
iterAccumCoder),
@@ -160,7 +160,11 @@ public class GroupCombineFunctions {
acc1.getOrDecode(iterAccumCoder),
acc2.getOrDecode(iterAccumCoder)),
iterAccumCoder));
-return accumulatedResult.mapToPair(i -> new Tuple2<>(i._1,
i._2.getOrDecode(iterAccumCoder)));
+return accumulatedResult.mapToPair(
+i ->
+new Tuple2<>(
+CoderHelpers.fromByteArray(i._1.getValue(), keyCoder),
+i._2.getOrDecode(iterAccumCoder)));
}
/** An implementation of {@link Reshuffle} for the Spark runner. */
diff --git
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
index 8186a87..35ac89a 100644
---
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
+++
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
@@ -27,6 +27,7 @@ import org.apache.beam.runners.core.StateInternals;
import org.apache.beam.runners.core.StateInternalsFactory;
import org.apache.beam.runners.spark.SparkRunner;
import org.apache.beam.runners.spark.coders.CoderHelpers;
+import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.runners.spark.util.SideInputBroadcast;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.transforms.DoFn;
@@ -154,9 +155,12 @@ public final class TranslationUtils {
/** Extract key from a {@link WindowedValue} {@link KV} into a pair. */
public static
- PairFunction>, K, WindowedValue>>
- toPairByKeyInWindowedValue() {
-return windowedKv -> new Tuple2<>(windowedKv.getValue().getKey(),
windowedKv);
+ PairFunction>, ByteArray, WindowedValue>>
+ toPairByKeyInWindowedValue(final Coder keyCoder) {
+return windowedKv ->
+new Tuple2<>(
+new
ByteArray(CoderHelpers.toByteArray(windowedKv.getValue().getKey(), keyCoder)),
+windowedKv);
}
/** Extract window from a {@link KV} with {@link WindowedValue} value. */
[beam] branch master updated: [BEAM-6774] Euphoria - replace Distinct.mapped with Distinct.projected
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d4e7c80 [BEAM-6774] Euphoria - replace Distinct.mapped with
Distinct.projected
new c230034 Merge pull request #7809: Euphoria: Fix distinct mapped
return type
d4e7c80 is described below
commit d4e7c805c1ac80e14e2b65b89de6d31c291918a6
Author: Jan Lukavsky
AuthorDate: Mon Feb 11 12:16:06 2019 +0100
[BEAM-6774] Euphoria - replace Distinct.mapped with Distinct.projected
---
.../euphoria/core/client/io/Collector.java | 2 +-
.../core/client/operator/AssignEventTime.java | 6 +-
.../euphoria/core/client/operator/CountByKey.java | 12 +-
.../euphoria/core/client/operator/Distinct.java| 296 +++--
.../euphoria/core/client/operator/Filter.java | 6 +-
.../euphoria/core/client/operator/FlatMap.java | 14 +-
.../euphoria/core/client/operator/FullJoin.java| 18 +-
.../euphoria/core/client/operator/Join.java| 26 +-
.../euphoria/core/client/operator/LeftJoin.java| 18 +-
.../euphoria/core/client/operator/MapElements.java | 14 +-
.../euphoria/core/client/operator/ReduceByKey.java | 30 +--
.../core/client/operator/ReduceWindow.java | 26 +-
.../euphoria/core/client/operator/RightJoin.java | 18 +-
.../euphoria/core/client/operator/SumByKey.java| 12 +-
.../euphoria/core/client/operator/TopPerKey.java | 28 +-
.../euphoria/core/client/operator/Union.java | 6 +-
.../core/client/operator/base/ShuffleOperator.java | 1 +
.../translate/BroadcastHashJoinTranslator.java | 4 +-
.../euphoria/core/translate/OperatorTransform.java | 2 +-
.../core/translate/ReduceByKeyTranslator.java | 4 +-
.../core/translate/TimestampExtractTransform.java | 118
.../provider/GenericTranslatorProvider.java| 4 +-
.../core/docs/DocumentationExamplesTest.java | 6 +-
.../euphoria/core/testkit/DistinctTest.java| 112 ++--
.../translate/TimestampExtractTransformTest.java | 44 +++
website/src/documentation/sdks/euphoria.md | 4 +-
26 files changed, 599 insertions(+), 232 deletions(-)
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/io/Collector.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/io/Collector.java
index f990ff4..d3f772a 100644
---
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/io/Collector.java
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/io/Collector.java
@@ -36,7 +36,7 @@ public interface Collector extends Environment {
/**
* Returns {@link Context} view of the collector. Since {@link Collector}
usually share the same
- * methods as {@link Context} it can be safely casted.
+ * methods as {@link Context} it can be safely cast.
*
* @return this instance as a context class
*/
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/AssignEventTime.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/AssignEventTime.java
index a660c05..bb65a1c 100644
---
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/AssignEventTime.java
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/AssignEventTime.java
@@ -110,9 +110,9 @@ public class AssignEventTime extends
Operator
@Override
public UsingBuilder of(PCollection input) {
@SuppressWarnings("unchecked")
- final Builder casted = (Builder) this;
- casted.input = input;
- return casted;
+ final Builder cast = (Builder) this;
+ cast.input = input;
+ return cast;
}
@Override
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
index 3ba8ea2..01c17d5 100644
---
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
@@ -185,10 +185,10 @@ public class CountByKey extends
ShuffleOperator WindowByBuilder keyBy(
UnaryFunction keyExtractor, @Nullable TypeDescriptor
keyType) {
@SuppressWarnings("unchecked")
- final Builder casted = (Builder) this;
- casted.keyExtractor = require
[beam] branch master updated: [BEAM-6589] include org.objenesis into shaded jar
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new dc3bc33 [BEAM-6589] include org.objenesis into shaded jar
new 228a7d5 Merge pull request #7732: [BEAM-6589] include org.objenesis
into shaded jar
dc3bc33 is described below
commit dc3bc33cf3de36953199f6410a13eefe689f624b
Author: Jan Lukavsky
AuthorDate: Tue Feb 5 15:04:05 2019 +0100
[BEAM-6589] include org.objenesis into shaded jar
---
sdks/java/extensions/kryo/build.gradle | 2 ++
1 file changed, 2 insertions(+)
diff --git a/sdks/java/extensions/kryo/build.gradle
b/sdks/java/extensions/kryo/build.gradle
index 305eb57..14878cc 100644
--- a/sdks/java/extensions/kryo/build.gradle
+++ b/sdks/java/extensions/kryo/build.gradle
@@ -28,9 +28,11 @@ applyJavaNature(
dependencies {
include(dependency('com.esotericsoftware:.*'))
include(dependency('org.ow2.asm:asm'))
+include(dependency('org.objenesis:objenesis'))
}
relocate 'com.esotericsoftware',
getJavaRelocatedPath('com.esotericsoftware')
relocate 'org.objectweb', getJavaRelocatedPath('org.objectweb')
+relocate 'org.objenesis', getJavaRelocatedPath('org.objenesis')
}
)
[beam] 01/01: Merge pull request #7399: [BEAM-6350] Reuse PCollectionView when created in translators
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 978034149747f2c6f13d4ceaec203b89e2e616b6 Merge: 213f6c9 1ba3efb Author: David Moravek AuthorDate: Mon Jan 14 14:56:53 2019 +0100 Merge pull request #7399: [BEAM-6350] Reuse PCollectionView when created in translators .../core/translate/AbstractJoinTranslator.java | 9 +- .../translate/BroadcastHashJoinTranslator.java | 61 ++-- .../euphoria/core/translate/JoinTranslator.java| 12 ++- .../translate/BroadcastHashJoinTranslatorTest.java | 106 + 4 files changed, 174 insertions(+), 14 deletions(-)
[beam] branch master updated (213f6c9 -> 9780341)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 213f6c9 [BEAM-6184] Enforce javadoc check on non-trivial public methods (#7454) add 452e0d1 [BEAM-6350] Reuse same PCollectionView when created in translators add 2b90576 [BEAM-6350] AbstractJoinTranslator extended in order to prevent code muliplication. PCollectionViews are now stored in BroadcastHashJoinTranslator. Key extractor is taken into consideration when looking for the same views. add 1ba3efb [BEAM-6350] Test added. new 9780341 Merge pull request #7399: [BEAM-6350] Reuse PCollectionView when created in translators The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../core/translate/AbstractJoinTranslator.java | 9 +- .../translate/BroadcastHashJoinTranslator.java | 61 ++-- .../euphoria/core/translate/JoinTranslator.java| 12 ++- .../translate/BroadcastHashJoinTranslatorTest.java | 106 + 4 files changed, 174 insertions(+), 14 deletions(-) create mode 100644 sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/BroadcastHashJoinTranslatorTest.java
[beam] 01/01: Merge pull request #7194: [BEAM-6174] Kryo dependency removed.
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 54214f8b19f0ce53c6fe551b27d3f647b25bd248 Merge: 3a348e8 34420cc Author: David Moravek AuthorDate: Tue Dec 4 13:24:14 2018 +0100 Merge pull request #7194: [BEAM-6174] Kryo dependency removed. sdks/java/extensions/euphoria/build.gradle | 5 - 1 file changed, 5 deletions(-)
[beam] branch master updated (3a348e8 -> 54214f8)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3a348e8 Merge pull request #7163 from markflyhigh/py-precommit-it add 34420cc [BEAM-6174] Kryo dependency removed. new 54214f8 Merge pull request #7194: [BEAM-6174] Kryo dependency removed. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/extensions/euphoria/build.gradle | 5 - 1 file changed, 5 deletions(-)
[beam] 01/01: Merge pull request #7103: Kryo: Annotate kryo as Experimental.
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 49e57be52b519809ecff3d5660ee75e362917974 Merge: 50e4e0d 89cad9c Author: David Moravek AuthorDate: Wed Nov 21 17:44:08 2018 +0100 Merge pull request #7103: Kryo: Annotate kryo as Experimental. .../src/main/java/org/apache/beam/sdk/extensions/kryo/KryoCoder.java| 2 ++ .../java/org/apache/beam/sdk/extensions/kryo/KryoCoderProvider.java | 2 ++ .../src/main/java/org/apache/beam/sdk/extensions/kryo/KryoOptions.java | 2 ++ .../main/java/org/apache/beam/sdk/extensions/kryo/KryoRegistrar.java| 2 ++ 4 files changed, 8 insertions(+)
[beam] branch master updated (50e4e0d -> 49e57be)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 50e4e0d Merge pull request #7093: [BEAM-6099] RedisIO support for PFADD operation add 89cad9c Kryo: Annotate kryo as Experimental. new 49e57be Merge pull request #7103: Kryo: Annotate kryo as Experimental. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/java/org/apache/beam/sdk/extensions/kryo/KryoCoder.java| 2 ++ .../java/org/apache/beam/sdk/extensions/kryo/KryoCoderProvider.java | 2 ++ .../src/main/java/org/apache/beam/sdk/extensions/kryo/KryoOptions.java | 2 ++ .../main/java/org/apache/beam/sdk/extensions/kryo/KryoRegistrar.java| 2 ++ 4 files changed, 8 insertions(+)
[beam] branch master updated (469c90a -> 8fc0667)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 469c90a Merge pull request #7039: [Website] Euphoria roadmap. add 045e1fe [Website] table stream joins to Euphoria roadmap add 8fc0667 Merge pull request #7100: [Website] table stream joins to Euphoria roadmap No new revisions were added by this update. Summary of changes: website/src/roadmap/euphoria.md | 4 1 file changed, 4 insertions(+)
[beam] 01/01: Merge pull request #7039: [Website] Euphoria roadmap.
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 469c90a2a9aa47e3424297e044cf9205eafb8c4e Merge: 4f24e27 66e5b5d Author: David Moravek AuthorDate: Wed Nov 21 10:46:46 2018 +0100 Merge pull request #7039: [Website] Euphoria roadmap. website/src/roadmap/euphoria.md | 22 +++--- 1 file changed, 19 insertions(+), 3 deletions(-)
[beam] branch master updated (4f24e27 -> 469c90a)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4f24e27 Merge pull request #7097: [BEAM-6103] Adds support for setting a HTTP write timeout for BigQueryIO add 66e5b5d [Website] Euphoria roadmap. new 469c90a Merge pull request #7039: [Website] Euphoria roadmap. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/roadmap/euphoria.md | 22 +++--- 1 file changed, 19 insertions(+), 3 deletions(-)
[beam] branch master updated (30139fa -> 847f3e7)
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 30139fa Merge pull request #7021: [BEAM-6011] Phrase triggering
nexmark
add e5cba30 [BEAM-6062] MetricsContainerStepMap#toString overwritten to
show Beam metrics in Spark Web Interface.
new 847f3e7 Merge pull request #7037: [BEAM-6062] Spark runner do not
show Beam metrics in Spark Web Interface
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../runners/spark/metrics/MetricsAccumulator.java | 2 +-
...aram.java => SparkMetricsContainerStepMap.java} | 24 +++---
2 files changed, 13 insertions(+), 13 deletions(-)
copy
runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/{MetricsAccumulatorParam.java
=> SparkMetricsContainerStepMap.java} (63%)
[beam] 01/01: Merge pull request #7037: [BEAM-6062] Spark runner do not show Beam metrics in Spark Web Interface
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 847f3e74f35bdd2163ed4fa00b31a8f6b3c0f673 Merge: 30139fa e5cba30 Author: David Moravek AuthorDate: Thu Nov 15 13:42:09 2018 +0100 Merge pull request #7037: [BEAM-6062] Spark runner do not show Beam metrics in Spark Web Interface .../runners/spark/metrics/MetricsAccumulator.java | 2 +- .../metrics/SparkMetricsContainerStepMap.java | 42 ++ 2 files changed, 43 insertions(+), 1 deletion(-)
[beam] branch master updated (a302edc -> ecb57dd)
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from a302edc Merge pull request #6738: Remove some unnecessary type
definitions
add ef029ab [BEAM-6054] Euphoria translation providers refactored.
add 4e98153 [BEAM-6054] Review fixes.
add ecb57dd Merge pull request #7019: [BEAM-6054] Euphoria translation
providers refactored.
No new revisions were added by this update.
Summary of changes:
.../core/client/operator/base/Operator.java| 7 +-
.../euphoria/core/translate/EuphoriaOptions.java | 12 +-
.../core/translate/OperatorTranslator.java | 6 +-
.../core/translate/SimpleTranslatorProvider.java | 89
.../euphoria/core/translate/UnionTranslator.java | 2 +-
.../core/translate/provider/CompositeProvider.java | 84 +++
.../provider/GenericTranslatorProvider.java| 246 +
.../core/translate/provider}/package-info.java | 5 +-
.../core/docs/DocumentationExamplesTest.java | 57 +
.../core/testkit/BroadcastHashJoinTest.java| 30 ++-
.../translate/provider/CompositeProviderTest.java | 109 +
.../provider/GenericTranslatorProviderTest.java| 143
.../core/translate/provider/ProviderTestUtils.java | 142
website/src/documentation/sdks/euphoria.md | 66 +-
14 files changed, 875 insertions(+), 123 deletions(-)
delete mode 100644
sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/SimpleTranslatorProvider.java
create mode 100644
sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/CompositeProvider.java
create mode 100644
sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/GenericTranslatorProvider.java
copy
{runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options
=>
sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider}/package-info.java
(81%)
create mode 100644
sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/CompositeProviderTest.java
create mode 100644
sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/GenericTranslatorProviderTest.java
create mode 100644
sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/ProviderTestUtils.java
[beam] branch master updated (8a88e72 -> f911be4)
This is an automated email from the ASF dual-hosted git repository.
dmvk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 8a88e72 Merge pull request #6988: [BEAM-6023] Remove Create.Values
translation from Spark Runner
add d52c4d2 [BEAM-5790] Euphoria: Remove Dataset abstraction.
add 6e13ed0 [BEAM-5790] Euphoria: Fix missing website link.
add 07faaa9 [BEAM-5790] Euphoria: Remove OutputBuilder where possible.
new f911be4 Merge pull request #6750: [BEAM-5790] Euphoria: Remove
Dataset abstraction.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../euphoria/core/client/dataset/Dataset.java | 109 ---
.../euphoria/core/client/dataset/package-info.java | 22 --
.../euphoria/core/client/lib/Euphoria.java | 51 ---
.../extensions/euphoria/core/client/lib/Split.java | 32 +-
.../core/client/operator/AssignEventTime.java | 40 ++-
.../core/client/operator/CompositeOperator.java| 6 +-
.../euphoria/core/client/operator/CountByKey.java | 45 ++-
.../euphoria/core/client/operator/Distinct.java| 42 ++-
.../euphoria/core/client/operator/Filter.java | 30 +-
.../euphoria/core/client/operator/FlatMap.java | 40 +--
.../euphoria/core/client/operator/FullJoin.java| 18 +-
.../euphoria/core/client/operator/Join.java| 64 ++--
.../euphoria/core/client/operator/LeftJoin.java| 18 +-
.../euphoria/core/client/operator/MapElements.java | 43 ++-
.../core/client/operator/OutputValues.java | 59
.../euphoria/core/client/operator/ReduceByKey.java | 54 ++--
.../core/client/operator/ReduceWindow.java | 40 ++-
.../euphoria/core/client/operator/RightJoin.java | 18 +-
.../euphoria/core/client/operator/SumByKey.java| 45 ++-
.../euphoria/core/client/operator/TopPerKey.java | 29 +-
.../euphoria/core/client/operator/Union.java | 52 ++-
.../core/client/operator/base/Builders.java| 10 +-
.../core/client/operator/base/ShuffleOperator.java | 2 +-
.../euphoria/core/client/type/TypeUtils.java | 23 --
.../util/PCollectionLists.java}| 15 +-
...lator.java => CompositeOperatorTranslator.java} | 33 +-
.../euphoria/core/translate/FlatMapTranslator.java | 3 +-
.../euphoria/core/translate/OperatorTransform.java | 32 +-
.../core/translate/ReduceByKeyTranslator.java | 5 +-
.../core/translate/SimpleTranslatorProvider.java | 12 +-
.../euphoria/core/client/lib/SplitTest.java| 26 +-
.../core/client/operator/CountByKeyTest.java | 37 +--
.../core/client/operator/DistinctTest.java | 30 +-
.../euphoria/core/client/operator/FilterTest.java | 16 +-
.../euphoria/core/client/operator/FlatMapTest.java | 30 +-
.../euphoria/core/client/operator/JoinTest.java| 186 +--
.../core/client/operator/MapElementsTest.java | 30 +-
.../core/client/operator/ReduceByKeyTest.java | 76 ++---
.../core/client/operator/ReduceWindowTest.java | 30 +-
.../core/client/operator/SumByKeyTest.java | 30 +-
.../{OperatorTestUtils.java => TestUtils.java} | 72 -
.../core/client/operator/TopPerKeyTest.java| 30 +-
.../euphoria/core/client/operator/UnionTest.java | 53 ++--
.../euphoria/core/client/util/IOUtilsTest.java | 6 +-
.../core/docs/DocumentationExamplesTest.java | 350 +
.../core/testkit/AbstractOperatorTest.java | 19 +-
.../core/testkit/BroadcastHashJoinTest.java| 16 +-
.../euphoria/core/testkit/CountByKeyTest.java | 6 +-
.../euphoria/core/testkit/DistinctTest.java| 8 +-
.../euphoria/core/testkit/FilterTest.java | 4 +-
.../euphoria/core/testkit/FlatMapTest.java | 6 +-
.../extensions/euphoria/core/testkit/JoinTest.java | 76 ++---
.../euphoria/core/testkit/MapElementsTest.java | 6 +-
.../euphoria/core/testkit/ReduceByKeyTest.java | 149 +
.../euphoria/core/testkit/ReduceWindowTest.java| 18 +-
.../euphoria/core/testkit/SumByKeyTest.java| 7 +-
.../euphoria/core/testkit/TopPerKeyTest.java | 12 +-
.../euphoria/core/testkit/UnionTest.java | 37 ++-
.../core/translate/BeamMetricsTranslationTest.java | 11 +-
.../euphoria/core/translate/EuphoriaTest.java | 103 --
website/src/documentation/sdks/euphoria.md | 154 -
61 files changed, 1071 insertions(+), 1555 deletions(-)
delete mode 100644
sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/dataset/Dataset.java
delete mode 100644
sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/dataset/package-inf
[beam] branch dsl-euphoria deleted (was 62f7c1a)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch dsl-euphoria in repository https://gitbox.apache.org/repos/asf/beam.git. was 62f7c1a Merge pull request #5550 from seznam/dsl-euphoria The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated (0f1f8f7 -> 8a88e72)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0f1f8f7 Merge pull request #6990: [BEAM-5931]Revert "Revert "[BEAM-5299] Define max timestamp for global window in proto"" add 9532721 [BEAM-6023] Remove Create.Values translation from Spark Runner add 8a88e72 Merge pull request #6988: [BEAM-6023] Remove Create.Values translation from Spark Runner No new revisions were added by this update. Summary of changes: .../spark/translation/EvaluationContext.java| 21 - .../spark/translation/TransformTranslator.java | 20 .../beam/runners/spark/SparkRunnerDebuggerTest.java | 2 +- 3 files changed, 1 insertion(+), 42 deletions(-)
