[beam] branch master updated: [BEAM-13233] Replace AWS API used to list shards from DescribeStream to ListShards
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 95ec22b [BEAM-13233] Replace AWS API used to list shards from
DescribeStream to ListShards
new 67bcf1e Merge pull request #15973 from
mosche/BEAM-13233-aws2-list-shards
95ec22b is described below
commit 95ec22b64eb5a12a8bf08068bfdbce614b1cf2ab
Author: mosche
AuthorDate: Mon Nov 15 12:24:01 2021 +0100
[BEAM-13233] Replace AWS API used to list shards from DescribeStream to
ListShards
---
CHANGES.md | 1 +
.../aws2/kinesis/DynamicCheckpointGenerator.java | 24 +-
.../io/aws2/kinesis/KinesisReaderCheckpoint.java | 5 +
.../beam/sdk/io/aws2/kinesis/ShardReadersPool.java | 46 +++-
.../sdk/io/aws2/kinesis/ShardRecordsIterator.java | 6 +-
.../io/aws2/kinesis/SimplifiedKinesisClient.java | 170 +---
.../beam/sdk/io/aws2/kinesis/StartingPoint.java| 7 +-
.../io/aws2/kinesis/StartingPointShardsFinder.java | 201 --
.../sdk/io/aws2/kinesis/AmazonKinesisMock.java | 73 +++--
.../kinesis/DynamicCheckpointGeneratorTest.java| 36 +--
.../sdk/io/aws2/kinesis/KinesisMockReadTest.java | 15 +-
.../sdk/io/aws2/kinesis/ShardReadersPoolTest.java | 8 +-
.../aws2/kinesis/SimplifiedKinesisClientTest.java | 306 +++--
.../kinesis/StartingPointShardsFinderTest.java | 292
14 files changed, 529 insertions(+), 661 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 06a9069..00192c1 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -83,6 +83,7 @@
* X behavior was changed
([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
* A new URN convention was adopted for cross-language transforms and existing
URNs were updated. This may break advanced use-cases, for example, if a custom
expansion service is used to connect diffrent Beam Java and Python versions.
([BEAM-12047](https://issues.apache.org/jira/browse/BEAM-12047)).
* The upgrade to Calcite 1.28.0 introduces a breaking change in the SUBSTRING
function in SqlTransform, when used with the Calcite dialect
([BEAM-13099](https://issues.apache.org/jira/browse/BEAM-13099),
[CALCITE-4427](https://issues.apache.org/jira/browse/CALCITE-4427)).
+* ListShards (with DescribeStreamSummary) is used instead of DescribeStream to
list shards in Kinesis streams (AWS SDK v2). Due to this change, as mentioned
in [AWS
documentation](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html),
for fine-grained IAM policies it is required to update them to allow calls to
ListShards and DescribeStreamSummary APIs. For more information, see
[Controlling Access to Amazon Kinesis Data
Streams](https://docs.aws.amazon.com/streams [...]
## Deprecations
diff --git
a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/DynamicCheckpointGenerator.java
b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/DynamicCheckpointGenerator.java
index cd81ea2..3b23349 100644
---
a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/DynamicCheckpointGenerator.java
+++
b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/DynamicCheckpointGenerator.java
@@ -17,9 +17,7 @@
*/
package org.apache.beam.sdk.io.aws2.kinesis;
-import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
-
-import java.util.Set;
+import java.util.List;
import java.util.stream.Collectors;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -34,35 +32,23 @@ class DynamicCheckpointGenerator implements
CheckpointGenerator {
private static final Logger LOG =
LoggerFactory.getLogger(DynamicCheckpointGenerator.class);
private final String streamName;
private final StartingPoint startingPoint;
- private final StartingPointShardsFinder startingPointShardsFinder;
public DynamicCheckpointGenerator(String streamName, StartingPoint
startingPoint) {
this.streamName = streamName;
this.startingPoint = startingPoint;
-this.startingPointShardsFinder = new StartingPointShardsFinder();
- }
-
- public DynamicCheckpointGenerator(
- String streamName,
- StartingPoint startingPoint,
- StartingPointShardsFinder startingPointShardsFinder) {
-this.streamName = checkNotNull(streamName, "streamName");
-this.startingPoint = checkNotNull(startingPoint, "startingPoint");
-this.startingPointShardsFinder =
-checkNotNull(startingPointShardsFinder, "startingPointShardsFinder");
}
@Override
public KinesisReaderCheckpoint generate(SimplifiedKinesisClient kinesis)
throws TransientKinesisException {
-Set shardsAtStartingPoint =
-startingPointShardsFinder.findSha
[beam] branch master updated: [BEAM-13476] Changed Timeout Error Text
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 935db8e [BEAM-13476] Changed Timeout Error Text new bf30ba0 Merge pull request #16282 from [BEAM-13476] [Playground] [Bugfix] Update timeout/error message to be more friendly 935db8e is described below commit 935db8e996b4aae47fcfe02d0aec5dc9178956d7 Author: Alexander AuthorDate: Sat Dec 18 15:07:41 2021 +0400 [BEAM-13476] Changed Timeout Error Text --- .../modules/editor/repository/code_repository/code_repository.dart | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/playground/frontend/lib/modules/editor/repository/code_repository/code_repository.dart b/playground/frontend/lib/modules/editor/repository/code_repository/code_repository.dart index 6f8a235..c4a3ec4 100644 --- a/playground/frontend/lib/modules/editor/repository/code_repository/code_repository.dart +++ b/playground/frontend/lib/modules/editor/repository/code_repository/code_repository.dart @@ -22,7 +22,11 @@ import 'package:playground/modules/editor/repository/code_repository/run_code_re import 'package:playground/modules/editor/repository/code_repository/run_code_result.dart'; const kPipelineCheckDelay = Duration(seconds: 1); -const kTimeoutErrorText = 'Code execution exceeded timeout'; +const kTimeoutErrorText = +'Pipeline exceeded Playground execution timeout and was terminated. ' +'We recommend installing Apache Beam ' +'https://beam.apache.org/get-started/downloads/ ' +'to try examples without timeout limitation.'; const kUnknownErrorText = 'Something went wrong. Please try again later or create a jira ticket';
[beam] branch master updated (bf30ba0 -> 660e698)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bf30ba0 Merge pull request #16282 from [BEAM-13476] [Playground] [Bugfix] Update timeout/error message to be more friendly add 660e698 Merge pull request #16281 from [BEAM-13475] [Playground] [Bugfix] Error message disappears too quick No new revisions were added by this update. Summary of changes: .../code_repository/code_repository.dart | 18 --- .../components/base_notification.dart | 3 +- .../notifications/components/notification.dart | 35 ++ playground/frontend/pubspec.yaml | 1 + 4 files changed, 31 insertions(+), 26 deletions(-)
[beam] branch master updated: [BEAM-13474] Changed 'Playground' logo text color
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 60d67e9 [BEAM-13474] Changed 'Playground' logo text color
new 6ebd420 Merge pull request #16280 from [BEAM-13474] [Playground]
[Bugfix] Darkmode - "Playground" should be white color per mocks
60d67e9 is described below
commit 60d67e9a9ad04e0d24dea4162629f15154d618d3
Author: Alexander
AuthorDate: Mon Dec 20 14:48:16 2021 +0400
[BEAM-13474] Changed 'Playground' logo text color
---
playground/frontend/lib/components/logo/logo_component.dart | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/playground/frontend/lib/components/logo/logo_component.dart
b/playground/frontend/lib/components/logo/logo_component.dart
index c1601e1..b14f096 100644
--- a/playground/frontend/lib/components/logo/logo_component.dart
+++ b/playground/frontend/lib/components/logo/logo_component.dart
@@ -51,7 +51,12 @@ class Logo extends StatelessWidget {
text: 'Beam',
style: TextStyle(color: theme.primaryColor),
),
- const TextSpan(text: ' Playground'),
+ TextSpan(
+text: ' Playground',
+style: TextStyle(
+ color: theme.textTheme.bodyText1?.color,
+),
+ ),
],
),
),
[beam] branch master updated (6ebd420 -> aec2efa)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6ebd420 Merge pull request #16280 from [BEAM-13474] [Playground] [Bugfix] Darkmode - "Playground" should be white color per mocks add aec2efa Merge pull request #16279 from [BEAM-13473] [Playground] [Bugfix] Reset should reset options when resetting example No new revisions were added by this update. Summary of changes: playground/frontend/lib/pages/playground/states/playground_state.dart | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (67bcf1e -> cdc9065)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 67bcf1e Merge pull request #15973 from mosche/BEAM-13233-aws2-list-shards add cd74e9f [BEAM-13443] Avoid blocking put to Kinesis records queue to shutdown readers faster add cdc9065 Merge pull request #16286: [BEAM-13443] Avoid blocking put to Kinesis records queue to shutdown readers faster No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws2/kinesis/ShardReadersPool.java | 16 +--- .../org/apache/beam/sdk/io/kinesis/ShardReadersPool.java | 16 +--- 2 files changed, 26 insertions(+), 6 deletions(-)
[beam] branch master updated (cdc9065 -> 8b5ec22)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cdc9065 Merge pull request #16286: [BEAM-13443] Avoid blocking put to Kinesis records queue to shutdown readers faster add 8b5ec22 Merge pull request #16278 from [BEAM-13479] [Playground] Change logic with Cancel method. No new revisions were added by this update. Summary of changes: playground/backend/cmd/server/controller.go| 6 +- .../backend/internal/code_processing/code_processing.go| 10 +- .../internal/code_processing/code_processing_test.go | 14 +- 3 files changed, 23 insertions(+), 7 deletions(-)
[beam] branch master updated: [BEAM-13485][Playground] Change the location where the `go.mod` and 'go.some' files are copied
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c3825c3 [BEAM-13485][Playground] Change the location where the
`go.mod` and 'go.some' files are copied
new 33def57 Merge pull request #16272 from [BEAM-13485][Playground]
[Bugfix] Move go.mod and go.sum to executable_files/{pipelineId}
c3825c3 is described below
commit c3825c3acc81ab172e02838f4e1454696c0e1cc3
Author: AydarZaynutdinov
AuthorDate: Fri Dec 17 13:11:52 2021 +0300
[BEAM-13485][Playground]
Change the location where the `go.mod` and 'go.some' files are copied
---
.../internal/setup_tools/life_cycle/life_cycle_setuper.go| 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git
a/playground/backend/internal/setup_tools/life_cycle/life_cycle_setuper.go
b/playground/backend/internal/setup_tools/life_cycle/life_cycle_setuper.go
index 35cbcab..a090c9d 100644
--- a/playground/backend/internal/setup_tools/life_cycle/life_cycle_setuper.go
+++ b/playground/backend/internal/setup_tools/life_cycle/life_cycle_setuper.go
@@ -56,7 +56,7 @@ func Setup(sdk pb.Sdk, code string, pipelineId uuid.UUID,
workingDir string, pre
// copy necessary files
switch sdk {
case pb.Sdk_SDK_GO:
- if err = prepareGoFiles(lc, preparedModDir, workingDir,
pipelineId); err != nil {
+ if err = prepareGoFiles(lc, preparedModDir, pipelineId); err !=
nil {
lc.DeleteFolders()
return nil, err
}
@@ -78,13 +78,13 @@ func Setup(sdk pb.Sdk, code string, pipelineId uuid.UUID,
workingDir string, pre
}
// prepareGoFiles prepares file for Go environment.
-// Copy go.mod and go.sum file from /path/to/preparedModDir to
/path/to/workingDir.
-func prepareGoFiles(lc *fs_tool.LifeCycle, preparedModDir, workingDir string,
pipelineId uuid.UUID) error {
- if err := lc.CopyFile(goModFileName, preparedModDir,
filepath.Join(workingDir, baseFileFolder)); err != nil {
+// Copy go.mod and go.sum file from /path/to/preparedModDir to
/path/to/workingDir/executable_files/{pipelineId}
+func prepareGoFiles(lc *fs_tool.LifeCycle, preparedModDir string, pipelineId
uuid.UUID) error {
+ if err := lc.CopyFile(goModFileName, preparedModDir,
lc.Folder.BaseFolder); err != nil {
logger.Errorf("%s: error during copying %s file: %s\n",
pipelineId, goModFileName, err.Error())
return err
}
- if err := lc.CopyFile(goSumFileName, preparedModDir,
filepath.Join(workingDir, baseFileFolder)); err != nil {
+ if err := lc.CopyFile(goSumFileName, preparedModDir,
lc.Folder.BaseFolder); err != nil {
logger.Errorf("%s: error during copying %s file: %s\n",
pipelineId, goSumFileName, err.Error())
return err
}
@@ -92,7 +92,7 @@ func prepareGoFiles(lc *fs_tool.LifeCycle, preparedModDir,
workingDir string, pi
}
// prepareJavaFiles prepares file for Java environment.
-// Copy log config file from /path/to/preparedModDir to
/path/to/workingDir/executable_files/{pipelineId}
+// Copy log config file from /path/to/workingDir to
/path/to/workingDir/executable_files/{pipelineId}
// and update this file according to pipeline.
func prepareJavaFiles(lc *fs_tool.LifeCycle, workingDir string, pipelineId
uuid.UUID) error {
err := lc.CopyFile(javaLogConfigFileName, workingDir,
lc.Folder.BaseFolder)
[beam] branch master updated (33def57 -> 4d8d115)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 33def57 Merge pull request #16272 from [BEAM-13485][Playground]
[Bugfix] Move go.mod and go.sum to executable_files/{pipelineId}
add 4d8d115 Merge pull request #16232 from [BEAM-13285][Playground] Add
steps to playgroundPrecommit gradle tasks
No new revisions were added by this update.
Summary of changes:
build.gradle.kts | 1 +
playground/build.gradle.kts | 10 ++
playground/frontend/build.gradle | 15 +--
3 files changed, 24 insertions(+), 2 deletions(-)
[beam] branch master updated (4d8d115 -> f0d6403)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4d8d115 Merge pull request #16232 from [BEAM-13285][Playground] Add steps to playgroundPrecommit gradle tasks add f0d6403 [BEAM-13399] Add check for dev versions of JARs to download code (#16228) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download.go | 5 + .../pkg/beam/core/runtime/xlangx/expansionx/download_test.go | 11 +++ 2 files changed, 16 insertions(+)
[beam] branch master updated (f0d6403 -> 81ed94f)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f0d6403 [BEAM-13399] Add check for dev versions of JARs to download code (#16228) add 81ed94f Merge pull request #16122 from [BEAM-13345] [Playground] add resizable split view for output and source areas No new revisions were added by this update. Summary of changes: .../frontend/assets/drag_horizontal.svg| 8 +- .../frontend/assets/drag_vertical.svg | 8 +- playground/frontend/assets/output_left.svg | 2 +- playground/frontend/assets/output_right.svg| 2 +- .../lib/components/split_view/split_view.dart | 177 + playground/frontend/lib/constants/assets.dart | 2 + .../components/playground_page_body.dart | 38 +++-- 7 files changed, 212 insertions(+), 25 deletions(-) copy .test-infra/metrics/sync/jenkins/README.md => playground/frontend/assets/drag_horizontal.svg (71%) copy .test-infra/metrics/sync/jenkins/README.md => playground/frontend/assets/drag_vertical.svg (74%) create mode 100644 playground/frontend/lib/components/split_view/split_view.dart
[beam] branch master updated (81ed94f -> 0fd6a04)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 81ed94f Merge pull request #16122 from [BEAM-13345] [Playground] add resizable split view for output and source areas add 0fd6a04 Merge pull request #16170 from [BEAM-13411][Playground] Add getting of logs for the precompiled objects No new revisions were added by this update. Summary of changes: playground/api/v1/api.proto| 48 +- playground/backend/cmd/server/controller.go| 18 +- playground/backend/internal/api/v1/api.pb.go | 828 ++--- playground/backend/internal/api/v1/api_grpc.pb.go | 64 +- .../internal/cloud_bucket/precompiled_objects.go | 11 + playground/frontend/lib/api/v1/api.pb.dart | 518 + playground/frontend/lib/api/v1/api.pbgrpc.dart | 279 --- playground/frontend/lib/api/v1/api.pbjson.dart | 112 ++- .../example_client/grpc_example_client.dart| 19 +- .../notifications/components/notification.dart | 1 - playground/frontend/pubspec.lock | 7 + 11 files changed, 1271 insertions(+), 634 deletions(-)
[beam] branch master updated (0fd6a04 -> afd2c41)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0fd6a04 Merge pull request #16170 from [BEAM-13411][Playground] Add getting of logs for the precompiled objects add afd2c41 Merge pull request #16172 from [BEAM-13417] [Playground] Add java preparator for run katas No new revisions were added by this update. Summary of changes: playground/backend/configs/SDK_JAVA.json | 2 +- .../internal/code_processing/code_processing.go| 18 +++-- .../internal/environment/environment_service.go| 20 - .../environment/environment_service_test.go| 13 ++-- playground/backend/internal/executors/executor.go | 6 +- playground/backend/internal/fs_tool/fs_test.go | 12 +-- playground/backend/internal/fs_tool/java_fs.go | 4 +- .../backend/internal/fs_tool/java_fs_test.go | 2 +- .../backend/internal/preparators/go_preparators.go | 18 +++-- .../internal/preparators/go_preparators_test.go| 2 +- .../internal/preparators/java_preparators.go | 91 -- .../internal/preparators/java_preparators_test.go | 51 +++- .../internal/setup_tools/builder/setup_builder.go | 7 ++ .../setup_tools/builder/setup_builder_test.go | 2 +- .../backend/internal/validators/java_validators.go | 35 +++-- .../internal/validators/java_validators_test.go| 70 ++--- .../backend/internal/validators/validator.go | 5 +- .../backend/internal/validators/validator_test.go | 2 + 18 files changed, 300 insertions(+), 60 deletions(-)
[beam] branch master updated (afd2c41 -> ceb30a3)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from afd2c41 Merge pull request #16172 from [BEAM-13417] [Playground] Add java preparator for run katas add ceb30a3 Merge pull request #16240 from [BEAM-13465][Playground] Change error messages which are sent to the client No new revisions were added by this update. Summary of changes: playground/backend/cmd/server/controller.go| 64 -- .../internal/code_processing/code_processing.go| 14 ++--- .../setup_tools/life_cycle/life_cycle_setuper.go | 11 ++-- 3 files changed, 48 insertions(+), 41 deletions(-)
[beam] branch master updated: [BEAM-13466]: sort categories and examples by name
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2a1174a [BEAM-13466]: sort categories and examples by name
new d01c6bb Merge pull request #16245 from [BEAM-13466] [Playground] sort
categories and examples by name
2a1174a is described below
commit 2a1174a129db233b91c2fcd7088d57497a58b061
Author: Aydar Farrakhov
AuthorDate: Wed Dec 15 17:39:20 2021 +0300
[BEAM-13466]: sort categories and examples by name
---
.../frontend/lib/modules/examples/models/category_model.dart | 7 ++-
playground/frontend/lib/modules/examples/models/example_model.dart | 7 ++-
.../examples/repositories/example_client/grpc_example_client.dart | 5 +++--
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git
a/playground/frontend/lib/modules/examples/models/category_model.dart
b/playground/frontend/lib/modules/examples/models/category_model.dart
index da9d0e4..9620592 100644
--- a/playground/frontend/lib/modules/examples/models/category_model.dart
+++ b/playground/frontend/lib/modules/examples/models/category_model.dart
@@ -18,9 +18,14 @@
import 'package:playground/modules/examples/models/example_model.dart';
-class CategoryModel {
+class CategoryModel with Comparable {
final String name;
final List examples;
const CategoryModel({required this.name, required this.examples});
+
+ @override
+ int compareTo(CategoryModel other) {
+return name.toLowerCase().compareTo(other.name.toLowerCase());
+ }
}
diff --git a/playground/frontend/lib/modules/examples/models/example_model.dart
b/playground/frontend/lib/modules/examples/models/example_model.dart
index a79f827..7286bff 100644
--- a/playground/frontend/lib/modules/examples/models/example_model.dart
+++ b/playground/frontend/lib/modules/examples/models/example_model.dart
@@ -38,7 +38,7 @@ extension ExampleTypeToString on ExampleType {
}
}
-class ExampleModel {
+class ExampleModel with Comparable {
final ExampleType type;
final String name;
final String path;
@@ -64,4 +64,9 @@ class ExampleModel {
setOutputs(String outputs) {
this.outputs = outputs;
}
+
+ @override
+ int compareTo(ExampleModel other) {
+return name.toLowerCase().compareTo(other.name.toLowerCase());
+ }
}
diff --git
a/playground/frontend/lib/modules/examples/repositories/example_client/grpc_example_client.dart
b/playground/frontend/lib/modules/examples/repositories/example_client/grpc_example_client.dart
index 7347f3e..34d51aa 100644
---
a/playground/frontend/lib/modules/examples/repositories/example_client/grpc_example_client.dart
+++
b/playground/frontend/lib/modules/examples/repositories/example_client/grpc_example_client.dart
@@ -153,13 +153,14 @@ class GrpcExampleClient implements ExampleClient {
for (var category in sdkMap.categories) {
List examples = category.precompiledObjects
.map((example) => _toExampleModel(example))
-.toList();
+.toList()
+ ..sort();
categoriesForSdk.add(CategoryModel(
name: category.categoryName,
examples: examples,
));
}
- entries.add(MapEntry(sdk, categoriesForSdk));
+ entries.add(MapEntry(sdk, categoriesForSdk..sort()));
}
sdkCategoriesMap.addEntries(entries);
return sdkCategoriesMap;
[beam] branch master updated (d01c6bb -> 3b492ad)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d01c6bb Merge pull request #16245 from [BEAM-13466] [Playground] sort categories and examples by name add 3b492ad Merge pull request #16234 from [BEAM-13461][Playground] [Bugfix] Error when try to set status after time-out No new revisions were added by this update. Summary of changes: .../internal/code_processing/code_processing.go| 60 +++--- 1 file changed, 30 insertions(+), 30 deletions(-)
[beam] branch master updated (3b492ad -> ee8e759)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3b492ad Merge pull request #16234 from [BEAM-13461][Playground] [Bugfix] Error when try to set status after time-out add ee8e759 Merge pull request #15489 from [BEAM-12865] Allow custom batch_duration in WriteToBigQuery No new revisions were added by this update. Summary of changes: CHANGES.md | 1 + sdks/python/apache_beam/io/gcp/bigquery.py | 41 + sdks/python/apache_beam/io/gcp/bigquery_test.py | 17 ++ 3 files changed, 47 insertions(+), 12 deletions(-)
[beam] branch master updated (ee8e759 -> f67670b)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ee8e759 Merge pull request #15489 from [BEAM-12865] Allow custom batch_duration in WriteToBigQuery add f67670b [BEAM-13483] Increase timeout of Java Examples Dataflow suite. (#16226) No new revisions were added by this update. Summary of changes: .test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow.groovy| 2 +- .test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (f67670b -> 5dac32d)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f67670b [BEAM-13483] Increase timeout of Java Examples Dataflow suite. (#16226) add 5dac32d Pin transitive log4j dependencies to 2.17.0 in :sdks:java:io:hcatalog and :sdks:java:extensions:sql:hcatalog (#16289) No new revisions were added by this update. Summary of changes: sdks/java/extensions/sql/hcatalog/build.gradle | 14 ++ sdks/java/io/hcatalog/build.gradle | 10 -- 2 files changed, 22 insertions(+), 2 deletions(-)
[beam] branch master updated (5dac32d -> 25326b2)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5dac32d Pin transitive log4j dependencies to 2.17.0 in :sdks:java:io:hcatalog and :sdks:java:extensions:sql:hcatalog (#16289) new a474db6 [BEAM-13088] Add parameter tempWriteDataset to BigQueryIO to store temp staging tables new d7cb814 [BEAM-13088] Make tempDataset final new 25326b2 Merge pull request #16076 from [BEAM-13088] Load BigQuery temp tables into different dataset The 33958 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/gcp/bigquery/BatchLoads.java | 11 +++-- .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 19 +++- .../beam/sdk/io/gcp/bigquery/WriteTables.java | 8 +++- .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 53 +- 4 files changed, 85 insertions(+), 6 deletions(-)
[beam] branch master updated: [BEAM-13497] Correct class name in Flink tests.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 930c885 [BEAM-13497] Correct class name in Flink tests. new 73b1ef4 Merge pull request #16294 from ibzib/BEAM-13497 930c885 is described below commit 930c88587673d015d7431bb5e8ff9cbf2902c672 Author: Kyle Weaver AuthorDate: Mon Dec 20 15:34:27 2021 -0800 [BEAM-13497] Correct class name in Flink tests. --- runners/flink/job-server/flink_job_server.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runners/flink/job-server/flink_job_server.gradle b/runners/flink/job-server/flink_job_server.gradle index f00f244..0859887 100644 --- a/runners/flink/job-server/flink_job_server.gradle +++ b/runners/flink/job-server/flink_job_server.gradle @@ -179,7 +179,7 @@ def portableValidatesRunnerTask(String name, Boolean streaming, Boolean checkpoi excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedPCollections' excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream' excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderedDelivery' - excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderInStage' + excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderInBundle' } } },
[beam] branch master updated: [BEAM-13434] Bump solr to 8.11.1 (#16296)
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0d407fe [BEAM-13434] Bump solr to 8.11.1 (#16296)
0d407fe is described below
commit 0d407fe249934b4322a7ee5f05d0762d1bcf56fb
Author: Brian Hulette
AuthorDate: Mon Dec 20 17:01:07 2021 -0800
[BEAM-13434] Bump solr to 8.11.1 (#16296)
---
sdks/java/io/solr/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/java/io/solr/build.gradle b/sdks/java/io/solr/build.gradle
index efdfde2..bceb11a 100644
--- a/sdks/java/io/solr/build.gradle
+++ b/sdks/java/io/solr/build.gradle
@@ -24,7 +24,7 @@ applyJavaNature(
description = "Apache Beam :: SDKs :: Java :: IO :: Solr"
ext.summary = "IO to read and write from/to Solr."
-String solrVersion = "8.5.2"
+String solrVersion = "8.11.1"
dependencies {
compile library.java.vendored_guava_26_0_jre
[beam] branch master updated: [BEAM-13496] Upgrade Flink runner to include log4j patches.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4e4fb27 [BEAM-13496] Upgrade Flink runner to include log4j patches.
new 05cbe24 Merge pull request #16293 from ibzib/flink-upgrade
4e4fb27 is described below
commit 4e4fb278905a931b88c73277620b35f1f0f7f6db
Author: Kyle Weaver
AuthorDate: Mon Dec 20 15:12:06 2021 -0800
[BEAM-13496] Upgrade Flink runner to include log4j patches.
---
runners/flink/1.11/build.gradle | 2 +-
runners/flink/1.12/build.gradle | 2 +-
runners/flink/1.13/build.gradle | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/runners/flink/1.11/build.gradle b/runners/flink/1.11/build.gradle
index c3736ed..81cc0a0 100644
--- a/runners/flink/1.11/build.gradle
+++ b/runners/flink/1.11/build.gradle
@@ -20,7 +20,7 @@ def basePath = '..'
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.11.4'
+ flink_version = '1.11.6'
// Version specific code overrides.
main_source_overrides = ['./src/main/java']
test_source_overrides = ['./src/test/java']
diff --git a/runners/flink/1.12/build.gradle b/runners/flink/1.12/build.gradle
index 31878af..ed62e0f 100644
--- a/runners/flink/1.12/build.gradle
+++ b/runners/flink/1.12/build.gradle
@@ -20,7 +20,7 @@ def basePath = '..'
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.12.5'
+ flink_version = '1.12.7'
// Version specific code overrides.
main_source_overrides = ["${basePath}/1.11/src/main/java", './src/main/java']
test_source_overrides = ["${basePath}/1.11/src/test/java", './src/test/java']
diff --git a/runners/flink/1.13/build.gradle b/runners/flink/1.13/build.gradle
index d6d04eb..d29d2f9 100644
--- a/runners/flink/1.13/build.gradle
+++ b/runners/flink/1.13/build.gradle
@@ -20,7 +20,7 @@ def basePath = '..'
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.13.2'
+ flink_version = '1.13.5'
// Version specific code overrides.
main_source_overrides = ["${basePath}/1.11/src/main/java",
"${basePath}/1.12/src/main/java", './src/main/java']
test_source_overrides = ["${basePath}/1.11/src/test/java",
"${basePath}/1.12/src/test/java", './src/test/java']
[beam] branch release-2.35.0 updated (edcfdc4 -> 7e25626)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch release-2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from edcfdc4 [BEAM-13388] Cherry-pick more changes for google cloud dlp update. (#16268) add 8c72327 Bump solr to 8.11.1 to get new log4j new 7e25626 [release-2.35.0][BEAM-13434] Bump solr to 8.11.1 #16299 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/solr/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: [release-2.35.0][BEAM-13434] Bump solr to 8.11.1 #16299
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch release-2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 7e25626231a49fe0a90a09cc9782ffb8cde669c7 Merge: edcfdc4 8c72327 Author: tvalentyn AuthorDate: Mon Dec 20 19:31:10 2021 -0800 [release-2.35.0][BEAM-13434] Bump solr to 8.11.1 #16299 sdks/java/io/solr/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch release-2.35.0 updated (7e25626 -> a45d570)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch release-2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from 7e25626 [release-2.35.0][BEAM-13434] Bump solr to 8.11.1 #16299 add f81095b [BEAM-13496] Upgrade Flink runner to include log4j patches. new a45d570 [BEAM-13496] [cp] Upgrade Flink runner to include log4j patches. #16300 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/flink/1.11/build.gradle | 2 +- runners/flink/1.12/build.gradle | 2 +- runners/flink/1.13/build.gradle | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] 01/01: [BEAM-13496] [cp] Upgrade Flink runner to include log4j patches. #16300
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch release-2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit a45d5706704d81a14e619955c79ee13a13a2d446 Merge: 7e25626 f81095b Author: tvalentyn AuthorDate: Mon Dec 20 19:31:32 2021 -0800 [BEAM-13496] [cp] Upgrade Flink runner to include log4j patches. #16300 runners/flink/1.11/build.gradle | 2 +- runners/flink/1.12/build.gradle | 2 +- runners/flink/1.13/build.gradle | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch nightly-refs/heads/master updated (18fa2a1 -> 05cbe24)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 18fa2a1 Merge pull request #16270: [adhoc] Fix guava imports in tests
add 95ec22b [BEAM-13233] Replace AWS API used to list shards from
DescribeStream to ListShards
add 67bcf1e Merge pull request #15973 from
mosche/BEAM-13233-aws2-list-shards
add cd74e9f [BEAM-13443] Avoid blocking put to Kinesis records queue to
shutdown readers faster
add cdc9065 Merge pull request #16286: [BEAM-13443] Avoid blocking put to
Kinesis records queue to shutdown readers faster
add 8b5ec22 Merge pull request #16278 from [BEAM-13479] [Playground]
Change logic with Cancel method.
add 935db8e [BEAM-13476] Changed Timeout Error Text
add bf30ba0 Merge pull request #16282 from [BEAM-13476] [Playground]
[Bugfix] Update timeout/error message to be more friendly
add 660e698 Merge pull request #16281 from [BEAM-13475] [Playground]
[Bugfix] Error message disappears too quick
add 60d67e9 [BEAM-13474] Changed 'Playground' logo text color
add 6ebd420 Merge pull request #16280 from [BEAM-13474] [Playground]
[Bugfix] Darkmode - "Playground" should be white color per mocks
add aec2efa Merge pull request #16279 from [BEAM-13473] [Playground]
[Bugfix] Reset should reset options when resetting example
add c3825c3 [BEAM-13485][Playground] Change the location where the
`go.mod` and 'go.some' files are copied
add 33def57 Merge pull request #16272 from [BEAM-13485][Playground]
[Bugfix] Move go.mod and go.sum to executable_files/{pipelineId}
add 4d8d115 Merge pull request #16232 from [BEAM-13285][Playground] Add
steps to playgroundPrecommit gradle tasks
add f0d6403 [BEAM-13399] Add check for dev versions of JARs to download
code (#16228)
add 81ed94f Merge pull request #16122 from [BEAM-13345] [Playground] add
resizable split view for output and source areas
add 0fd6a04 Merge pull request #16170 from [BEAM-13411][Playground] Add
getting of logs for the precompiled objects
add afd2c41 Merge pull request #16172 from [BEAM-13417] [Playground] Add
java preparator for run katas
add ceb30a3 Merge pull request #16240 from [BEAM-13465][Playground]
Change error messages which are sent to the client
add 2a1174a [BEAM-13466]: sort categories and examples by name
add d01c6bb Merge pull request #16245 from [BEAM-13466] [Playground] sort
categories and examples by name
add 3b492ad Merge pull request #16234 from [BEAM-13461][Playground]
[Bugfix] Error when try to set status after time-out
add ee8e759 Merge pull request #15489 from [BEAM-12865] Allow custom
batch_duration in WriteToBigQuery
add f67670b [BEAM-13483] Increase timeout of Java Examples Dataflow
suite. (#16226)
add 5dac32d Pin transitive log4j dependencies to 2.17.0 in
:sdks:java:io:hcatalog and :sdks:java:extensions:sql:hcatalog (#16289)
add a474db6 [BEAM-13088] Add parameter tempWriteDataset to BigQueryIO to
store temp staging tables
add d7cb814 [BEAM-13088] Make tempDataset final
add 25326b2 Merge pull request #16076 from [BEAM-13088] Load BigQuery
temp tables into different dataset
add 930c885 [BEAM-13497] Correct class name in Flink tests.
add 73b1ef4 Merge pull request #16294 from ibzib/BEAM-13497
add 0d407fe [BEAM-13434] Bump solr to 8.11.1 (#16296)
add 4e4fb27 [BEAM-13496] Upgrade Flink runner to include log4j patches.
add 05cbe24 Merge pull request #16293 from ibzib/flink-upgrade
No new revisions were added by this update.
Summary of changes:
.../job_PreCommit_Java_Examples_Dataflow.groovy| 2 +-
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +-
CHANGES.md | 2 +
build.gradle.kts | 1 +
playground/api/v1/api.proto| 48 +-
playground/backend/cmd/server/controller.go| 82 +-
playground/backend/configs/SDK_JAVA.json | 2 +-
playground/backend/internal/api/v1/api.pb.go | 828 ++---
playground/backend/internal/api/v1/api_grpc.pb.go | 64 +-
.../internal/cloud_bucket/precompiled_objects.go | 11 +
.../internal/code_processing/code_processing.go| 100 +--
.../code_processing/code_processing_test.go| 14 +-
.../internal/environment/environment_service.go| 20 +-
.../environment/environment_service_test.go| 13 +-
playground/backend/internal/executors/executor.go | 6 +-
playground/backend/internal/fs_tool/fs_test.go | 12 +-
playground/backend/internal/fs_tool/java_fs.go | 4 +-
.../backend/internal/fs_tool/java_fs_test.go | 2 +-
.../backend/internal/preparators/go_preparators.go | 18 +-
.../internal/preparators/go_preparators_test.go| 2 +-
.../internal/preparat
[beam] branch master updated: Use a patched shadow 6.1.0 plugin using Log4j 2.16.0 (#16269)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c0921f7 Use a patched shadow 6.1.0 plugin using Log4j 2.16.0 (#16269)
c0921f7 is described below
commit c0921f7b90e3dabe66601e04d3347034b84a39cd
Author: Ahmet Altay
AuthorDate: Mon Dec 20 20:19:37 2021 -0800
Use a patched shadow 6.1.0 plugin using Log4j 2.16.0 (#16269)
* Use a patched shadow jar using Log4j 2.16.0
* Use a better version identifier
* update runtime dependency too
---
buildSrc/build.gradle.kts | 5 +++--
buildSrc/jarlibs/shadow-6.1.0-log4jfix-SNAPSHOT.jar | Bin 0 -> 5628292 bytes
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/buildSrc/build.gradle.kts b/buildSrc/build.gradle.kts
index 60d037b..7ec9537 100644
--- a/buildSrc/build.gradle.kts
+++ b/buildSrc/build.gradle.kts
@@ -31,13 +31,14 @@ repositories {
url = uri("https://repo.spring.io/plugins-release/";)
content { includeGroup("io.spring.gradle") }
}
+ flatDir { dirs("jarlibs") }
}
// Dependencies on other plugins used when this plugin is invoked
dependencies {
compile(gradleApi())
compile(localGroovy())
- compile("com.github.jengelman.gradle.plugins:shadow:6.1.0")
+ compile("com.github.jengelman.gradle.plugins:shadow:6.1.0-log4jfix-SNAPSHOT")
compile("gradle.plugin.com.github.spotbugs.snom:spotbugs-gradle-plugin:4.5.0")
runtime("net.ltgt.gradle:gradle-apt-plugin:0.21")
// Enable a Java annotation processor
@@ -48,7 +49,7 @@ dependencies {
runtime("gradle.plugin.com.github.blindpirate:gogradle:0.11.4")
// Enable Go code compilation
runtime("gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.22.0")
// Enable building Docker containers
runtime("gradle.plugin.com.dorongold.plugins:task-tree:1.5")
// Adds a 'taskTree' task to print task dependency tree
- runtime("com.github.jengelman.gradle.plugins:shadow:6.1.0")
// Enable shading Java dependencies
+
runtime("com.github.jengelman.gradle.plugins:shadow:6.1.0-log4jfix-SNAPSHOT")
// Enable shading Java dependencies
runtime("ca.coglinc:javacc-gradle-plugin:2.4.0")
// Enable the JavaCC parser generator
runtime("net.linguica.gradle:maven-settings-plugin:0.5")
runtime("gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.5.0")
// Enable creating an offline repository
diff --git a/buildSrc/jarlibs/shadow-6.1.0-log4jfix-SNAPSHOT.jar
b/buildSrc/jarlibs/shadow-6.1.0-log4jfix-SNAPSHOT.jar
new file mode 100644
index 000..8ebd239
Binary files /dev/null and
b/buildSrc/jarlibs/shadow-6.1.0-log4jfix-SNAPSHOT.jar differ
[beam] branch master updated (c0921f7 -> 3fb40fe)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c0921f7 Use a patched shadow 6.1.0 plugin using Log4j 2.16.0 (#16269) add 3fb40fe [BEAM-12830] Replace GoGradle plugin with Shell Scripts. (#16291) No new revisions were added by this update. Summary of changes: .github/workflows/local_env_tests.yml | 9 ++- .../org/apache/beam/gradle/BeamModulePlugin.groovy | 56 --- sdks/go/build.gradle | 40 --- sdks/go/container/build.gradle | 39 +- sdks/go/examples/build.gradle | 82 +++--- sdks/go/run_with_go_version.sh | 75 sdks/go/test/build.gradle | 42 --- sdks/go/test/load/build.gradle | 54 +- sdks/go/test/run_validatesrunner_tests.sh | 2 +- sdks/java/container/build.gradle | 34 + sdks/python/container/build.gradle | 34 + 11 files changed, 131 insertions(+), 336 deletions(-) create mode 100755 sdks/go/run_with_go_version.sh
