[beam] tag nightly-master updated (3de6bee -> 95d3ac6)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/beam.git.
*** WARNING: tag nightly-master was modified! ***
from 3de6bee (commit)
to 95d3ac6 (commit)
from 3de6bee Merge pull request #15544 from udim/master
add dd37ac8 fixing test link
add 853fa36 fix linter issues
add cac6b50 Merge pull request #15504: Fixing test link in
TestPipeline.java
add 628b2df Add DataFrame changes to CHANGES.md
add 2bcfb68 Merge pull request #15631 from
TheNeuralBit/dataframe-changes-update
add 9bc8c96 [BEAM-10913] - Forcing update of YAML file by running kubectl
apply (#15663)
add a0b933c [BEAM-10955] Flink Java Runner test flake: Could not find
Flink job (#15664)
add 512fb80 [BEAM-10114] Bump Pub/Sub Lite version (#15640)
add 421b538 Don't use run_pytest.sh for pyarrow tests
add 3bfeebc Fail in run_pytest.sh if -m is specified
add c8af600 [BEAM-12985] Call pytest directly rather than run_pytest.sh
to avoid running pyarrow tests twice (#15693)
add a1ad167 [BEAM-12964]: add code editor component
add 5e277f0 [BEAM-12966]: load initial example on page load
add 7572337 [BEAM-12964]: fix editor scrolling
add 90fe52a [BEAM-12964]: add dark theme support
add b156970 [BEAM-12964]: add licence to example repository file
add e61225e [BEAM-12964]: extract playground page providers
add 9db38a0 [BEAM-12964]: refactor styles
add 95d3ac6 Merge pull request #15649 from [BEAM-12964] [Playground] Add
code editor text field
No new revisions were added by this update.
Summary of changes:
.test-infra/metrics/build.gradle | 5 ++
CHANGES.md | 8 ++-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 3 +-
.../toggle_theme_button/toggle_theme_button.dart | 4 +-
playground/frontend/lib/config/theme.dart | 47 +++---
playground/frontend/lib/constants/sizes.dart | 8 +--
.../modules/editor/components/editor_textarea.dart | 72 +-
.../examples/models/example_model.dart}| 11 ++--
.../examples/repositories/example_repository.dart} | 36 +--
.../lib/modules/sdk/components/sdk_selector.dart | 4 +-
.../components/editor_textarea_wrapper.dart| 58 +
.../components/playground_page_providers.dart | 57 +
.../lib/pages/playground/playground_page.dart | 16 ++---
.../examples_state.dart} | 16 +++--
.../playground/{ => states}/playground_state.dart | 30 -
playground/frontend/pubspec.lock | 30 -
playground/frontend/pubspec.yaml | 1 +
.../pages/playground/playground_state_test.dart| 2 +-
.../beam/runners/flink/FlinkSavepointTest.java | 8 ++-
.../org/apache/beam/sdk/testing/TestPipeline.java | 5 +-
sdks/java/io/google-cloud-platform/build.gradle| 8 +++
sdks/python/scripts/run_pytest.sh | 5 ++
sdks/python/tox.ini| 5 +-
23 files changed, 364 insertions(+), 75 deletions(-)
copy playground/frontend/lib/{pages/playground/playground_state.dart =>
modules/examples/models/example_model.dart} (83%)
copy playground/frontend/lib/{pages/playground/playground_state.dart =>
modules/examples/repositories/example_repository.dart} (56%)
create mode 100644
playground/frontend/lib/pages/playground/components/editor_textarea_wrapper.dart
create mode 100644
playground/frontend/lib/pages/playground/components/playground_page_providers.dart
copy playground/frontend/lib/pages/playground/{playground_state.dart =>
states/examples_state.dart} (67%)
rename playground/frontend/lib/pages/playground/{ =>
states}/playground_state.dart (60%)
[beam] branch master updated (c8af600 -> 95d3ac6)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from c8af600 [BEAM-12985] Call pytest directly rather than run_pytest.sh
to avoid running pyarrow tests twice (#15693)
new a1ad167 [BEAM-12964]: add code editor component
new 5e277f0 [BEAM-12966]: load initial example on page load
new 7572337 [BEAM-12964]: fix editor scrolling
new 90fe52a [BEAM-12964]: add dark theme support
new b156970 [BEAM-12964]: add licence to example repository file
new e61225e [BEAM-12964]: extract playground page providers
new 9db38a0 [BEAM-12964]: refactor styles
new 95d3ac6 Merge pull request #15649 from [BEAM-12964] [Playground] Add
code editor text field
The 33193 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../toggle_theme_button/toggle_theme_button.dart | 4 +-
playground/frontend/lib/config/theme.dart | 47 +++---
playground/frontend/lib/constants/sizes.dart | 8 +--
.../modules/editor/components/editor_textarea.dart | 72 +-
.../examples/models/example_model.dart}| 11 ++--
.../examples/repositories/example_repository.dart} | 36 +--
.../lib/modules/sdk/components/sdk_selector.dart | 4 +-
.../components/editor_textarea_wrapper.dart| 58 +
.../components/playground_page_providers.dart | 57 +
.../lib/pages/playground/playground_page.dart | 16 ++---
.../examples_state.dart} | 16 +++--
.../playground/{ => states}/playground_state.dart | 30 -
playground/frontend/pubspec.lock | 30 -
playground/frontend/pubspec.yaml | 1 +
.../pages/playground/playground_state_test.dart| 2 +-
15 files changed, 327 insertions(+), 65 deletions(-)
copy playground/frontend/lib/{pages/playground/playground_state.dart =>
modules/examples/models/example_model.dart} (83%)
copy playground/frontend/lib/{pages/playground/playground_state.dart =>
modules/examples/repositories/example_repository.dart} (56%)
create mode 100644
playground/frontend/lib/pages/playground/components/editor_textarea_wrapper.dart
create mode 100644
playground/frontend/lib/pages/playground/components/playground_page_providers.dart
copy playground/frontend/lib/pages/playground/{playground_state.dart =>
states/examples_state.dart} (67%)
rename playground/frontend/lib/pages/playground/{ =>
states}/playground_state.dart (60%)
[beam] branch master updated (512fb80 -> c8af600)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 512fb80 [BEAM-10114] Bump Pub/Sub Lite version (#15640) add 421b538 Don't use run_pytest.sh for pyarrow tests add 3bfeebc Fail in run_pytest.sh if -m is specified add c8af600 [BEAM-12985] Call pytest directly rather than run_pytest.sh to avoid running pyarrow tests twice (#15693) No new revisions were added by this update. Summary of changes: sdks/python/scripts/run_pytest.sh | 5 + sdks/python/tox.ini | 5 +++-- 2 files changed, 8 insertions(+), 2 deletions(-)
[beam] 01/01: Revert "[BEAM-12908] Change to use PubsubSignal for information propagation so this works on dataflow"
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch revert-15607-pubsub-signal-psl-it
in repository https://gitbox.apache.org/repos/asf/beam.git
commit cb09eb8f314e44023897ddbf49b0c9df8cb6f7e5
Author: Kyle Weaver
AuthorDate: Fri Oct 8 15:06:18 2021 -0700
Revert "[BEAM-12908] Change to use PubsubSignal for information propagation
so this works on dataflow"
---
.../beam/sdk/io/gcp/pubsublite/ReadWriteIT.java| 88 +++---
1 file changed, 59 insertions(+), 29 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java
index bd15310..80c362a 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.io.gcp.pubsublite;
import static org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull;
+import static org.junit.Assert.fail;
import com.google.cloud.pubsublite.AdminClient;
import com.google.cloud.pubsublite.AdminClientSettings;
@@ -35,18 +36,19 @@ import com.google.cloud.pubsublite.proto.Subscription;
import
com.google.cloud.pubsublite.proto.Subscription.DeliveryConfig.DeliveryRequirement;
import com.google.cloud.pubsublite.proto.Topic;
import com.google.cloud.pubsublite.proto.Topic.PartitionConfig.Capacity;
+import com.google.errorprone.annotations.concurrent.GuardedBy;
import com.google.protobuf.ByteString;
import java.util.ArrayDeque;
+import java.util.ArrayList;
import java.util.Deque;
-import java.util.Set;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
-import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.StreamingOptions;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -55,11 +57,12 @@ import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.FlatMapElements;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PCollection;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.After;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -68,12 +71,12 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@RunWith(JUnit4.class)
+@Ignore("https://issues.apache.org/jira/browse/BEAM-12908";)
public class ReadWriteIT {
private static final Logger LOG = LoggerFactory.getLogger(ReadWriteIT.class);
private static final CloudZone ZONE = CloudZone.parse("us-central1-b");
private static final int MESSAGE_COUNT = 90;
- @Rule public transient TestPubsubSignal signal = TestPubsubSignal.create();
@Rule public transient TestPipeline pipeline = TestPipeline.create();
private static ProjectId getProject(PipelineOptions options) {
@@ -205,21 +208,28 @@ public class ReadWriteIT {
"dedupeMessages",
PubsubLiteIO.deduplicate(UuidDeduplicationOptions.newBuilder().build()));
}
- public static SimpleFunction extractIds() {
-return new SimpleFunction() {
- @Override
- public Integer apply(SequencedMessage input) {
-return Integer.parseInt(input.getMessage().getData().toStringUtf8());
- }
-};
+ // This static out of band communication is needed to retain serializability.
+ @GuardedBy("ReadWriteIT.class")
+ private static final List received = new ArrayList<>();
+
+ private static synchronized void addMessageReceived(SequencedMessage
message) {
+received.add(message);
}
- public static SerializableFunction, Boolean> testIds() {
-return ids -> {
- LOG.info("Ids are: {}", ids);
- Set target = IntStream.range(0,
MESSAGE_COUNT).boxed().collect(Collectors.toSet());
- return target.equals(ids);
-};
+ private static synchronized List
getTestQuickstartReceived() {
+return ImmutableList.copyOf(received);
+ }
+
+ private static PTransform,
PCollection>
+ collectTestQuickstart() {
+return MapElements.via(
+new SimpleFunction() {
+ @Override
+ public Void apply(SequencedMessage input) {
+a
[beam] branch revert-15607-pubsub-signal-psl-it created (now cb09eb8)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch revert-15607-pubsub-signal-psl-it in repository https://gitbox.apache.org/repos/asf/beam.git. at cb09eb8 Revert "[BEAM-12908] Change to use PubsubSignal for information propagation so this works on dataflow" This branch includes the following new commits: new cb09eb8 Revert "[BEAM-12908] Change to use PubsubSignal for information propagation so this works on dataflow" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (a0b933c -> 512fb80)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a0b933c [BEAM-10955] Flink Java Runner test flake: Could not find Flink job (#15664) add 512fb80 [BEAM-10114] Bump Pub/Sub Lite version (#15640) No new revisions were added by this update. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 3 ++- sdks/java/io/google-cloud-platform/build.gradle | 8 2 files changed, 10 insertions(+), 1 deletion(-)
[beam] branch master updated (9bc8c96 -> a0b933c)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9bc8c96 [BEAM-10913] - Forcing update of YAML file by running kubectl apply (#15663) add a0b933c [BEAM-10955] Flink Java Runner test flake: Could not find Flink job (#15664) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/flink/FlinkSavepointTest.java| 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (2bcfb68 -> 9bc8c96)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2bcfb68 Merge pull request #15631 from TheNeuralBit/dataframe-changes-update add 9bc8c96 [BEAM-10913] - Forcing update of YAML file by running kubectl apply (#15663) No new revisions were added by this update. Summary of changes: .test-infra/metrics/build.gradle | 5 + 1 file changed, 5 insertions(+)
[beam] branch master updated: Add DataFrame changes to CHANGES.md
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 628b2df Add DataFrame changes to CHANGES.md new 2bcfb68 Merge pull request #15631 from TheNeuralBit/dataframe-changes-update 628b2df is described below commit 628b2df1c84e397c88e519aec244f583148559ef Author: Brian Hulette AuthorDate: Thu Sep 30 16:16:57 2021 -0700 Add DataFrame changes to CHANGES.md --- CHANGES.md | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 1c132df..fa78b52 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -67,6 +67,10 @@ * X feature added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). * Upgrade to Calcite 1.26.0 ([BEAM-9379](https://issues.apache.org/jira/browse/BEAM-9379)). +* Added a new `dataframe` extra to the Python SDK that tracks `pandas` versions + we've verified compatibility with. We now recommend installing Beam with `pip + install apache-beam[dataframe]` when you intend to use the DataFrame API + ([BEAM-12906](https://issues.apache.org/jira/browse/BEAM-12906)). ## Breaking Changes @@ -80,8 +84,8 @@ ## Bugfixes * Fixed X (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). -* Fixed error while writing multiple DeferredFrames to csv (Python) -([BEAM-12701](https://issues.apache.org/jira/browse/BEAM-12701)). +* Fixed error while writing multiple DeferredFrames to csv (Python) ([BEAM-12701](https://issues.apache.org/jira/browse/BEAM-12701)). +* Fixed error when importing the DataFrame API with pandas 1.0.x installed ([BEAM-12945](https://issues.apache.org/jira/browse/BEAM-12945)). * Fixed top.SmallestPerKey implementation in the Go SDK ([BEAM-12946](https://issues.apache.org/jira/browse/BEAM-12946)). ## Known Issues
[beam] branch master updated (3de6bee -> cac6b50)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3de6bee Merge pull request #15544 from udim/master add dd37ac8 fixing test link add 853fa36 fix linter issues add cac6b50 Merge pull request #15504: Fixing test link in TestPipeline.java No new revisions were added by this update. Summary of changes: .../core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
