[beam] branch master updated: Add Beam Schema Options to changelog
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d42b49c Add Beam Schema Options to changelog new 86b7ca8 Merge pull request #11321 from Add Beam Schema Options to changelog d42b49c is described below commit d42b49ceef411ff031089289380cb4b020d17429 Author: Alex Van Boxel AuthorDate: Mon Apr 6 11:12:27 2020 +0200 Add Beam Schema Options to changelog --- CHANGES.md | 11 +++ 1 file changed, 11 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index 76e3da7..e14ed80 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -68,12 +68,23 @@ and an upcoming [blog post](https://beam.apache.org/blog/python/typing/2020/03/06/python-typing.html). +* Java SDK: Introducing the concept of options in Beam Schema’s. These options add extra +context to fields and schemas. This replaces the current Beam metadata that is present +in a FieldType only, options are available in fields and row schemas. Schema options are +fully typed and can contain complex rows. *Remark: Schema aware is still experimental.* +([BEAM-9035](https://issues.apache.org/jira/browse/BEAM-9035)) +* Java SDK: The protobuf extension is fully schema aware and also includes protobuf option +conversion to beam schema options. *Remark: Schema aware is still experimental.* +([BEAM-9044](https://issues.apache.org/jira/browse/BEAM-9044)) + ## Breaking Changes * HBaseIO.ReadAll now requires a PCollection of HBaseIO.Read objects instead of HBaseQuery objects ([BEAM-9279](https://issues.apache.org/jira/browse/BEAM-9279)). * ProcessContext.updateWatermark has been removed in favor of using a WatermarkEstimator ([BEAM-9430](https://issues.apache.org/jira/browse/BEAM-9430)). ## Deprecations +* Java SDK: Beam Schema FieldType.getMetadata is now deprecated and is replaced by the Beam +Schema Options, it will be removed in version `2.23.0`. ([BEAM-9704](https://issues.apache.org/jira/browse/BEAM-9704)) ## Known Issues
[beam] branch master updated: [BEAM-9704] Deprecate FieldType metadata
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b88705b [BEAM-9704] Deprecate FieldType metadata
new b55eb93 Merge pull request #11320 from [BEAM-9704] Deprecate
FieldType metadata
b88705b is described below
commit b88705b30c2d2420554f618bbc9df3561cfac1a9
Author: Alex Van Boxel
AuthorDate: Mon Apr 6 11:38:08 2020 +0200
[BEAM-9704] Deprecate FieldType metadata
---
.../java/org/apache/beam/sdk/schemas/Schema.java | 36 +++---
1 file changed, 32 insertions(+), 4 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index c7f5b46..d961627 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -591,8 +591,13 @@ public class Schema implements Serializable {
@Nullable
public abstract Schema getRowSchema();
-/** Returns optional extra metadata. */
+/**
+ * Returns optional extra metadata.
+ *
+ * @deprecated use schema options instead.
+ */
@SuppressWarnings("mutable")
+@Deprecated
abstract Map getMetadata();
public abstract FieldType.Builder toBuilder();
@@ -631,6 +636,8 @@ public class Schema implements Serializable {
abstract Builder setRowSchema(@Nullable Schema rowSchema);
+ /** @deprecated use schema options instead. */
+ @Deprecated
abstract Builder setMetadata(Map metadata);
abstract FieldType build();
@@ -718,7 +725,12 @@ public class Schema implements Serializable {
return
FieldType.forTypeName(TypeName.LOGICAL_TYPE).setLogicalType(logicalType).build();
}
-/** Set the metadata map for the type, overriding any existing metadata..
*/
+/**
+ * Set the metadata map for the type, overriding any existing metadata.
+ *
+ * @deprecated use schema options instead.
+ */
+@Deprecated
public FieldType withMetadata(Map metadata) {
Map wrapped =
metadata.entrySet().stream()
@@ -727,7 +739,12 @@ public class Schema implements Serializable {
return toBuilder().setMetadata(wrapped).build();
}
-/** Returns a copy of the descriptor with metadata set for the given key.
*/
+/**
+ * Returns a copy of the descriptor with metadata set for the given key.
+ *
+ * @deprecated use schema options instead.
+ */
+@Deprecated
public FieldType withMetadata(String key, byte[] metadata) {
Map newMetadata =
ImmutableMap.builder()
@@ -737,22 +754,33 @@ public class Schema implements Serializable {
return toBuilder().setMetadata(newMetadata).build();
}
-/** Returns a copy of the descriptor with metadata set for the given key.
*/
+/**
+ * Returns a copy of the descriptor with metadata set for the given key.
+ *
+ * @deprecated use schema options instead.
+ */
+@Deprecated
public FieldType withMetadata(String key, String metadata) {
return withMetadata(key, metadata.getBytes(StandardCharsets.UTF_8));
}
+/** @deprecated use schema options instead. */
+@Deprecated
public Map getAllMetadata() {
return getMetadata().entrySet().stream()
.collect(Collectors.toMap(e -> e.getKey(), e -> e.getValue().array));
}
+/** @deprecated use schema options instead. */
+@Deprecated
@Nullable
public byte[] getMetadata(String key) {
ByteArrayWrapper metadata = getMetadata().get(key);
return (metadata != null) ? metadata.array : null;
}
+/** @deprecated use schema options instead. */
+@Deprecated
public String getMetadataString(String key) {
ByteArrayWrapper metadata = getMetadata().get(key);
if (metadata != null) {
[beam] branch master updated: [BEAM-9044] Protobuf options to Schema options
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e741380 [BEAM-9044] Protobuf options to Schema options
new 06a120c Merge pull request #10529 from [BEAM-9044] Protobuf options
to Schema options
e741380 is described below
commit e741380c7924d50a48e813ae992d2c3c722a8342
Author: Alex Van Boxel
AuthorDate: Thu Jan 2 08:39:37 2020 +0100
[BEAM-9044] Protobuf options to Schema options
Convert protobuf options to Row schema options. It gets the options
from a proto message descriptor and converts them into beam schema
options on the Schema. It does the same from proto field descriptors,
converting them into beam field options.
The converter creates typed options as the proto options are also
fully typed. For the convertion it uses the ProtoDynamicSchema
to resolve the types.
---
.../beam/sdk/schemas/logicaltypes/OneOfType.java | 4 +-
.../extensions/protobuf/ProtoByteBuddyUtils.java | 4 +-
.../beam/sdk/extensions/protobuf/ProtoDomain.java | 82 +-
.../protobuf/ProtoDynamicMessageSchema.java| 80 +++---
.../extensions/protobuf/ProtoSchemaTranslator.java | 143 +--
.../sdk/extensions/protobuf/ProtoDomainTest.java | 12 +-
.../protobuf/ProtoDynamicMessageSchemaTest.java| 8 +-
.../protobuf/ProtoMessageSchemaTest.java | 6 +-
.../protobuf/ProtoSchemaTranslatorTest.java| 126 ++
.../sdk/extensions/protobuf/TestProtoSchemas.java | 275 ++---
.../src/test/proto/proto3_schema_messages.proto| 46
.../src/test/proto/proto3_schema_options.proto | 93 +++
12 files changed, 604 insertions(+), 275 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/OneOfType.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/OneOfType.java
index 79c1819..37c643a 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/OneOfType.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/OneOfType.java
@@ -55,9 +55,7 @@ public class OneOfType implements
LogicalType {
private OneOfType(List fields, @Nullable Map
enumMap) {
List nullableFields =
-fields.stream()
-.map(f -> Field.nullable(f.getName(), f.getType()))
-.collect(Collectors.toList());
+fields.stream().map(f ->
f.withNullable(true)).collect(Collectors.toList());
if (enumMap != null) {
nullableFields.stream().forEach(f ->
checkArgument(enumMap.containsKey(f.getName(;
enumerationType = EnumerationType.create(enumMap);
diff --git
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
index b681452..9ac7325 100644
---
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
+++
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
@@ -922,7 +922,7 @@ class ProtoByteBuddyUtils {
fieldValueTypeSupplier.get(clazz,
oneOfType.getOneOfSchema()).stream()
.collect(Collectors.toMap(FieldValueTypeInformation::getName, f
-> f));
for (Field oneOfField : oneOfType.getOneOfSchema().getFields()) {
-int protoFieldIndex = getFieldNumber(oneOfField.getType());
+int protoFieldIndex = getFieldNumber(oneOfField);
FieldValueGetter oneOfFieldGetter =
createGetter(
oneOfFieldTypes.get(oneOfField.getName()),
@@ -993,7 +993,7 @@ class ProtoByteBuddyUtils {
TreeMap> oneOfSetters =
Maps.newTreeMap();
for (Field oneOfField : oneOfType.getOneOfSchema().getFields()) {
FieldValueSetter setter = getProtoFieldValueSetter(oneOfField,
methods, builderClass);
-oneOfSetters.put(getFieldNumber(oneOfField.getType()), setter);
+oneOfSetters.put(getFieldNumber(oneOfField), setter);
}
return createOneOfSetter(field.getName(), oneOfSetters, builderClass);
} else {
diff --git
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
index c13cf88..3da07cc 100644
---
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
+++
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
@@ -17,8 +17,20 @@
*/
package org.apache.beam.sdk.extensions.protobuf;
+import com.google.protob
[beam] branch master updated: [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6ec49ce [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option
builder
new 894bcb0 Merge pull request #11249 from [BEAM-9605] BIP-1: Rename
setRowOption to setOption on Option builder
6ec49ce is described below
commit 6ec49ce3633efee4274643ee176f1786571b5a25
Author: Alex Van Boxel
AuthorDate: Fri Mar 27 11:48:42 2020 +0100
[BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
Added missing rename.
---
.../apache/beam/runners/core/construction/SchemaTranslationTest.java | 2 +-
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java | 4 ++--
.../src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
index bfb9adb..c80bbd6 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
@@ -74,7 +74,7 @@ public class SchemaTranslationTest {
.setOption(
"field_option_map", FieldType.map(FieldType.STRING,
FieldType.INT32), optionMap)
.setOption("field_option_array",
FieldType.array(FieldType.STRING), optionList)
- .setRowOption("field_option_row", optionRow)
+ .setOption("field_option_row", optionRow)
.setOption("field_option_value", FieldType.STRING, "other");
return ImmutableList.builder()
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index d0a8645..c7f5b46 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -1119,7 +1119,7 @@ public class Schema implements Serializable {
this(new HashMap<>());
}
- public Builder setRowOption(String optionName, Row value) {
+ public Builder setOption(String optionName, Row value) {
setOption(optionName, FieldType.row(value.getSchema()), value);
return this;
}
@@ -1209,7 +1209,7 @@ public class Schema implements Serializable {
}
public static Options.Builder setOption(String optionName, Row value) {
- return Options.builder().setRowOption(optionName, value);
+ return Options.builder().setOption(optionName, value);
}
}
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
index b8bb0b0..81897a8 100644
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
@@ -239,7 +239,7 @@ public class SchemaOptionsTest {
.setOption("field_option_double", FieldType.DOUBLE, 12.0)
.setOption("field_option_map", FieldType.map(FieldType.INT32,
FieldType.STRING), TEST_MAP)
.setOption("field_option_array", FieldType.array(FieldType.STRING),
TEST_LIST)
-.setRowOption("field_option_row", TEST_ROW)
+.setOption("field_option_row", TEST_ROW)
.setOption("field_option_value", FieldType.STRING, "other");
}
[beam] branch master updated: [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3aa488d [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option
builder
new 9597c5b Merge pull request #11218 from [BEAM-9605] BIP-1: Rename
setRowOption to setOption on Option builder
3aa488d is described below
commit 3aa488d4a752b5ec3cd210e7ae9b7db4d80124d4
Author: Alex Van Boxel
AuthorDate: Wed Mar 25 14:16:33 2020 +0100
[BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
Rename setRowOption to setOption on Option builder as setRowOption
name is too confusing.
It sets an option as a Row, not an option on a Row. Using setOption
is better and doesn't conflict with the other setOption with 3
parameters and explicit type.
This commit also removed left over @Nullable annotations.
---
.../core/src/main/java/org/apache/beam/sdk/schemas/Schema.java | 10 +++---
.../java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java| 2 +-
2 files changed, 4 insertions(+), 8 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index e64f123..d0a8645 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -336,7 +336,7 @@ public class Schema implements Serializable {
SAME,
WEAKEN,
IGNORE
- };
+ }
/** Returns true if two Schemas have the same fields, but possibly in
different orders. */
public boolean equivalent(Schema other) {
@@ -389,7 +389,7 @@ public class Schema implements Serializable {
builder.append("Options:");
builder.append(options);
return builder.toString();
- };
+ }
@Override
public int hashCode() {
@@ -1171,7 +1171,6 @@ public class Schema implements Serializable {
/** Get the value of an option. If the option is not found null is
returned. */
@SuppressWarnings("TypeParameterUnusedInFormals")
-@Nullable
public T getValue(String optionName) {
Option option = options.get(optionName);
if (option != null) {
@@ -1182,13 +1181,11 @@ public class Schema implements Serializable {
}
/** Get the value of an option. If the option is not found null is
returned. */
-@Nullable
public T getValue(String optionName, Class valueClass) {
return getValue(optionName);
}
/** Get the value of an option. If the option is not found the default
value is returned. */
-@Nullable
public T getValueOrDefault(String optionName, T defaultValue) {
Option option = options.get(optionName);
if (option != null) {
@@ -1198,7 +1195,6 @@ public class Schema implements Serializable {
}
/** Get the type of an option. */
-@Nullable
public FieldType getType(String optionName) {
Option option = options.get(optionName);
if (option != null) {
@@ -1212,7 +1208,7 @@ public class Schema implements Serializable {
return Options.builder().setOption(optionName, fieldType, value);
}
-public static Options.Builder setRowOption(String optionName, Row value) {
+public static Options.Builder setOption(String optionName, Row value) {
return Options.builder().setRowOption(optionName, value);
}
}
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
index f3e3685..b8bb0b0 100644
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaOptionsTest.java
@@ -169,7 +169,7 @@ public class SchemaOptionsTest {
@Test
public void testRowOption() {
-Schema.Options options = Schema.Options.setRowOption(OPTION_NAME,
TEST_ROW).build();
+Schema.Options options = Schema.Options.setOption(OPTION_NAME,
TEST_ROW).build();
assertEquals(TEST_ROW, options.getValue(OPTION_NAME));
assertEquals(FieldType.row(TEST_ROW.getSchema()),
options.getType(OPTION_NAME));
}
[beam] branch master updated (88c7aa2 -> 42115ee)
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 88c7aa2 [BEAM-9371] Add SideInputLoadTest to Java SDK (#10949) add de01ff6 Add notes to change log. add 42115ee Merge pull request apache/220-changelog No new revisions were added by this update. Summary of changes: CHANGES.md | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated: [BEAM-9035] BIP-1: Typed options for Row Schema and Field
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d851e6c [BEAM-9035] BIP-1: Typed options for Row Schema and Field
new 03c8544 Merge pull request #10413 from [BEAM-9035] Typed options for
Row Schema and Field
d851e6c is described below
commit d851e6cae36d9acee845ca24a4cf50ff5ce704b4
Author: Alex Van Boxel
AuthorDate: Sun Dec 1 12:21:01 2019 +0100
[BEAM-9035] BIP-1: Typed options for Row Schema and Field
This is the first PR of a multipart commit: this ticket implements the
basic infrastructure of options on row and field. Options in Beam
Schema add extra context to fields and schema. In contracts to
metadata, options would be added to fields, logical types and rows.
Options are key/typed value combination. The type system is using
the beam schema itself and the value can be any type that is supported
by the beam schema, including row.
---
model/pipeline/src/main/proto/schema.proto | 8 +
.../core/construction/CoderTranslators.java| 2 +-
.../runners/core/construction/CommonCoderTest.java | 3 +-
.../core/construction/SchemaTranslationTest.java | 81 -
.../util/SchemaCoderCloudObjectTranslator.java | 2 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 306 --
.../apache/beam/sdk/schemas/SchemaTranslation.java | 150 +++--
.../main/java/org/apache/beam/sdk/values/Row.java | 228 +-
.../apache/beam/sdk/values/SchemaVerification.java | 255 +++
.../apache/beam/sdk/schemas/SchemaOptionsTest.java | 347 +
10 files changed, 1107 insertions(+), 275 deletions(-)
diff --git a/model/pipeline/src/main/proto/schema.proto
b/model/pipeline/src/main/proto/schema.proto
index 2cf404e..dcf75ca 100644
--- a/model/pipeline/src/main/proto/schema.proto
+++ b/model/pipeline/src/main/proto/schema.proto
@@ -31,6 +31,7 @@ option java_outer_classname = "SchemaApi";
message Schema {
repeated Field fields = 1;
string id = 2;
+ repeated Option options = 3;
}
message Field {
@@ -39,6 +40,7 @@ message Field {
FieldType type = 3;
int32 id = 4;
int32 encoding_position = 5;
+ repeated Option options = 6;
}
message FieldType {
@@ -91,6 +93,12 @@ message LogicalType {
FieldValue argument = 5;
}
+message Option {
+ string name = 1;
+ FieldType type = 2;
+ FieldValue value = 3;
+}
+
message Row {
repeated FieldValue values = 1;
}
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
index 81b7922..d7abf4e 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
@@ -163,7 +163,7 @@ class CoderTranslators {
components.isEmpty(), "Expected empty component list, but
received: " + components);
Schema schema;
try {
- schema =
SchemaTranslation.fromProto(SchemaApi.Schema.parseFrom(payload));
+ schema =
SchemaTranslation.schemaFromProto(SchemaApi.Schema.parseFrom(payload));
} catch (InvalidProtocolBufferException e) {
throw new RuntimeException("Unable to parse schema for RowCoder: ",
e);
}
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
index 4ec1a7d..2ac9db5 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
@@ -302,7 +302,8 @@ public class CommonCoderTest {
} else if (s.equals(getUrn(StandardCoders.Enum.ROW))) {
Schema schema;
try {
-schema =
SchemaTranslation.fromProto(SchemaApi.Schema.parseFrom(coderSpec.getPayload()));
+schema =
+
SchemaTranslation.schemaFromProto(SchemaApi.Schema.parseFrom(coderSpec.getPayload()));
} catch (InvalidProtocolBufferException e) {
throw new RuntimeException("Failed to parse schema payload for row
coder", e);
}
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
index 6022d70..bfb9adb 100644
---
a/runners/core-co
[beam] branch master updated: BEAM-8924 - Update Apache Tika to 1.24
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 82081d8 BEAM-8924 - Update Apache Tika to 1.24
new 49cd4d6 Merge pull request #11159 from BEAM-8924 - Update Apache Tika
to 1.24
82081d8 is described below
commit 82081d8bc70af7d405eb47009822ebed47b0631a
Author: Colm O hEigeartaigh
AuthorDate: Wed Mar 18 17:25:38 2020 +
BEAM-8924 - Update Apache Tika to 1.24
---
sdks/java/io/tika/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/java/io/tika/build.gradle b/sdks/java/io/tika/build.gradle
index 1c161fb..384b80e 100644
--- a/sdks/java/io/tika/build.gradle
+++ b/sdks/java/io/tika/build.gradle
@@ -22,7 +22,7 @@ applyJavaNature(automaticModuleName:
'org.apache.beam.sdk.io.tika')
description = "Apache Beam :: SDKs :: Java :: IO :: Tika"
ext.summary = "Tika Input to parse files."
-def tika_version = "1.23"
+def tika_version = "1.24"
def bndlib_version = "1.43.0"
dependencies {
[beam] branch master updated: [BEAM-9453] Changed new string creation to use StandardCharsets.UTF_8
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 589d274 [BEAM-9453] Changed new string creation to use
StandardCharsets.UTF_8
new 2085f1b Merge pull request #11017 from [BEAM-9453] Fix potential
UnsupportedEncodingException
589d274 is described below
commit 589d274145690cfbee695f9d0c6dbee208193f43
Author: Hk-tang
AuthorDate: Mon Mar 2 13:44:20 2020 -0700
[BEAM-9453] Changed new string creation to use StandardCharsets.UTF_8
---
.../src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
index 6028839..bffce77 100644
---
a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
+++
b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
@@ -28,6 +28,7 @@ import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
+import java.nio.charset.StandardCharsets;
import javax.annotation.Nullable;
/**
@@ -79,7 +80,7 @@ public class RabbitMqMessage implements Serializable {
if (value instanceof LongString) {
LongString longString = (LongString) value;
byte[] bytes = longString.getBytes();
- String s = new String(bytes, "UTF-8");
+ String s = new String(bytes, StandardCharsets.UTF_8);
value = s;
} else {
throw new RuntimeException(String.format("no transformation
defined for %s", value));
[beam] branch master updated (b4f3ec3 -> 8612571)
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b4f3ec3 [BEAM-8328] Disable community metrics integration test in 'test' task (#11024) add a6edfde Update lostluck's info on the Go SDK roadmap add 347ccdf Switch contact email to apache.org. add 8612571 Merge pull request #11036 from Update lostluck's info on the Go SDK roadmap No new revisions were added by this update. Summary of changes: website/src/roadmap/go-sdk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated: [BEAM-9360] Fix equivalence check for FieldType
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d68d433 [BEAM-9360] Fix equivalence check for FieldType
new 8d253ac Merge pull request #10943 from [BEAM-9360] Fix equivalence
check for FieldType
d68d433 is described below
commit d68d433fcfde22ee1b8159130c33d4b0c50fe38d
Author: Jozef Vilcek
AuthorDate: Sat Feb 22 10:03:53 2020 +0100
[BEAM-9360] Fix equivalence check for FieldType
---
.../main/java/org/apache/beam/sdk/schemas/Schema.java| 3 ---
.../java/org/apache/beam/sdk/schemas/SchemaTest.java | 16
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index fc6eb5e..2ca53bd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -815,9 +815,6 @@ public class Schema implements Serializable {
if (!getTypeName().equals(other.getTypeName())) {
return false;
}
- if (!Objects.equals(getMetadata(), other.getMetadata())) {
-return false;
- }
switch (getTypeName()) {
case ROW:
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaTest.java
index 1369f59..cc47678 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaTest.java
@@ -237,6 +237,22 @@ public class SchemaTest {
}
@Test
+ public void testFieldsWithDifferentMetadataAreEquivalent() {
+Field foo = Field.of("foo", FieldType.STRING);
+Field fooWithMetadata = Field.of("foo",
FieldType.STRING.withMetadata("key", "value"));
+
+Schema schema1 = Schema.builder().addField(foo).build();
+Schema schema2 = Schema.builder().addField(foo).build();
+assertEquals(schema1, schema2);
+assertTrue(schema1.equivalent(schema2));
+
+schema1 = Schema.builder().addField(foo).build();
+schema2 = Schema.builder().addField(fooWithMetadata).build();
+assertNotEquals(schema1, schema2);
+assertTrue(schema1.equivalent(schema2));
+ }
+
+ @Test
public void testNestedNotEquivalent() {
Schema nestedSchema1 = Schema.builder().addInt64Field("foo").build();
Schema nestedSchema2 = Schema.builder().addStringField("foo").build();
[beam] branch master updated: Fix non correctly formatted class in sdks/java/core
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1d7453d Fix non correctly formatted class in sdks/java/core
new 753fd3f Merge pull request #11010 from Fix non correctly formatted
class in sdks/java/core
1d7453d is described below
commit 1d7453da96c858553bb0a401c13ddfa1209a09ae
Author: Ismaël Mejía
AuthorDate: Sun Mar 1 07:48:39 2020 +0100
Fix non correctly formatted class in sdks/java/core
---
.../test/java/org/apache/beam/sdk/transforms/ParDoTest.java| 10 --
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index fa1e892..8b243f4 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -514,9 +514,7 @@ public class ParDoTest implements Serializable {
SingleOutput, String> parDo = ParDo.of(fn);
// Use the parDo in a pipeline to cause state coders to be inferred.
- pipeline
- .apply(Create.of(KV.of("input", "value")))
- .apply(parDo);
+ pipeline.apply(Create.of(KV.of("input", "value"))).apply(parDo);
DisplayData displayData = DisplayData.from(parDo);
assertThat(
@@ -534,9 +532,9 @@ public class ParDoTest implements Serializable {
hasKey("state_map"),
hasType(DisplayData.Type.STRING),
hasValue(
- "MapState"),
+ "MapState"),
hasLabel("State \"map\"";
}
[beam] branch master updated (6e69b26 -> 3b26ebf)
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6e69b26 Merge pull request #11001: [BEAM-9405] Fix post-commit error about create_job_service add 3b26ebf Adds DisplayData for StateSpecs used by stateful ParDos No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/coders/SerializableCoder.java | 5 +++ .../java/org/apache/beam/sdk/transforms/ParDo.java | 47 -- .../org/apache/beam/sdk/transforms/ParDoTest.java | 47 ++ 3 files changed, 96 insertions(+), 3 deletions(-)
[beam] branch master updated: [BEAM-9394] DynamicMessage handling of empty map violates schema nullability
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 34d7c85 [BEAM-9394] DynamicMessage handling of empty map violates
schema nullability
new 777a401 Merge pull request #10984 from [BEAM-9394] DynamicMessage
handling of empty map violates nullability
34d7c85 is described below
commit 34d7c85d7c1cb894b620cd7bb903152a5d43dbe7
Author: Alex Van Boxel
AuthorDate: Thu Feb 27 08:40:18 2020 +0100
[BEAM-9394] DynamicMessage handling of empty map violates schema nullability
Fixed the handling of empty maps. It runned NULL, but should return and
emtpy
map in the Row. Added tests for Maps and Array. Only Map had the incorrect
behaviour.
---
.../protobuf/ProtoDynamicMessageSchema.java| 4 +--
.../protobuf/ProtoDynamicMessageSchemaTest.java| 35 ++
.../protobuf/ProtoMessageSchemaTest.java | 32
.../sdk/extensions/protobuf/TestProtoSchemas.java | 34 +
4 files changed, 103 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
index 93bf002..ec8537c 100644
---
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
+++
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
@@ -561,10 +561,10 @@ public class ProtoDynamicMessageSchema implements
Serializable {
@Override
Map getFromProtoMessage(Message message) {
List list = (List)
message.getField(getFieldDescriptor(message));
+ Map rowMap = new HashMap<>();
if (list.size() == 0) {
-return null;
+return rowMap;
}
- Map rowMap = new HashMap<>();
list.forEach(
entryMessage -> {
Descriptors.Descriptor entryDescriptor =
entryMessage.getDescriptorForType();
diff --git
a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchemaTest.java
b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchemaTest.java
index 14bd131..0b7984d 100644
---
a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchemaTest.java
+++
b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchemaTest.java
@@ -24,6 +24,10 @@ import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.MAP_PRIMI
import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NESTED_PROTO;
import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NESTED_ROW;
import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NESTED_SCHEMA;
+import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NULL_MAP_PRIMITIVE_PROTO;
+import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NULL_MAP_PRIMITIVE_ROW;
+import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NULL_REPEATED_PROTO;
+import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.NULL_REPEATED_ROW;
import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.ONEOF_PROTO_BOOL;
import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.ONEOF_PROTO_INT32;
import static
org.apache.beam.sdk.extensions.protobuf.TestProtoSchemas.ONEOF_PROTO_PRIMITIVE;
@@ -126,6 +130,22 @@ public class ProtoDynamicMessageSchemaTest {
assertEquals(REPEATED_PROTO.toString(),
fromRow.apply(REPEATED_ROW).toString());
}
+ @Test
+ public void testNullRepeatedProtoToRow() throws
InvalidProtocolBufferException {
+ProtoDynamicMessageSchema schemaProvider =
+schemaFromDescriptor(RepeatPrimitive.getDescriptor());
+SerializableFunction toRow =
schemaProvider.getToRowFunction();
+assertEquals(NULL_REPEATED_ROW,
toRow.apply(toDynamic(NULL_REPEATED_PROTO)));
+ }
+
+ @Test
+ public void testNullRepeatedRowToProto() {
+ProtoDynamicMessageSchema schemaProvider =
+schemaFromDescriptor(RepeatPrimitive.getDescriptor());
+SerializableFunction fromRow =
schemaProvider.getFromRowFunction();
+assertEquals(NULL_REPEATED_PROTO.toString(),
fromRow.apply(NULL_REPEATED_ROW).toString());
+ }
+
// Test map type
@Test
public void testMapSchema() {
@@ -149,6 +169,21 @@ public class ProtoDynamicMessageSchemaTest {
}
@Test
+ public void testNullMapProtoToRow() throws InvalidProtocolBufferException {
+ProtoDynamicMessageSchema schemaProvider =
schem
[beam] branch master updated: Refine access level for `sdks/java/extensions/protobuf`
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c4d006c Refine access level for `sdks/java/extensions/protobuf`
new 4e35fb6 Merge pull request #10796 from Refine access level for
`sdks/java/extensions/protobuf`
c4d006c is described below
commit c4d006c290e23cb100e19f307f8a696c76b47ee3
Author: Ismaël Mejía
AuthorDate: Thu Feb 6 23:36:49 2020 +0100
Refine access level for `sdks/java/extensions/protobuf`
---
.../beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java | 2 +-
.../beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java | 11 ---
2 files changed, 5 insertions(+), 8 deletions(-)
diff --git
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
index 2b0af35..3b670ba 100644
---
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
+++
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtils.java
@@ -111,7 +111,7 @@ import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
-public class ProtoByteBuddyUtils {
+class ProtoByteBuddyUtils {
private static final ByteBuddy BYTE_BUDDY = new ByteBuddy();
private static final TypeDescriptor BYTE_STRING_TYPE_DESCRIPTOR =
TypeDescriptor.of(ByteString.class);
diff --git
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java
index bc7503f..2981090 100644
---
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java
+++
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java
@@ -27,8 +27,6 @@ import com.google.protobuf.Message;
import java.util.List;
import java.util.Map;
import java.util.Set;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.Fixed32;
import org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.Fixed64;
import
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaLogicalTypes.SFixed32;
@@ -127,8 +125,7 @@ import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
* label is specified.
*
*/
-@Experimental(Kind.SCHEMAS)
-public class ProtoSchemaTranslator {
+class ProtoSchemaTranslator {
/** This METADATA tag is used to store the field number of a proto tag. */
public static final String PROTO_NUMBER_METADATA_TAG = "PROTO_NUMBER";
@@ -140,12 +137,12 @@ public class ProtoSchemaTranslator {
"PROTO_MAP_VALUE_MESSAGE_NAME";
/** Attach a proto field number to a type. */
- public static FieldType withFieldNumber(FieldType fieldType, int index) {
+ static FieldType withFieldNumber(FieldType fieldType, int index) {
return fieldType.withMetadata(PROTO_NUMBER_METADATA_TAG,
Long.toString(index));
}
/** Return the proto field number for a type. */
- public static int getFieldNumber(FieldType fieldType) {
+ static int getFieldNumber(FieldType fieldType) {
return
Integer.parseInt(fieldType.getMetadataString(PROTO_NUMBER_METADATA_TAG));
}
@@ -180,7 +177,7 @@ public class ProtoSchemaTranslator {
}
/** Return a Beam scheam representing a proto class. */
- public static Schema getSchema(Class clazz) {
+ static Schema getSchema(Class clazz) {
return getSchema(ProtobufUtil.getDescriptorForClass(clazz));
}
[beam] branch master updated (cf3ca68 -> bb63260)
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cf3ca68 [BEAM-9247] Integrate GCP Vision API functionality (#10959) add bb63260 Fix kotlin warnings (#10976) No new revisions were added by this update. Summary of changes: .../examples/kotlin/cookbook/BigQueryTornadoes.kt | 10 +- .../examples/kotlin/cookbook/FilterExamples.kt | 8 +- .../beam/examples/kotlin/snippets/Snippets.kt | 58 +++--- .../examples/DebuggingWordCountTestKotlin.java | 57 -- .../beam/examples/MinimalWordCountTestKotlin.java | 91 - .../beam/examples/WindowedWordCountITKotlin.kt | 225 - .../apache/beam/examples/WordCountITKotlin.java| 77 --- examples/kotlin/src/test/resources/LICENSE | 202 -- 8 files changed, 30 insertions(+), 698 deletions(-) delete mode 100644 examples/kotlin/src/test/java/org/apache/beam/examples/DebuggingWordCountTestKotlin.java delete mode 100644 examples/kotlin/src/test/java/org/apache/beam/examples/MinimalWordCountTestKotlin.java delete mode 100644 examples/kotlin/src/test/java/org/apache/beam/examples/WindowedWordCountITKotlin.kt delete mode 100644 examples/kotlin/src/test/java/org/apache/beam/examples/WordCountITKotlin.java delete mode 100644 examples/kotlin/src/test/resources/LICENSE
[beam] branch master updated (85259c2 -> 7990bc7)
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 85259c2 Merge pull request #10931 from ibzib/flink-test-log add 7a0fe93 [BEAM-7274] Add DynamicMessage Schema support add 7990bc7 Merge pull request #10502: [BEAM-7274] Add DynamicMessage Schema support No new revisions were added by this update. Summary of changes: .../beam/sdk/schemas/logicaltypes/OneOfType.java | 9 +- .../beam/sdk/extensions/protobuf/ProtoDomain.java | 12 +- .../protobuf/ProtoDynamicMessageSchema.java| 844 + .../extensions/protobuf/ProtoSchemaTranslator.java | 66 +- .../protobuf/ProtoDynamicMessageSchemaTest.java| 282 +++ .../sdk/extensions/protobuf/TestProtoSchemas.java | 86 ++- 6 files changed, 1268 insertions(+), 31 deletions(-) create mode 100644 sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java create mode 100644 sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchemaTest.java
[beam] branch master updated: Fix typo on runners/extensions-java label for github PR autolabeler
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new eb010be Fix typo on runners/extensions-java label for github PR autolabeler new 0bc4b3f Merge pull request #10877 from Fix typo on runners/extensions-java label for github PR autolabeler eb010be is described below commit eb010be3c5aa0da38ff5a38a761d6be67abab8e4 Author: Ismaël Mejía AuthorDate: Sat Feb 15 16:47:24 2020 +0100 Fix typo on runners/extensions-java label for github PR autolabeler --- .github/autolabeler.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/autolabeler.yml b/.github/autolabeler.yml index df504e4..48e0657 100644 --- a/.github/autolabeler.yml +++ b/.github/autolabeler.yml @@ -77,7 +77,7 @@ io: ["sdks/go/pkg/beam/io/*", "sdks/java/io/*", "sdks/python/apache_beam/io/*"] "core": ["runners/core-construction-java/*", "runners/core-java/*"] "dataflow": ["runners/google-cloud-dataflow-java/*", "sdks/go/pkg/beam/runners/dataflow/*", "sdks/python/runners/dataflow/*"] "direct": ["runners/direct-java/*", "sdks/go/pkg/beam/runners/direct/*", "sdks/python/runners/direct/*"] -"extenstions-java": ["runners/extenstions-java/*"] +"extensions-java": ["runners/extensions-java/*"] "flink": ["runners/flink/*", "sdks/go/pkg/beam/runners/flink/*", "sdks/python/runners/portability/flink*"] "fn-execution": ["runners/java-fn-execution/*"] "gearpump": ["runners/gearpump/*"]
[beam] branch master updated: Refine labels/categories for PR autolabeling
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 2a8db6c Refine labels/categories for PR autolabeling new 346f716 Merge pull request #10845 from Refine labels/categories for PR autolabeling 2a8db6c is described below commit 2a8db6c1ca8982ecfd9d8b1a2535648672d8f007 Author: Ismaël Mejía AuthorDate: Wed Feb 12 22:27:04 2020 +0100 Refine labels/categories for PR autolabeling --- .github/autolabeler.yml | 32 +--- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/.github/autolabeler.yml b/.github/autolabeler.yml index 304a898..df504e4 100644 --- a/.github/autolabeler.yml +++ b/.github/autolabeler.yml @@ -18,12 +18,13 @@ # Please keep the entries sorted lexicographically in each category. # General -build: ["buildSrc/*", "release/*", "ownership/*", ".github/*", "gradle/*"] -docker: ["sdks/python/container/*", "sdks/go/container/*", "sdks/java/container/*", "runners/flink/job-server-container/*", "runners/spark/job-server/container/*"] -examples: ["examples/*"] +build: ["assembly.xml", "build.gradle", "buildSrc/*", ".gitattributes", ".github/*", ".gitignore", "gradle/*", ".mailmap", "ownership/*", "release/*", "sdks/java/build-tools/*", "settings.gradle"] +docker: ["runners/flink/job-server-container/*", "runners/spark/job-server/container/*", "sdks/go/container/*", "sdks/java/container/*", "sdks/python/container/*"] +examples: ["examples/*", "sdks/go/examples/*", "sdks/python/apache_beam/examples/*"] go: ["sdks/go/*"] infra: [".test-infra/*"] -java: ["sdks/java/*"] +java: ["examples/java/*", "sdks/java/*"] +kotlin: ["examples/kotlin/*"] learning: ["learning/*"] model: ["model/*"] python: ["sdks/python/*"] @@ -44,14 +45,14 @@ extensions: ["sdks/java/extensions/*"] "zetasketch": ["sdks/java/extensions/zetasketch/*"] # IO -io: ["sdks/java/io/*"] +io: ["sdks/go/pkg/beam/io/*", "sdks/java/io/*", "sdks/python/apache_beam/io/*"] "amqp": ["sdks/java/io/amqp/*"] -"aws": ["sdks/java/io/amazon-web-services/*","sdks/java/io/amazon-web-services2/*"] +"aws": ["sdks/java/io/amazon-web-services/*", "sdks/java/io/amazon-web-services2/*", "sdks/python/apache_beam/io/aws/*"] "cassandra": ["sdks/java/io/cassandra/*"] "clickhouse": ["sdks/java/io/clickhouse/*"] -"elasticsearch": ["sdks/java/io/elasticsearch/*","sdks/java/io/elasticsearch-tests/*"] -"gcp": ["sdks/java/io/bigquery-io-perf-tests/*","sdks/java/io/google-cloud-platform/*"] -"hadoop": ["sdks/java/io/hadoop-common/*","sdks/java/io/hadoop-file-system/*","sdks/java/io/hadoop-format/*"] +"elasticsearch": ["sdks/java/io/elasticsearch/*", "sdks/java/io/elasticsearch-tests/*"] +"gcp": ["sdks/go/pkg/beam/io/bigqueryio/*", "sdks/go/pkg/beam/io/datastoreio/*", "sdks/go/pkg/beam/io/pubsubio/*", "sdks/java/io/bigquery-io-perf-tests/*", "sdks/java/io/google-cloud-platform/*", "sdks/python/apache_beam/io/gcp/*"] +"hadoop": ["sdks/java/io/hadoop-common/*", "sdks/java/io/hadoop-file-system/*", "sdks/java/io/hadoop-format/*"] "hbase": ["sdks/java/io/hbase/*"] "hcatalog": ["sdks/java/io/hcatalog/*"] "jdbc": ["sdks/java/io/jdbc/*"] @@ -71,17 +72,18 @@ io: ["sdks/java/io/*"] "xml": ["sdks/java/io/xml/*"] # Runners -"runners": ["runners/*"] +"runners": ["runners/*", "sdks/go/pkg/beam/runners/*", "sdks/python/runners/*"] "apex": ["runners/apex/*"] -"core": ["runners/core-construction-java/*","runners/core-java/*"] -"dataflow": ["runners/google-cloud-dataflow-java/*"] -"direct": ["runners/direct-java/*"] +"core": ["runners/core-construction-java/*", "runners/core-java/*"] +"dataflow": ["runners/google-cloud-dataflow-java/*", "sdks/go/pkg/beam/runners/dataflow/*", "sdks/python/runners/dataflow/*"] +"direct": ["runners/direct-java/*", "sdks/go/pkg/beam/runners/direct/*", "sdks/python/runners/direct/*"] "extenstions-java": ["runners/extenstions-java/*"] -"flink": ["runners/flink/*"] +"flink": ["runners/flink/*", "sdks/go/pkg/beam/runners/flink/*", "sdks/python/runners/portability/flink*"] "fn-execution": ["runners/java-fn-execution/*"] "gearpump": ["runners/gearpump/*"] +"interactive": ["sdks/python/runners/interactive/*"] "jet": ["runners/jet/*"] "local": ["runners/local-java/*"] "portability": ["runners/portability/*"] "samza": ["runners/samza/*"] -"spark": ["runners/spark/*"] +"spark": ["runners/spark/*", "sdks/go/pkg/beam/runners/spark/*", "sdks/python/runners/portability/spark*"]
[beam] branch feature/autolabeler created (now bcb0d8d)
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a change to branch feature/autolabeler in repository https://gitbox.apache.org/repos/asf/beam.git. at bcb0d8d Added newline at the end of the file This branch includes the following new commits: new bcb0d8d Added newline at the end of the file The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Added newline at the end of the file
This is an automated email from the ASF dual-hosted git repository. alexvanboxel pushed a commit to branch feature/autolabeler in repository https://gitbox.apache.org/repos/asf/beam.git commit bcb0d8da909963a19492aeadec1bc5a0eb7465b6 Author: Alex Van Boxel AuthorDate: Tue Feb 11 08:38:21 2020 +0100 Added newline at the end of the file --- .github/autolabeler.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/autolabeler.yml b/.github/autolabeler.yml index e8d7754..8ea389f 100644 --- a/.github/autolabeler.yml +++ b/.github/autolabeler.yml @@ -23,4 +23,4 @@ java: ["sdks/java/*"] python: ["sdks/python/*"] runner: ["runners/*"] examples: ["examples/*", "learning/*"] -model: ["model/*"] \ No newline at end of file +model: ["model/*"]
[beam] branch master updated (aa1142a -> 7fc5fb5)
This is an automated email from the ASF dual-hosted git repository.
alexvanboxel pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from aa1142a Merge pull request #10798 from robertwb/provision
add aab5ae5 [BEAM-9241] Fix inconsistent proto nullability
add 7fc5fb5 Merge pull request #10746: [BEAM-9241] Fix inconsistent proto
nullability
No new revisions were added by this update.
Summary of changes:
.../extensions/protobuf/ProtoSchemaTranslator.java | 39 --
.../protobuf/ProtoMessageSchemaTest.java | 50 +++-
.../protobuf/ProtoSchemaTranslatorTest.java| 28 +
.../sdk/extensions/protobuf/TestProtoSchemas.java | 136 ++---
...messages.proto => proto2_schema_messages.proto} | 38 +++---
5 files changed, 216 insertions(+), 75 deletions(-)
copy
sdks/java/extensions/protobuf/src/test/proto/{proto2_coder_test_messages.proto
=> proto2_schema_messages.proto} (59%)
