[beam] 02/02: fix type error
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a7501289a23e2d8a3a7e5a3b9f5e920702d4d634 Author: mingmxu AuthorDate: Mon Feb 26 23:39:08 2018 -0800 fix type error --- .../java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java index 01e45c1..4589d39 100644 --- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java +++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java @@ -98,7 +98,7 @@ public class BeamSqlDslUdfUdafTest extends BeamSqlDslBase { RowType subStrRowType = RowSqlType.builder() .withIntegerField("f_int") -.withIntegerField("sub_string") +.withVarcharField("sub_string") .build(); Row subStrRow = Row.withRowType(subStrRowType).addValues(2, "s").build(); PAssert.that(result3).containsInAnyOrder(subStrRow); -- To stop receiving notification emails like this one, please contact xumingm...@apache.org.
[beam] branch master updated (8a39c80 -> a750128)
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8a39c80 Merge pull request #4695: Add To/From Proto Round Trip for ExecutableStage new 880caef @Parameter annotation does not work for UDFs in Beam SQL new a750128 fix type error The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/extensions/sql/BeamSqlUdf.java | 2 +- .../sql/impl/interpreter/BeamSqlFnExecutor.java| 4 ...pression.java => BeamSqlDefaultExpression.java} | 15 +++-- .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 26 +- 4 files changed, 33 insertions(+), 14 deletions(-) copy sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/{BeamSqlInputRefExpression.java => BeamSqlDefaultExpression.java} (74%) -- To stop receiving notification emails like this one, please contact xumingm...@apache.org.
[beam] 01/02: @Parameter annotation does not work for UDFs in Beam SQL
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 880caef3fc17295f817648ccdffe369c2dc63fa1 Author: mingmxu AuthorDate: Mon Feb 26 15:42:55 2018 -0800 @Parameter annotation does not work for UDFs in Beam SQL --- .../apache/beam/sdk/extensions/sql/BeamSqlUdf.java | 2 +- .../sql/impl/interpreter/BeamSqlFnExecutor.java| 4 +++ .../operator/BeamSqlDefaultExpression.java}| 38 ++ .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 26 ++- 4 files changed, 47 insertions(+), 23 deletions(-) diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java index 91bad20..5df046a 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java @@ -35,7 +35,7 @@ import org.apache.beam.sdk.annotations.Experimental; * } * * The first parameter is named "s" and is mandatory, - * and the second parameter is named "n" and is optional. + * and the second parameter is named "n" and is optional(always NULL if not specified). */ @Experimental public interface BeamSqlUdf extends Serializable { diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java index ae65c2b..dbdd03d 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java @@ -23,6 +23,7 @@ import java.util.Calendar; import java.util.List; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlCaseExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlCastExpression; +import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlDefaultExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlInputRefExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive; @@ -383,6 +384,9 @@ public class BeamSqlFnExecutor implements BeamSqlExpressionExecutor { case "DATETIME_PLUS": return new BeamSqlDatetimePlusExpression(subExps); +//DEFAULT keyword for UDF with optional parameter +case "DEFAULT": + return new BeamSqlDefaultExpression(); case "CASE": ret = new BeamSqlCaseExpression(subExps); diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDefaultExpression.java similarity index 52% copy from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java copy to sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDefaultExpression.java index 91bad20..0557600 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDefaultExpression.java @@ -15,29 +15,25 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.beam.sdk.extensions.sql; +package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator; -import java.io.Serializable; -import org.apache.beam.sdk.annotations.Experimental; +import org.apache.beam.sdk.transforms.windowing.BoundedWindow; +import org.apache.beam.sdk.values.Row; +import org.apache.calcite.sql.type.SqlTypeName; /** - * Interface to create a UDF in Beam SQL. - * - * A static method {@code eval} is required. Here is an example: - * - * - * public static class MyLeftFunction { - * public String eval( - * @Parameter(name = "s") String s, - * @Parameter(name = "n", optional = true) Integer n) { - * return s.substring(0, n == null ? 1 : n); - * } - * } - * - * The first parameter is named "s" and is mandatory, - * and the second parameter is named "n" and is optional. + * DEFAULT keyword for UDF with optional parameter. */ -@Experimental -public interface BeamSqlUdf extends Serializable { - String UDF_METHOD = "eval&quo
[beam] 01/02: add setup/teardown for BeamSqlSeekableTable.
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit b0991a47a0e1a7bba416be1c0b3781c298966731 Author: mingmxu AuthorDate: Sun Feb 25 22:14:25 2018 -0800 add setup/teardown for BeamSqlSeekableTable. --- .../apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java | 10 ++ .../extensions/sql/impl/transform/BeamJoinTransforms.java| 12 +++- .../sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java | 8 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java index d274dd9..95165a5 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java @@ -29,7 +29,17 @@ import org.apache.beam.sdk.values.Row; @Experimental public interface BeamSqlSeekableTable extends Serializable{ /** + * prepare the instance. + */ + void setup(); + + /** * return a list of {@code Row} with given key set. */ List seekRow(Row lookupSubRow); + + /** + * cleanup resources of the instance. + */ + void teardown(); } diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java index 648b973..2b44814 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java @@ -219,7 +219,12 @@ public class BeamJoinTransforms { @Override public PCollection expand(PCollection input) { - return input.apply("join_as_lookup", ParDo.of(new DoFn() { + return input.apply("join_as_lookup", ParDo.of(new DoFn(){ +@Setup +public void setup(){ + seekableTable.setup(); +} + @ProcessElement public void processElement(ProcessContext context) { Row factRow = context.element(); @@ -230,6 +235,11 @@ public class BeamJoinTransforms { } } +@Teardown +public void teardown(){ + seekableTable.teardown(); +} + private Row extractJoinSubRow(Row factRow) { List joinSubsetValues = factJoinIdx diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java index ca639bd..6a06123 100644 --- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java +++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java @@ -131,6 +131,14 @@ public class BeamJoinRelUnboundedVsBoundedTest extends BaseRelTest { public List seekRow(Row lookupSubRow) { return Arrays.asList(Row.withRowType(getRowType()).addValues(1, "SITE1").build()); } + +@Override +public void setup() { +} + +@Override +public void teardown() { +} } @Test -- To stop receiving notification emails like this one, please contact xumingm...@apache.org.
[beam] 02/02: rename method as suggested and declare as default methods.
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit dbaf09838c5d064fdd6d953b91ad9a1ce61af72b Author: mingmxu AuthorDate: Mon Feb 12 08:02:31 2018 -0800 rename method as suggested and declare as default methods. --- .../org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java | 4 ++-- .../sdk/extensions/sql/impl/transform/BeamJoinTransforms.java | 4 ++-- .../sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java | 8 3 files changed, 4 insertions(+), 12 deletions(-) diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java index 95165a5..7b256c8 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java @@ -31,7 +31,7 @@ public interface BeamSqlSeekableTable extends Serializable{ /** * prepare the instance. */ - void setup(); + default void setUp(){}; /** * return a list of {@code Row} with given key set. @@ -41,5 +41,5 @@ public interface BeamSqlSeekableTable extends Serializable{ /** * cleanup resources of the instance. */ - void teardown(); + default void tearDown(){}; } diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java index 2b44814..1be47b4 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java @@ -222,7 +222,7 @@ public class BeamJoinTransforms { return input.apply("join_as_lookup", ParDo.of(new DoFn(){ @Setup public void setup(){ - seekableTable.setup(); + seekableTable.setUp(); } @ProcessElement @@ -237,7 +237,7 @@ public class BeamJoinTransforms { @Teardown public void teardown(){ - seekableTable.teardown(); + seekableTable.tearDown(); } private Row extractJoinSubRow(Row factRow) { diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java index 6a06123..ca639bd 100644 --- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java +++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java @@ -131,14 +131,6 @@ public class BeamJoinRelUnboundedVsBoundedTest extends BaseRelTest { public List seekRow(Row lookupSubRow) { return Arrays.asList(Row.withRowType(getRowType()).addValues(1, "SITE1").build()); } - -@Override -public void setup() { -} - -@Override -public void teardown() { -} } @Test -- To stop receiving notification emails like this one, please contact xumingm...@apache.org.
[beam] branch master updated (2b70d60 -> dbaf098)
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2b70d60 Merge pull request #4731: [BEAM-3732] Fix broken maven profiles new b0991a4 add setup/teardown for BeamSqlSeekableTable. new dbaf098 rename method as suggested and declare as default methods. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java | 10 ++ .../extensions/sql/impl/transform/BeamJoinTransforms.java| 12 +++- 2 files changed, 21 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact xumingm...@apache.org.
[beam] branch master updated (c8585fa -> 35570f1)
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c8585fa This closes #4198 add 19e8159 convert JOIN as LOOKUP add c84e3f2 add BeamSqlSeekableTable to SurfaceTest new 35570f1 Merge pull request #4196 from XuMingmin/BEAM-3171 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/extensions/sql/BeamSqlSeekableTable.java} | 18 ++--- .../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 45 +++- .../sql/impl/transform/BeamJoinTransforms.java | 81 ++ .../sdk/extensions/sql/BeamSqlApiSurfaceTest.java | 1 + .../rel/BeamJoinRelUnboundedVsBoundedTest.java | 63 + 5 files changed, 197 insertions(+), 11 deletions(-) copy sdks/java/{core/src/main/java/org/apache/beam/sdk/metrics/Metric.java => extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java} (66%) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4196 from XuMingmin/BEAM-3171
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 35570f1accfd2e9ef04421c9f02b6cdfcb7c95f4 Merge: c8585fa c84e3f2 Author: James Xu AuthorDate: Sat Dec 2 16:53:17 2017 +0800 Merge pull request #4196 from XuMingmin/BEAM-3171 [BEAM-3171] convert a join into lookup .../sdk/extensions/sql/BeamSqlSeekableTable.java | 35 ++ .../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 45 +++- .../sql/impl/transform/BeamJoinTransforms.java | 81 ++ .../sdk/extensions/sql/BeamSqlApiSurfaceTest.java | 1 + .../rel/BeamJoinRelUnboundedVsBoundedTest.java | 63 + 5 files changed, 223 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] 01/02: [BEAM-3238][SQL] Move Coders to map in BeamRecordSqlType
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e2c3de092b0a7a5bbfcb5b7797984d5f4bd10138 Author: Anton Kedin AuthorDate: Wed Nov 22 16:04:05 2017 -0800 [BEAM-3238][SQL] Move Coders to map in BeamRecordSqlType Improve readability. All coders are supposed to be thread safe and currently are backed by the static instances. --- .../beam/sdk/extensions/sql/BeamRecordSqlType.java | 111 + 1 file changed, 47 insertions(+), 64 deletions(-) diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java index 982494a..1784ec1 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java @@ -17,13 +17,13 @@ */ package org.apache.beam.sdk.extensions.sql; +import com.google.common.collect.ImmutableMap; import java.math.BigDecimal; import java.sql.Types; import java.util.ArrayList; import java.util.Collections; import java.util.Date; import java.util.GregorianCalendar; -import java.util.HashMap; import java.util.List; import java.util.Map; import org.apache.beam.sdk.coders.BigDecimalCoder; @@ -50,26 +50,39 @@ import org.apache.beam.sdk.values.BeamRecordType; * */ public class BeamRecordSqlType extends BeamRecordType { - private static final Map SQL_TYPE_TO_JAVA_CLASS = new HashMap<>(); - static { -SQL_TYPE_TO_JAVA_CLASS.put(Types.TINYINT, Byte.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.SMALLINT, Short.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.INTEGER, Integer.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.BIGINT, Long.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.FLOAT, Float.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.DOUBLE, Double.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.DECIMAL, BigDecimal.class); - -SQL_TYPE_TO_JAVA_CLASS.put(Types.BOOLEAN, Boolean.class); - -SQL_TYPE_TO_JAVA_CLASS.put(Types.CHAR, String.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.VARCHAR, String.class); - -SQL_TYPE_TO_JAVA_CLASS.put(Types.TIME, GregorianCalendar.class); - -SQL_TYPE_TO_JAVA_CLASS.put(Types.DATE, Date.class); -SQL_TYPE_TO_JAVA_CLASS.put(Types.TIMESTAMP, Date.class); - } + private static final Map JAVA_CLASSES = ImmutableMap + .builder() + .put(Types.TINYINT, Byte.class) + .put(Types.SMALLINT, Short.class) + .put(Types.INTEGER, Integer.class) + .put(Types.BIGINT, Long.class) + .put(Types.FLOAT, Float.class) + .put(Types.DOUBLE, Double.class) + .put(Types.DECIMAL, BigDecimal.class) + .put(Types.BOOLEAN, Boolean.class) + .put(Types.CHAR, String.class) + .put(Types.VARCHAR, String.class) + .put(Types.TIME, GregorianCalendar.class) + .put(Types.DATE, Date.class) + .put(Types.TIMESTAMP, Date.class) + .build(); + + private static final Map CODERS = ImmutableMap + .builder() + .put(Types.TINYINT, ByteCoder.of()) + .put(Types.SMALLINT, ShortCoder.of()) + .put(Types.INTEGER, BigEndianIntegerCoder.of()) + .put(Types.BIGINT, BigEndianLongCoder.of()) + .put(Types.FLOAT, FloatCoder.of()) + .put(Types.DOUBLE, DoubleCoder.of()) + .put(Types.DECIMAL, BigDecimalCoder.of()) + .put(Types.BOOLEAN, BooleanCoder.of()) + .put(Types.CHAR, StringUtf8Coder.of()) + .put(Types.VARCHAR, StringUtf8Coder.of()) + .put(Types.TIME, TimeCoder.of()) + .put(Types.DATE, DateCoder.of()) + .put(Types.TIMESTAMP, DateCoder.of()) + .build(); public List fieldTypes; @@ -84,54 +97,24 @@ public class BeamRecordSqlType extends BeamRecordType { } public static BeamRecordSqlType create(List fieldNames, - List fieldTypes) { + List fieldTypes) { if (fieldNames.size() != fieldTypes.size()) { throw new IllegalStateException("the sizes of 'dataType' and 'fieldTypes' must match."); } + List fieldCoders = new ArrayList<>(fieldTypes.size()); + for (int idx = 0; idx < fieldTypes.size(); ++idx) { - switch (fieldTypes.get(idx)) { - case Types.INTEGER: -fieldCoders.add(BigEndianIntegerCoder.of()); -break; - case Types.SMALLINT: -fieldCoders.add(ShortCoder.of()); -break; - case Types.TINYINT: -fieldCoders.add(ByteCoder.of()); -break; - case Types.DOUBLE: -fieldCoders.add(DoubleCoder.of()); -break; - case Types.FLOAT: -fieldCoders.add(FloatCoder.of()); -break; - case Types.DECIMAL: -fieldCoders.add(BigDecimalCoder.of()); -break; - case Types.BIGINT: -
[beam] branch master updated (27f505e -> 2eb7de0)
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 27f505e Merge pull request #4095 from rniemo-g/master new e2c3de0 [BEAM-3238][SQL] Move Coders to map in BeamRecordSqlType new 2eb7de0 [BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/extensions/sql/BeamRecordSqlType.java | 190 ++--- .../sdk/extensions/sql/BeamRecordSqlTypeTest.java | 115 + ...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 56 +++--- 3 files changed, 273 insertions(+), 88 deletions(-) create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlTypeTest.java -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 02/02: [BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder
This is an automated email from the ASF dual-hosted git repository. xumingming pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2eb7de0fe6e96da9805fc827294da1e1329ff716 Author: Anton Kedin AuthorDate: Wed Nov 22 19:17:21 2017 -0800 [BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder To improve readability of creating BeamRecordSqlTypes. --- .../beam/sdk/extensions/sql/BeamRecordSqlType.java | 85 ++- .../sdk/extensions/sql/BeamRecordSqlTypeTest.java | 115 + ...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 56 +- 3 files changed, 229 insertions(+), 27 deletions(-) diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java index 1784ec1..a6b23b6 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java @@ -17,6 +17,7 @@ */ package org.apache.beam.sdk.extensions.sql; +import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import java.math.BigDecimal; import java.sql.Types; @@ -108,8 +109,8 @@ public class BeamRecordSqlType extends BeamRecordType { Integer fieldType = fieldTypes.get(idx); if (!CODERS.containsKey(fieldType)) { - throw new UnsupportedOperationException( - "Data type: " + fieldType + " not supported yet!"); +throw new UnsupportedOperationException( +"Data type: " + fieldType + " not supported yet!"); } fieldCoders.add(CODERS.get(fieldType)); @@ -166,4 +167,84 @@ public class BeamRecordSqlType extends BeamRecordType { return "BeamRecordSqlType [fieldNames=" + getFieldNames() + ", fieldTypes=" + fieldTypes + "]"; } + + public static Builder builder() { +return new Builder(); + } + + /** + * Builder class to construct {@link BeamRecordSqlType}. + */ + public static class Builder { + +private ImmutableList.Builder fieldNames; +private ImmutableList.Builder fieldTypes; + +public Builder withField(String fieldName, Integer fieldType) { + fieldNames.add(fieldName); + fieldTypes.add(fieldType); + return this; +} + +public Builder withTinyIntField(String fieldName) { + return withField(fieldName, Types.TINYINT); +} + +public Builder withSmallIntField(String fieldName) { + return withField(fieldName, Types.SMALLINT); +} + +public Builder withIntegerField(String fieldName) { + return withField(fieldName, Types.INTEGER); +} + +public Builder withBigIntField(String fieldName) { + return withField(fieldName, Types.BIGINT); +} + +public Builder withFloatField(String fieldName) { + return withField(fieldName, Types.FLOAT); +} + +public Builder withDoubleField(String fieldName) { + return withField(fieldName, Types.DOUBLE); +} + +public Builder withDecimalField(String fieldName) { + return withField(fieldName, Types.DECIMAL); +} + +public Builder withBooleanField(String fieldName) { + return withField(fieldName, Types.BOOLEAN); +} + +public Builder withCharField(String fieldName) { + return withField(fieldName, Types.CHAR); +} + +public Builder withVarcharField(String fieldName) { + return withField(fieldName, Types.VARCHAR); +} + +public Builder withTimeField(String fieldName) { + return withField(fieldName, Types.TIME); +} + +public Builder withDateField(String fieldName) { + return withField(fieldName, Types.DATE); +} + +public Builder withTimestampField(String fieldName) { + return withField(fieldName, Types.TIMESTAMP); +} + +private Builder() { + this.fieldNames = ImmutableList.builder(); + this.fieldTypes = ImmutableList.builder(); +} + +public BeamRecordSqlType build() { + return create(fieldNames.build(), fieldTypes.build()); +} + } } diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlTypeTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlTypeTest.java new file mode 100644 index 000..78ff221 --- /dev/null +++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlTypeTest.java @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "Lic
[2/2] beam git commit: This closes #4115
This closes #4115 Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2b4a6b5d Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2b4a6b5d Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2b4a6b5d Branch: refs/heads/master Commit: 2b4a6b5d5cb455fe73971fde4caa7d1446cd55aa Parents: acfab89 92b3a9a Author: James Xu Authored: Mon Nov 13 14:42:08 2017 +0800 Committer: James Xu Committed: Mon Nov 13 14:42:08 2017 +0800 -- .../sql/meta/provider/kafka/BeamKafkaTable.java | 39 +++- 1 file changed, 30 insertions(+), 9 deletions(-) --
[1/2] beam git commit: support topicPartition in BeamKafkaTable
Repository: beam Updated Branches: refs/heads/master acfab8965 -> 2b4a6b5d5 support topicPartition in BeamKafkaTable Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/92b3a9ac Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/92b3a9ac Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/92b3a9ac Branch: refs/heads/master Commit: 92b3a9acf04450313c3c2340f6921bf433c2dc04 Parents: acfab89 Author: mingmxu Authored: Fri Nov 10 19:51:37 2017 -0800 Committer: James Xu Committed: Mon Nov 13 14:41:29 2017 +0800 -- .../sql/meta/provider/kafka/BeamKafkaTable.java | 39 +++- 1 file changed, 30 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/beam/blob/92b3a9ac/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java -- diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java index 50f7496..8f663a3 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java @@ -34,6 +34,7 @@ import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PBegin; import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.PDone; +import org.apache.kafka.common.TopicPartition; import org.apache.kafka.common.serialization.ByteArrayDeserializer; import org.apache.kafka.common.serialization.ByteArraySerializer; @@ -45,6 +46,7 @@ import org.apache.kafka.common.serialization.ByteArraySerializer; public abstract class BeamKafkaTable extends BaseBeamTable implements Serializable { private String bootstrapServers; private List topics; + private List topicPartitions; private Map configUpdates; protected BeamKafkaTable(BeamRecordSqlType beamSqlRowType) { @@ -58,6 +60,13 @@ public abstract class BeamKafkaTable extends BaseBeamTable implements Serializab this.topics = topics; } + public BeamKafkaTable(BeamRecordSqlType beamSqlRowType, + List topicPartitions, String bootstrapServers) { +super(beamSqlRowType); +this.bootstrapServers = bootstrapServers; +this.topicPartitions = topicPartitions; + } + public BeamKafkaTable updateConsumerProperties(Map configUpdates) { this.configUpdates = configUpdates; return this; @@ -76,15 +85,27 @@ public abstract class BeamKafkaTable extends BaseBeamTable implements Serializab @Override public PCollection buildIOReader(Pipeline pipeline) { -return PBegin.in(pipeline).apply("read", -KafkaIO.read() -.withBootstrapServers(bootstrapServers) -.withTopics(topics) -.updateConsumerProperties(configUpdates) -.withKeyDeserializerAndCoder(ByteArrayDeserializer.class, ByteArrayCoder.of()) -.withValueDeserializerAndCoder(ByteArrayDeserializer.class, ByteArrayCoder.of()) -.withoutMetadata()) -.apply("in_format", getPTransformForInput()); +KafkaIO.Read kafkaRead = null; +if (topics != null) { + kafkaRead = KafkaIO.read() + .withBootstrapServers(bootstrapServers) + .withTopics(topics) + .updateConsumerProperties(configUpdates) + .withKeyDeserializerAndCoder(ByteArrayDeserializer.class, ByteArrayCoder.of()) + .withValueDeserializerAndCoder(ByteArrayDeserializer.class, ByteArrayCoder.of()); +} else if (topicPartitions != null) { + kafkaRead = KafkaIO.read() + .withBootstrapServers(bootstrapServers) + .withTopicPartitions(topicPartitions) + .updateConsumerProperties(configUpdates) + .withKeyDeserializerAndCoder(ByteArrayDeserializer.class, ByteArrayCoder.of()) + .withValueDeserializerAndCoder(ByteArrayDeserializer.class, ByteArrayCoder.of()); +} else { + throw new IllegalArgumentException("One of topics and topicPartitions must be configurated."); +} + +return PBegin.in(pipeline).apply("read", kafkaRead.withoutMetadata()) +.apply("in_format", getPTransformForInput()); } @Override
[2/2] beam git commit: This closes #4101
This closes #4101 Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4451d556 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4451d556 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4451d556 Branch: refs/heads/master Commit: 4451d5566d6ffd73d2fd6f946a11ffd1eebd90c6 Parents: 867d816 4a5741f Author: James Xu Authored: Thu Nov 9 13:43:09 2017 +0800 Committer: James Xu Committed: Thu Nov 9 13:43:09 2017 +0800 -- .../beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --
[1/2] beam git commit: change `withAllowedTimestampSkew`
Repository: beam Updated Branches: refs/heads/master 867d81684 -> 4451d5566 change `withAllowedTimestampSkew` Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4a5741f2 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4a5741f2 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4a5741f2 Branch: refs/heads/master Commit: 4a5741f2f1aabac098e41cfe453b52876a1bcdb0 Parents: 867d816 Author: mingmxu Authored: Wed Nov 8 13:44:18 2017 -0800 Committer: James Xu Committed: Thu Nov 9 13:42:04 2017 +0800 -- .../beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/beam/blob/4a5741f2/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java -- diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java index e49e79c..6ed30b6 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java @@ -80,7 +80,8 @@ public class BeamAggregationRel extends Aggregate implements BeamRelNode { BeamSqlRelUtils.getBeamRelInput(input).buildBeamPipeline(inputPCollections, sqlEnv); if (windowFieldIdx != -1) { upstream = upstream.apply(stageName + "assignEventTimestamp", WithTimestamps - .of(new BeamAggregationTransforms.WindowTimestampFn(windowFieldIdx))) + .of(new BeamAggregationTransforms.WindowTimestampFn(windowFieldIdx)) + .withAllowedTimestampSkew(new Duration(Long.MAX_VALUE))) .setCoder(upstream.getCoder()); }
[1/2] beam git commit: [BEAM-2203] Implement 'timestamp - interval'
Repository: beam Updated Branches: refs/heads/master 299429394 -> e0166ceb6 [BEAM-2203] Implement 'timestamp - interval' Move TIMESTAMPDIFF implementation into BeamSqlTimestampMinusTimestampExpression, add BeamSqlTimestampMinusIntervalExpression Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a63eb21 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a63eb21 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a63eb21 Branch: refs/heads/master Commit: 0a63eb2195575f6cc5ee228d4edb1ed9571f9f5f Parents: 29942939 Author: Anton Kedin Authored: Sun Nov 5 23:26:10 2017 -0800 Committer: James Xu Committed: Wed Nov 8 11:58:34 2017 +0800 -- .../date/BeamSqlDatetimeMinusExpression.java| 104 - ...BeamSqlTimestampMinusIntervalExpression.java | 83 +++ ...eamSqlTimestampMinusTimestampExpression.java | 97 .../BeamSqlDatetimeMinusExpressionTest.java | 189 --- ...SqlTimestampMinusIntervalExpressionTest.java | 163 + ...qlTimestampMinusTimestampExpressionTest.java | 233 +++ .../BeamSqlDateFunctionsIntegrationTest.java| 18 ++ 7 files changed, 691 insertions(+), 196 deletions(-) -- http://git-wip-us.apache.org/repos/asf/beam/blob/0a63eb21/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java -- diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java index 6a5cbb1..6948ba1 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java @@ -28,80 +28,80 @@ import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimi import org.apache.beam.sdk.transforms.windowing.BoundedWindow; import org.apache.beam.sdk.values.BeamRecord; import org.apache.calcite.sql.type.SqlTypeName; -import org.joda.time.DateTime; import org.joda.time.DurationFieldType; -import org.joda.time.Period; -import org.joda.time.PeriodType; /** * Infix '-' operation for timestamps. * - * Currently this implementation is specific to how Calcite implements 'TIMESTAMPDIFF(..)'. - * It converts the TIMESTAMPDIFF() call into infix minus and normalizes it - * with corresponding TimeUnit's multiplier. + * Implements 2 SQL subtraction operations at the moment: + * 'timestampdiff(timeunit, timestamp, timestamp)', and 'timestamp - interval' * - * In addition to this TIMESTAMPDIFF(..) implementation, Calcite also supports infix - * operations 'interval - interval' and 'timestamp - interval'. - * These are not implemented yet. + * Calcite converts both of the above into infix '-' expression, with different operands and + * return types. + * + * This class delegates evaluation to specific implementation of one of the above operations, + * see {@link BeamSqlTimestampMinusTimestampExpression} + * and {@link BeamSqlTimestampMinusIntervalExpression} + * + * Calcite supports one more subtraction kind: 'interval - interval', + * but it is not implemented yet. */ public class BeamSqlDatetimeMinusExpression extends BeamSqlExpression { - private SqlTypeName intervalType; - private static final Map INTERVALS_DURATIONS_TYPES = + static final Map INTERVALS_DURATIONS_TYPES = ImmutableMap.builder() - .put(SqlTypeName.INTERVAL_SECOND, DurationFieldType.seconds()) - .put(SqlTypeName.INTERVAL_MINUTE, DurationFieldType.minutes()) - .put(SqlTypeName.INTERVAL_HOUR, DurationFieldType.hours()) - .put(SqlTypeName.INTERVAL_DAY, DurationFieldType.days()) - .put(SqlTypeName.INTERVAL_MONTH, DurationFieldType.months()) - .put(SqlTypeName.INTERVAL_YEAR, DurationFieldType.years()) - .build(); - - public BeamSqlDatetimeMinusExpression( - List operands, SqlTypeName intervalType) { -super(operands, SqlTypeName.BIGINT); -this.intervalType = intervalType; + .put(SqlTypeName.INTERVAL_SECOND, DurationFieldType.seconds()) + .put(SqlTypeName.INTERVAL_MINUTE, DurationFieldType.minutes()) + .put(SqlTypeName.INTERVAL_HOUR, DurationFieldType.hours()) + .put(SqlTypeName.INTERVAL_DAY, DurationFieldType.days()) + .put(SqlTypeName.INTERVAL_MONTH, DurationFieldType.months()) + .put(SqlTypeName.INTERVAL_YEAR, DurationFieldType.years()) +
[2/2] beam git commit: This closes #4082
This closes #4082 Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e0166ceb Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e0166ceb Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e0166ceb Branch: refs/heads/master Commit: e0166ceb657b393e4037ac7c178797a864379745 Parents: 29942939 0a63eb2 Author: James Xu Authored: Wed Nov 8 11:59:07 2017 +0800 Committer: James Xu Committed: Wed Nov 8 11:59:07 2017 +0800 -- .../date/BeamSqlDatetimeMinusExpression.java| 104 - ...BeamSqlTimestampMinusIntervalExpression.java | 83 +++ ...eamSqlTimestampMinusTimestampExpression.java | 97 .../BeamSqlDatetimeMinusExpressionTest.java | 189 --- ...SqlTimestampMinusIntervalExpressionTest.java | 163 + ...qlTimestampMinusTimestampExpressionTest.java | 233 +++ .../BeamSqlDateFunctionsIntegrationTest.java| 18 ++ 7 files changed, 691 insertions(+), 196 deletions(-) --
[1/2] beam git commit: [BEAM-2203] Implement TIMESTAMPDIFF()
Repository: beam Updated Branches: refs/heads/master 269bf8946 -> f80bf2fdf [BEAM-2203] Implement TIMESTAMPDIFF() Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4fe7480a Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4fe7480a Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4fe7480a Branch: refs/heads/master Commit: 4fe7480a749699e0d1418dbe0b29e6ecefd08497 Parents: 269bf89 Author: Anton Kedin Authored: Mon Oct 30 17:36:21 2017 -0700 Committer: James Xu Committed: Tue Nov 7 23:09:20 2017 +0800 -- sdks/java/extensions/sql/pom.xml| 8 + .../sql/impl/interpreter/BeamSqlFnExecutor.java | 9 +- .../operator/BeamSqlReinterpretExpression.java | 55 - .../date/BeamSqlDatetimeMinusExpression.java| 107 + .../BeamSqlReinterpretExpression.java | 58 + .../DatetimeReinterpretConversions.java | 59 + .../IntegerReinterpretConversions.java | 44 .../reinterpret/ReinterpretConversion.java | 114 + .../operator/reinterpret/Reinterpreter.java | 101 .../operator/reinterpret/package-info.java | 22 ++ .../impl/interpreter/BeamSqlFnExecutorTest.java | 14 ++ .../BeamSqlReinterpretExpressionTest.java | 116 +++--- .../BeamSqlDatetimeMinusExpressionTest.java | 232 +++ .../DatetimeReinterpretConversionsTest.java | 73 ++ .../IntegerReinterpretConversionsTest.java | 81 +++ .../reinterpret/ReinterpretConversionTest.java | 116 ++ .../operator/reinterpret/ReinterpreterTest.java | 155 + .../BeamSqlDateFunctionsIntegrationTest.java| 52 + 18 files changed, 1330 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/beam/blob/4fe7480a/sdks/java/extensions/sql/pom.xml -- diff --git a/sdks/java/extensions/sql/pom.xml b/sdks/java/extensions/sql/pom.xml index 742d3b6..34ff670 100644 --- a/sdks/java/extensions/sql/pom.xml +++ b/sdks/java/extensions/sql/pom.xml @@ -37,6 +37,7 @@ -MM-dd HH:mm 1.13.0 1.10.0 +1.9.5 @@ -271,5 +272,12 @@ hamcrest-all test + + + org.mockito + mockito-all + ${mockito.version} + test + http://git-wip-us.apache.org/repos/asf/beam/blob/4fe7480a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java -- diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java index 8770055..31d5022 100644 --- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java +++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java @@ -27,7 +27,6 @@ import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlCastE import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlInputRefExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive; -import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlReinterpretExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlUdfExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlWindowEndExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlWindowExpression; @@ -50,6 +49,7 @@ import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSql import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlCurrentTimestampExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlDateCeilExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlDateFloorExpression; +import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlDatetimeMinusExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlDatetimePlusExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlExtractExpression; import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlIntervalMultiplyExpression; @@ -79,6 +79,7 @@ import org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.math.BeamSql import org.apache.beam.s
[2/2] beam git commit: This closes #4065
This closes #4065 Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f80bf2fd Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f80bf2fd Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f80bf2fd Branch: refs/heads/master Commit: f80bf2fdff655b62deb458b1e57d867bfbf03f41 Parents: 269bf89 4fe7480 Author: James Xu Authored: Tue Nov 7 23:11:03 2017 +0800 Committer: James Xu Committed: Tue Nov 7 23:11:03 2017 +0800 -- sdks/java/extensions/sql/pom.xml| 8 + .../sql/impl/interpreter/BeamSqlFnExecutor.java | 9 +- .../operator/BeamSqlReinterpretExpression.java | 55 - .../date/BeamSqlDatetimeMinusExpression.java| 107 + .../BeamSqlReinterpretExpression.java | 58 + .../DatetimeReinterpretConversions.java | 59 + .../IntegerReinterpretConversions.java | 44 .../reinterpret/ReinterpretConversion.java | 114 + .../operator/reinterpret/Reinterpreter.java | 101 .../operator/reinterpret/package-info.java | 22 ++ .../impl/interpreter/BeamSqlFnExecutorTest.java | 14 ++ .../BeamSqlReinterpretExpressionTest.java | 116 +++--- .../BeamSqlDatetimeMinusExpressionTest.java | 232 +++ .../DatetimeReinterpretConversionsTest.java | 73 ++ .../IntegerReinterpretConversionsTest.java | 81 +++ .../reinterpret/ReinterpretConversionTest.java | 116 ++ .../operator/reinterpret/ReinterpreterTest.java | 155 + .../BeamSqlDateFunctionsIntegrationTest.java| 52 + 18 files changed, 1330 insertions(+), 86 deletions(-) --
[GitHub] beam pull request #4041: [BEAM-2528] BeamSQL DDL :: CreateTable
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/4041 [BEAM-2528] BeamSQL DDL :: CreateTable I started this PR as an initial attempt to implement the BeamSQL `create table` statement. The implementation might not be so mature, but I hope this could be a place we can discuss deeper about the create table. I will introduce this PR in the following 3 aspects: * MetaStore * TableProvider * Grammar ## MetaStore Metastore is responsible for handling the CRUD of table during a session. e.g. create a table, query all tables, query a table by the specified name etc. When a table is created, the table meta info can be persisted by the metastore, but the default `InMemoryMetaStore` will only store the meta info in memory, so it will NOT be persisted, but user can implement the `MetaStore` interface to make a persistent implementation. The table names in MetaStore need to be unique. ## TableProvider The tables in MetaStore can come from many different sources, the construction of a usable table is the responsibility of a `TableProvider`, TableProvider have the similar interface like `MetaStore`, but it only handles a specific type of table, e.g. `TextTableProvider` only handle text tables, while `KafakaTableProvider` only handle kafka tables. In this PR, only `TextTableProvider` and `KafakaTableProvider` are implemented as example. ## Grammar The grammar for create a TEXT table is: ```sql CREATE TABLE ORDERS( ID INT PRIMARY KEY COMMENT 'this is the primary key', NAME VARCHAR(127) COMMENT 'this is the name' ) COMMENT 'this is the table orders' LOCATION 'text://home/admin/orders' TBLPROPERTIES '{"format": "Excel"}' ``` `LOCATION` dictates where the data of the table is stored. The scheme of the LOCATION dictate the table type, e.g. in the above example, the table type is `text`, using the table type we can find the corresponding `TextTableProvider` using the ServiceLoader merchanism. `TBLPROPERTIES` is used to specify some other properties of the table, in the above example, we specified the format of each line of text file: `Excel`(one variant of CSV format). The grammar for create a KAFKA table is: ```sql CREATE TABLE ORDERS( ID INT PRIMARY KEY COMMENT 'this is the primary key', NAME VARCHAR(127) COMMENT 'this is the name' ) COMMENT 'this is the table orders' LOCATION 'kafka://localhost:2181/brokers?topic=test' TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", "topic2"]}' ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2528-create-table-from-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/4041.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4041 commit ab5ee759116c7e8139d7d17aca631f018a48fb40 Author: James Xu Date: 2017-09-13T12:36:37Z [BEAM-2528] create table commit 813b9a7a5e7232d3028bc6b859d96c0d856c1517 Author: James Xu Date: 2017-10-26T12:17:09Z minor ---
[GitHub] beam pull request #3481: [BEAM-2528] BeamSql: DDL: create table
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3481 ---
[GitHub] beam pull request #3708: [BEAM-2740] introduce the new InnerBeamSqlEnv/BeamS...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3708 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3702: [BEAM-2744] rename BeamRecordType#size()
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3702 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3700: [BEAM-2745] add BeamRecordSqlType.getFieldTypeByInd...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3700 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3708: [BEAM-2740] introduce the new InnerBeamSqlEnv/BeamS...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3708 [BEAM-2740] introduce the new InnerBeamSqlEnv/BeamSqlEnv Summary: * Simplified `BeamSqlEnv`: now you can only register UDF/UDAF in `BeamSqlEnv`, and `BeamSqlEnv` just record the udfName -> udfClass mapping, actually work will be done by `InnerBeamSqlEnv`. * `BeamSqlEnv` is only used by the surface API classes. * `InnerBeamSqlEnv` is actually the old `BeamSqlEnv`, it holds schema and planner. * `InnerBeamSqlEnv` is only used by the implementation classes. * `BeamSqlEnv` is converted to `InnerBeamSqlEnv` automatically in `BeamSql` and `BeamSqlCli`. * `BeamSqlEnv` and `InnerBeamSqlEnv` are quite different, so can not share the same interface. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2740 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3708.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3708 commit 80491a26456e1a4a1ffc0e4e00290de63b9617e4 Author: James Xu Date: 2017-08-09T10:54:54Z [BEAM-2740] introduce InnerBeamSqlEnv to extract implementation related logic from BeamSqlEnv commit 5074f830347e07ef0ba8396bc9a464a37eb089ec Author: James Xu Date: 2017-08-09T11:04:46Z add licence declaration for BeamRelTest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3699: [BEAM-2742] change Field type from primitives to bo...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3699 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3704: [BEAM-2733] update javadoc for BeamSql
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3704 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3704: [BEAM-2733] update javadoc for BeamSql
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3704 [BEAM-2733] update javadoc for BeamSql Summary: * `Exception` declaration are removed from `BeamSql.query ()` * Added javadoc to explain when the sql query use a non-exist tableãa subset of tables from upstream, what happens. * Added javadoc for BeamSql.query and QueryTransform to describe the lifetime of tables in PCollectionTuple. * SimpleQueryTransform and QueryTransform are actually user-facing (they are return types of query() and simpleQuery()), so they are kept `public`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2733-BeamSql Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3704.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3704 commit df4d24f2104b61b65bbfc027d005415d4282c992 Author: James Xu Date: 2017-08-08T11:52:53Z [BEAM-2733] update javadoc for BeamSql --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3702: [BEAM-2744] rename BeamRecordType#size()
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3702 [BEAM-2744] rename BeamRecordType#size() Summary: * BeamRecord.size() -> BeamRecord.getFieldCount() * BeamRecordType.size() -> BeamRecordType.getFieldCount() * remove BeamRecord.valueInString() You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2744-rename-BeamRecordType-size Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3702.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3702 commit 6466a9fcfa0dcc4c73d05a98ffd8727584d190b4 Author: James Xu Date: 2017-08-08T07:15:59Z [BEAM-2744] rename BeamRecordType#size() commit ff10e1a5a64e9b7c607b965a9b5bc318465f1995 Author: James Xu Date: 2017-08-08T07:27:48Z remove BeamRecord#valueInString() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3700: [BEAM-2745] add BeamRecordSqlType.getFieldTypeByInd...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3700 [BEAM-2745] add BeamRecordSqlType.getFieldTypeByIndex() Summary: * BeamSqlRecordType -> BeamRecordSqlType * fieldsName -> fieldNames * fieldsType -> fieldTypes * add getFieldTypeByIndex() You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2745-add-getFieldTypeByIndex Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3700.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3700 commit cec323b1ddde3185082794dd420904eb89374bdb Author: James Xu Date: 2017-08-08T06:53:18Z [BEAM-2745] add BeamRecordSqlType.getFieldTypeByIndex() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3699: [BEAM-2742] change Field type from primitives to bo...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3699 [BEAM-2742] change Field type from primitives to boxed types change Field type of `BeamRecord` from primitive types to boxed types, and removed `isNull` method. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2742-switch-to-boxed-types Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3699.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3699 commit 189959a65f5fc1f7835dcd2a99a76c7717860e07 Author: James Xu Date: 2017-08-08T06:26:09Z [BEAM-2742] change Field type from primitives to boxed types --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3666: [BEAM-2441] move dsls/sql to sdks/java/extensions/s...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3666 [BEAM-2441] move dsls/sql to sdks/java/extensions/sql R: @XuMingmin @takidau You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2441 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3666.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3666 commit ba493f85a5a18665cd6ae4eb61e0a86fab1a6c07 Author: James Xu Date: 2017-07-30T15:39:37Z move dsls/sql to sdks/java/extensions/sql commit c1b5482d3fb13a14926f0ffc23d0810b3105ed24 Author: James Xu Date: 2017-07-30T15:58:02Z rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql commit febd044ae306a28fa3797a1663e54c1d7fbe43ce Author: James Xu Date: 2017-07-31T09:11:53Z move all implementation classes/packages into impl package --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3583: [BEAM-2613] add integration test for comparison ope...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3583 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3571: [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3571 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3570: [BEAM-2565] add integration test for conditional fu...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3570 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3547: [BEAM-2561] add integration test for date functions
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3547 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3583: [BEAM-2613] add integration test for comparison ope...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3583 [BEAM-2613] add integration test for comparison operators Summary: 1. added integration test for comparison operators. 2. refactored comparison operator expressions into a separate package: `comparison`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2613-integration-test-for-comparison-operators Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3583.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3583 commit f5044071c0ff0e909f2bd24ab3a8ea5921161c4b Author: James Xu Date: 2017-07-13T10:24:55Z [BEAM-2613] add integration test for comparison operators --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3571: [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3571 [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType @XuMingmin @jbonofre @takidau You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2621-rename-BeamSqlRecordType Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3571.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3571 commit 54b939445839a4b2b81d01cb50f75d76ee0fc175 Author: James Xu Date: 2017-07-17T09:55:56Z BeamSqlRecordType -> BeamSqlRowType --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3570: [BEAM-2565] add integration test for conditional fu...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3570 [BEAM-2565] add integration test for conditional functions @XuMingmin @jbonofre @takidau You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2565-integration-test-for-case-operator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3570.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3570 commit 04f50114960266ec194a75b5d2c4ded53259367f Author: James Xu Date: 2017-07-17T09:34:05Z [BEAM-2565] add integration test for conditional functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3557: [BEAM-2560] Add integration test for arithmetic ope...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3557 [BEAM-2560] Add integration test for arithmetic operators. Re-implemented the arithmetic operators & refactored string functions integration test to utilize `ExpressionChecker`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2560-integration-test-for-arithmetic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3557.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3557 commit 715615438ac72b2d4bf154e2816a65d2ccdd46ea Author: James Xu Date: 2017-07-07T03:04:46Z [BEAM-2560] Add integration test for arithmetic operators. And also refactor BeamSqlStringFunctionsIntegrationTest to use ExpressionChecker --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3532: [BEAM-2564] add integration test for string functio...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3532 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3506: [BEAM-2550] add UnitTest for JOIN in DSL
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3506 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3547: [BEAM-2561] add integration test for date functions
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3547 [BEAM-2561] add integration test for date functions R: @XuMingmin @takidau there are some duplications with PR #3532 , will be some refactor after #3532 is merged. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2561-integration-test-for-date-functions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3547.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3547 commit 3a0b1f57ddca7a08c9ee7ee3a6ea0a760b9a0dc0 Author: James Xu Date: 2017-07-11T09:49:04Z [BEAM-2561] add integration test for date functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3532: [BEAM-2564] add integration test for string functio...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3532 [BEAM-2564] add integration test for string functions Add integration test for string functions and fixed a bug in `BeamSqlTrimExpression`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2564-integration-test-for-string-operators Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3532.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3532 commit 709768f98fa1bd4daa3080bfe1aaec31b78f Author: James Xu Date: 2017-07-10T11:58:21Z [BEAM-2564] add integration test for string functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3506: [BEAM-2550] add UnitTest for JOIN in DSL
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3506 [BEAM-2550] add UnitTest for JOIN in DSL @XuMingmin @takidau I have only tested `bounded vs bounded` join scenario, since we already have enumerated every cases like `bounded vs bounded`, `unbounded vs unbounded` in tests for `BeamJoinRel`, test here is mainly to test the integration of JOIN with the DSL methods. Also I kinda fixed a related bug in `BeamIOSourceRel` -- which have not setCoder for the rows. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2550-test-join-in-dsl Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3506.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3506 commit 5885d1cad12d28c453603e12c8a11f369d1d6d65 Author: James Xu Date: 2017-07-06T03:29:41Z [BEAM-2550] add UnitTest for JOIN in DSL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3478: [BEAM-2515] BeamSql: refactor the MockedBeamSqlTabl...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3478 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3481: [BEAM-2528] BeamSql: DDL: create table
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3481 [BEAM-2528] BeamSql: DDL: create table I started this PR as an initial attempt to implement the `create table` statement. The implementation might not be so mature, but I hope this could be a place we can discuss deeper about the create table. I will introduce this PR in the following 3 aspects: * MetaStore * TableProvider * Grammar ## MetaStore Metastore is responsible for handling the CRUD of table during a session. e.g. create a table, query all tables, query a table by the specified name etc. When a table is created, the table meta info can be persisted by the metastore, but the default `InMemoryMetaStore` will only store the meta info in memory, so it will NOT be persisted, but user can implement the `MetaStore` interface to make a persistent implementation. The table names in MetaStore need to be unique. ## TableProvider The tables in MetaStore can come from many different sources, the construction of a usable table is the responsibility of a `TableProvider`, TableProvider have the similar interface like `MetaStore`, but it only handles a specific type of table, e.g. `TextTableProvider` only handle text tables, while `KafakaTableProvider` only handle kafka tables. ## Grammar The grammar for create table is: ```sql CREATE TABLE ORDERS( ID INT PRIMARY KEY COMMENT 'this is the primary key', NAME VARCHAR(127) COMMENT 'this is the name' ) COMMENT 'this is the table orders' LOCATION 'text://home/admin/orders' TBLPROPERTIES '{"format": "Excel"}' ``` `LOCATION` dictates where the data of the table is stored. The scheme of the LOCATION dictate the table type, e.g. in the above example, the table type is `text`, using the table type we can find the corresponding `TextTableProvider` using the ServiceLoader merchanism. `TBLPROPERTIES` is used to specify some other properties of the table, in the above example, we specified the format of each line of text file: `Excel`(one variant of CSV format). You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2528-create-table Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3481.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3481 commit 9bbb08f5bdabbb903491c9ec0bfc32145632d63b Author: James Xu Date: 2017-06-17T11:30:03Z [BEAM-2528] BeamSql: DDL: create table --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3478: [BEAM-2515] BeamSql: refactor the MockedBeamSqlTabl...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3478 [BEAM-2515] BeamSql: refactor the MockedBeamSqlTable and related tests Summary: * rename `MockedBeamSqlTable` to `MockedBoundedTable` * Use the `RowsBuilder` to build rows in Unit Test. * use `PAssert` rather than `assertEquals` when test PCollections. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2515-refactor-mock Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3478.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3478 commit 1c03838b88b45e2bb32ddf0a8bfe022de68cafc0 Author: James Xu Date: 2017-06-27T02:42:40Z use PAssert rather than assertEquals commit 5b48fc288127429355cc727cadbc657cba719320 Author: James Xu Date: 2017-06-27T09:08:39Z minor commit 40b8962e2f5a66d279801f8d4604c442634d902a Author: James Xu Date: 2017-06-30T06:54:26Z MockedBeamSqlTable -> MockedBoundedTable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3277: [BEAM-2193] implement join
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3277 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3377: [BEAM-2443] apply AutoValue to BeamSqlRecordType
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3377 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3377: [BEAM-2443] apply AutoValue to BeamSqlRecordType
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3377 [BEAM-2443] apply AutoValue to BeamSqlRecordType You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2443-autovalues Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3377.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3377 commit f6343235e2e769a9cee80b0b844db8b8761ed884 Author: James Xu Date: 2017-06-16T06:31:55Z [BEAM-2443] apply AutoValue to BeamSqlRecordType --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3357: [BEAM-2442] BeamSql api surface test
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3357 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3365: [BEAM-2444] BeamSql: use java standard exception
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3365 [BEAM-2444] BeamSql: use java standard exception R: @XuMingmin R: @takidau You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2444-use-java-standard-exception Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3365.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3365 commit 83e6c2046571c14d104e622b2e2ad656143e09aa Author: James Xu Date: 2017-06-15T02:57:10Z [BEAM-2444] BeamSql: use java standard exception --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3364: [BEAM-2440] BeamSql: reduce visibility
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3364 [BEAM-2440] BeamSql: reduce visibility Since the packages are cross-import each other, many of them can't be set to `package private` access level. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2440-reduce-visibility Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3364.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3364 commit 3625dbd9e00db28d2175c6ac000792fd5c5d8f65 Author: Jean-Baptiste Onofré Date: 2016-08-09T09:17:32Z Add DSLs module commit 7867ce62e43bef7bfd8011c605379df05494dfcf Author: mingmxu Date: 2017-04-10T02:49:08Z [BEAM-301] Initial skeleton for Beam SQL commit f1c2b6540652156ad7c51ec89a933aacb8c6dab6 Author: Jean-Baptiste Onofré Date: 2017-04-12T20:00:53Z [BEAM-301] This closes #2479 commit 529bc9d9375ae162d70de274975c547a878053a3 Author: mingmxu Date: 2017-04-13T21:46:28Z checkstyle and rename package correct package from org.beam.dsls.sql to org.apache.beam.dsls.sql update with checkstyle commit ca8760373b37b540219301f1e719633956a787dc Author: Davor Bonaci Date: 2017-04-18T21:40:40Z This closes #2530 commit aa07a1d41b71c810c4968bd35e5ca69dc56b8c30 Author: Dan Halperin Date: 2017-04-19T19:14:05Z This closes #2584 commit 464cc275952305bf51ecfdf056e784441c9c2272 Author: mingmxu Date: 2017-04-23T05:20:25Z redesign BeamSqlExpression to execute Calcite SQL expression. Changes: 1. revert BEAM dependency to 0.6.0 to avoid impact of changes in master branch; 2. updates as discussion during review; refine BeamSqlRowCoder commit a6845a35fc2a4091dcef24f231c6d3a418561e6f Author: Davor Bonaci Date: 2017-05-04T07:12:55Z This closes #2656 commit cf95571d90329287b92946065c93724780f31685 Author: Davor Bonaci Date: 2017-05-04T07:16:34Z This closes #2878 commit dd70852f6fa811a410938e5baa51cb8e602c931b Author: James Xu Date: 2017-05-04T03:44:14Z Fix inconsistent mapping for SQL FLOAT commit f7ee8d33e13506a17314720c8c23fff486bff76c Author: Davor Bonaci Date: 2017-05-07T02:24:04Z This closes #2886 commit a376f37b908828fa0d44b5d71822d51b9c47f971 Author: James Xu Date: 2017-05-04T14:15:02Z [BEAM-2158] Implement the arithmetic operators commit 4ea38d82359ed443c906de9caed8488d1dbf199a Author: Jean-Baptiste Onofré Date: 2017-05-09T08:24:20Z [BEAM-2158] This closes #2893 commit b12c7b49b8eab7b208f2701c412123635813ee28 Author: James Xu Date: 2017-05-05T17:20:54Z [BEAM-2161] Add support for String operators commit 3e678a75a746c9e40fdbe6599d20969718c34882 Author: Jean-Baptiste Onofré Date: 2017-05-09T12:17:53Z [BEAM-2161] This closes #2919 commit 59abc7640866f58427817c026af15df30947f3a6 Author: James Xu Date: 2017-05-04T03:44:14Z [BEAM-2079] Support TextIO as SQL source/sink commit 7e918a76e409e5c849f2878702746935f4d7bf90 Author: Jean-Baptiste Onofré Date: 2017-05-10T05:24:40Z [BEAM-2079] This closes #2942 commit 3c6d60fb2f473b1860cfeca0f740ec817499d085 Author: mingmxu Date: 2017-05-10T06:04:17Z [BEAM-2006] window support Add support for aggregation: global, HOP, TUMBLE, SESSION, only aggregation function COUNT fix typo commit 6729a027daea04d0ad38f8a83ad1baafa465169f Author: Dan Halperin Date: 2017-05-10T17:03:17Z This closes #3039 commit f728fbe5c7153341ace046fa8b2465ef8844be1b Author: mingmxu Date: 2017-05-10T23:38:13Z Support common-used aggregation functions in SQL, including: COUNT,SUM,AVG,MAX,MIN rename BeamAggregationTransform to BeamAggregationTransforms update comments commit 523482be0501a7bce79087f47c7752b900178a00 Author: Dan Halperin Date: 2017-05-13T00:47:06Z This closes #3067 commit 12dd804678bfaad4705c8f1d50eaf03a086f6daf Author: James Xu Date: 2017-05-10T09:19:48Z [BEAM-2195] Implement conditional operator (CASE) commit 8bb59840be32ef945cfb7151ab7f8368fabb19f0 Author: Jean-Baptiste Onofré Date: 2017-05-13T06:09:08Z [BEAM-2195] This closes #3042 commit d74a5c34f0732825c052d4c32096e5b91096a72c Author: mingmxu Date: 2017-05-13T04:14:29Z [BEAM-2234] Change return type of buildBeamPipeline to `PCollection` commit 868bcbdad30a2232889d7d439b5190862ae48421 Author: Jean-Baptiste Onofré Date: 2017-05-14T05:52:39Z [BEAM-2234] This closes #3135 commit 8044e59e1bfa175e9d686a4c332be85be22850f1 Author: James Xu Date: 2017-05-12T17:07:01Z [BEAM-2149] Improved kafka table implemention. 1. use robust CSV library to parse & print. 2. support different data types rather than just `String`. 3. a little cleanup for TextTable (to extra
[GitHub] beam pull request #3218: [BEAM-2356] add NOT operator & some refactoring
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3218 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3357: [BEAM-2442] BeamSql api surface test
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3357 [BEAM-2442] BeamSql api surface test Summary: * The surface api of BeamSql includes the following: * BeamSql * BeamSqlCli * BeamSqlEnv * All the classes in package `org.apache.beam.dsls.sql.schema` * Now `BeamSql` and `BeamSqlCli` both extends from `BeamSqlEnv` to avoid expose `SchemaPlus` and `BeamQueryPlanner`. * Calcite realted methods are encapsulated into `CalciteUtils`(which is not part of surface api) to avoid exposure. * Created a new `BeamSqlTable` which abstracts the beam `table` concept. * `RelDataType`, `RelProtoDataType` are all removed from surface api, `BeamSqlRecordType` is the only class which represents the schema of a table. * java.sql.Types is used to represent sql type instead of Calcite `SqlTypeName`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2442-api-surface-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3357.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3357 commit 507e0451584d20bd7d2fe59a5ee410cc8959f5d8 Author: James Xu Date: 2017-06-14T15:47:10Z [BEAM-2442] BeamSql surface api test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3183: [BEAM-2325] support Set operator: intersect, except...
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3183 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3207: [BEAM-2247] implement date functions
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3207 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3254: [BEAM-2375] upgrade beam-version to 2.0.0
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3254 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3277: [BEAM-2193] implement join
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3277 [BEAM-2193] implement join implement join You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2193-join Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3277.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3277 commit 4313eede78a541b5739f2bdd1588776213dfcf9f Author: James Xu Date: 2017-05-29T03:11:34Z [BEAM-2375] upgrade beam-version to 2.0.0 commit 23750b46f32b3d854f75d7b8a3bdfcb8a44887bd Author: James Xu Date: 2017-05-29T13:59:58Z [BEAM-2193] implement join --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3267: [BEAM-2378] support FULL OUTER JOIN
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3267 [BEAM-2378] support FULL OUTER JOIN @davorbonaci can you take a look at this please? You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2378-full-outer-join Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3267.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3267 commit 86555d68b86f7b62cea8523e1a97aa99b677d7eb Author: James Xu Date: 2017-05-31T02:28:55Z [BEAM-2378] support FULL OUTER JOIN --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3257: fix javadoc of View
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3257 fix javadoc of View While reading javadoc of `org.apache.beam.sdk.transforms.View`, find the sample code of `View` seems not correct, hope the PR will fix it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam fix-javadoc-of-View Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3257.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3257 commit 4cef2cbc595af1bbd68255d5404fabf8f57c699b Author: James Xu Date: 2017-05-30T03:27:32Z fix javadoc of View --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3254: [BEAM-2375] upgrade beam-version to 2.0.0
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3254 [BEAM-2375] upgrade beam-version to 2.0.0 Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam DSL_SQL Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3254.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3254 commit 4313eede78a541b5739f2bdd1588776213dfcf9f Author: James Xu Date: 2017-05-29T03:11:34Z [BEAM-2375] upgrade beam-version to 2.0.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3253: rebase master for feature branch DSL_SQL
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3253 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3248: rebasing DSL_SQL branch from master
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3248 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3253: rebase master for feature branch DSL_SQL
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3253 rebase master for feature branch DSL_SQL Since it is just a rebase, so didn't create a dedicated issue for it. And since it is a rebase, it cann't be merged automatically, need to merge manually with `--force`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam DSL_SQL_3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3253.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3253 commit 42066431b45ced1931787650d2210be724fb7f5c Author: Eugene Kirpichov Date: 2017-05-12T17:57:03Z This closes #3113 commit 6a90b9d164d98d916d0e0af2be39a740ce769076 Author: Aviem Zur Date: 2017-05-12T13:02:54Z [BEAM-2277] Fix URI_SCHEME_PATTERN in FileSystems commit 0acf3620d90a970f4024ae95b909c3bb4ef827ad Author: Dan Halperin Date: 2017-05-12T18:10:48Z This closes #3115 commit abe62feb16f6eb59e294c37abf408f84c9280616 Author: Aviem Zur Date: 2017-05-12T19:18:02Z [BEAM-2279] Add HDFS support to Spark runner profiles in archetypes and examples commit c2a36284b510461942205b627a019dfbe83f8b37 Author: Davor Bonaci Date: 2017-05-12T21:54:00Z This closes #3126 commit a6a5ff7be387ef295fc7f921de36a3ea77327bc1 Author: Dan Halperin Date: 2017-05-12T16:20:34Z [BEAM-2277] Add ResourceIdTester and test existing ResourceId implementations A first cut at some of the parts of the ResourceId spec. commit 15df211c758e7c8f05c3136f25bbe18e3f394321 Author: Dan Halperin Date: 2017-05-12T18:11:06Z [BEAM-2277] HadoopFileSystem: normalize implementation * Drop empty authority always * Resolve directories commit fbb0de129d881b6b6a1ae37fc8d75075c9d8df86 Author: Dan Halperin Date: 2017-05-12T18:13:00Z Remove '/' entirely from determining FileSystem scheme commit ec956c85efa16d00c5e218ee1257b8ee62a2013d Author: Dan Halperin Date: 2017-05-12T18:38:54Z Mark FileSystem and related as Experimental commit f3540d47f10c18859340a738a7e93643ee57f604 Author: Dan Halperin Date: 2017-05-12T18:46:16Z Rename FileSystems.setDefaultConfigInWorkers And document that it's not for users. commit 3921163829d2a99b0524bf3fa1b85a4cde826f3a Author: Dan Halperin Date: 2017-05-12T20:57:59Z Fix shading of guava testlib commit 9f81fd299bd32e0d6056a7da9fa994cf74db0ed9 Author: Dan Halperin Date: 2017-05-12T21:59:14Z This closes #3121 commit f53ceecfb18a52aeeab8f1869c189e4aa2f56d35 Author: Ahmet Altay Date: 2017-05-12T21:54:33Z internal comments commit 89289e498e71d6e9fbd24702032471fa13fbd46c Author: Ahmet Altay Date: 2017-05-12T22:08:16Z This closes #3128 commit 7bade36eb18a48e01a8401f97f10a70a492172c4 Author: Dan Halperin Date: 2017-05-12T22:52:51Z [BEAM-2279] Fix archetype breakages * Add version for hadoop-file-system dependency. * Add DirectRunner in test scope in java8, otherwise mvn package fails when another runner profile is enabled. commit 9a43da74a9005507acdc80524edc634c8318674d Author: Dan Halperin Date: 2017-05-12T22:57:21Z This closes #3132 commit 504bd6a8036946a796a7290853e45bf20d32d07d Author: Rune Fevang Date: 2017-05-12T20:24:32Z [BEAM-2290] Fix issue where timestamps weren't set when using CompressedSource commit d4df9079039d57e37cdb1666c8c3120f2fa2909a Author: Dan Halperin Date: 2017-05-15T16:52:38Z This closes #3131 commit 8060d66083973cdab91f68085652ab68e3dee596 Author: Thomas Groh Date: 2017-05-15T17:00:45Z Reduce Log Level of PubsubUnboundedSource commit 6e8d8d65708e3d5b0525a7b7f97597d4af71d51e Author: Thomas Groh Date: 2017-05-15T17:07:14Z This closes #3145 commit de1adefda958ae152eae4b7b94b3b78563e130aa Author: Neville Li Date: 2017-05-15T18:55:09Z readAvros should't have proto Message upper bound commit 88ec00860032078dcd33e07d94642912f6aa7bf7 Author: Eugene Kirpichov Date: 2017-05-15T19:22:27Z This closes #3149 commit 15ec2cd83d4291957a75d9f2ac838ff867dcbe0c Author: Dan Halperin Date: 2017-05-15T17:57:24Z Fix GcsResourceIdTest in postcommits This test is not a @ValidatesRunner test, so it should not use TestPipelineOptions. commit 926157d42db572dacb9142665d5c922dcad589f0 Author: Luke Cwik Date: 2017-05-15T20:16:05Z Fix GcsResourceIdTest in postcommits This closes #3147 commit 911128b287909e2e2240e0674223df7c66e952f8 Author: Ahmet Altay Date: 2017-05-15T16:59:29Z Moving the data file for trigger tests to testing/data commit 9b5a8e62ca695b0d0def27844084237102f0055a Author: Ahmet Altay Date: 2017-05-15T20:34:09Z This closes #3144 commit 96906aee9d0cdbb9fae67dd5bd2af0710a1b67bf Author: Charles
[GitHub] beam pull request #3244: DSL_SQL: merge master into DSL_SQL feature branch
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3244 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3248: rebasing DSL_SQL branch from master
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3248 rebasing DSL_SQL branch from master first time do this(rebase feature branch from master), not sure whether it is the right way. @jbonofre can you take a look at it? You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam DSL_SQL Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3248.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3248 commit e8d829817e3f7e07884a91737b61bfa5133d5724 Author: Borisa Zivkovic Date: 2017-05-12T07:44:20Z BigtableIO should use AutoValue for read and write commit 42066431b45ced1931787650d2210be724fb7f5c Author: Eugene Kirpichov Date: 2017-05-12T17:57:03Z This closes #3113 commit 6a90b9d164d98d916d0e0af2be39a740ce769076 Author: Aviem Zur Date: 2017-05-12T13:02:54Z [BEAM-2277] Fix URI_SCHEME_PATTERN in FileSystems commit 0acf3620d90a970f4024ae95b909c3bb4ef827ad Author: Dan Halperin Date: 2017-05-12T18:10:48Z This closes #3115 commit abe62feb16f6eb59e294c37abf408f84c9280616 Author: Aviem Zur Date: 2017-05-12T19:18:02Z [BEAM-2279] Add HDFS support to Spark runner profiles in archetypes and examples commit c2a36284b510461942205b627a019dfbe83f8b37 Author: Davor Bonaci Date: 2017-05-12T21:54:00Z This closes #3126 commit a6a5ff7be387ef295fc7f921de36a3ea77327bc1 Author: Dan Halperin Date: 2017-05-12T16:20:34Z [BEAM-2277] Add ResourceIdTester and test existing ResourceId implementations A first cut at some of the parts of the ResourceId spec. commit 15df211c758e7c8f05c3136f25bbe18e3f394321 Author: Dan Halperin Date: 2017-05-12T18:11:06Z [BEAM-2277] HadoopFileSystem: normalize implementation * Drop empty authority always * Resolve directories commit fbb0de129d881b6b6a1ae37fc8d75075c9d8df86 Author: Dan Halperin Date: 2017-05-12T18:13:00Z Remove '/' entirely from determining FileSystem scheme commit ec956c85efa16d00c5e218ee1257b8ee62a2013d Author: Dan Halperin Date: 2017-05-12T18:38:54Z Mark FileSystem and related as Experimental commit f3540d47f10c18859340a738a7e93643ee57f604 Author: Dan Halperin Date: 2017-05-12T18:46:16Z Rename FileSystems.setDefaultConfigInWorkers And document that it's not for users. commit 3921163829d2a99b0524bf3fa1b85a4cde826f3a Author: Dan Halperin Date: 2017-05-12T20:57:59Z Fix shading of guava testlib commit 9f81fd299bd32e0d6056a7da9fa994cf74db0ed9 Author: Dan Halperin Date: 2017-05-12T21:59:14Z This closes #3121 commit f53ceecfb18a52aeeab8f1869c189e4aa2f56d35 Author: Ahmet Altay Date: 2017-05-12T21:54:33Z internal comments commit 89289e498e71d6e9fbd24702032471fa13fbd46c Author: Ahmet Altay Date: 2017-05-12T22:08:16Z This closes #3128 commit 7bade36eb18a48e01a8401f97f10a70a492172c4 Author: Dan Halperin Date: 2017-05-12T22:52:51Z [BEAM-2279] Fix archetype breakages * Add version for hadoop-file-system dependency. * Add DirectRunner in test scope in java8, otherwise mvn package fails when another runner profile is enabled. commit 9a43da74a9005507acdc80524edc634c8318674d Author: Dan Halperin Date: 2017-05-12T22:57:21Z This closes #3132 commit 504bd6a8036946a796a7290853e45bf20d32d07d Author: Rune Fevang Date: 2017-05-12T20:24:32Z [BEAM-2290] Fix issue where timestamps weren't set when using CompressedSource commit d4df9079039d57e37cdb1666c8c3120f2fa2909a Author: Dan Halperin Date: 2017-05-15T16:52:38Z This closes #3131 commit 8060d66083973cdab91f68085652ab68e3dee596 Author: Thomas Groh Date: 2017-05-15T17:00:45Z Reduce Log Level of PubsubUnboundedSource commit 6e8d8d65708e3d5b0525a7b7f97597d4af71d51e Author: Thomas Groh Date: 2017-05-15T17:07:14Z This closes #3145 commit de1adefda958ae152eae4b7b94b3b78563e130aa Author: Neville Li Date: 2017-05-15T18:55:09Z readAvros should't have proto Message upper bound commit 88ec00860032078dcd33e07d94642912f6aa7bf7 Author: Eugene Kirpichov Date: 2017-05-15T19:22:27Z This closes #3149 commit 15ec2cd83d4291957a75d9f2ac838ff867dcbe0c Author: Dan Halperin Date: 2017-05-15T17:57:24Z Fix GcsResourceIdTest in postcommits This test is not a @ValidatesRunner test, so it should not use TestPipelineOptions. commit 926157d42db572dacb9142665d5c922dcad589f0 Author: Luke Cwik Date: 2017-05-15T20:16:05Z Fix GcsResourceIdTest in postcommits This closes #3147 commit 911128b287909e2e2240e0674223df7c66e952f8 Author: Ahmet Altay Date: 2017-05-15T16:59:29Z Moving the data file for trigger tests to testing/data commit 9b5a8e62ca695b0d0def27844084237102f0055a Author: Ahmet Altay Date: 2017-05-15T2
[GitHub] beam pull request #3247: rebasing test
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3247 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3247: rebasing test
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3247 rebasing test Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam DSL_SQL Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3247.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3247 commit 42066431b45ced1931787650d2210be724fb7f5c Author: Eugene Kirpichov Date: 2017-05-12T17:57:03Z This closes #3113 commit 6a90b9d164d98d916d0e0af2be39a740ce769076 Author: Aviem Zur Date: 2017-05-12T13:02:54Z [BEAM-2277] Fix URI_SCHEME_PATTERN in FileSystems commit 0acf3620d90a970f4024ae95b909c3bb4ef827ad Author: Dan Halperin Date: 2017-05-12T18:10:48Z This closes #3115 commit abe62feb16f6eb59e294c37abf408f84c9280616 Author: Aviem Zur Date: 2017-05-12T19:18:02Z [BEAM-2279] Add HDFS support to Spark runner profiles in archetypes and examples commit c2a36284b510461942205b627a019dfbe83f8b37 Author: Davor Bonaci Date: 2017-05-12T21:54:00Z This closes #3126 commit a6a5ff7be387ef295fc7f921de36a3ea77327bc1 Author: Dan Halperin Date: 2017-05-12T16:20:34Z [BEAM-2277] Add ResourceIdTester and test existing ResourceId implementations A first cut at some of the parts of the ResourceId spec. commit 15df211c758e7c8f05c3136f25bbe18e3f394321 Author: Dan Halperin Date: 2017-05-12T18:11:06Z [BEAM-2277] HadoopFileSystem: normalize implementation * Drop empty authority always * Resolve directories commit fbb0de129d881b6b6a1ae37fc8d75075c9d8df86 Author: Dan Halperin Date: 2017-05-12T18:13:00Z Remove '/' entirely from determining FileSystem scheme commit ec956c85efa16d00c5e218ee1257b8ee62a2013d Author: Dan Halperin Date: 2017-05-12T18:38:54Z Mark FileSystem and related as Experimental commit f3540d47f10c18859340a738a7e93643ee57f604 Author: Dan Halperin Date: 2017-05-12T18:46:16Z Rename FileSystems.setDefaultConfigInWorkers And document that it's not for users. commit 3921163829d2a99b0524bf3fa1b85a4cde826f3a Author: Dan Halperin Date: 2017-05-12T20:57:59Z Fix shading of guava testlib commit 9f81fd299bd32e0d6056a7da9fa994cf74db0ed9 Author: Dan Halperin Date: 2017-05-12T21:59:14Z This closes #3121 commit f53ceecfb18a52aeeab8f1869c189e4aa2f56d35 Author: Ahmet Altay Date: 2017-05-12T21:54:33Z internal comments commit 89289e498e71d6e9fbd24702032471fa13fbd46c Author: Ahmet Altay Date: 2017-05-12T22:08:16Z This closes #3128 commit 7bade36eb18a48e01a8401f97f10a70a492172c4 Author: Dan Halperin Date: 2017-05-12T22:52:51Z [BEAM-2279] Fix archetype breakages * Add version for hadoop-file-system dependency. * Add DirectRunner in test scope in java8, otherwise mvn package fails when another runner profile is enabled. commit 9a43da74a9005507acdc80524edc634c8318674d Author: Dan Halperin Date: 2017-05-12T22:57:21Z This closes #3132 commit 504bd6a8036946a796a7290853e45bf20d32d07d Author: Rune Fevang Date: 2017-05-12T20:24:32Z [BEAM-2290] Fix issue where timestamps weren't set when using CompressedSource commit d4df9079039d57e37cdb1666c8c3120f2fa2909a Author: Dan Halperin Date: 2017-05-15T16:52:38Z This closes #3131 commit 8060d66083973cdab91f68085652ab68e3dee596 Author: Thomas Groh Date: 2017-05-15T17:00:45Z Reduce Log Level of PubsubUnboundedSource commit 6e8d8d65708e3d5b0525a7b7f97597d4af71d51e Author: Thomas Groh Date: 2017-05-15T17:07:14Z This closes #3145 commit de1adefda958ae152eae4b7b94b3b78563e130aa Author: Neville Li Date: 2017-05-15T18:55:09Z readAvros should't have proto Message upper bound commit 88ec00860032078dcd33e07d94642912f6aa7bf7 Author: Eugene Kirpichov Date: 2017-05-15T19:22:27Z This closes #3149 commit 15ec2cd83d4291957a75d9f2ac838ff867dcbe0c Author: Dan Halperin Date: 2017-05-15T17:57:24Z Fix GcsResourceIdTest in postcommits This test is not a @ValidatesRunner test, so it should not use TestPipelineOptions. commit 926157d42db572dacb9142665d5c922dcad589f0 Author: Luke Cwik Date: 2017-05-15T20:16:05Z Fix GcsResourceIdTest in postcommits This closes #3147 commit 911128b287909e2
[GitHub] beam pull request #3246: rebasing DSL_SQL branch from master
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3246 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3246: rebasing DSL_SQL branch from master
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3246 rebasing DSL_SQL branch from master Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam DSL_SQL Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3246.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3246 commit 6a90b9d164d98d916d0e0af2be39a740ce769076 Author: Aviem Zur Date: 2017-05-12T13:02:54Z [BEAM-2277] Fix URI_SCHEME_PATTERN in FileSystems commit 0acf3620d90a970f4024ae95b909c3bb4ef827ad Author: Dan Halperin Date: 2017-05-12T18:10:48Z This closes #3115 commit abe62feb16f6eb59e294c37abf408f84c9280616 Author: Aviem Zur Date: 2017-05-12T19:18:02Z [BEAM-2279] Add HDFS support to Spark runner profiles in archetypes and examples commit c2a36284b510461942205b627a019dfbe83f8b37 Author: Davor Bonaci Date: 2017-05-12T21:54:00Z This closes #3126 commit a6a5ff7be387ef295fc7f921de36a3ea77327bc1 Author: Dan Halperin Date: 2017-05-12T16:20:34Z [BEAM-2277] Add ResourceIdTester and test existing ResourceId implementations A first cut at some of the parts of the ResourceId spec. commit 15df211c758e7c8f05c3136f25bbe18e3f394321 Author: Dan Halperin Date: 2017-05-12T18:11:06Z [BEAM-2277] HadoopFileSystem: normalize implementation * Drop empty authority always * Resolve directories commit fbb0de129d881b6b6a1ae37fc8d75075c9d8df86 Author: Dan Halperin Date: 2017-05-12T18:13:00Z Remove '/' entirely from determining FileSystem scheme commit ec956c85efa16d00c5e218ee1257b8ee62a2013d Author: Dan Halperin Date: 2017-05-12T18:38:54Z Mark FileSystem and related as Experimental commit f3540d47f10c18859340a738a7e93643ee57f604 Author: Dan Halperin Date: 2017-05-12T18:46:16Z Rename FileSystems.setDefaultConfigInWorkers And document that it's not for users. commit 3921163829d2a99b0524bf3fa1b85a4cde826f3a Author: Dan Halperin Date: 2017-05-12T20:57:59Z Fix shading of guava testlib commit 9f81fd299bd32e0d6056a7da9fa994cf74db0ed9 Author: Dan Halperin Date: 2017-05-12T21:59:14Z This closes #3121 commit f53ceecfb18a52aeeab8f1869c189e4aa2f56d35 Author: Ahmet Altay Date: 2017-05-12T21:54:33Z internal comments commit 89289e498e71d6e9fbd24702032471fa13fbd46c Author: Ahmet Altay Date: 2017-05-12T22:08:16Z This closes #3128 commit 7bade36eb18a48e01a8401f97f10a70a492172c4 Author: Dan Halperin Date: 2017-05-12T22:52:51Z [BEAM-2279] Fix archetype breakages * Add version for hadoop-file-system dependency. * Add DirectRunner in test scope in java8, otherwise mvn package fails when another runner profile is enabled. commit 9a43da74a9005507acdc80524edc634c8318674d Author: Dan Halperin Date: 2017-05-12T22:57:21Z This closes #3132 commit 504bd6a8036946a796a7290853e45bf20d32d07d Author: Rune Fevang Date: 2017-05-12T20:24:32Z [BEAM-2290] Fix issue where timestamps weren't set when using CompressedSource commit d4df9079039d57e37cdb1666c8c3120f2fa2909a Author: Dan Halperin Date: 2017-05-15T16:52:38Z This closes #3131 commit 8060d66083973cdab91f68085652ab68e3dee596 Author: Thomas Groh Date: 2017-05-15T17:00:45Z Reduce Log Level of PubsubUnboundedSource commit 6e8d8d65708e3d5b0525a7b7f97597d4af71d51e Author: Thomas Groh Date: 2017-05-15T17:07:14Z This closes #3145 commit de1adefda958ae152eae4b7b94b3b78563e130aa Author: Neville Li Date: 2017-05-15T18:55:09Z readAvros should't have proto Message upper bound commit 88ec00860032078dcd33e07d94642912f6aa7bf7 Author: Eugene Kirpichov Date: 2017-05-15T19:22:27Z This closes #3149 commit 15ec2cd83d4291957a75d9f2ac838ff867dcbe0c Author: Dan Halperin Date: 2017-05-15T17:57:24Z Fix GcsResourceIdTest in postcommits This test is not a @ValidatesRunner test, so it should not use TestPipelineOptions. commit 926157d42db572dacb9142665d5c922dcad589f0 Author: Luke Cwik Date: 2017-05-15T20:16:05Z Fix GcsResourceIdTest in postcommits This closes #3147 commit 911128b287909e2e2240e0674223df7c66e952f8 Author: Ahmet Altay Date: 2017-05-15T16:59:29Z Moving the data file for trig
[GitHub] beam pull request #3244: DSL_SQL: merge master into DSL_SQL feature branch
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3244 DSL_SQL: merge master into DSL_SQL feature branch Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x ] Make sure tests pass via `mvn clean verify`. - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/beam master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3244.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3244 commit 16a3fff18fdd34ebd85fe7d436f8ce3a429666d0 Author: Robert Bradshaw Date: 2017-05-11T19:09:28Z [BEAM-1345] Mark apache_beam/internal as internal. commit 7881fd5e6b53f6a26e23f877e47a410f6a515ae7 Author: Robert Bradshaw Date: 2017-05-11T19:54:13Z [BEAM-1345] Clearly delineate public api in apache_beam/typehints. commit 4a62fabdb30de3f9d1a539ee7e685b5151ed9ca1 Author: Robert Bradshaw Date: 2017-05-11T22:15:13Z Closes #3094 commit 0b7356e329f060c3824991cbe3b43bce89042763 Author: Charles Chen Date: 2017-05-11T22:07:30Z Move assert_that, equal_to, is_empty to apache_beam.testing.util commit 1b31167fb8bac634d40e5ab5b86b0e928e00f309 Author: Robert Bradshaw Date: 2017-05-11T22:27:59Z Closes #3086 commit 98e685d85a4cf75faee8156516d7d3aff60077f3 Author: Robert Bradshaw Date: 2017-05-10T22:55:46Z Remove some internal details from the public API. commit fe625678f55d9c9cc288b644749d7916015cf080 Author: Ahmet Altay Date: 2017-05-11T22:43:21Z This closes #3065 commit d76349beeab5cf6e0513d6272d635ac0b4662239 Author: Dan Halperin Date: 2017-05-11T21:03:08Z Don't deploy jdk1.8-tests module commit 37e0b437e14991b6627efa8c3fc5b215e8925f81 Author: Dan Halperin Date: 2017-05-11T22:49:39Z This closes #3097 commit fc3e28de37acc9de597518cf9fe793d1ddebcf2a Author: Robert Bradshaw Date: 2017-05-11T22:47:54Z Fix due to GBKO name change. commit 07e5536bc128a324e213eec1a6bd97beb950dcc6 Author: Robert Bradshaw Date: 2017-05-11T22:50:35Z Closes #3101 commit a45d63f6362e88d034704a4749cef7eedc76a691 Author: Sourabh Bajaj Date: 2017-05-11T23:41:06Z Remove unused test data commit fc77ca7cb18f5dc93dca9427112a3eaaf1494788 Author: Ahmet Altay Date: 2017-05-12T00:01:38Z This closes #3106 commit c39c9aecf859095bfde87267e7eb9d9a4fd1682d Author: Thomas Groh Date: 2017-05-12T00:16:53Z Enable SerializableCoder to Serialize with Generic Types A TypeToken that contains generics is not serializable. However, the TypeDescriptor does not need to be transmitted via the serialized form, so mark it as transient. commit 652fcb7e9e427b3ae44dbfcdd537298d9dd07cb5 Author: Thomas Groh Date: 2017-05-12T00:55:50Z This closes #3109 commit 2f177820c4e72fe92dc47d595847ddecdae7afcd Author: Sourabh Bajaj Date: 2017-05-12T00:23:40Z Rename filesink to filebasedsink commit 36a6cd69aa01b7e122e6e84ff1218adc7ba059c9 Author: Ahmet Altay Date: 2017-05-12T01:04:24Z This closes #3111 commit db7d71aec5d6c4b7a15474c7ef9db28460fb0043 Author: Thomas Groh Date: 2017-05-11T23:17:54Z Mark More values methods Internal PCollectionTuple#ofPrimitiveOutputsInternal is internal. TaggedPValue is internal. commit d6ac39a23e50e82c013e5052b48cb6ffbf929195 Author: Luke Cwik Date: 2017-05-12T01:34:15Z [BEAM-1345] Mark More values methods Internal This closes #3104 commit 5228f7d1f48bf39c0f97c28ab64064089e5f8df9 Author: Borisa Zivkovic Date: 2017-05-12T13:03:02Z [BEAM-2153] Move connection management in JmsIO.write() to setup/teardown methods commit 13c06bf79a45aa35e099730569331aa4d04d25de Author: Jean-Baptiste Onofré Date: 2017-05-12T14:49:20Z [BEAM-2153] This closes #3116 commit e8d829817e3f7e07884a91737b61bfa5133d5724 Author: Borisa Zivkovic Date: 2017-05-12T07:44:20Z BigtableIO should use AutoValue for read and write commit 42066431b45ced1931787650d2210be724fb7f5c Author: Eugene Kirpichov Date: 2017-05-12T17:57:03Z This closes #3113 commit 6a90b9d164d98d916d0e0af2be39a740ce769076 Author: Aviem Zur Date: 2017-05-12T13:02:54Z [BEAM-2277] Fix URI_SCHEME_PATTERN in FileSystems commit 0acf3620d90a970f4024ae95b909c3bb4ef827ad Author: Dan Halperin Date: 2017-05-12T18:10:48Z This closes #3115 commit abe62feb16f6eb59e294c37abf408f84c9280616 Author: Aviem Zur
[GitHub] beam pull request #3168: [BEAM-2310] support new data type: DECIMAL
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3168 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3166: [BEAM-2309] implement VALUES
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3166 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3218: [BEAM-2356] add NOT operator & some refactoring
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3218 [BEAM-2356] add NOT operator & some refactoring Seems only `NOT` operator is missing, other operators will be implicitly converted to `AND`, `OR`, `NOT`, during parsing phase, e.g. `IS FALSE` is converted to `NOT`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2356-logical-operator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3218.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3218 commit c0502cb74edbdda6fe65168a821ecc0949f82cf5 Author: James Xu Date: 2017-05-24T04:41:47Z add NOT operator & some refactoring --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3207: [BEAM-2247] implement date functions
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3207 [BEAM-2247] implement date functions @XuMingmin @jbonofre can you guys take a look at this one? You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2247-date-functions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3207.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3207 commit 5783fc09545058a8c35a110fcc7284626e9927f9 Author: James Xu Date: 2017-05-17T03:48:11Z init implementation(require Values) commit 48bac0325507c9265d8f6e94ea3747954b31bbc5 Author: James Xu Date: 2017-05-23T13:38:06Z implement date functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3162: [BEAM-2297] support UNION
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3162 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3183: [BEAM-2325] support Set operator: intersect & excep...
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3183 [BEAM-2325] support Set operator: intersect & except R: @XuMingmin R: @jbonofre guys, can you take a look at this one? You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2325-intersect-except Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3183.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3183 commit c6d909baa2fa6624d47893e50264d75a535dbfe6 Author: James Xu Date: 2017-05-19T13:47:10Z support Set operator: intersect & except --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3168: [BEAM-2310] support new data type: DECIMAL
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3168 [BEAM-2310] support new data type: DECIMAL 1. support new data type: DECIMAL 2. support encoding/decoding for TIME 3. add tests all the supported data types. R: @XuMingmin You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2310-decimal Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3168.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3168 commit 9a0739dcdfc3b865ea3e74f47bd1188a18abe60a Author: James Xu Date: 2017-05-17T14:48:00Z support encoding/decoding of TIME commit 12331d5af6482f18ddfaec2b1e22e192a1605c52 Author: James Xu Date: 2017-05-17T14:52:08Z support new data type: DECIMAL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3166: [BEAM-2309] implement VALUES
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3166 [BEAM-2309] implement VALUES 1. implement VALUES 2. add support for data type CHAR(to be able to test VALUES) R: @XuMingmin You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2309-values Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3166.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3166 commit 72217a0ff8d75842e33bf98b5cf91b258725f1c8 Author: James Xu Date: 2017-05-17T06:41:18Z implement VALUES 1. implement VALUES 2. add support for data type CHAR(to be able to test VALUES) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3082: [BEAM-2255] implement ORDER BY
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3082 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3162: [BEAM-2297] support UNION
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3162 [BEAM-2297] support UNION support UNION You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2297-union Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3162 commit d540d78359f28a8e7a38af22d69f1136c778bd74 Author: James Xu Date: 2017-05-16T05:10:56Z support UNION --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3122: [BEAM-2149] Improved kafka table implemention
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3122 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3042: [BEAM-2195] support conditional operators
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/3042 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3122: [BEAM-2149] Improved kafka table implemention
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3122 [BEAM-2149] Improved kafka table implemention 1. use robust CSV library to parse & print. 2. support different data types rather than just `String`. 3. a little cleanup for TextTable (to extract common methods). You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2149-kafka Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3122.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3122 commit 2074343ff4afc402a8ff50ecc4c99dbbd05281cb Author: James Xu Date: 2017-05-12T17:07:01Z Improved kafka table implemention. 1. use robust CSV library to parse & print. 2. support different data types rather than just `String`. 3. a little cleanup for TextTable (to extract common methods). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3082: [BEAM-2255] implement ORDER BY
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3082 [BEAM-2255] implement ORDER BY You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2255-order-by Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3082.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3082 commit 95ee7578f77577c60859776539e54a41103f94a4 Author: James Xu Date: 2017-05-11T17:07:18Z implement ORDER BY --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3077: Fix typo in org.apache.beam.sdk.transforms.Top
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3077 Fix typo in org.apache.beam.sdk.transforms.Top Hope I understand the method comments correctly:) You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam fix-typo-in-Top Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3077.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3077 commit a0377c9e8264ff58bafa6438d19a533f8a17 Author: James Xu Date: 2017-05-11T08:07:10Z minor typo fix in comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3042: [BEAM-2195] support conditional operators
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/3042 [BEAM-2195] support conditional operators Support conditional operators(CASE, NULLIF, COALESCE). @XuMingmin can you please take a look? You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2195-support-conditional-operators Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3042.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3042 commit 3625dbd9e00db28d2175c6ac000792fd5c5d8f65 Author: Jean-Baptiste Onofré Date: 2016-08-09T09:17:32Z Add DSLs module commit 7867ce62e43bef7bfd8011c605379df05494dfcf Author: mingmxu Date: 2017-04-10T02:49:08Z [BEAM-301] Initial skeleton for Beam SQL commit f1c2b6540652156ad7c51ec89a933aacb8c6dab6 Author: Jean-Baptiste Onofré Date: 2017-04-12T20:00:53Z [BEAM-301] This closes #2479 commit 529bc9d9375ae162d70de274975c547a878053a3 Author: mingmxu Date: 2017-04-13T21:46:28Z checkstyle and rename package correct package from org.beam.dsls.sql to org.apache.beam.dsls.sql update with checkstyle commit ca8760373b37b540219301f1e719633956a787dc Author: Davor Bonaci Date: 2017-04-18T21:40:40Z This closes #2530 commit aa07a1d41b71c810c4968bd35e5ca69dc56b8c30 Author: Dan Halperin Date: 2017-04-19T19:14:05Z This closes #2584 commit 464cc275952305bf51ecfdf056e784441c9c2272 Author: mingmxu Date: 2017-04-23T05:20:25Z redesign BeamSqlExpression to execute Calcite SQL expression. Changes: 1. revert BEAM dependency to 0.6.0 to avoid impact of changes in master branch; 2. updates as discussion during review; refine BeamSqlRowCoder commit a6845a35fc2a4091dcef24f231c6d3a418561e6f Author: Davor Bonaci Date: 2017-05-04T07:12:55Z This closes #2656 commit cf95571d90329287b92946065c93724780f31685 Author: Davor Bonaci Date: 2017-05-04T07:16:34Z This closes #2878 commit dd70852f6fa811a410938e5baa51cb8e602c931b Author: James Xu Date: 2017-05-04T03:44:14Z Fix inconsistent mapping for SQL FLOAT commit f7ee8d33e13506a17314720c8c23fff486bff76c Author: Davor Bonaci Date: 2017-05-07T02:24:04Z This closes #2886 commit a376f37b908828fa0d44b5d71822d51b9c47f971 Author: James Xu Date: 2017-05-04T14:15:02Z [BEAM-2158] Implement the arithmetic operators commit 4ea38d82359ed443c906de9caed8488d1dbf199a Author: Jean-Baptiste Onofré Date: 2017-05-09T08:24:20Z [BEAM-2158] This closes #2893 commit b12c7b49b8eab7b208f2701c412123635813ee28 Author: James Xu Date: 2017-05-05T17:20:54Z [BEAM-2161] Add support for String operators commit 3e678a75a746c9e40fdbe6599d20969718c34882 Author: Jean-Baptiste Onofré Date: 2017-05-09T12:17:53Z [BEAM-2161] This closes #2919 commit 59abc7640866f58427817c026af15df30947f3a6 Author: James Xu Date: 2017-05-04T03:44:14Z [BEAM-2079] Support TextIO as SQL source/sink commit 7e918a76e409e5c849f2878702746935f4d7bf90 Author: Jean-Baptiste Onofré Date: 2017-05-10T05:24:40Z [BEAM-2079] This closes #2942 commit 302117400fa309bb4e136b225dacd1f2cba37749 Author: James Xu Date: 2017-05-10T09:19:48Z implement conditional operator(CASE) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2942: [BEAM-2079] Support TextIO as SQL source/sink
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/2942 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2919: [BEAM-2161] support string operators
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/2919 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2893: [BEAM-2158] implement the arithmetic operators
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/2893 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2942: [BEAM-2079] Support TextIO as SQL source/sink
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/2942 [BEAM-2079] Support TextIO as SQL source/sink Support TextIO as SQL source/sink You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2079 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2942.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2942 commit 3625dbd9e00db28d2175c6ac000792fd5c5d8f65 Author: Jean-Baptiste Onofré Date: 2016-08-09T09:17:32Z Add DSLs module commit 7867ce62e43bef7bfd8011c605379df05494dfcf Author: mingmxu Date: 2017-04-10T02:49:08Z [BEAM-301] Initial skeleton for Beam SQL commit f1c2b6540652156ad7c51ec89a933aacb8c6dab6 Author: Jean-Baptiste Onofré Date: 2017-04-12T20:00:53Z [BEAM-301] This closes #2479 commit 529bc9d9375ae162d70de274975c547a878053a3 Author: mingmxu Date: 2017-04-13T21:46:28Z checkstyle and rename package correct package from org.beam.dsls.sql to org.apache.beam.dsls.sql update with checkstyle commit ca8760373b37b540219301f1e719633956a787dc Author: Davor Bonaci Date: 2017-04-18T21:40:40Z This closes #2530 commit aa07a1d41b71c810c4968bd35e5ca69dc56b8c30 Author: Dan Halperin Date: 2017-04-19T19:14:05Z This closes #2584 commit 464cc275952305bf51ecfdf056e784441c9c2272 Author: mingmxu Date: 2017-04-23T05:20:25Z redesign BeamSqlExpression to execute Calcite SQL expression. Changes: 1. revert BEAM dependency to 0.6.0 to avoid impact of changes in master branch; 2. updates as discussion during review; refine BeamSqlRowCoder commit a6845a35fc2a4091dcef24f231c6d3a418561e6f Author: Davor Bonaci Date: 2017-05-04T07:12:55Z This closes #2656 commit cf95571d90329287b92946065c93724780f31685 Author: Davor Bonaci Date: 2017-05-04T07:16:34Z This closes #2878 commit dd70852f6fa811a410938e5baa51cb8e602c931b Author: James Xu Date: 2017-05-04T03:44:14Z Fix inconsistent mapping for SQL FLOAT commit f7ee8d33e13506a17314720c8c23fff486bff76c Author: Davor Bonaci Date: 2017-05-07T02:24:04Z This closes #2886 commit bc9f9f3daffe47426f22f2b944f860b8cd2889da Author: James Xu Date: 2017-05-04T03:44:14Z Support TextIO as SQL source/sink --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2886: [BEAM-2151]Fix inconsistent mapping for SQL FLOAT
Github user xumingming closed the pull request at: https://github.com/apache/beam/pull/2886 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2919: [BEAM-2161] support string operators
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/2919 [BEAM-2161] support string operators https://issues.apache.org/jira/browse/BEAM-2161 @XuMingmin You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2161-support-string-operators Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2919.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2919 commit bc0506b28aaa41da0795facb78ab6dc936d87d3d Author: James Xu Date: 2017-05-05T17:20:54Z implement string operators --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2893: [BEAM-2158] implement the arithmetic operators
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/2893 [BEAM-2158] implement the arithmetic operators implement the arithmetic operators: https://issues.apache.org/jira/browse/BEAM-2158 @XuMingmin can you take a look at this? You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2158-support-arithmetic-operators Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2893.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2893 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #2886: [BEAM-2151]Fix inconsistent mapping for SQL FLOAT
GitHub user xumingming opened a pull request: https://github.com/apache/beam/pull/2886 [BEAM-2151]Fix inconsistent mapping for SQL FLOAT This pull request is mainly to fix the SQL FLOAT mapping, previously it is mapped to Float in `BeamSQLRow`, but mapped to `Double` in `BeamSQLRowCoder`, now we consistently map SQL FLOAT to java Float. When try to unit test this, found another bug in BeamSQLRow: the VARCHAR field is not encode/decoded correctly(mainly because OUTTER Context vs NESTED Context), so fixed this bug at the same time. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xumingming/beam BEAM-2151 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/2886.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2886 commit 02ffb848cf536cf631696f68051fecded2907931 Author: James Xu Date: 2017-05-04T03:44:14Z Fix inconsistent mapping for SQL FLOAT --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---