[jira] [Commented] (BEAM-2636) user_score on DataflowRunner is broken

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092630#comment-16092630
 ] 

Ahmet Altay commented on BEAM-2636:
---

Post commit test for prevent failures like this.

> user_score on DataflowRunner is broken
> --
>
> Key: BEAM-2636
> URL: https://issues.apache.org/jira/browse/BEAM-2636
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner 
> has a special code handling transforms with that name, this will break for 
> all user transforms that has this name.
> We can either:
> - Handle this correctly
> - Or document this as a reserved keyword and change the example.
> cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2532.
--
   Resolution: Fixed
 Assignee: Neville Li  (was: Chamikara Jayalath)
Fix Version/s: 2.2.0

> BigQueryIO source should avoid expensive JSON schema parsing for every record
> -
>
> Key: BEAM-2532
> URL: https://issues.apache.org/jira/browse/BEAM-2532
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Marian Dvorsky
>Assignee: Neville Li
>Priority: Minor
> Fix For: 2.2.0
>
>
> BigQueryIO source converts the schema from JSON for every input row, here:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L159
> This is the performance bottleneck in a simple pipeline with BigQueryIO 
> source.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2637) Post commit test for mobile gaming examples

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2637:
-

 Summary: Post commit test for mobile gaming examples
 Key: BEAM-2637
 URL: https://issues.apache.org/jira/browse/BEAM-2637
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay


We need post commit test for mobile gaming examples to prevent failures beyond 
DirectRunner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092625#comment-16092625
 ] 

ASF GitHub Bot commented on BEAM-2532:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3584


> BigQueryIO source should avoid expensive JSON schema parsing for every record
> -
>
> Key: BEAM-2532
> URL: https://issues.apache.org/jira/browse/BEAM-2532
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Marian Dvorsky
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> BigQueryIO source converts the schema from JSON for every input row, here:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L159
> This is the performance bottleneck in a simple pipeline with BigQueryIO 
> source.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3584: [BEAM-2532] add a Serializable TableSchema Supplier...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3584


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2532] Memoizes TableSchema in BigQuerySourceBase

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master d5101750e -> 7fde976d1


[BEAM-2532] Memoizes TableSchema in BigQuerySourceBase

Instead of parsing the JSON schema for every record.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e86c004d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e86c004d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e86c004d

Branch: refs/heads/master
Commit: e86c004de5d4b5f8bd0c3c53207cf3c1760f5d8e
Parents: d510175
Author: Neville Li 
Authored: Tue Jul 18 09:07:21 2017 -0400
Committer: Eugene Kirpichov 
Committed: Tue Jul 18 22:28:57 2017 -0700

--
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java  | 19 +--
 1 file changed, 17 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e86c004d/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
index 2de60a2..2b1eafe 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
@@ -29,11 +29,16 @@ import com.google.api.services.bigquery.model.JobReference;
 import com.google.api.services.bigquery.model.TableReference;
 import com.google.api.services.bigquery.model.TableRow;
 import com.google.api.services.bigquery.model.TableSchema;
+import com.google.common.base.Function;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import java.io.IOException;
+import java.io.Serializable;
 import java.util.List;
 import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.io.AvroSource;
@@ -168,10 +173,12 @@ abstract class BigQuerySourceBase extends 
BoundedSource {
 
 SerializableFunction function =
 new SerializableFunction() {
+  private Supplier schema = Suppliers.memoize(
+  Suppliers.compose(new TableSchemaFunction(), 
Suppliers.ofInstance(jsonSchema)));
+
   @Override
   public TableRow apply(GenericRecord input) {
-return BigQueryAvroUtils.convertGenericRecordToTableRow(
-input, BigQueryHelpers.fromJsonString(jsonSchema, 
TableSchema.class));
+return BigQueryAvroUtils.convertGenericRecordToTableRow(input, 
schema.get());
   }};
 
 List avroSources = Lists.newArrayList();
@@ -182,6 +189,14 @@ abstract class BigQuerySourceBase extends 
BoundedSource {
 return ImmutableList.copyOf(avroSources);
   }
 
+  private static class TableSchemaFunction implements Serializable, 
Function {
+@Nullable
+@Override
+public TableSchema apply(@Nullable String input) {
+  return BigQueryHelpers.fromJsonString(input, TableSchema.class);
+}
+  }
+
   protected static class BigQueryReader extends BoundedReader {
 private final BigQuerySourceBase source;
 private final BigQueryServices.BigQueryJsonReader reader;

[2/2] beam git commit: This closes #3584: [BEAM-2532] add a Serializable TableSchema Supplier in BigQuerySourceBase

[jkff]

This closes #3584: [BEAM-2532] add a Serializable TableSchema Supplier in 
BigQuerySourceBase


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7fde976d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7fde976d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7fde976d

Branch: refs/heads/master
Commit: 7fde976d14fe697dd88d2b161540c73d5cb01517
Parents: d510175 e86c004
Author: Eugene Kirpichov 
Authored: Tue Jul 18 22:33:58 2017 -0700
Committer: Eugene Kirpichov 
Committed: Tue Jul 18 22:33:58 2017 -0700

--
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java  | 19 +--
 1 file changed, 17 insertions(+), 2 deletions(-)
--

[jira] [Updated] (BEAM-2636) user_score on DataflowRunner is broken

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-2636:
--
Description: 
UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner has 
a special code handling transforms with that name, this will break for all user 
transforms that has this name.

We can either:
- Handle this correctly
- Or document this as a reserved keyword and change the example.

cc: [~chamikara]

  was:
UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner has 
a special code handling transform with that name, this will break for all user 
transforms that has this name.

We can either:
- Handle this correctly
- Or document this as a reserved keyword and change the example.

cc: [~chamikara]


> user_score on DataflowRunner is broken
> --
>
> Key: BEAM-2636
> URL: https://issues.apache.org/jira/browse/BEAM-2636
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner 
> has a special code handling transforms with that name, this will break for 
> all user transforms that has this name.
> We can either:
> - Handle this correctly
> - Or document this as a reserved keyword and change the example.
> cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2636) user_score on DataflowRunner is broken

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2636:
-

 Summary: user_score on DataflowRunner is broken
 Key: BEAM-2636
 URL: https://issues.apache.org/jira/browse/BEAM-2636
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Affects Versions: 2.1.0
Reporter: Ahmet Altay
Assignee: Sourabh Bajaj


UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner has 
a special code handling transform with that name, this will break for all user 
transforms that has this name.

We can either:
- Handle this correctly
- Or document this as a reserved keyword and change the example.

cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3608

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4405

[Apache Jenkins Server]

See 


Changes:

[pei] [BEAM-2630] TestPipeline: construct job/app names based on Description

--
[...truncated 3.42 MB...]
2017-07-19T04:43:29.498 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/bval/bval-core/0.5/bval-core-0.5.jar
2017-07-19T04:43:29.551 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/bval/bval-core/0.5/bval-core-0.5.jar
 (61 KB at 219.2 KB/sec)
2017-07-19T04:43:29.551 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apex/apex-bufferserver/3.6.0/apex-bufferserver-3.6.0.jar
2017-07-19T04:43:29.599 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/bval/bval-jsr303/0.5/bval-jsr303-0.5.jar
 (242 KB at 744.6 KB/sec)
2017-07-19T04:43:29.599 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
2017-07-19T04:43:29.600 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apex/apex-bufferserver/3.6.0/apex-bufferserver-3.6.0.jar
 (104 KB at 319.2 KB/sec)
2017-07-19T04:43:29.600 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar
2017-07-19T04:43:29.686 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apex/apex-engine/3.6.0/apex-engine-3.6.0.jar
 (1168 KB at 2839.9 KB/sec)
2017-07-19T04:43:29.686 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/sun/jersey/contribs/jersey-apache-client4/1.9/jersey-apache-client4-1.9.jar
2017-07-19T04:43:29.687 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar
 (277 KB at 670.3 KB/sec)
2017-07-19T04:43:29.687 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-client/2.6.0/hadoop-yarn-client-2.6.0.jar
2017-07-19T04:43:29.715 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
 (579 KB at 1313.9 KB/sec)
2017-07-19T04:43:29.715 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.0/hadoop-annotations-2.6.0.jar
2017-07-19T04:43:29.717 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/sun/jersey/contribs/jersey-apache-client4/1.9/jersey-apache-client4-1.9.jar
 (15 KB at 33.2 KB/sec)
2017-07-19T04:43:29.717 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6.0/hadoop-yarn-api-2.6.0.jar
2017-07-19T04:43:29.730 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-client/2.6.0/hadoop-yarn-client-2.6.0.jar
 (125 KB at 274.7 KB/sec)
2017-07-19T04:43:29.731 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.6.0/hadoop-yarn-common-2.6.0.jar
2017-07-19T04:43:29.743 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.0/hadoop-annotations-2.6.0.jar
 (17 KB at 35.5 KB/sec)
2017-07-19T04:43:29.743 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/jline/jline/2.11/jline-2.11.jar
2017-07-19T04:43:29.813 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apex/apex-shaded-ning19/1.0.0/apex-shaded-ning19-1.0.0.jar
 (2164 KB at 4022.2 KB/sec)
2017-07-19T04:43:29.814 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.9.2/ant-1.9.2.jar
2017-07-19T04:43:29.814 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/jline/jline/2.11/jline-2.11.jar (204 KB at 
378.3 KB/sec)
2017-07-19T04:43:29.815 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.9.2/ant-launcher-1.9.2.jar
2017-07-19T04:43:29.844 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.9.2/ant-launcher-1.9.2.jar
 (18 KB at 31.5 KB/sec)
2017-07-19T04:43:29.844 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/engio/mbassador/1.1.9/mbassador-1.1.9.jar
2017-07-19T04:43:29.885 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/engio/mbassador/1.1.9/mbassador-1.1.9.jar
 (77 KB at 125.5 KB/sec)
2017-07-19T04:43:29.885 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar
2017-07-19T04:43:29.944 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar
 (129 KB at 191.8 KB/sec)
2017-07-19T04:43:29.944 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5-shaded-4.3.jar
2017-07-19T04:43:30.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5-shaded-4.3.jar
 (142 KB at 193.8 KB/sec)
2017-07-19T04:43:30.004 [INFO] Downloading: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2659

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3607

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3433

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-2630) TestPipeline.getAppName() doesn't work properly when TestPipeline is used as @Rule

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He closed BEAM-2630.

Resolution: Fixed

> TestPipeline.getAppName() doesn't work properly when TestPipeline is used as 
> @Rule
> --
>
> Key: BEAM-2630
> URL: https://issues.apache.org/jira/browse/BEAM-2630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: 2.2.0
>
>
> Currently, TestPipeline.getAppName() looks at the stacktraces. However, when 
> @Rule is used, it is invoked from TestPipeline.init(). And, job names of 
> ValidatesRunnerTests all becomes pardo-init for tests in ParDo. (Same for 
> other ValidatesRunnerTests)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3582: [BEAM-2630] TestPipeline: construct job/app names b...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3582


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2630] TestPipeline: construct job/app names based on Description in junit TestRule.

[pei]

Repository: beam
Updated Branches:
  refs/heads/master 0d927ef6a -> d5101750e


[BEAM-2630] TestPipeline: construct job/app names based on Description in junit 
TestRule.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bdf5bd6e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bdf5bd6e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bdf5bd6e

Branch: refs/heads/master
Commit: bdf5bd6e50fa9f44ad7560714cd41ac3f346d124
Parents: 0d927ef
Author: Pei He 
Authored: Mon Jul 17 23:34:27 2017 +0800
Committer: Pei He 
Committed: Wed Jul 19 11:30:12 2017 +0800

--
 .../apache/beam/sdk/testing/TestPipeline.java   | 63 
 .../beam/sdk/testing/TestPipelineTest.java  | 38 +---
 2 files changed, 13 insertions(+), 88 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bdf5bd6e/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index 9206e04..34f1c83 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -31,9 +31,7 @@ import com.google.common.base.Predicate;
 import com.google.common.base.Predicates;
 import com.google.common.base.Strings;
 import com.google.common.collect.FluentIterable;
-import com.google.common.collect.Iterators;
 import java.io.IOException;
-import java.lang.reflect.Method;
 import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.LinkedList;
@@ -307,6 +305,7 @@ public class TestPipeline extends Pipeline implements 
TestRule {
 
   @Override
   public void evaluate() throws Throwable {
+
options.as(ApplicationNameOptions.class).setAppName(getAppName(description));
 
 setDeducedEnforcementLevel();
 
@@ -402,7 +401,6 @@ public class TestPipeline extends Pipeline implements 
TestRule {
   MAPPER.readValue(beamTestPipelineOptions, String[].class))
   .as(TestPipelineOptions.class);
 
-  options.as(ApplicationNameOptions.class).setAppName(getAppName());
   // If no options were specified, set some reasonable defaults
   if (Strings.isNullOrEmpty(beamTestPipelineOptions)) {
 // If there are no provided options, check to see if a dummy runner 
should be used.
@@ -450,56 +448,17 @@ public class TestPipeline extends Pipeline implements 
TestRule {
 }
   }
 
-  /** Returns the class + method name of the test, or a default name. */
-  private static String getAppName() {
-Optional stackTraceElement = findCallersStackTrace();
-if (stackTraceElement.isPresent()) {
-  String methodName = stackTraceElement.get().getMethodName();
-  String className = stackTraceElement.get().getClassName();
-  if (className.contains(".")) {
-className = className.substring(className.lastIndexOf(".") + 1);
-  }
-  return className + "-" + methodName;
-}
-return "UnitTest";
-  }
-
-  /** Returns the {@link StackTraceElement} of the calling class. */
-  private static Optional findCallersStackTrace() {
-Iterator elements =
-Iterators.forArray(Thread.currentThread().getStackTrace());
-// First find the TestPipeline class in the stack trace.
-while (elements.hasNext()) {
-  StackTraceElement next = elements.next();
-  if (TestPipeline.class.getName().equals(next.getClassName())) {
-break;
-  }
-}
-// Then find the first instance after that is not the TestPipeline
-Optional firstInstanceAfterTestPipeline = 
Optional.absent();
-while (elements.hasNext()) {
-  StackTraceElement next = elements.next();
-  if (!TestPipeline.class.getName().equals(next.getClassName())) {
-if (!firstInstanceAfterTestPipeline.isPresent()) {
-  firstInstanceAfterTestPipeline = Optional.of(next);
-}
-try {
-  Class nextClass = Class.forName(next.getClassName());
-  for (Method method : nextClass.getMethods()) {
-if (method.getName().equals(next.getMethodName())) {
-  if (method.isAnnotationPresent(org.junit.Test.class)) {
-return Optional.of(next);
-  } else if (method.isAnnotationPresent(org.junit.Before.class)) {
-break;
-  }
-}
-  }
-} catch (Throwable t) {
-  break;
-}
-  }
+  /** Returns the class + method name of the test. */
+  private String getAppName(Description description) {
+String methodName = 

[2/2] beam git commit: This closes #3582

[pei]

This closes #3582


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d5101750
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d5101750
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d5101750

Branch: refs/heads/master
Commit: d5101750e76460b4ad057103069abbd3833bce96
Parents: 0d927ef bdf5bd6
Author: Pei He 
Authored: Wed Jul 19 11:31:32 2017 +0800
Committer: Pei He 
Committed: Wed Jul 19 11:31:32 2017 +0800

--
 .../apache/beam/sdk/testing/TestPipeline.java   | 63 
 .../beam/sdk/testing/TestPipelineTest.java  | 38 +---
 2 files changed, 13 insertions(+), 88 deletions(-)
--

[jira] [Commented] (BEAM-2621) rename BeamSqlRecordType to BeamSqlRowType

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092493#comment-16092493
 ] 

ASF GitHub Bot commented on BEAM-2621:
--

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3571


> rename BeamSqlRecordType to BeamSqlRowType
> --
>
> Key: BEAM-2621
> URL: https://issues.apache.org/jira/browse/BEAM-2621
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3571: [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType

[xumingming]

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3571


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2658

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3432

[Apache Jenkins Server]

See 

[1/3] beam git commit: [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType

[lzljs3620320]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL a1f7cf6de -> d4d615a72


http://git-wip-us.apache.org/repos/asf/beam/blob/a9c8a8a1/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
index 41a786f..9ed56b4 100644
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
@@ -18,8 +18,8 @@
 
 package org.apache.beam.dsls.sql.schema.text;
 
-import org.apache.beam.dsls.sql.schema.BeamSqlRecordType;
 import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -46,13 +46,13 @@ public class BeamTextCSVTable extends BeamTextTable {
   /**
* CSV table with {@link CSVFormat#DEFAULT DEFAULT} format.
*/
-  public BeamTextCSVTable(BeamSqlRecordType beamSqlRecordType, String 
filePattern)  {
-this(beamSqlRecordType, filePattern, CSVFormat.DEFAULT);
+  public BeamTextCSVTable(BeamSqlRowType beamSqlRowType, String filePattern)  {
+this(beamSqlRowType, filePattern, CSVFormat.DEFAULT);
   }
 
-  public BeamTextCSVTable(BeamSqlRecordType beamSqlRecordType, String 
filePattern,
+  public BeamTextCSVTable(BeamSqlRowType beamSqlRowType, String filePattern,
   CSVFormat csvFormat) {
-super(beamSqlRecordType, filePattern);
+super(beamSqlRowType, filePattern);
 this.csvFormat = csvFormat;
   }
 
@@ -60,11 +60,11 @@ public class BeamTextCSVTable extends BeamTextTable {
   public PCollection buildIOReader(Pipeline pipeline) {
 return PBegin.in(pipeline).apply("decodeRecord", 
TextIO.read().from(filePattern))
 .apply("parseCSVLine",
-new BeamTextCSVTableIOReader(beamSqlRecordType, filePattern, 
csvFormat));
+new BeamTextCSVTableIOReader(beamSqlRowType, filePattern, 
csvFormat));
   }
 
   @Override
   public PTransform, PDone> buildIOWriter() {
-return new BeamTextCSVTableIOWriter(beamSqlRecordType, filePattern, 
csvFormat);
+return new BeamTextCSVTableIOWriter(beamSqlRowType, filePattern, 
csvFormat);
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/a9c8a8a1/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
index ef0a465..874c3e4 100644
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
@@ -21,9 +21,8 @@ package org.apache.beam.dsls.sql.schema.text;
 import static 
org.apache.beam.dsls.sql.schema.BeamTableUtils.csvLine2BeamSqlRow;
 
 import java.io.Serializable;
-
-import org.apache.beam.dsls.sql.schema.BeamSqlRecordType;
 import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
@@ -37,13 +36,13 @@ public class BeamTextCSVTableIOReader
 extends PTransform
 implements Serializable {
   private String filePattern;
-  protected BeamSqlRecordType beamSqlRecordType;
+  protected BeamSqlRowType beamSqlRowType;
   protected CSVFormat csvFormat;
 
-  public BeamTextCSVTableIOReader(BeamSqlRecordType beamSqlRecordType, String 
filePattern,
+  public BeamTextCSVTableIOReader(BeamSqlRowType beamSqlRowType, String 
filePattern,
   CSVFormat csvFormat) {
 this.filePattern = filePattern;
-this.beamSqlRecordType = beamSqlRecordType;
+this.beamSqlRowType = beamSqlRowType;
 this.csvFormat = csvFormat;
   }
 
@@ -53,7 +52,7 @@ public class BeamTextCSVTableIOReader
   @ProcessElement
   public void processElement(ProcessContext ctx) {
 String str = ctx.element();
-ctx.output(csvLine2BeamSqlRow(csvFormat, str, beamSqlRecordType));
+ctx.output(csvLine2BeamSqlRow(csvFormat, str, beamSqlRowType));
   }
 }));
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/a9c8a8a1/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOWriter.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOWriter.java
 

[2/3] beam git commit: [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType

[lzljs3620320]

[BEAM-2621] BeamSqlRecordType -> BeamSqlRowType


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a9c8a8a1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a9c8a8a1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a9c8a8a1

Branch: refs/heads/DSL_SQL
Commit: a9c8a8a1e1f21d31c973e12b3c05c118e21fa43b
Parents: a1f7cf6
Author: James Xu 
Authored: Mon Jul 17 17:55:56 2017 +0800
Committer: JingsongLi 
Committed: Wed Jul 19 10:05:17 2017 +0800

--
 .../org/apache/beam/dsls/sql/BeamSqlEnv.java| 12 +++---
 .../beam/dsls/sql/example/BeamSqlExample.java   |  4 +-
 .../interpreter/BeamSqlExpressionExecutor.java  |  2 +-
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java |  4 +-
 .../operator/BeamSqlCaseExpression.java |  8 ++--
 .../operator/BeamSqlCastExpression.java | 22 +--
 .../operator/BeamSqlCompareExpression.java  |  6 +--
 .../interpreter/operator/BeamSqlExpression.java |  2 +-
 .../operator/BeamSqlInputRefExpression.java |  4 +-
 .../operator/BeamSqlIsNotNullExpression.java|  4 +-
 .../operator/BeamSqlIsNullExpression.java   |  4 +-
 .../interpreter/operator/BeamSqlPrimitive.java  |  2 +-
 .../operator/BeamSqlReinterpretExpression.java  |  6 +--
 .../operator/BeamSqlUdfExpression.java  |  4 +-
 .../operator/BeamSqlWindowEndExpression.java|  4 +-
 .../operator/BeamSqlWindowExpression.java   |  4 +-
 .../operator/BeamSqlWindowStartExpression.java  |  4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java |  6 +--
 .../date/BeamSqlCurrentDateExpression.java  |  2 +-
 .../date/BeamSqlCurrentTimeExpression.java  |  2 +-
 .../date/BeamSqlCurrentTimestampExpression.java |  2 +-
 .../date/BeamSqlDateCeilExpression.java |  4 +-
 .../date/BeamSqlDateFloorExpression.java|  4 +-
 .../operator/date/BeamSqlExtractExpression.java |  4 +-
 .../operator/logical/BeamSqlAndExpression.java  |  4 +-
 .../operator/logical/BeamSqlNotExpression.java  |  4 +-
 .../operator/logical/BeamSqlOrExpression.java   |  4 +-
 .../math/BeamSqlMathBinaryExpression.java   |  4 +-
 .../math/BeamSqlMathUnaryExpression.java|  4 +-
 .../operator/math/BeamSqlPiExpression.java  |  2 +-
 .../string/BeamSqlCharLengthExpression.java |  4 +-
 .../string/BeamSqlConcatExpression.java |  6 +--
 .../string/BeamSqlInitCapExpression.java|  4 +-
 .../operator/string/BeamSqlLowerExpression.java |  4 +-
 .../string/BeamSqlOverlayExpression.java| 10 ++---
 .../string/BeamSqlPositionExpression.java   |  8 ++--
 .../string/BeamSqlSubstringExpression.java  |  8 ++--
 .../operator/string/BeamSqlTrimExpression.java  | 10 ++---
 .../operator/string/BeamSqlUpperExpression.java |  4 +-
 .../beam/dsls/sql/rel/BeamAggregationRel.java   | 22 +--
 .../apache/beam/dsls/sql/rel/BeamFilterRel.java |  2 +-
 .../beam/dsls/sql/rel/BeamIOSourceRel.java  |  2 +-
 .../apache/beam/dsls/sql/rel/BeamJoinRel.java   | 12 +++---
 .../beam/dsls/sql/rel/BeamProjectRel.java   |  4 +-
 .../apache/beam/dsls/sql/rel/BeamSortRel.java   |  2 +-
 .../apache/beam/dsls/sql/rel/BeamValuesRel.java |  9 ++---
 .../beam/dsls/sql/schema/BaseBeamTable.java | 10 ++---
 .../dsls/sql/schema/BeamPCollectionTable.java   |  8 ++--
 .../beam/dsls/sql/schema/BeamSqlRecordType.java | 40 
 .../apache/beam/dsls/sql/schema/BeamSqlRow.java | 10 ++---
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   |  6 +--
 .../beam/dsls/sql/schema/BeamSqlRowType.java| 40 
 .../beam/dsls/sql/schema/BeamSqlTable.java  |  2 +-
 .../beam/dsls/sql/schema/BeamTableUtils.java| 10 ++---
 .../sql/schema/kafka/BeamKafkaCSVTable.java | 29 +++---
 .../dsls/sql/schema/kafka/BeamKafkaTable.java   | 11 +++---
 .../dsls/sql/schema/text/BeamTextCSVTable.java  | 14 +++
 .../schema/text/BeamTextCSVTableIOReader.java   | 11 +++---
 .../schema/text/BeamTextCSVTableIOWriter.java   |  9 ++---
 .../dsls/sql/schema/text/BeamTextTable.java |  6 +--
 .../transform/BeamAggregationTransforms.java| 26 ++---
 .../dsls/sql/transform/BeamJoinTransforms.java  |  6 +--
 .../dsls/sql/transform/BeamSqlProjectFn.java| 16 
 .../beam/dsls/sql/utils/CalciteUtils.java   | 12 +++---
 .../dsls/sql/BeamSqlDslAggregationTest.java | 14 +++
 .../apache/beam/dsls/sql/BeamSqlDslBase.java| 26 ++---
 .../beam/dsls/sql/BeamSqlDslJoinTest.java   | 10 ++---
 .../beam/dsls/sql/BeamSqlDslProjectTest.java| 10 ++---
 .../beam/dsls/sql/BeamSqlDslUdfUdafTest.java|  6 +--
 .../org/apache/beam/dsls/sql/TestUtils.java | 28 +++---
 ...mSqlBuiltinFunctionsIntegrationTestBase.java |  6 +--
 .../interpreter/BeamSqlFnExecutorTestBase.java  |  8 ++--
 .../beam/dsls/sql/mock/MockedBoundedTable.java  | 14 +++
 

[3/3] beam git commit: This closes #3571

[lzljs3620320]

This closes #3571


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d4d615a7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d4d615a7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d4d615a7

Branch: refs/heads/DSL_SQL
Commit: d4d615a7237729f5e59085ae2bd77541c6ea7ddb
Parents: a1f7cf6 a9c8a8a
Author: JingsongLi 
Authored: Wed Jul 19 10:06:04 2017 +0800
Committer: JingsongLi 
Committed: Wed Jul 19 10:06:04 2017 +0800

--
 .../org/apache/beam/dsls/sql/BeamSqlEnv.java| 12 +++---
 .../beam/dsls/sql/example/BeamSqlExample.java   |  4 +-
 .../interpreter/BeamSqlExpressionExecutor.java  |  2 +-
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java |  4 +-
 .../operator/BeamSqlCaseExpression.java |  8 ++--
 .../operator/BeamSqlCastExpression.java | 22 +--
 .../operator/BeamSqlCompareExpression.java  |  6 +--
 .../interpreter/operator/BeamSqlExpression.java |  2 +-
 .../operator/BeamSqlInputRefExpression.java |  4 +-
 .../operator/BeamSqlIsNotNullExpression.java|  4 +-
 .../operator/BeamSqlIsNullExpression.java   |  4 +-
 .../interpreter/operator/BeamSqlPrimitive.java  |  2 +-
 .../operator/BeamSqlReinterpretExpression.java  |  6 +--
 .../operator/BeamSqlUdfExpression.java  |  4 +-
 .../operator/BeamSqlWindowEndExpression.java|  4 +-
 .../operator/BeamSqlWindowExpression.java   |  4 +-
 .../operator/BeamSqlWindowStartExpression.java  |  4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java |  6 +--
 .../date/BeamSqlCurrentDateExpression.java  |  2 +-
 .../date/BeamSqlCurrentTimeExpression.java  |  2 +-
 .../date/BeamSqlCurrentTimestampExpression.java |  2 +-
 .../date/BeamSqlDateCeilExpression.java |  4 +-
 .../date/BeamSqlDateFloorExpression.java|  4 +-
 .../operator/date/BeamSqlExtractExpression.java |  4 +-
 .../operator/logical/BeamSqlAndExpression.java  |  4 +-
 .../operator/logical/BeamSqlNotExpression.java  |  4 +-
 .../operator/logical/BeamSqlOrExpression.java   |  4 +-
 .../math/BeamSqlMathBinaryExpression.java   |  4 +-
 .../math/BeamSqlMathUnaryExpression.java|  4 +-
 .../operator/math/BeamSqlPiExpression.java  |  2 +-
 .../string/BeamSqlCharLengthExpression.java |  4 +-
 .../string/BeamSqlConcatExpression.java |  6 +--
 .../string/BeamSqlInitCapExpression.java|  4 +-
 .../operator/string/BeamSqlLowerExpression.java |  4 +-
 .../string/BeamSqlOverlayExpression.java| 10 ++---
 .../string/BeamSqlPositionExpression.java   |  8 ++--
 .../string/BeamSqlSubstringExpression.java  |  8 ++--
 .../operator/string/BeamSqlTrimExpression.java  | 10 ++---
 .../operator/string/BeamSqlUpperExpression.java |  4 +-
 .../beam/dsls/sql/rel/BeamAggregationRel.java   | 22 +--
 .../apache/beam/dsls/sql/rel/BeamFilterRel.java |  2 +-
 .../beam/dsls/sql/rel/BeamIOSourceRel.java  |  2 +-
 .../apache/beam/dsls/sql/rel/BeamJoinRel.java   | 12 +++---
 .../beam/dsls/sql/rel/BeamProjectRel.java   |  4 +-
 .../apache/beam/dsls/sql/rel/BeamSortRel.java   |  2 +-
 .../apache/beam/dsls/sql/rel/BeamValuesRel.java |  9 ++---
 .../beam/dsls/sql/schema/BaseBeamTable.java | 10 ++---
 .../dsls/sql/schema/BeamPCollectionTable.java   |  8 ++--
 .../beam/dsls/sql/schema/BeamSqlRecordType.java | 40 
 .../apache/beam/dsls/sql/schema/BeamSqlRow.java | 10 ++---
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   |  6 +--
 .../beam/dsls/sql/schema/BeamSqlRowType.java| 40 
 .../beam/dsls/sql/schema/BeamSqlTable.java  |  2 +-
 .../beam/dsls/sql/schema/BeamTableUtils.java| 10 ++---
 .../sql/schema/kafka/BeamKafkaCSVTable.java | 29 +++---
 .../dsls/sql/schema/kafka/BeamKafkaTable.java   | 11 +++---
 .../dsls/sql/schema/text/BeamTextCSVTable.java  | 14 +++
 .../schema/text/BeamTextCSVTableIOReader.java   | 11 +++---
 .../schema/text/BeamTextCSVTableIOWriter.java   |  9 ++---
 .../dsls/sql/schema/text/BeamTextTable.java |  6 +--
 .../transform/BeamAggregationTransforms.java| 26 ++---
 .../dsls/sql/transform/BeamJoinTransforms.java  |  6 +--
 .../dsls/sql/transform/BeamSqlProjectFn.java| 16 
 .../beam/dsls/sql/utils/CalciteUtils.java   | 12 +++---
 .../dsls/sql/BeamSqlDslAggregationTest.java | 14 +++
 .../apache/beam/dsls/sql/BeamSqlDslBase.java| 26 ++---
 .../beam/dsls/sql/BeamSqlDslJoinTest.java   | 10 ++---
 .../beam/dsls/sql/BeamSqlDslProjectTest.java| 10 ++---
 .../beam/dsls/sql/BeamSqlDslUdfUdafTest.java|  6 +--
 .../org/apache/beam/dsls/sql/TestUtils.java | 28 +++---
 ...mSqlBuiltinFunctionsIntegrationTestBase.java |  6 +--
 .../interpreter/BeamSqlFnExecutorTestBase.java  |  8 ++--
 .../beam/dsls/sql/mock/MockedBoundedTable.java  | 14 +++
 

[jira] [Closed] (BEAM-2160) add support Math functions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2160.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> add support Math functions 
> ---
>
> Key: BEAM-2160
> URL: https://issues.apache.org/jira/browse/BEAM-2160
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Tarush Grover
>  Labels: bundle, dsl_sql_merge, split
> Fix For: Not applicable
>
>
> All Math functions listed as below:
> {code}
> POWER(numeric1, numeric2) 
> ABS(numeric) 
> MOD(numeric, numeric) 
> SQRT(numeric) 
> LN(numeric)   
> LOG10(numeric) 
> EXP(numeric)  
> CEIL(numeric) 
> FLOOR(numeric)
> RAND([seed])  
> RAND_INTEGER([seed, ] numeric)
> ACOS(numeric)
> ASIN(numeric) 
> ATAN(numeric) 
> ATAN2(numeric, numeric)   
> COS(numeric)  
> COT(numeric)  
> DEGREES(numeric)  
> PI()  
> RADIANS(numeric)  
> ROUND(numeric1, numeric2) 
> SIGN(numeric) 
> SIN(numeric)  
> TAN(numeric)  
> TRUNCATE(numeric1, numeric2)  
> {code}
> See  
> https://calcite.apache.org/docs/reference.html#arithmetic-operators-and-functions
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2585) add math function RAND_INTEGER and RAND

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2585.

   Resolution: Fixed
Fix Version/s: Not applicable

> add math function RAND_INTEGER and RAND
> ---
>
> Key: BEAM-2585
> URL: https://issues.apache.org/jira/browse/BEAM-2585
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2585) add math function RAND_INTEGER and RAND

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092442#comment-16092442
 ] 

ASF GitHub Bot commented on BEAM-2585:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3552


> add math function RAND_INTEGER and RAND
> ---
>
> Key: BEAM-2585
> URL: https://issues.apache.org/jira/browse/BEAM-2585
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3552: [BEAM-2585] add math function RAND_INTEGER and RAND

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3552


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3606

[Apache Jenkins Server]

See 

[1/2] beam git commit: rebased, add RAND/RAND_INTEGER

[takidau]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL caf647daf -> a1f7cf6de


rebased, add RAND/RAND_INTEGER

update as commented


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/97c9b075
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/97c9b075
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/97c9b075

Branch: refs/heads/DSL_SQL
Commit: 97c9b07576f9baa1bf10aaf1817f731137c1a1a0
Parents: caf647d
Author: mingmxu 
Authored: Tue Jul 18 00:09:39 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Jul 18 17:54:43 2017 -0700

--
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java |  8 +++
 .../operator/math/BeamSqlRandExpression.java| 54 
 .../math/BeamSqlRandIntegerExpression.java  | 58 +
 .../BeamSqlMathFunctionsIntegrationTest.java| 67 
 4 files changed, 187 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/97c9b075/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/BeamSqlFnExecutor.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/BeamSqlFnExecutor.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/BeamSqlFnExecutor.java
index 08d124f..64bc880 100644
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/BeamSqlFnExecutor.java
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/BeamSqlFnExecutor.java
@@ -68,6 +68,8 @@ import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlLogExpression;
 import org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlPiExpression;
 import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlPowerExpression;
 import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlRadiansExpression;
+import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlRandExpression;
+import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlRandIntegerExpression;
 import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlRoundExpression;
 import 
org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlSignExpression;
 import org.apache.beam.dsls.sql.interpreter.operator.math.BeamSqlSinExpression;
@@ -299,6 +301,12 @@ public class BeamSqlFnExecutor implements 
BeamSqlExpressionExecutor {
 case "TRUNCATE":
   ret = new BeamSqlTruncateExpression(subExps);
   break;
+case "RAND":
+  ret = new BeamSqlRandExpression(subExps);
+  break;
+case "RAND_INTEGER":
+  ret = new BeamSqlRandIntegerExpression(subExps);
+  break;
 
 // string operators
 case "||":

http://git-wip-us.apache.org/repos/asf/beam/blob/97c9b075/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlRandExpression.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlRandExpression.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlRandExpression.java
new file mode 100644
index 000..944936b
--- /dev/null
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlRandExpression.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.math;
+
+import java.util.List;
+import java.util.Random;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlMathUnaryExpression} for 'RAND([seed])' function.
+ */
+public class BeamSqlRandExpression extends BeamSqlExpression {
+  private Random rand = new Random();
+  private Integer seed = null;
+
+  public BeamSqlRandExpression(List 

[2/2] beam git commit: [BEAM-2585] This closes #3552

[takidau]

[BEAM-2585] This closes #3552


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a1f7cf6d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a1f7cf6d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a1f7cf6d

Branch: refs/heads/DSL_SQL
Commit: a1f7cf6deaa2e7344e695c25234d82086194c560
Parents: caf647d 97c9b07
Author: Tyler Akidau 
Authored: Tue Jul 18 17:55:26 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Jul 18 17:55:26 2017 -0700

--
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java |  8 +++
 .../operator/math/BeamSqlRandExpression.java| 54 
 .../math/BeamSqlRandIntegerExpression.java  | 58 +
 .../BeamSqlMathFunctionsIntegrationTest.java| 67 
 4 files changed, 187 insertions(+)
--

[jira] [Resolved] (BEAM-2619) Add a GroupByKeyTest where the input is windowed into sessions

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2619.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Add a GroupByKeyTest where the input is windowed into sessions
> --
>
> Key: BEAM-2619
> URL: https://issues.apache.org/jira/browse/BEAM-2619
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: Not applicable
>
>
> This demonstrates the merging behavior.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2618) Add a GroupByKeyTest where the inputs are windowed into SlidingWindows

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2618.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Add a GroupByKeyTest where the inputs are windowed into SlidingWindows
> --
>
> Key: BEAM-2618
> URL: https://issues.apache.org/jira/browse/BEAM-2618
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2618) Add a GroupByKeyTest where the inputs are windowed into SlidingWindows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092427#comment-16092427
 ] 

ASF GitHub Bot commented on BEAM-2618:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3576


> Add a GroupByKeyTest where the inputs are windowed into SlidingWindows
> --
>
> Key: BEAM-2618
> URL: https://issues.apache.org/jira/browse/BEAM-2618
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: This closes #3576

[tgroh]

This closes #3576


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0d927ef6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0d927ef6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0d927ef6

Branch: refs/heads/master
Commit: 0d927ef6ab0fa5dd03a6b38ea9fe9bfeacd8
Parents: be5b934 1e94704
Author: Thomas Groh 
Authored: Tue Jul 18 17:52:56 2017 -0700
Committer: Thomas Groh 
Committed: Tue Jul 18 17:52:56 2017 -0700

--
 .../beam/sdk/transforms/GroupByKeyTest.java | 156 +++
 1 file changed, 122 insertions(+), 34 deletions(-)
--

[GitHub] beam pull request #3576: [BEAM-2618][BEAM-2619] Add GroupByKey tests for Mul...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3576


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add GroupByKey tests for Multiple & Merging windows

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master be5b9347b -> 0d927ef6a


Add GroupByKey tests for Multiple & Merging windows

This gives explicit coverage to a GroupByKey where the elements are in
multiple windows, or in merging windows.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1e947045
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1e947045
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1e947045

Branch: refs/heads/master
Commit: 1e947045a54bd59b449fd56f8f5f50879b6d9c4c
Parents: be5b934
Author: Thomas Groh 
Authored: Mon Jul 17 13:38:11 2017 -0700
Committer: Thomas Groh 
Committed: Tue Jul 18 17:52:55 2017 -0700

--
 .../beam/sdk/transforms/GroupByKeyTest.java | 156 +++
 1 file changed, 122 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1e947045/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
index 4b5d5f5..8fcb4c0 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
@@ -23,18 +23,20 @@ import static org.hamcrest.CoreMatchers.equalTo;
 import static org.hamcrest.CoreMatchers.hasItem;
 import static org.hamcrest.Matchers.empty;
 import static 
org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
-import static org.hamcrest.core.Is.is;
 import static org.junit.Assert.assertThat;
 
 import com.google.common.base.Function;
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Iterables;
 import java.io.DataInputStream;
 import java.io.DataOutputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ThreadLocalRandom;
@@ -56,9 +58,12 @@ import org.apache.beam.sdk.testing.ValidatesRunner;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.InvalidWindows;
 import org.apache.beam.sdk.transforms.windowing.Repeatedly;
 import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
 import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.values.KV;
@@ -67,6 +72,7 @@ import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TimestampedValue;
 import org.apache.beam.sdk.values.TypeDescriptor;
 import org.apache.beam.sdk.values.WindowingStrategy;
+import org.hamcrest.Matcher;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.junit.Assert;
@@ -82,13 +88,13 @@ import org.junit.runners.JUnit4;
  */
 @RunWith(JUnit4.class)
 @SuppressWarnings({"rawtypes", "unchecked"})
-public class GroupByKeyTest {
+public class GroupByKeyTest implements Serializable {
 
   @Rule
-  public final TestPipeline p = TestPipeline.create();
+  public transient TestPipeline p = TestPipeline.create();
 
   @Rule
-  public ExpectedException thrown = ExpectedException.none();
+  public transient ExpectedException thrown = ExpectedException.none();
 
   @Test
   @Category(ValidatesRunner.class)
@@ -109,27 +115,18 @@ public class GroupByKeyTest {
 PCollection> output =
 input.apply(GroupByKey.create());
 
-PAssert.that(output)
-.satisfies(new AssertThatHasExpectedContentsForTestGroupByKey());
+SerializableFunction>, Void> 
checker =
+containsKvs(
+kv("k1", 3, 4),
+kv("k5", Integer.MIN_VALUE, Integer.MAX_VALUE),
+kv("k2", 66, -33),
+kv("k3", 0));
+PAssert.that(output).satisfies(checker);
+PAssert.that(output).inWindow(GlobalWindow.INSTANCE).satisfies(checker);
 
 p.run();
   }
 
-  static class AssertThatHasExpectedContentsForTestGroupByKey
-  implements SerializableFunction>,
-  Void> {
-@Override
-  

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092381#comment-16092381
 ] 

ASF GitHub Bot commented on BEAM-1542:
--

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3591

[BEAM-1542] Introduced `SpannerIO.readFn`

`SpannerIO.readFn` allows to pipe reading from Cloud Spanner to other 
computations. The motivating use case is exporting data from a Spanner 
database, when list of the tables is generated in a previous transform.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam readrefactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3591


commit 393f50874d4b5cf8f3ced2dcc248d1a0920bf031
Author: Mairbek Khadikov 
Date:   2017-07-17T19:37:47Z

Inline buildSpannerOptions method

commit a12a5526901eb5a6b69bbc342abcbbcff46b3e41
Author: Mairbek Khadikov 
Date:   2017-07-17T19:39:45Z

Specify user agent.

commit 426a1fd98a9626c880feb6921028da876b56a1f4
Author: Mairbek Khadikov 
Date:   2017-07-18T21:40:24Z

Changed format to be Apache_Beam_Java/%version%

commit 4a5d2242f5653fa0cda709a359cd0b092ab3a3ff
Author: Mairbek Khadikov 
Date:   2017-07-18T23:23:58Z

Introduced a ReadOperation abstraction.

commit 16737e9476a8599d387c7752e21bad56883265f6
Author: Mairbek Khadikov 
Date:   2017-07-18T23:36:10Z

SpannerIO.readFn

commit bc5f33c88805c925b388fc6de365a7baa4ff
Author: Mairbek Khadikov 
Date:   2017-07-19T00:10:08Z

Documentation




> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3591: [BEAM-1542] Introduced `SpannerIO.readFn`

[mairbek]

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3591

[BEAM-1542] Introduced `SpannerIO.readFn`

`SpannerIO.readFn` allows to pipe reading from Cloud Spanner to other 
computations. The motivating use case is exporting data from a Spanner 
database, when list of the tables is generated in a previous transform.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam readrefactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3591


commit 393f50874d4b5cf8f3ced2dcc248d1a0920bf031
Author: Mairbek Khadikov 
Date:   2017-07-17T19:37:47Z

Inline buildSpannerOptions method

commit a12a5526901eb5a6b69bbc342abcbbcff46b3e41
Author: Mairbek Khadikov 
Date:   2017-07-17T19:39:45Z

Specify user agent.

commit 426a1fd98a9626c880feb6921028da876b56a1f4
Author: Mairbek Khadikov 
Date:   2017-07-18T21:40:24Z

Changed format to be Apache_Beam_Java/%version%

commit 4a5d2242f5653fa0cda709a359cd0b092ab3a3ff
Author: Mairbek Khadikov 
Date:   2017-07-18T23:23:58Z

Introduced a ReadOperation abstraction.

commit 16737e9476a8599d387c7752e21bad56883265f6
Author: Mairbek Khadikov 
Date:   2017-07-18T23:36:10Z

SpannerIO.readFn

commit bc5f33c88805c925b388fc6de365a7baa4ff
Author: Mairbek Khadikov 
Date:   2017-07-19T00:10:08Z

Documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2657

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3605

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3431

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4402

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2544) AvroIOTest is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092304#comment-16092304
 ] 

ASF GitHub Bot commented on BEAM-2544:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3475


> AvroIOTest is flaky
> ---
>
> Key: BEAM-2544
> URL: https://issues.apache.org/jira/browse/BEAM-2544
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Alex Filatov
>Assignee: Eugene Kirpichov
>Priority: Minor
>
> "Write then read" tests randomly fail.
> Steps to reproduce:
> cd /runners/direct-java
> mvn clean compile
> mvn surefire:test@validates-runner-tests -Dtest=AvroIOTest
> Repeat last step until a failure (on my machine failure rate is approx 1/3).
> Example:
> [ERROR] 
> testAvroIOWriteAndReadSchemaUpgrade(org.apache.beam.sdk.io.AvroIOTest)  Time 
> elapsed: 0.198 s  <<< ERROR!
> java.lang.RuntimeException: java.io.FileNotFoundException: 
> /var/folders/1c/sl733g5s1g7_4mq61_qmbjx4gn/T/junit3332447750239941326/output.avro
>  (No such file or directory)
>   at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:340)
>   at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:302)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:201)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:64)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:283)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:340)
>   at 
> org.apache.beam.sdk.io.AvroIOTest.testAvroIOWriteAndReadSchemaUpgrade(AvroIOTest.java:275)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
>   at 
> 

[2/2] beam git commit: This closes #3475: [BEAM-2544] Fix flaky AvroIOTest

[jkff]

This closes #3475: [BEAM-2544] Fix flaky AvroIOTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/be5b9347
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/be5b9347
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/be5b9347

Branch: refs/heads/master
Commit: be5b9347bc44de1f042c76e1ba3f47a13772c72b
Parents: dd9e866 911edba
Author: Eugene Kirpichov 
Authored: Tue Jul 18 15:49:54 2017 -0700
Committer: Eugene Kirpichov 
Committed: Tue Jul 18 15:49:54 2017 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 46 +++-
 1 file changed, 25 insertions(+), 21 deletions(-)
--

[GitHub] beam pull request #3475: [BEAM-2544] Fix flaky AvroIOTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3475


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2544] Fix flaky AvroIOTest by eliminating race condition in "write then read" tests.

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master dd9e866e0 -> be5b9347b


[BEAM-2544] Fix flaky AvroIOTest by eliminating race condition in "write then 
read" tests.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/911edbad
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/911edbad
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/911edbad

Branch: refs/heads/master
Commit: 911edbade388a63626e0ad6f8b7c2ad7a9f9b7c2
Parents: dd9e866
Author: Alex Filatov 
Authored: Thu Jun 29 23:23:04 2017 +0300
Committer: Eugene Kirpichov 
Committed: Tue Jul 18 15:49:44 2017 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 46 +++-
 1 file changed, 25 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/911edbad/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
index 4a1386c..4380c57 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
@@ -90,7 +90,11 @@ import org.junit.runners.JUnit4;
 @RunWith(JUnit4.class)
 public class AvroIOTest {
 
-  @Rule public TestPipeline p = TestPipeline.create();
+  @Rule
+  public TestPipeline writePipeline = TestPipeline.create();
+
+  @Rule
+  public TestPipeline readPipeline = TestPipeline.create();
 
   @Rule public TemporaryFolder tmpFolder = new TemporaryFolder();
 
@@ -144,15 +148,15 @@ public class AvroIOTest {
 ImmutableList.of(new GenericClass(3, "hi"), new GenericClass(5, 
"bar"));
 File outputFile = tmpFolder.newFile("output.avro");
 
-p.apply(Create.of(values))
+writePipeline.apply(Create.of(values))
 
.apply(AvroIO.write(GenericClass.class).to(outputFile.getAbsolutePath()).withoutSharding());
-p.run();
+writePipeline.run().waitUntilFinish();
 
 PCollection input =
-
p.apply(AvroIO.read(GenericClass.class).from(outputFile.getAbsolutePath()));
+
readPipeline.apply(AvroIO.read(GenericClass.class).from(outputFile.getAbsolutePath()));
 
 PAssert.that(input).containsInAnyOrder(values);
-p.run();
+readPipeline.run();
   }
 
   @Test
@@ -163,19 +167,19 @@ public class AvroIOTest {
 ImmutableList.of(new GenericClass(3, "hi"), new GenericClass(5, 
"bar"));
 File outputFile = tmpFolder.newFile("output.avro");
 
-p.apply(Create.of(values))
+writePipeline.apply(Create.of(values))
 .apply(
 AvroIO.write(GenericClass.class)
 .to(outputFile.getAbsolutePath())
 .withoutSharding()
 .withCodec(CodecFactory.deflateCodec(9)));
-p.run();
+writePipeline.run().waitUntilFinish();
 
 PCollection input =
-
p.apply(AvroIO.read(GenericClass.class).from(outputFile.getAbsolutePath()));
+
readPipeline.apply(AvroIO.read(GenericClass.class).from(outputFile.getAbsolutePath()));
 
 PAssert.that(input).containsInAnyOrder(values);
-p.run();
+readPipeline.run();
 DataFileStream dataFileStream =
 new DataFileStream(new FileInputStream(outputFile), new 
GenericDatumReader());
 assertEquals("deflate", dataFileStream.getMetaString("avro.codec"));
@@ -189,19 +193,19 @@ public class AvroIOTest {
 ImmutableList.of(new GenericClass(3, "hi"), new GenericClass(5, 
"bar"));
 File outputFile = tmpFolder.newFile("output.avro");
 
-p.apply(Create.of(values))
+writePipeline.apply(Create.of(values))
 .apply(
 AvroIO.write(GenericClass.class)
 .to(outputFile.getAbsolutePath())
 .withoutSharding()
 .withCodec(CodecFactory.nullCodec()));
-p.run();
+writePipeline.run().waitUntilFinish();
 
 PCollection input =
-
p.apply(AvroIO.read(GenericClass.class).from(outputFile.getAbsolutePath()));
+
readPipeline.apply(AvroIO.read(GenericClass.class).from(outputFile.getAbsolutePath()));
 
 PAssert.that(input).containsInAnyOrder(values);
-p.run();
+readPipeline.run();
 DataFileStream dataFileStream =
 new DataFileStream(new FileInputStream(outputFile), new 
GenericDatumReader());
 assertEquals("null", dataFileStream.getMetaString("avro.codec"));
@@ -261,18 +265,18 @@ public class AvroIOTest {
 ImmutableList.of(new GenericClass(3, "hi"), new GenericClass(5, 
"bar"));
 File outputFile = tmpFolder.newFile("output.avro");
 
-p.apply(Create.of(values))
+writePipeline.apply(Create.of(values))
 

[jira] [Closed] (BEAM-2524) Update Google Cloud Console URL returned by DataflowRunner to support regions.

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke closed BEAM-2524.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> Update Google Cloud Console URL returned by DataflowRunner to support regions.
> --
>
> Key: BEAM-2524
> URL: https://issues.apache.org/jira/browse/BEAM-2524
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Robert Burke
>Assignee: Robert Burke
> Fix For: 2.2.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Both the Java and Python Dataflow Runners need to be updated with a 
> regionalized form of the Google Cloud Console URL to support multiple 
> Dataflow Regions.
> The new URL format will be:
> https://console.cloud.corp.google.com/dataflow/jobsDetail/locations//jobs/?project=



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2196) support UDF

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2196:
--
Fix Version/s: (was: 2.1.0)
   Not applicable

> support UDF
> ---
>
> Key: BEAM-2196
> URL: https://issues.apache.org/jira/browse/BEAM-2196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2196) support UDF

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2196:
---

> support UDF
> ---
>
> Key: BEAM-2196
> URL: https://issues.apache.org/jira/browse/BEAM-2196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2196) support UDF

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2196.
-
Resolution: Fixed

> support UDF
> ---
>
> Key: BEAM-2196
> URL: https://issues.apache.org/jira/browse/BEAM-2196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2375) upgrade beam-version to 2.0.0

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2375.
-
Resolution: Fixed

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2424) CAST operator supporting numeric, date and timestamp types

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2424:
--
Fix Version/s: (was: 2.1.0)
   Not applicable

> CAST operator supporting numeric, date and timestamp types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2424) CAST operator supporting numeric, date and timestamp types

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2424.
-
Resolution: Fixed

> CAST operator supporting numeric, date and timestamp types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2461) LN Function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2461.
-
Resolution: Fixed

> LN Function
> ---
>
> Key: BEAM-2461
> URL: https://issues.apache.org/jira/browse/BEAM-2461
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2383) ROUND math function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2383:
---

> ROUND math function
> ---
>
> Key: BEAM-2383
> URL: https://issues.apache.org/jira/browse/BEAM-2383
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2424) CAST operator supporting numeric, date and timestamp types

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2424:
---

> CAST operator supporting numeric, date and timestamp types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2461) LN Function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2461:
--
Fix Version/s: (was: 2.2.0)
   Not applicable

> LN Function
> ---
>
> Key: BEAM-2461
> URL: https://issues.apache.org/jira/browse/BEAM-2461
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2375) upgrade beam-version to 2.0.0

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2375:
--
Fix Version/s: (was: 2.1.0)
   Not applicable

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2383) ROUND math function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2383:
--
Fix Version/s: (was: 2.1.0)
   Not applicable

> ROUND math function
> ---
>
> Key: BEAM-2383
> URL: https://issues.apache.org/jira/browse/BEAM-2383
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2383) ROUND math function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2383.
-
Resolution: Fixed

> ROUND math function
> ---
>
> Key: BEAM-2383
> URL: https://issues.apache.org/jira/browse/BEAM-2383
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2375) upgrade beam-version to 2.0.0

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2375:
---

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2461) LN Function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2461.
-
Resolution: Fixed

> LN Function
> ---
>
> Key: BEAM-2461
> URL: https://issues.apache.org/jira/browse/BEAM-2461
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2461) LN Function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2461:
--
Fix Version/s: (was: 2.1.0)
   2.2.0

> LN Function
> ---
>
> Key: BEAM-2461
> URL: https://issues.apache.org/jira/browse/BEAM-2461
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2480) ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN, LOG10, EXP Functions

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2480:
--
Fix Version/s: (was: 2.1.0)
   Not applicable

> ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN, LOG10, EXP 
> Functions
> --
>
> Key: BEAM-2480
> URL: https://issues.apache.org/jira/browse/BEAM-2480
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2461) LN Function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2461:
---

> LN Function
> ---
>
> Key: BEAM-2461
> URL: https://issues.apache.org/jira/browse/BEAM-2461
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2480) ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN, LOG10, EXP Functions

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2480:
---

> ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN, LOG10, EXP 
> Functions
> --
>
> Key: BEAM-2480
> URL: https://issues.apache.org/jira/browse/BEAM-2480
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2480) ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN, LOG10, EXP Functions

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2480.
-
Resolution: Fixed

> ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN, LOG10, EXP 
> Functions
> --
>
> Key: BEAM-2480
> URL: https://issues.apache.org/jira/browse/BEAM-2480
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2171) Power function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2171:
--
Fix Version/s: (was: 2.1.0)
   Not applicable

> Power function
> --
>
> Key: BEAM-2171
> URL: https://issues.apache.org/jira/browse/BEAM-2171
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2171) Power function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-2171.
-
Resolution: Fixed

> Power function
> --
>
> Key: BEAM-2171
> URL: https://issues.apache.org/jira/browse/BEAM-2171
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2171) Power function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-2171:
---

> Power function
> --
>
> Key: BEAM-2171
> URL: https://issues.apache.org/jira/browse/BEAM-2171
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2656

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3430

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2524) Update Google Cloud Console URL returned by DataflowRunner to support regions.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092243#comment-16092243
 ] 

ASF GitHub Bot commented on BEAM-2524:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3455


> Update Google Cloud Console URL returned by DataflowRunner to support regions.
> --
>
> Key: BEAM-2524
> URL: https://issues.apache.org/jira/browse/BEAM-2524
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Robert Burke
>Assignee: Robert Burke
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Both the Java and Python Dataflow Runners need to be updated with a 
> regionalized form of the Google Cloud Console URL to support multiple 
> Dataflow Regions.
> The new URL format will be:
> https://console.cloud.corp.google.com/dataflow/jobsDetail/locations//jobs/?project=



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3455: [BEAM-2524] Update Dataflow FE URLs with regionaliz...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3455


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Accept Region in Dataflow Monitoring Page URL

[tgroh]

Accept Region in Dataflow Monitoring Page URL

Update Google Cloud Dataflow FE URLs from the Dataflow Runners to
regionalized paths.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/111603a9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/111603a9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/111603a9

Branch: refs/heads/master
Commit: 111603a9952f415fa1386046f7a2d3bde5b6532d
Parents: 2d5b6d7
Author: Robert Burke 
Authored: Tue Jun 27 15:41:56 2017 -0700
Committer: Thomas Groh 
Committed: Tue Jul 18 14:49:56 2017 -0700

--
 .../beam/runners/dataflow/DataflowPipelineJob.java  | 14 --
 .../beam/runners/dataflow/DataflowRunner.java   |  3 ++-
 .../beam/runners/dataflow/util/MonitoringUtil.java  | 16 +---
 .../dataflow/BatchStatefulParDoOverridesTest.java   |  1 +
 .../dataflow/DataflowPipelineTranslatorTest.java|  1 +
 .../runners/dataflow/internal/apiclient.py  |  7 +--
 .../runners/dataflow/test_dataflow_runner.py|  5 +++--
 7 files changed, 37 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/111603a9/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index e30d426..e736373 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -169,6 +169,13 @@ public class DataflowPipelineJob implements PipelineResult 
{
   }
 
   /**
+   * Get the region this job exists in.
+   */
+  public String getRegion() {
+return dataflowOptions.getRegion();
+  }
+
+  /**
* Returns a new {@link DataflowPipelineJob} for the job that replaced this 
one, if applicable.
*
* @throws IllegalStateException if called before the job has terminated or 
if the job terminated
@@ -344,7 +351,9 @@ public class DataflowPipelineJob implements PipelineResult {
   getJobId(),
   getReplacedByJob().getJobId(),
   MonitoringUtil.getJobMonitoringPageURL(
-  getReplacedByJob().getProjectId(), 
getReplacedByJob().getJobId()));
+  getReplacedByJob().getProjectId(),
+  getRegion(),
+  getReplacedByJob().getJobId()));
   break;
 default:
   LOG.info("Job {} failed with status {}.", getJobId(), state);
@@ -422,7 +431,8 @@ public class DataflowPipelineJob implements PipelineResult {
 "Failed to cancel job in state %s, "
 + "please go to the Developers Console to cancel it 
manually: %s",
 state,
-MonitoringUtil.getJobMonitoringPageURL(getProjectId(), 
getJobId()));
+MonitoringUtil.getJobMonitoringPageURL(
+getProjectId(), getRegion(), getJobId()));
 LOG.warn(errorMsg);
 throw new IOException(errorMsg, e);
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/111603a9/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 8935759..57a5ea5 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -679,7 +679,8 @@ public class DataflowRunner extends 
PipelineRunner {
 }
 
 LOG.info("To access the Dataflow monitoring console, please navigate to 
{}",
-MonitoringUtil.getJobMonitoringPageURL(options.getProject(), 
jobResult.getId()));
+MonitoringUtil.getJobMonitoringPageURL(
+  options.getProject(), options.getRegion(), jobResult.getId()));
 System.out.println("Submitted job: " + jobResult.getId());
 
 LOG.info("To cancel the job using the 'gcloud' tool, run:\n> {}",

http://git-wip-us.apache.org/repos/asf/beam/blob/111603a9/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java

[1/2] beam git commit: This closes #3455

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 2d5b6d745 -> dd9e866e0


This closes #3455


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dd9e866e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dd9e866e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dd9e866e

Branch: refs/heads/master
Commit: dd9e866e087351e395e464e07d512b2e8db107c4
Parents: 2d5b6d7 111603a
Author: Thomas Groh 
Authored: Tue Jul 18 14:49:56 2017 -0700
Committer: Thomas Groh 
Committed: Tue Jul 18 14:49:56 2017 -0700

--
 .../beam/runners/dataflow/DataflowPipelineJob.java  | 14 --
 .../beam/runners/dataflow/DataflowRunner.java   |  3 ++-
 .../beam/runners/dataflow/util/MonitoringUtil.java  | 16 +---
 .../dataflow/BatchStatefulParDoOverridesTest.java   |  1 +
 .../dataflow/DataflowPipelineTranslatorTest.java|  1 +
 .../runners/dataflow/internal/apiclient.py  |  7 +--
 .../runners/dataflow/test_dataflow_runner.py|  5 +++--
 7 files changed, 37 insertions(+), 10 deletions(-)
--

[jira] [Updated] (BEAM-2463) EXP Function

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2463:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> EXP Function
> 
>
> Key: BEAM-2463
> URL: https://issues.apache.org/jira/browse/BEAM-2463
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] branch asf-site updated (c18b828 -> 7ccef23)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from c18b828  Prepare repository for deployment.
 add e6dcf8d  [BEAM-2500] List Amazon S3 File System as a planned I/O.
 add 6a6b93a  This closes #257
 new 7ccef23  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html | 4 
 src/documentation/io/built-in.md | 4 
 2 files changed, 8 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7ccef23ba52a6ac7024c59ac29ef7bc4e48803fe
Author: Mergebot 
AuthorDate: Tue Jul 18 21:47:53 2017 +

Prepare repository for deployment.
---
 content/documentation/io/built-in/index.html | 4 
 1 file changed, 4 insertions(+)

diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index 90e4c35..25368d0 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -204,6 +204,10 @@
 NameLanguageJIRA
   
   
+Amazon S3 File SystemJava
+https://issues.apache.org/jira/browse/BEAM-2500;>BEAM-2500
+  
+  
 Apache DistributedLogJava
 https://issues.apache.org/jira/browse/BEAM-607;>BEAM-607
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Updated] (BEAM-2310) support data type: DECIMAL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2310:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> support data type: DECIMAL
> --
>
> Key: BEAM-2310
> URL: https://issues.apache.org/jira/browse/BEAM-2310
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2309) support relation operator: Values

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2309:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> support relation operator: Values
> -
>
> Key: BEAM-2309
> URL: https://issues.apache.org/jira/browse/BEAM-2309
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: 2.2.0
>
>
> Values operator corresponds to SQL like:
> {code:sql}
> select LOCALTIME
> {code}
> which does not query from any table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2462) LOG10 Function

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2462:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> LOG10 Function
> --
>
> Key: BEAM-2462
> URL: https://issues.apache.org/jira/browse/BEAM-2462
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] branch mergebot updated (992a3fc -> 6a6b93a)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 992a3fc  This closes #259
 add c18b828  Prepare repository for deployment.
 new e6dcf8d  [BEAM-2500] List Amazon S3 File System as a planned I/O.
 new 6a6b93a  This closes #257

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/capability-matrix/index.html   | 72 +-
 src/documentation/io/built-in.md   |  4 ++
 2 files changed, 75 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/02: [BEAM-2500] List Amazon S3 File System as a planned I/O.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e6dcf8d6f5efdb7db5fa1f9fb08d4b67350ee2cb
Author: Luke Cwik 
AuthorDate: Wed Jul 12 12:17:16 2017 -0700

[BEAM-2500] List Amazon S3 File System as a planned I/O.
---
 src/documentation/io/built-in.md | 4 
 1 file changed, 4 insertions(+)

diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 0927d10..2db36e5 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -72,6 +72,10 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 NameLanguageJIRA
   
   
+Amazon S3 File SystemJava
+https://issues.apache.org/jira/browse/BEAM-2500;>BEAM-2500
+  
+  
 Apache DistributedLogJava
 https://issues.apache.org/jira/browse/BEAM-607;>BEAM-607
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Updated] (BEAM-2195) support conditional functions & operators

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2195:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> support conditional functions & operators
> -
>
> Key: BEAM-2195
> URL: https://issues.apache.org/jira/browse/BEAM-2195
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: 2.2.0
>
>
> https://calcite.apache.org/docs/reference.html#conditional-functions-and-operators



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2255) support ORDER BY clause

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2255:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> support ORDER BY clause
> ---
>
> Key: BEAM-2255
> URL: https://issues.apache.org/jira/browse/BEAM-2255
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: 2.2.0
>
>
> Currently Beam does not have `global sort` support[1]. So we will implement a 
> limited version of `ORDER BY` using `org.apache.beam.sdk.transforms.Top` for 
> now. So it will support sql like:
> {code:sql}
> select * from t order by id desc limit 10;
> {code}
> but not support `ORDER BY` without a `limit`:
> {code:sql}
> select * from t order by id desc
> {code}
> and also not support `ORDER BY` with a `offset`:
> {code:sql}
> select * from t order by id desc limit 10, 10;
> {code}
> [1]. 
> https://lists.apache.org/thread.html/bc0e65a3bb653b8fd0db96bcd4c9da5af71a71af5a5639a472167808@1464278191@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] 02/02: This closes #257

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6a6b93a876a342e652ba1e12e0beed585e275d3f
Merge: c18b828 e6dcf8d
Author: Mergebot 
AuthorDate: Tue Jul 18 21:45:15 2017 +

This closes #257

 src/documentation/io/built-in.md | 4 
 1 file changed, 4 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Updated] (BEAM-2160) add support Math functions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2160:
-
Fix Version/s: 2.2.0

> add support Math functions 
> ---
>
> Key: BEAM-2160
> URL: https://issues.apache.org/jira/browse/BEAM-2160
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Tarush Grover
>  Labels: bundle, dsl_sql_merge, split
> Fix For: 2.2.0
>
>
> All Math functions listed as below:
> {code}
> POWER(numeric1, numeric2) 
> ABS(numeric) 
> MOD(numeric, numeric) 
> SQRT(numeric) 
> LN(numeric)   
> LOG10(numeric) 
> EXP(numeric)  
> CEIL(numeric) 
> FLOOR(numeric)
> RAND([seed])  
> RAND_INTEGER([seed, ] numeric)
> ACOS(numeric)
> ASIN(numeric) 
> ATAN(numeric) 
> ATAN2(numeric, numeric)   
> COS(numeric)  
> COT(numeric)  
> DEGREES(numeric)  
> PI()  
> RADIANS(numeric)  
> ROUND(numeric1, numeric2) 
> SIGN(numeric) 
> SIN(numeric)  
> TAN(numeric)  
> TRUNCATE(numeric1, numeric2)  
> {code}
> See  
> https://calcite.apache.org/docs/reference.html#arithmetic-operators-and-functions
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2149) BeamKafkaCSVTable: support column type other than string

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2149:
-
Fix Version/s: (was: 2.1.0)
   2.2.0

> BeamKafkaCSVTable: support column type other than string
> 
>
> Key: BEAM-2149
> URL: https://issues.apache.org/jira/browse/BEAM-2149
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>Priority: Minor
> Fix For: 2.2.0
>
>
> Currently, BeamKafkaCSVTable only supports String column, need to support 
> other types.
> Also, use a more robust library to parse CSV.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2289) add unit tests for BeamSQLFilterFn and BeamSQLProjectFn

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2289.

Resolution: Fixed

> add unit tests for BeamSQLFilterFn and BeamSQLProjectFn
> ---
>
> Key: BEAM-2289
> URL: https://issues.apache.org/jira/browse/BEAM-2289
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2289) add unit tests for BeamSQLFilterFn and BeamSQLProjectFn

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2289:
--

> add unit tests for BeamSQLFilterFn and BeamSQLProjectFn
> ---
>
> Key: BEAM-2289
> URL: https://issues.apache.org/jira/browse/BEAM-2289
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4401

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2625) Pull request template could use a little more detail

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2625.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Pull request template could use a little more detail
> 
>
> Key: BEAM-2625
> URL: https://issues.apache.org/jira/browse/BEAM-2625
> Project: Beam
>  Issue Type: Bug
>  Components: project-management
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: Not applicable
>
>
> In the mergebot world where we might have contributors curate good commits 
> more frequently, adding that to the PR checklist seems nice.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4400

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2470) Inconsistent behavior on the functioning of the dataflow templates?

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2470:
-

Assignee: Thomas Groh  (was: Kenneth Knowles)

> Inconsistent behavior on the functioning of the dataflow templates?
> ---
>
> Key: BEAM-2470
> URL: https://issues.apache.org/jira/browse/BEAM-2470
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>Assignee: Thomas Groh
>
> 0
> down vote
> favorite
> When i create a dataflow template, the characteristics of Runtime parameters 
> are not persisted in the template file. At runtime, if i try to pass a value 
> for this parameter, i take a 400 error
> I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
> My parameters are the following :
> {code:java}
> trait MyParameters extends PipelineOptions {
>   def getInput: ValueProvider[String]
>   def setInput(value: ValueProvider[String]): Unit
> }
> {code}
> They are registred with this code
> {code:java}
> val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: 
> _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
> PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
> implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
> {code}
> To create the template i call sbt with this parameters :
> {code:java}
> run-main jobs.XmlImportJob--runner=DataflowRunner --project=MyProject  
> --templateLocation=gs://myBucket/XmlImportTemplate  
> --tempLocation=gs://myBucket/staging --instance=myInstance
> {code}
> If i pass explicitly --input, it becomes a StaticValue instead of 
> RuntimeValue, and this time, i can see it in the template file.
> The template is called from a google function watching a bucket storage 
> (inspired from 
> https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/)
>  :
> {code:java}
> ...
> dataflow.projects.templates.create({
> projectId: projectId,
> resource: {
> parameters: {
> input: `gs://${file.bucket}/${file.name}`
> },
> jobName: jobs[job].name,
> gcsPath: 'gs://MyBucket/MyTemplate'
> }
> }
> ...
> {code}
> The 400 error :
> {code:java}
> problem running dataflow template, error was: { Error: (109c1c52dc52fec7): 
> The workflow could not be created. Causes: (109c1c52dc52fb8e): Found 
> unexpected parameters: ['input' (perhaps you meant 'runner')] at 
> Request._callback 
> (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
>  at Request.self.callback 
> (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22) 
> at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at 
> Request.  at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at 
> IncomingMessage. 
> (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12) 
> at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 
> 400, errors: [ { message: '(109c1c52dc52fec7): The workflow could not be 
> created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: [\'input\' 
> (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ] }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2470) Inconsistent behavior on the functioning of the dataflow templates?

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092215#comment-16092215
 ] 

Kenneth Knowles commented on BEAM-2470:
---

Sending to component owner for DataflowRunner.

> Inconsistent behavior on the functioning of the dataflow templates?
> ---
>
> Key: BEAM-2470
> URL: https://issues.apache.org/jira/browse/BEAM-2470
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>Assignee: Thomas Groh
>
> 0
> down vote
> favorite
> When i create a dataflow template, the characteristics of Runtime parameters 
> are not persisted in the template file. At runtime, if i try to pass a value 
> for this parameter, i take a 400 error
> I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
> My parameters are the following :
> {code:java}
> trait MyParameters extends PipelineOptions {
>   def getInput: ValueProvider[String]
>   def setInput(value: ValueProvider[String]): Unit
> }
> {code}
> They are registred with this code
> {code:java}
> val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: 
> _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
> PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
> implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
> {code}
> To create the template i call sbt with this parameters :
> {code:java}
> run-main jobs.XmlImportJob--runner=DataflowRunner --project=MyProject  
> --templateLocation=gs://myBucket/XmlImportTemplate  
> --tempLocation=gs://myBucket/staging --instance=myInstance
> {code}
> If i pass explicitly --input, it becomes a StaticValue instead of 
> RuntimeValue, and this time, i can see it in the template file.
> The template is called from a google function watching a bucket storage 
> (inspired from 
> https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/)
>  :
> {code:java}
> ...
> dataflow.projects.templates.create({
> projectId: projectId,
> resource: {
> parameters: {
> input: `gs://${file.bucket}/${file.name}`
> },
> jobName: jobs[job].name,
> gcsPath: 'gs://MyBucket/MyTemplate'
> }
> }
> ...
> {code}
> The 400 error :
> {code:java}
> problem running dataflow template, error was: { Error: (109c1c52dc52fec7): 
> The workflow could not be created. Causes: (109c1c52dc52fb8e): Found 
> unexpected parameters: ['input' (perhaps you meant 'runner')] at 
> Request._callback 
> (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
>  at Request.self.callback 
> (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22) 
> at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at 
> Request.  at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at 
> IncomingMessage. 
> (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12) 
> at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 
> 400, errors: [ { message: '(109c1c52dc52fec7): The workflow could not be 
> created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: [\'input\' 
> (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ] }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2333) Rehydrate Pipeline from Runner API proto

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2333:
--
Issue Type: New Feature  (was: Bug)

> Rehydrate Pipeline from Runner API proto
> 
>
> Key: BEAM-2333
> URL: https://issues.apache.org/jira/browse/BEAM-2333
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2470) Inconsistent behavior on the functioning of the dataflow templates?

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2470:
--
Component/s: (was: runner-core)
 runner-dataflow

> Inconsistent behavior on the functioning of the dataflow templates?
> ---
>
> Key: BEAM-2470
> URL: https://issues.apache.org/jira/browse/BEAM-2470
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>Assignee: Kenneth Knowles
>
> 0
> down vote
> favorite
> When i create a dataflow template, the characteristics of Runtime parameters 
> are not persisted in the template file. At runtime, if i try to pass a value 
> for this parameter, i take a 400 error
> I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
> My parameters are the following :
> {code:java}
> trait MyParameters extends PipelineOptions {
>   def getInput: ValueProvider[String]
>   def setInput(value: ValueProvider[String]): Unit
> }
> {code}
> They are registred with this code
> {code:java}
> val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: 
> _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
> PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
> implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
> {code}
> To create the template i call sbt with this parameters :
> {code:java}
> run-main jobs.XmlImportJob--runner=DataflowRunner --project=MyProject  
> --templateLocation=gs://myBucket/XmlImportTemplate  
> --tempLocation=gs://myBucket/staging --instance=myInstance
> {code}
> If i pass explicitly --input, it becomes a StaticValue instead of 
> RuntimeValue, and this time, i can see it in the template file.
> The template is called from a google function watching a bucket storage 
> (inspired from 
> https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/)
>  :
> {code:java}
> ...
> dataflow.projects.templates.create({
> projectId: projectId,
> resource: {
> parameters: {
> input: `gs://${file.bucket}/${file.name}`
> },
> jobName: jobs[job].name,
> gcsPath: 'gs://MyBucket/MyTemplate'
> }
> }
> ...
> {code}
> The 400 error :
> {code:java}
> problem running dataflow template, error was: { Error: (109c1c52dc52fec7): 
> The workflow could not be created. Causes: (109c1c52dc52fb8e): Found 
> unexpected parameters: ['input' (perhaps you meant 'runner')] at 
> Request._callback 
> (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
>  at Request.self.callback 
> (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22) 
> at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at 
> Request.  at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at 
> IncomingMessage. 
> (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12) 
> at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 
> 400, errors: [ { message: '(109c1c52dc52fec7): The workflow could not be 
> created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: [\'input\' 
> (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ] }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)