[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962383#comment-15962383
 ] 

ASF GitHub Bot commented on BEAM-301:
-

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2479

[BEAM-301] Add a Beam SQL DSL

The initial PR to setup skeleton of Beam SQL. 
Pass `mvn verify` locally, a Jenkins job is required later for pre-commit 
check.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam sql_init_pr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2479


commit 07904d383bb2b14badb031be2bf5222e2d2c8d03
Author: mingmxu 
Date:   2017-04-10T02:49:08Z

initial PR to setup Beam SQL skeleton

initial PR to setup Beam SQL skeleton

format fix in pom.xml




> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2479: [BEAM-301] Add a Beam SQL DSL

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2479

[BEAM-301] Add a Beam SQL DSL

The initial PR to setup skeleton of Beam SQL. 
Pass `mvn verify` locally, a Jenkins job is required later for pre-commit 
check.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam sql_init_pr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2479


commit 07904d383bb2b14badb031be2bf5222e2d2c8d03
Author: mingmxu 
Date:   2017-04-10T02:49:08Z

initial PR to setup Beam SQL skeleton

initial PR to setup Beam SQL skeleton

format fix in pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2478: Dsl sql init

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/2478


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2478: Dsl sql init

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2478

Dsl sql init

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam dsl_sql_init

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2478.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2478


commit 57fddbd8e3705254a52e5e478a0dcf9493142203
Author: mingmxu 
Date:   2017-03-09T05:09:03Z

save work: BaseBeamTable, BeamRelNode, BeamRule

commit 673c25871904124010fe59eed0bcb0c9a9161100
Author: mingmxu 
Date:   2017-03-09T19:27:50Z

save work: BeamSQLSpELExecutor as an executor for relational operators;

commit 2bce67f5727ecad9fe9973fa4af6022f4f4cd6f5
Author: mingmxu 
Date:   2017-03-09T19:49:57Z

save work, update buildBeamPipeline()

commit 64f8e89e169151fd323cf1bb45432281a19cedef
Author: mingmxu 
Date:   2017-03-09T22:30:10Z

complete first query: SELECT  FROM  WHERE 

commit 616484c58a265fee31c64d66360689cd3ba71355
Author: mingmxu 
Date:   2017-03-10T21:48:49Z

save work: support INSERT

commit 1272621c37ba76fd7a1997ba43f19d281e8030aa
Author: mingmxu 
Date:   2017-03-10T22:22:04Z

update license

commit e11dfcf9e1b0aae0eb26ebd2246a914957b8e712
Author: mingmxu 
Date:   2017-03-11T01:45:46Z

save work: add BeamSqlRunner as the tier1 interface;

commit 1d80bb5b7813949ca3bb05fd92bdd706d4e43084
Author: mingmxu 
Date:   2017-03-11T07:22:25Z

remove test code;

commit b51ce8d51c2df1be822e7d28400c4a80145fa3ac
Author: mingmxu 
Date:   2017-03-11T19:15:49Z

fix checkstyle 1

commit 700ab00abf79e31b9a14683f8d708ace0bda6923
Author: XuMingmin 
Date:   2017-03-11T19:17:29Z

Merge pull request #1 from apache/master

sync up latest code

commit f1afcfe7fe81ac1427a18d367c0536933cf85e63
Author: mingmxu 
Date:   2017-03-12T07:30:27Z

refine BaseBeamTable, and add one example BeamSqlExample.

commit 3c53db8aa941c3be8ec8171c98cb4405febfac40
Author: mingmxu 
Date:   2017-03-13T02:44:53Z

add e2e test cases for planner.

commit 09df9b5c15dd5b409d9da937225caa073a663255
Author: XuMingmin 
Date:   2017-03-15T21:45:17Z

Merge pull request #3 from apache/master

sync-up code

commit d5fe32755b626f79abfd950a71e6b46550fa0d2b
Author: mingmxu 
Date:   2017-03-30T18:48:50Z

save work

commit 83ea09ea195315afac94116a248b4995231e2d72
Author: XuMingmin 
Date:   2017-04-03T22:28:03Z

Merge pull request #6 from apache/master

rebase

commit 8b8347268035ddb37bfc5865306368c5dc2cad67
Author: XuMingmin 
Date:   2017-04-05T21:50:06Z

Merge pull request #7 from apache/master

rebase

commit 38f4b6168f5928d01c01bcffc17a7bee6de596cb
Author: XuMingmin 
Date:   2017-04-07T06:21:44Z

Merge pull request #8 from apache/master

rebase

commit d038df411aff0a68df7c73d6a4f67757923b1ae4
Author: XuMingmin 
Date:   2017-04-07T18:28:21Z

Merge pull request #9 from XuMingmin/BEAM-301

rebase from BEAM-301

commit c96fa7bd1da706031cb99f81911c3f01eee86028
Author: mingmxu 
Date:   2017-04-09T04:10:31Z

refine BeamSQLRow;

commit dd266b5d5b297149b22c9bff4b9621c9075e9e93
Author: mingmxu 
Date:   2017-04-10T00:14:03Z

Merge remote-tracking branch 'apache/DSL_SQL' into sql-rowrecord

Conflicts:
dsls/pom.xml

commit 4c2fc05dabfad47779ea1f164828f2228abe9994
Author: mingmxu 
Date:   2017-04-10T00:24:02Z

revert files that're not impacted

commit 1f55b6d1c6171c3ced46148ef6015a9ae2f85edf
Author: mingmxu 
Date:   2017-04-10T00:35:23Z

code cleanup: reformat imports

commit a2014ba09ad2903835229f3f3432fdd0094944f6
Author: XuMingmin 
Date:   

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3232

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1574

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Dataflow #289

[Apache Jenkins Server]

See 


Changes:

[aviemzur] [BEAM-1294] Long running UnboundedSource Readers

[jbonofre] [BEAM-1921] Expose connection properties in JdbcIO

--
[...truncated 245.25 KB...]
 ! 4e0c9d4...2c40dbc refs/pull/2033/merge -> origin/pr/2033/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2057/merge: No such 
file or directory
 ! 796aa16...a1a6d0a refs/pull/2057/merge -> origin/pr/2057/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2113/merge: No such 
file or directory
 ! 14a3303...1db71bf refs/pull/2113/merge -> origin/pr/2113/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2154/merge: No such 
file or directory
 ! c50274b...bdc6681 refs/pull/2154/merge -> origin/pr/2154/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2166/merge: No such 
file or directory
 ! 4ba2353...8710f00 refs/pull/2166/merge -> origin/pr/2166/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2204/merge: No such 
file or directory
 ! 9f75a1f...303a474 refs/pull/2204/merge -> origin/pr/2204/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2207/merge: No such 
file or directory
 ! 42f3ccd...fa13934 refs/pull/2207/merge -> origin/pr/2207/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2216/merge: No such 
file or directory
 ! ac7e11c...de74b7c refs/pull/2216/merge -> origin/pr/2216/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2263/merge: No such 
file or directory
 ! 366a64c...0d58195b refs/pull/2263/merge -> origin/pr/2263/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2266/merge: No such 
file or directory
 ! 3dd088a...530f7f1 refs/pull/2266/merge -> origin/pr/2266/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2273/merge: No such 
file or directory
 ! 6884de4...7214954 refs/pull/2273/merge -> origin/pr/2273/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2275/merge: No such 
file or directory
 ! 77bf99d...735cd94 refs/pull/2275/merge -> origin/pr/2275/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2286/merge: No such 
file or directory
 ! 4c95c1a...1521d3c refs/pull/2286/merge -> origin/pr/2286/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2331/merge: No such 
file or directory
 ! 96215ef...1d3e400 refs/pull/2331/merge -> origin/pr/2331/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2339/merge: No such 
file or directory
 ! 10007be...f2dcb59 refs/pull/2339/merge -> origin/pr/2339/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2340/merge: No such 
file or directory
 ! 75500d8...6ed8493 refs/pull/2340/merge -> origin/pr/2340/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2341/merge: No such 
file or directory
 ! cb42008...785e8c2 refs/pull/2341/merge -> origin/pr/2341/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2344/merge: No such 
file or directory
 ! 1cb169a...87a5839 refs/pull/2344/merge -> origin/pr/2344/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2358/merge: No such 
file or directory
 ! f311c97...c73c0b4 refs/pull/2358/merge -> origin/pr/2358/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2363/merge: No such 
file or directory
 ! 54530ea...41f60a4 refs/pull/2363/merge -> origin/pr/2363/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2401/merge: No such 
file or directory
 ! d0b3c3a...27d2beb refs/pull/2401/merge -> origin/pr/2401/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2406/merge: No such 
file or directory
 ! 60b3cea...382cd34 refs/pull/2406/merge -> origin/pr/2406/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2417/merge: No such 
file or directory
 ! 7317b34...6d079a4 refs/pull/2417/merge -> origin/pr/2417/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2421/merge: No such 
file or directory
 ! c987f51...272146e refs/pull/2421/merge -> origin/pr/2421/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2423/merge: No such 
file or directory
 ! 30b6885...5b1da5f 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3231

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1573

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1921) expose connectionProperties in JdbcIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-1921.

   Resolution: Fixed
Fix Version/s: First stable release

> expose connectionProperties in JdbcIO
> -
>
> Key: BEAM-1921
> URL: https://issues.apache.org/jira/browse/BEAM-1921
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: First stable release
>
>
> expose `connectionProperties` for developers to add connection properties



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1921) expose connectionProperties in JdbcIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962253#comment-15962253
 ] 

ASF GitHub Bot commented on BEAM-1921:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2477


> expose connectionProperties in JdbcIO
> -
>
> Key: BEAM-1921
> URL: https://issues.apache.org/jira/browse/BEAM-1921
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: First stable release
>
>
> expose `connectionProperties` for developers to add connection properties



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2477: [BEAM-1921] expose connectionProperties in JdbcIO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2477


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: [BEAM-1921] This closes #2477

[jbonofre]

[BEAM-1921] This closes #2477


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/713701dd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/713701dd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/713701dd

Branch: refs/heads/master
Commit: 713701ddc43c3987e261ecd08b34131ea80c3999
Parents: efd785f 6da0b27
Author: Jean-Baptiste Onofré 
Authored: Sun Apr 9 22:16:17 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Sun Apr 9 22:16:17 2017 +0200

--
 .../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 21 
 1 file changed, 21 insertions(+)
--

[1/2] beam git commit: [BEAM-1921] Expose connection properties in JdbcIO

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master efd785f88 -> 713701ddc


[BEAM-1921] Expose connection properties in JdbcIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6da0b27a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6da0b27a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6da0b27a

Branch: refs/heads/master
Commit: 6da0b27aa56e8a133e206b5eb84b9ab937a2b17e
Parents: efd785f
Author: mingmxu 
Authored: Sun Apr 9 11:34:26 2017 -0700
Committer: Jean-Baptiste Onofré 
Committed: Sun Apr 9 22:14:42 2017 +0200

--
 .../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 21 
 1 file changed, 21 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6da0b27a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
--
diff --git 
a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java 
b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 4754c98..dc31130 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -174,6 +174,7 @@ public class JdbcIO {
 @Nullable abstract String getUrl();
 @Nullable abstract String getUsername();
 @Nullable abstract String getPassword();
+@Nullable abstract String getConnectionProperties();
 @Nullable abstract DataSource getDataSource();
 
 abstract Builder builder();
@@ -184,6 +185,7 @@ public class JdbcIO {
   abstract Builder setUrl(String url);
   abstract Builder setUsername(String username);
   abstract Builder setPassword(String password);
+  abstract Builder setConnectionProperties(String connectionProperties);
   abstract Builder setDataSource(DataSource dataSource);
   abstract DataSourceConfiguration build();
 }
@@ -217,6 +219,22 @@ public class JdbcIO {
   return builder().setPassword(password).build();
 }
 
+/**
+ * Sets the connection properties passed to driver.connect(...).
+ * Format of the string must be [propertyName=property;]*
+ *
+ * NOTE - The "user" and "password" properties can be add via {@link 
#withUsername(String)},
+ * {@link #withPassword(String)}, so they do not need to be included here.
+ * @param connectionProperties
+ * @return
+ */
+public DataSourceConfiguration withConnectionProperties(String 
connectionProperties) {
+  checkArgument(connectionProperties != null, 
"DataSourceConfiguration.create(driver, url)"
+  + ".withConnectionProperties(connectionProperties) "
+  + "called with null connectionProperties");
+  return builder().setConnectionProperties(connectionProperties).build();
+}
+
 private void populateDisplayData(DisplayData.Builder builder) {
   if (getDataSource() != null) {
 builder.addIfNotNull(DisplayData.item("dataSource", 
getDataSource().getClass().getName()));
@@ -238,6 +256,9 @@ public class JdbcIO {
 basicDataSource.setUrl(getUrl());
 basicDataSource.setUsername(getUsername());
 basicDataSource.setPassword(getPassword());
+if (getConnectionProperties() != null) {
+  basicDataSource.setConnectionProperties(getConnectionProperties());
+}
 return basicDataSource.getConnection();
   }
 }

[jira] [Resolved] (BEAM-1900) create feature branch 'dsls_sql' for Beam SQL

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-1900.

   Resolution: Fixed
Fix Version/s: Not applicable

> create feature branch 'dsls_sql' for Beam SQL
> -
>
> Key: BEAM-1900
> URL: https://issues.apache.org/jira/browse/BEAM-1900
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1900) create feature branch 'dsls_sql' for Beam SQL

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962247#comment-15962247
 ] 

Jean-Baptiste Onofré commented on BEAM-1900:


I created the {{DSL_SQL}} branch, ready to receive PR:

https://github.com/apache/beam/tree/DSL_SQL

> create feature branch 'dsls_sql' for Beam SQL
> -
>
> Key: BEAM-1900
> URL: https://issues.apache.org/jira/browse/BEAM-1900
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1294) Long running UnboundedSource Readers

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur resolved BEAM-1294.
-
   Resolution: Implemented
Fix Version/s: First stable release

> Long running UnboundedSource Readers
> 
>
> Key: BEAM-1294
> URL: https://issues.apache.org/jira/browse/BEAM-1294
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
> Fix For: First stable release
>
>
> When reading from an UnboundedSource, current implementation will cause each 
> split to create a new Reader every micro-batch.
> As long as the overhead of creating a reader is relatively low, it's 
> reasonable (though I'd still be happy to get rid of), but in cases where the 
> creation overhead is large it becomes unreasonable forcing large batches.
> One way to solve this could be to create a pool of lazy-init readers to serve 
> each executor, maybe via Broadcast variables. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1294) Long running UnboundedSource Readers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962245#comment-15962245
 ] 

ASF GitHub Bot commented on BEAM-1294:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2033


> Long running UnboundedSource Readers
> 
>
> Key: BEAM-1294
> URL: https://issues.apache.org/jira/browse/BEAM-1294
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> When reading from an UnboundedSource, current implementation will cause each 
> split to create a new Reader every micro-batch.
> As long as the overhead of creating a reader is relatively low, it's 
> reasonable (though I'd still be happy to get rid of), but in cases where the 
> creation overhead is large it becomes unreasonable forcing large batches.
> One way to solve this could be to create a pool of lazy-init readers to serve 
> each executor, maybe via Broadcast variables. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2033: [BEAM-1294] Long running UnboundedSource Readers

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2033


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1294] Long running UnboundedSource Readers

[aviemzur]

Repository: beam
Updated Branches:
  refs/heads/master a0cfccda4 -> efd785f88


[BEAM-1294] Long running UnboundedSource Readers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d958796b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d958796b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d958796b

Branch: refs/heads/master
Commit: d958796b525861764318f0c022e4987aa64ac300
Parents: a0cfccd
Author: Aviem Zur 
Authored: Fri Feb 17 12:35:49 2017 +0200
Committer: Aviem Zur 
Committed: Sun Apr 9 22:42:57 2017 +0300

--
 .../beam/runners/spark/io/MicrobatchSource.java | 113 ---
 .../beam/runners/spark/io/SourceDStream.java|  11 +-
 .../spark/stateful/StateSpecFunctions.java  |   6 +-
 .../ResumeFromCheckpointStreamingTest.java  |  14 ++-
 4 files changed, 118 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d958796b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
index ff818a1..002eb34 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
@@ -19,11 +19,18 @@
 package org.apache.beam.runners.spark.io;
 
 import com.google.api.client.util.BackOff;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
 import com.google.common.util.concurrent.Uninterruptibles;
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.NoSuchElementException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.io.BoundedSource;
@@ -49,29 +56,34 @@ import org.slf4j.LoggerFactory;
 public class MicrobatchSource
 extends BoundedSource {
   private static final Logger LOG = 
LoggerFactory.getLogger(MicrobatchSource.class);
+  private static volatile Cache 
readerCache;
 
   private final UnboundedSource source;
   private final Duration maxReadTime;
   private final int numInitialSplits;
   private final long maxNumRecords;
   private final int sourceId;
+  private final double readerCacheInterval;
 
   // each split of the underlying UnboundedSource is associated with a 
(consistent) id
   // to match it's corresponding CheckpointMark state.
   private final int splitId;
 
-  MicrobatchSource(UnboundedSource source,
-   Duration maxReadTime,
-   int numInitialSplits,
-   long maxNumRecords,
-   int splitId,
-   int sourceId) {
+  MicrobatchSource(
+  UnboundedSource source,
+  Duration maxReadTime,
+  int numInitialSplits,
+  long maxNumRecords,
+  int splitId,
+  int sourceId,
+  double readerCacheInterval) {
 this.source = source;
 this.maxReadTime = maxReadTime;
 this.numInitialSplits = numInitialSplits;
 this.maxNumRecords = maxNumRecords;
 this.splitId = splitId;
 this.sourceId = sourceId;
+this.readerCacheInterval = readerCacheInterval;
   }
 
   /**
@@ -101,7 +113,8 @@ public class MicrobatchSource(splits.get(i), maxReadTime, 1, 
numRecords[i], i, sourceId));
+  result.add(new MicrobatchSource<>(splits.get(i), maxReadTime, 1, 
numRecords[i], i, sourceId,
+  readerCacheInterval));
 }
 return result;
   }
@@ -113,12 +126,30 @@ public class MicrobatchSource

[jira] [Assigned] (BEAM-1922) DataSource in JdbcIO is not closed

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1922:
--

Assignee: Xu Mingmin  (was: Jean-Baptiste Onofré)

> DataSource in JdbcIO is not closed
> --
>
> Key: BEAM-1922
> URL: https://issues.apache.org/jira/browse/BEAM-1922
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> The line below is not closed properly, 
> {code}
> BasicDataSource basicDataSource = new BasicDataSource();
> {code}
> It's a potential issue for the other usage, although {{DataSource}} doesn't 
> have a {{close()}} method.
> {code}
> public static DataSourceConfiguration create(DataSource dataSource)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1922) DataSource in JdbcIO is not closed

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962242#comment-15962242
 ] 

Jean-Baptiste Onofré commented on BEAM-1922:


Yes, it's the easiest path to start with.

> DataSource in JdbcIO is not closed
> --
>
> Key: BEAM-1922
> URL: https://issues.apache.org/jira/browse/BEAM-1922
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> The line below is not closed properly, 
> {code}
> BasicDataSource basicDataSource = new BasicDataSource();
> {code}
> It's a potential issue for the other usage, although {{DataSource}} doesn't 
> have a {{close()}} method.
> {code}
> public static DataSourceConfiguration create(DataSource dataSource)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1922) DataSource in JdbcIO is not closed

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962241#comment-15962241
 ] 

Xu Mingmin commented on BEAM-1922:
--

would fix {{BasicDataSource}} first

> DataSource in JdbcIO is not closed
> --
>
> Key: BEAM-1922
> URL: https://issues.apache.org/jira/browse/BEAM-1922
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>
> The line below is not closed properly, 
> {code}
> BasicDataSource basicDataSource = new BasicDataSource();
> {code}
> It's a potential issue for the other usage, although {{DataSource}} doesn't 
> have a {{close()}} method.
> {code}
> public static DataSourceConfiguration create(DataSource dataSource)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1922) DataSource in JdbcIO is not closed

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1922:
--

Assignee: Jean-Baptiste Onofré  (was: Xu Mingmin)

> DataSource in JdbcIO is not closed
> --
>
> Key: BEAM-1922
> URL: https://issues.apache.org/jira/browse/BEAM-1922
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>
> The line below is not closed properly, 
> {code}
> BasicDataSource basicDataSource = new BasicDataSource();
> {code}
> It's a potential issue for the other usage, although {{DataSource}} doesn't 
> have a {{close()}} method.
> {code}
> public static DataSourceConfiguration create(DataSource dataSource)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1922) DataSource in JdbcIO is not closed

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1922:


 Summary: DataSource in JdbcIO is not closed
 Key: BEAM-1922
 URL: https://issues.apache.org/jira/browse/BEAM-1922
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Xu Mingmin
Assignee: Xu Mingmin


The line below is not closed properly, 
{code}
BasicDataSource basicDataSource = new BasicDataSource();
{code}

It's a potential issue for the other usage, although {{DataSource}} doesn't 
have a {{close()}} method.
{code}
public static DataSourceConfiguration create(DataSource dataSource)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

beam git commit: Add DSLs module

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL [created] 870e42dec


Add DSLs module


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/870e42de
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/870e42de
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/870e42de

Branch: refs/heads/DSL_SQL
Commit: 870e42dec6870d0170724a7eff849e95f9ff266b
Parents: a0cfccd
Author: Jean-Baptiste Onofré 
Authored: Tue Aug 9 11:17:32 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Sun Apr 9 21:22:02 2017 +0200

--
 dsls/pom.xml | 56 +++
 pom.xml  |  1 +
 2 files changed, 57 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/870e42de/dsls/pom.xml
--
diff --git a/dsls/pom.xml b/dsls/pom.xml
new file mode 100644
index 000..6e00171
--- /dev/null
+++ b/dsls/pom.xml
@@ -0,0 +1,56 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+  4.0.0
+
+  
+org.apache.beam
+beam-parent
+0.7.0-SNAPSHOT
+../pom.xml
+  
+
+  beam-dsls-parent
+  Apache Beam :: DSLs
+
+  
+
+  
+
+  
+
+  
+
+
+  org.apache.maven.plugins
+  maven-jar-plugin
+  
+
+  default-test-jar
+  
+test-jar
+  
+
+  
+
+  
+
+  
+
+
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/beam/blob/870e42de/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 09f3985..ef312c1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -152,6 +152,7 @@
 sdks/java/build-tools
 sdks
 runners
+dsls
 examples
 
 sdks/java/javadoc

[jira] [Commented] (BEAM-1921) expose connectionProperties in JdbcIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962236#comment-15962236
 ] 

ASF GitHub Bot commented on BEAM-1921:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2477

[BEAM-1921] expose connectionProperties in JdbcIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam JdbcIO_Para

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2477


commit 09df9b5c15dd5b409d9da937225caa073a663255
Author: XuMingmin 
Date:   2017-03-15T21:45:17Z

Merge pull request #3 from apache/master

sync-up code

commit 83ea09ea195315afac94116a248b4995231e2d72
Author: XuMingmin 
Date:   2017-04-03T22:28:03Z

Merge pull request #6 from apache/master

rebase

commit 8b8347268035ddb37bfc5865306368c5dc2cad67
Author: XuMingmin 
Date:   2017-04-05T21:50:06Z

Merge pull request #7 from apache/master

rebase

commit 38f4b6168f5928d01c01bcffc17a7bee6de596cb
Author: XuMingmin 
Date:   2017-04-07T06:21:44Z

Merge pull request #8 from apache/master

rebase

commit 49da33daf0e87c823f04a8b4bcfb03511026461c
Author: mingmxu 
Date:   2017-04-09T18:34:26Z

add `connectionProperties`




> expose connectionProperties in JdbcIO
> -
>
> Key: BEAM-1921
> URL: https://issues.apache.org/jira/browse/BEAM-1921
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>
> expose `connectionProperties` for developers to add connection properties



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2477: [BEAM-1921] expose connectionProperties in JdbcIO

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2477

[BEAM-1921] expose connectionProperties in JdbcIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam JdbcIO_Para

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2477


commit 09df9b5c15dd5b409d9da937225caa073a663255
Author: XuMingmin 
Date:   2017-03-15T21:45:17Z

Merge pull request #3 from apache/master

sync-up code

commit 83ea09ea195315afac94116a248b4995231e2d72
Author: XuMingmin 
Date:   2017-04-03T22:28:03Z

Merge pull request #6 from apache/master

rebase

commit 8b8347268035ddb37bfc5865306368c5dc2cad67
Author: XuMingmin 
Date:   2017-04-05T21:50:06Z

Merge pull request #7 from apache/master

rebase

commit 38f4b6168f5928d01c01bcffc17a7bee6de596cb
Author: XuMingmin 
Date:   2017-04-07T06:21:44Z

Merge pull request #8 from apache/master

rebase

commit 49da33daf0e87c823f04a8b4bcfb03511026461c
Author: mingmxu 
Date:   2017-04-09T18:34:26Z

add `connectionProperties`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1921) expose connectionProperties in JdbcIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-1921:
-
Summary: expose connectionProperties in JdbcIO  (was: expose 
`withConnectionProperties` in JdbcIO)

> expose connectionProperties in JdbcIO
> -
>
> Key: BEAM-1921
> URL: https://issues.apache.org/jira/browse/BEAM-1921
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>
> expose `connectionProperties` for developers to add connection properties



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1921) expose `withConnectionProperties` in JdbcIO

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1921:


 Summary: expose `withConnectionProperties` in JdbcIO
 Key: BEAM-1921
 URL: https://issues.apache.org/jira/browse/BEAM-1921
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Xu Mingmin
Assignee: Xu Mingmin
Priority: Minor


expose `connectionProperties` for developers to add connection properties



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #288

[Apache Jenkins Server]

See 


--
[...truncated 228.91 KB...]
 x [deleted] (none) -> origin/pr/971/head
 x [deleted] (none) -> origin/pr/971/merge
 x [deleted] (none) -> origin/pr/972/head
 x [deleted] (none) -> origin/pr/973/head
 x [deleted] (none) -> origin/pr/974/head
 x [deleted] (none) -> origin/pr/974/merge
 x [deleted] (none) -> origin/pr/975/head
 x [deleted] (none) -> origin/pr/975/merge
 x [deleted] (none) -> origin/pr/976/head
 x [deleted] (none) -> origin/pr/976/merge
 x [deleted] (none) -> origin/pr/977/head
 x [deleted] (none) -> origin/pr/977/merge
 x [deleted] (none) -> origin/pr/978/head
 x [deleted] (none) -> origin/pr/978/merge
 x [deleted] (none) -> origin/pr/979/head
 x [deleted] (none) -> origin/pr/979/merge
 x [deleted] (none) -> origin/pr/98/head
 x [deleted] (none) -> origin/pr/980/head
 x [deleted] (none) -> origin/pr/980/merge
 x [deleted] (none) -> origin/pr/981/head
 x [deleted] (none) -> origin/pr/982/head
 x [deleted] (none) -> origin/pr/982/merge
 x [deleted] (none) -> origin/pr/983/head
 x [deleted] (none) -> origin/pr/983/merge
 x [deleted] (none) -> origin/pr/984/head
 x [deleted] (none) -> origin/pr/984/merge
 x [deleted] (none) -> origin/pr/985/head
 x [deleted] (none) -> origin/pr/985/merge
 x [deleted] (none) -> origin/pr/986/head
 x [deleted] (none) -> origin/pr/986/merge
 x [deleted] (none) -> origin/pr/987/head
 x [deleted] (none) -> origin/pr/988/head
 x [deleted] (none) -> origin/pr/988/merge
 x [deleted] (none) -> origin/pr/989/head
 x [deleted] (none) -> origin/pr/989/merge
 x [deleted] (none) -> origin/pr/99/head
 x [deleted] (none) -> origin/pr/99/merge
 x [deleted] (none) -> origin/pr/990/head
 x [deleted] (none) -> origin/pr/990/merge
 x [deleted] (none) -> origin/pr/991/head
 x [deleted] (none) -> origin/pr/991/merge
 x [deleted] (none) -> origin/pr/992/head
 x [deleted] (none) -> origin/pr/992/merge
 x [deleted] (none) -> origin/pr/993/head
 x [deleted] (none) -> origin/pr/993/merge
 x [deleted] (none) -> origin/pr/994/head
 x [deleted] (none) -> origin/pr/994/merge
 x [deleted] (none) -> origin/pr/995/head
 x [deleted] (none) -> origin/pr/995/merge
 x [deleted] (none) -> origin/pr/996/head
 x [deleted] (none) -> origin/pr/996/merge
 x [deleted] (none) -> origin/pr/997/head
 x [deleted] (none) -> origin/pr/997/merge
 x [deleted] (none) -> origin/pr/998/head
 x [deleted] (none) -> origin/pr/999/head
 x [deleted] (none) -> origin/pr/999/merge
remote: Counting objects: 1488, done.
remote: Compressing objects:   1% (1/61)   remote: Compressing objects: 
  3% (2/61)   remote: Compressing objects:   4% (3/61)   
remote: Compressing objects:   6% (4/61)   remote: Compressing objects: 
  8% (5/61)   remote: Compressing objects:   9% (6/61)   
remote: Compressing objects:  11% (7/61)   remote: Compressing objects: 
 13% (8/61)   remote: Compressing objects:  14% (9/61)   
remote: Compressing objects:  16% (10/61)   remote: Compressing 
objects:  18% (11/61)   remote: Compressing objects:  19% (12/61)   
remote: Compressing objects:  21% (13/61)   remote: Compressing 
objects:  22% (14/61)   remote: Compressing objects:  24% (15/61)   
remote: Compressing objects:  26% (16/61)   remote: Compressing 
objects:  27% (17/61)   remote: Compressing objects:  29% (18/61)   
remote: Compressing objects:  31% (19/61)   remote: Compressing 
objects:  32% (20/61)   remote: Compressing objects:  34% (21/61)   
remote: Compressing objects:  36% (22/61)   remote: Compressing 
objects:  37% (23/61)   remote: Compressing objects:  39% (24/61)   
remote: Compressing objects:  40% (25/61)   remote: Compressing 
objects:  42% (26/61)   remote: Compressing objects:  44% (27/61)   
remote: Compressing objects:  45% (28/61)   remote: Compressing 
objects:  47% (29/61)   remote: Compressing objects:  49% (30/61)   
remote: Compressing objects:  50% (31/61)   remote: Compressing 
objects:  52% (32/61)   remote: Compressing objects:  54% (33/61)   
remote: 

[jira] [Assigned] (BEAM-1692) Convert to Codahale Metric interfaces where possible in WithMetricsSupport MetricRegistry

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1692:
---

Assignee: (was: Aviem Zur)

> Convert to Codahale Metric interfaces where possible in WithMetricsSupport 
> MetricRegistry
> -
>
> Key: BEAM-1692
> URL: https://issues.apache.org/jira/browse/BEAM-1692
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Aviem Zur
>Priority: Minor
>
> Currently in Spark runner all Beam metrics are reported by Codahale metrics 
> as Codahale {{Gauge}} metrics.
> Some of the metrics are formatted to look like their counterparts in 
> Codahale's {{GraphiteReporter}} but this is a hack and should be avoided if 
> possible.
> Instead, convert Beam metrics to their Codahale counterparts in 
> {{WithMetricsSupport}} getters where possible.
> Since Beam metrics are not 100% compatible with Codahale metrics, consider 
> where to make tradeoffs and stay with a simple {{Gauge}}.
> Also, rename {{WithMetricsSupport}} something like {{SparkBeamMetricRegistry}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1482) Support SetState in Spark runner

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1482:
---

Assignee: Aviem Zur

> Support SetState in Spark runner
> 
>
> Key: BEAM-1482
> URL: https://issues.apache.org/jira/browse/BEAM-1482
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Aviem Zur
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1477) Support MapState in Spark runner

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1477:
---

Assignee: Aviem Zur

> Support MapState in Spark runner
> 
>
> Key: BEAM-1477
> URL: https://issues.apache.org/jira/browse/BEAM-1477
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Aviem Zur
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1035) Support for new State API in SparkRunner

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1035:
---

Assignee: Aviem Zur  (was: Amit Sela)

> Support for new State API in SparkRunner
> 
>
> Key: BEAM-1035
> URL: https://issues.apache.org/jira/browse/BEAM-1035
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Aviem Zur
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #2786

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1723) FlinkRunner should deduplicate when an UnboundedSource requires Deduping

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962136#comment-15962136
 ] 

ASF GitHub Bot commented on BEAM-1723:
--

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2476

[BEAM-1723] deduplication of UnboundedSource in Flink runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1723

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2476


commit 2dc6ea51bb60f8f4beb397bc7179e1829be30d77
Author: JingsongLi 
Date:   2017-04-09T13:16:50Z

[BEAM-1723] deduplication of UnboundedSource in Flink runner




> FlinkRunner should deduplicate when an UnboundedSource requires Deduping
> 
>
> Key: BEAM-1723
> URL: https://issues.apache.org/jira/browse/BEAM-1723
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Groh
>Assignee: Jingsong Lee
>
> UnboundedSource implementations can require deduping, and the FlinkRunner 
> currently logs a warning that this is not supported.
> https://github.com/apache/beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java#L139



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2476: [BEAM-1723] deduplication of UnboundedSource in Fli...

[JingsongLi]

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2476

[BEAM-1723] deduplication of UnboundedSource in Flink runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1723

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2476


commit 2dc6ea51bb60f8f4beb397bc7179e1829be30d77
Author: JingsongLi 
Date:   2017-04-09T13:16:50Z

[BEAM-1723] deduplication of UnboundedSource in Flink runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1723) FlinkRunner should deduplicate when an UnboundedSource requires Deduping

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962125#comment-15962125
 ] 

Jingsong Lee commented on BEAM-1723:


I think it is necessary to be configurable because the deduplication window is 
related to the checkpoint interval.

> FlinkRunner should deduplicate when an UnboundedSource requires Deduping
> 
>
> Key: BEAM-1723
> URL: https://issues.apache.org/jira/browse/BEAM-1723
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Groh
>Assignee: Jingsong Lee
>
> UnboundedSource implementations can require deduping, and the FlinkRunner 
> currently logs a warning that this is not supported.
> https://github.com/apache/beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java#L139



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #287

[Apache Jenkins Server]

See 


Changes:

[amitsela33] [BEAM-1737] Implement a Single-output ParDo as a Multi-output 
ParDo with

--
[...truncated 245.05 KB...]
 ! 40858af...437487e refs/pull/1851/merge -> origin/pr/1851/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2033/merge: No such 
file or directory
 ! e0496ef...05b742f refs/pull/2033/merge -> origin/pr/2033/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2057/merge: No such 
file or directory
 ! 9ccc8df...796aa16 refs/pull/2057/merge -> origin/pr/2057/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2093/merge: No such 
file or directory
 ! 62211ac...a81278c refs/pull/2093/merge -> origin/pr/2093/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2113/merge: No such 
file or directory
 ! d9b5625...14a3303 refs/pull/2113/merge -> origin/pr/2113/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2154/merge: No such 
file or directory
 ! 3b1488f...c50274b refs/pull/2154/merge -> origin/pr/2154/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2166/merge: No such 
file or directory
 ! 45a939f...4ba2353 refs/pull/2166/merge -> origin/pr/2166/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2204/merge: No such 
file or directory
 ! 511b015...9f75a1f refs/pull/2204/merge -> origin/pr/2204/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2207/merge: No such 
file or directory
 ! bbbaf62...42f3ccd refs/pull/2207/merge -> origin/pr/2207/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2216/merge: No such 
file or directory
 ! e12dbc7...ac7e11c refs/pull/2216/merge -> origin/pr/2216/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2263/merge: No such 
file or directory
 ! aabf077...366a64c refs/pull/2263/merge -> origin/pr/2263/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2266/merge: No such 
file or directory
 ! 9bf6436...3dd088a refs/pull/2266/merge -> origin/pr/2266/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2273/merge: No such 
file or directory
 ! a5c366a...6884de4 refs/pull/2273/merge -> origin/pr/2273/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2286/merge: No such 
file or directory
 ! d936aa7...4c95c1a refs/pull/2286/merge -> origin/pr/2286/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2331/merge: No such 
file or directory
 ! a698a71...96215ef refs/pull/2331/merge -> origin/pr/2331/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2339/merge: No such 
file or directory
 ! 031c471...10007be refs/pull/2339/merge -> origin/pr/2339/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2340/merge: No such 
file or directory
 ! 6119a35...75500d8 refs/pull/2340/merge -> origin/pr/2340/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2341/merge: No such 
file or directory
 ! f2b22c2...cb42008 refs/pull/2341/merge -> origin/pr/2341/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2344/merge: No such 
file or directory
 ! 2af764c...1cb169a refs/pull/2344/merge -> origin/pr/2344/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2358/merge: No such 
file or directory
 ! 1a6b3cd...f311c97 refs/pull/2358/merge -> origin/pr/2358/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2363/merge: No such 
file or directory
 ! 157a9c5...54530ea refs/pull/2363/merge -> origin/pr/2363/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2401/merge: No such 
file or directory
 ! 855ebc6...d0b3c3a refs/pull/2401/merge -> origin/pr/2401/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2406/merge: No such 
file or directory
 ! 889bad2...60b3cea refs/pull/2406/merge -> origin/pr/2406/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2417/merge: No such 
file or directory
 ! ea1995c...7317b34 refs/pull/2417/merge -> origin/pr/2417/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2420/merge: No such 
file or directory
 ! 5347532...1bea85f refs/pull/2420/merge -> origin/pr/2420/merge  

[jira] [Commented] (BEAM-1920) Add Spark 2.x support in Spark runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962081#comment-15962081
 ] 

Jean-Baptiste Onofré commented on BEAM-1920:


Good idea (for now, I did the opposite: 1.6.3 as default and 2 with profile). 
Let me do the change ! Thanks !

> Add Spark 2.x support in Spark runner
> -
>
> Key: BEAM-1920
> URL: https://issues.apache.org/jira/browse/BEAM-1920
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> I have a branch working with both Spark 1 and Spark 2 backend.
> I'm preparing a pull request about that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1723) FlinkRunner should deduplicate when an UnboundedSource requires Deduping

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962077#comment-15962077
 ] 

Jingsong Lee commented on BEAM-1723:


I understand. The reason for the duplication is that {{PubSubIO}} use Pull-Ack 
model, {{acknowledge()}} in {{finalizeCheckpoint()}} may be fail, while Kafka 
use offset to restore.

> FlinkRunner should deduplicate when an UnboundedSource requires Deduping
> 
>
> Key: BEAM-1723
> URL: https://issues.apache.org/jira/browse/BEAM-1723
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Groh
>Assignee: Jingsong Lee
>
> UnboundedSource implementations can require deduping, and the FlinkRunner 
> currently logs a warning that this is not supported.
> https://github.com/apache/beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java#L139



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1920) Add Spark 2.x support in Spark runner

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962076#comment-15962076
 ] 

Amit Sela commented on BEAM-1920:
-

Sweet! 
I'm considering Spark 2.x as default and {{1.6.3}} as the profile-enabled 
version because there are many issues with flaky tests now that I believe that 
should be resolved as a part of a bunch of bug fixes introduced in Spark 
{{2.0.0}} and {{2.0.1}}.

The only Con I can think of for that is that Spark 2.x dependency would provide 
an easy path to new, incompatible, APIs in Spark so we'd have to think 
carefully how we test for {{1.6.3}} breaking changes.  

> Add Spark 2.x support in Spark runner
> -
>
> Key: BEAM-1920
> URL: https://issues.apache.org/jira/browse/BEAM-1920
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> I have a branch working with both Spark 1 and Spark 2 backend.
> I'm preparing a pull request about that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1920) Add Spark 2.x support in Spark runner

[JIRA]

Jean-Baptiste Onofré created BEAM-1920:
--

 Summary: Add Spark 2.x support in Spark runner
 Key: BEAM-1920
 URL: https://issues.apache.org/jira/browse/BEAM-1920
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


I have a branch working with both Spark 1 and Spark 2 backend.

I'm preparing a pull request about that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #2785

[Apache Jenkins Server]

See 


--
[...truncated 515.76 KB...]
 x [deleted] (none) -> origin/pr/913/merge
 x [deleted] (none) -> origin/pr/914/head
 x [deleted] (none) -> origin/pr/914/merge
 x [deleted] (none) -> origin/pr/915/head
 x [deleted] (none) -> origin/pr/915/merge
 x [deleted] (none) -> origin/pr/916/head
 x [deleted] (none) -> origin/pr/916/merge
 x [deleted] (none) -> origin/pr/917/head
 x [deleted] (none) -> origin/pr/917/merge
 x [deleted] (none) -> origin/pr/918/head
 x [deleted] (none) -> origin/pr/918/merge
 x [deleted] (none) -> origin/pr/919/head
 x [deleted] (none) -> origin/pr/919/merge
 x [deleted] (none) -> origin/pr/92/head
 x [deleted] (none) -> origin/pr/92/merge
 x [deleted] (none) -> origin/pr/920/head
 x [deleted] (none) -> origin/pr/920/merge
 x [deleted] (none) -> origin/pr/921/head
 x [deleted] (none) -> origin/pr/921/merge
 x [deleted] (none) -> origin/pr/922/head
 x [deleted] (none) -> origin/pr/922/merge
 x [deleted] (none) -> origin/pr/923/head
 x [deleted] (none) -> origin/pr/924/head
 x [deleted] (none) -> origin/pr/925/head
 x [deleted] (none) -> origin/pr/925/merge
 x [deleted] (none) -> origin/pr/926/head
 x [deleted] (none) -> origin/pr/926/merge
 x [deleted] (none) -> origin/pr/927/head
 x [deleted] (none) -> origin/pr/927/merge
 x [deleted] (none) -> origin/pr/928/head
 x [deleted] (none) -> origin/pr/929/head
 x [deleted] (none) -> origin/pr/93/head
 x [deleted] (none) -> origin/pr/930/head
 x [deleted] (none) -> origin/pr/930/merge
 x [deleted] (none) -> origin/pr/931/head
 x [deleted] (none) -> origin/pr/931/merge
 x [deleted] (none) -> origin/pr/932/head
 x [deleted] (none) -> origin/pr/932/merge
 x [deleted] (none) -> origin/pr/933/head
 x [deleted] (none) -> origin/pr/933/merge
 x [deleted] (none) -> origin/pr/934/head
 x [deleted] (none) -> origin/pr/934/merge
 x [deleted] (none) -> origin/pr/935/head
 x [deleted] (none) -> origin/pr/936/head
 x [deleted] (none) -> origin/pr/936/merge
 x [deleted] (none) -> origin/pr/937/head
 x [deleted] (none) -> origin/pr/937/merge
 x [deleted] (none) -> origin/pr/938/head
 x [deleted] (none) -> origin/pr/939/head
 x [deleted] (none) -> origin/pr/94/head
 x [deleted] (none) -> origin/pr/940/head
 x [deleted] (none) -> origin/pr/940/merge
 x [deleted] (none) -> origin/pr/941/head
 x [deleted] (none) -> origin/pr/941/merge
 x [deleted] (none) -> origin/pr/942/head
 x [deleted] (none) -> origin/pr/942/merge
 x [deleted] (none) -> origin/pr/943/head
 x [deleted] (none) -> origin/pr/943/merge
 x [deleted] (none) -> origin/pr/944/head
 x [deleted] (none) -> origin/pr/945/head
 x [deleted] (none) -> origin/pr/945/merge
 x [deleted] (none) -> origin/pr/946/head
 x [deleted] (none) -> origin/pr/946/merge
 x [deleted] (none) -> origin/pr/947/head
 x [deleted] (none) -> origin/pr/947/merge
 x [deleted] (none) -> origin/pr/948/head
 x [deleted] (none) -> origin/pr/948/merge
 x [deleted] (none) -> origin/pr/949/head
 x [deleted] (none) -> origin/pr/949/merge
 x [deleted] (none) -> origin/pr/95/head
 x [deleted] (none) -> origin/pr/95/merge
 x [deleted] (none) -> origin/pr/950/head
 x [deleted] (none) -> origin/pr/951/head
 x [deleted] (none) -> origin/pr/951/merge
 x [deleted] (none) -> origin/pr/952/head
 x [deleted] (none) -> origin/pr/952/merge
 x [deleted] (none) -> origin/pr/953/head
 x [deleted] (none) -> origin/pr/954/head
 x [deleted] (none) -> origin/pr/954/merge
 x [deleted] (none) -> origin/pr/955/head
 x [deleted] (none) -> origin/pr/955/merge
 x [deleted] (none) -> origin/pr/956/head
 x [deleted] (none) -> origin/pr/957/head
 x [deleted] (none) -> origin/pr/958/head
 x [deleted] (none) -> origin/pr/959/head
 x [deleted] (none) -> origin/pr/959/merge
 x [deleted] (none) -> origin/pr/96/head
 x [deleted] (none) -> origin/pr/96/merge
 x [deleted] (none) -> 

[jira] [Resolved] (BEAM-1737) Implement a Single-output ParDo as a Multi-output ParDo with a single output

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1737.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Implement a Single-output ParDo as a Multi-output ParDo with a single output
> 
>
> Key: BEAM-1737
> URL: https://issues.apache.org/jira/browse/BEAM-1737
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Thomas Groh
>Assignee: Amit Sela
>Priority: Minor
> Fix For: First stable release
>
>
> This is the cause of having a separate path and implementation for 
> single-output ParDos, even though both go through the same translator.
> Partial stacktrace:
> Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.946 sec <<< 
> FAILURE! - in 
> org.apache.beam.runners.spark.translation.streaming.CreateStreamTest  
>[8233/41535]
> testLateDataAccumulating(org.apache.beam.runners.spark.translation.streaming.CreateStreamTest)
>   Time elapsed: 3.593 sec  <<< ERROR!
> java.lang.RuntimeException: 
> java.io.NotSerializableException: DStream checkpointing has been enabled but 
> the DStreams with their functions are not serializable
> org.apache.beam.runners.spark.translation.EvaluationContext
> Serialization stack:
> - object not serializable (class: 
> org.apache.beam.runners.spark.translation.EvaluationContext, value: 
> org.apache.beam.runners.spark.translation.EvaluationContext@a8c55d7)
> - field (class: 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1,
>  name: val$context, type: class 
> org.apache.beam.runners.spark.translation.EvaluationContext)
> - object (class 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1,
>  
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1@44f50940)
> - field (class: 
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1,
>  name: transformFunc$3, type: interface 
> org.apache.spark.api.java.function.Function)
> - object (class 
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1,
>  )
> - field (class: 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21,
>  name: cleanedF$2, type: interface scala.Function1)
> - object (class 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21,
>  )
> - field (class: 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5, 
> name: cleanedF$3, type: interface scala.Function2)
> - object (class 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5, 
> )
> - writeObject data (class: 
> org.apache.spark.streaming.dstream.DStreamCheckpointData)
> - object (class 
> org.apache.spark.streaming.dstream.DStreamCheckpointData, [
> 0 checkpoint files 
> ])
> 
> at 
> org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:60)
> at 
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:77)
> at 
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:113)
> at 
> org.apache.beam.runners.spark.TestSparkRunner.awaitWatermarksOrTimeout(TestSparkRunner.java:195)
> at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:127)
> at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:82)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
> ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1737) Implement a Single-output ParDo as a Multi-output ParDo with a single output

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962066#comment-15962066
 ] 

ASF GitHub Bot commented on BEAM-1737:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2420


> Implement a Single-output ParDo as a Multi-output ParDo with a single output
> 
>
> Key: BEAM-1737
> URL: https://issues.apache.org/jira/browse/BEAM-1737
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Thomas Groh
>Assignee: Amit Sela
>Priority: Minor
>
> This is the cause of having a separate path and implementation for 
> single-output ParDos, even though both go through the same translator.
> Partial stacktrace:
> Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.946 sec <<< 
> FAILURE! - in 
> org.apache.beam.runners.spark.translation.streaming.CreateStreamTest  
>[8233/41535]
> testLateDataAccumulating(org.apache.beam.runners.spark.translation.streaming.CreateStreamTest)
>   Time elapsed: 3.593 sec  <<< ERROR!
> java.lang.RuntimeException: 
> java.io.NotSerializableException: DStream checkpointing has been enabled but 
> the DStreams with their functions are not serializable
> org.apache.beam.runners.spark.translation.EvaluationContext
> Serialization stack:
> - object not serializable (class: 
> org.apache.beam.runners.spark.translation.EvaluationContext, value: 
> org.apache.beam.runners.spark.translation.EvaluationContext@a8c55d7)
> - field (class: 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1,
>  name: val$context, type: class 
> org.apache.beam.runners.spark.translation.EvaluationContext)
> - object (class 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1,
>  
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1@44f50940)
> - field (class: 
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1,
>  name: transformFunc$3, type: interface 
> org.apache.spark.api.java.function.Function)
> - object (class 
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1,
>  )
> - field (class: 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21,
>  name: cleanedF$2, type: interface scala.Function1)
> - object (class 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21,
>  )
> - field (class: 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5, 
> name: cleanedF$3, type: interface scala.Function2)
> - object (class 
> org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5, 
> )
> - writeObject data (class: 
> org.apache.spark.streaming.dstream.DStreamCheckpointData)
> - object (class 
> org.apache.spark.streaming.dstream.DStreamCheckpointData, [
> 0 checkpoint files 
> ])
> 
> at 
> org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:60)
> at 
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:77)
> at 
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:113)
> at 
> org.apache.beam.runners.spark.TestSparkRunner.awaitWatermarksOrTimeout(TestSparkRunner.java:195)
> at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:127)
> at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:82)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
> ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2420: [BEAM-1737] Interpreting a Single-output ParDo as a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2420


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2420

[amitsela]

This closes #2420


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a0cfccda
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a0cfccda
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a0cfccda

Branch: refs/heads/master
Commit: a0cfccda4f1fd08ae04aef41c1cc0bc14f00e112
Parents: 810db7f 9e294dc
Author: Amit Sela 
Authored: Sun Apr 9 10:06:08 2017 +0300
Committer: Amit Sela 
Committed: Sun Apr 9 10:06:08 2017 +0300

--
 .../runners/spark/translation/DoFnFunction.java | 130 ---
 .../spark/translation/TransformTranslator.java  |  74 ---
 .../streaming/StreamingTransformTranslator.java |  78 +++
 3 files changed, 47 insertions(+), 235 deletions(-)
--

[1/2] beam git commit: [BEAM-1737] Implement a Single-output ParDo as a Multi-output ParDo with a single output.

[amitsela]

Repository: beam
Updated Branches:
  refs/heads/master 810db7f99 -> a0cfccda4


[BEAM-1737] Implement a Single-output ParDo as a Multi-output ParDo with a 
single output.

remove use of EvaluationContext in DStream lambda, it is not serializable and 
also redundant in this
case.

implement pardo as multido.

cache only if this is an actual MultiDo.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9e294dc0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9e294dc0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9e294dc0

Branch: refs/heads/master
Commit: 9e294dc05c1f1ac2aa2ac15f364394698a2767a7
Parents: 810db7f
Author: Amit Sela 
Authored: Tue Apr 4 11:46:38 2017 +0300
Committer: Amit Sela 
Committed: Sun Apr 9 10:05:31 2017 +0300

--
 .../runners/spark/translation/DoFnFunction.java | 130 ---
 .../spark/translation/TransformTranslator.java  |  74 ---
 .../streaming/StreamingTransformTranslator.java |  78 +++
 3 files changed, 47 insertions(+), 235 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9e294dc0/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
deleted file mode 100644
index 11761b6..000
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
+++ /dev/null
@@ -1,130 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.runners.spark.translation;
-
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import org.apache.beam.runners.core.DoFnRunner;
-import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.spark.aggregators.NamedAggregators;
-import org.apache.beam.runners.spark.aggregators.SparkAggregators;
-import org.apache.beam.runners.spark.metrics.SparkMetricsContainer;
-import org.apache.beam.runners.spark.util.SideInputBroadcast;
-import org.apache.beam.runners.spark.util.SparkSideInputReader;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.spark.Accumulator;
-import org.apache.spark.api.java.function.FlatMapFunction;
-
-
-/**
- * Beam's Do functions correspond to Spark's FlatMap functions.
- *
- * @param   Input element type.
- * @param  Output element type.
- */
-public class DoFnFunction
-implements FlatMapFunction, 
WindowedValue> {
-
-  private final Accumulator aggregatorsAccum;
-  private final Accumulator metricsAccum;
-  private final String stepName;
-  private final DoFn doFn;
-  private final SparkRuntimeContext runtimeContext;
-  private final Map> sideInputs;
-  private final WindowingStrategy windowingStrategy;
-
-  /**
-   * @param aggregatorsAccum  The Spark {@link Accumulator} that backs the 
Beam Aggregators.
-   * @param doFn  The {@link DoFn} to be wrapped.
-   * @param runtimeContextThe {@link SparkRuntimeContext}.
-   * @param sideInputsSide inputs used in this {@link DoFn}.
-   * @param windowingStrategy Input {@link WindowingStrategy}.
-   */
-  public DoFnFunction(
-  Accumulator aggregatorsAccum,
-  Accumulator metricsAccum,
-  String stepName,
-  DoFn doFn,
-  SparkRuntimeContext runtimeContext,
-  Map> 
sideInputs,
-  WindowingStrategy windowingStrategy) {
-this.aggregatorsAccum = 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1567

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962044#comment-15962044
 ] 

ASF GitHub Bot commented on BEAM-802:
-

GitHub user mariapython opened a pull request:

https://github.com/apache/beam/pull/2475

[BEAM-802] Skip query metrics for template creation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam ppp_fix_equal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2475


commit d4cddc83b04ae87b18be16795f4a9bc069f39050
Author: Maria Garcia Herrero 
Date:   2017-04-09T06:08:44Z

Skip query metrics when creating a template




> Support Dynamic PipelineOptions for python
> --
>
> Key: BEAM-802
> URL: https://issues.apache.org/jira/browse/BEAM-802
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline 
> options)
> Background: Currently, the Runner creates the JSON pipeline description which 
> can be sent to the worker as is, since everything is already defined there 
> (with links to gs:// for input and binaries). With the parametrized approach, 
> those descriptions are empty and filled by the user or defaulted, so the 
> pipeline needs to be stored somewhere first until the values become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that 
> needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic 
> key/value pairs provided at runtime (dynamic)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2475: [BEAM-802] Skip query metrics for template creation

[mariapython]

GitHub user mariapython opened a pull request:

https://github.com/apache/beam/pull/2475

[BEAM-802] Skip query metrics for template creation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam ppp_fix_equal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2475


commit d4cddc83b04ae87b18be16795f4a9bc069f39050
Author: Maria Garcia Herrero 
Date:   2017-04-09T06:08:44Z

Skip query metrics when creating a template




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PerformanceTests_Dataflow #286

[Apache Jenkins Server]

See 


--
[...truncated 249.00 KB...]
 ! aaf02a7...da67322 refs/pull/1925/merge -> origin/pr/1925/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2009/merge: No such 
file or directory
 ! 10abae8...ea29773 refs/pull/2009/merge -> origin/pr/2009/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2069/merge: No such 
file or directory
 ! 9fbdc6f...46fd721 refs/pull/2069/merge -> origin/pr/2069/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2099/merge: No such 
file or directory
 ! e751d67...a6eeb60 refs/pull/2099/merge -> origin/pr/2099/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2110/merge: No such 
file or directory
 ! 7fc9e69...9767628 refs/pull/2110/merge -> origin/pr/2110/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2137/merge: No such 
file or directory
 ! 4adc08e...1a85857 refs/pull/2137/merge -> origin/pr/2137/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2169/merge: No such 
file or directory
 ! 6841376...61faa9a refs/pull/2169/merge -> origin/pr/2169/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2174/merge: No such 
file or directory
 ! b7474f1...ef18fc3 refs/pull/2174/merge -> origin/pr/2174/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2175/merge: No such 
file or directory
 ! f1f7865...a88e3df refs/pull/2175/merge -> origin/pr/2175/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2176/merge: No such 
file or directory
 ! 4eae45a...7f2dd61 refs/pull/2176/merge -> origin/pr/2176/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2179/merge: No such 
file or directory
 ! cff837e...d2f0ca9 refs/pull/2179/merge -> origin/pr/2179/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2195/merge: No such 
file or directory
 ! 8d4913b...f2859e1 refs/pull/2195/merge -> origin/pr/2195/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2227/merge: No such 
file or directory
 ! c3c991c...e15e874 refs/pull/2227/merge -> origin/pr/2227/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2236/merge: No such 
file or directory
 ! a6cc28a...e9dac14 refs/pull/2236/merge -> origin/pr/2236/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2249/merge: No such 
file or directory
 ! 51dbdeb...fe73128 refs/pull/2249/merge -> origin/pr/2249/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2256/merge: No such 
file or directory
 ! db9b993...8eca896 refs/pull/2256/merge -> origin/pr/2256/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2261/merge: No such 
file or directory
 ! 5a4271d...b3a80ae refs/pull/2261/merge -> origin/pr/2261/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2267/merge: No such 
file or directory
 ! c6eaaa1...b2e34c3 refs/pull/2267/merge -> origin/pr/2267/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2301/merge: No such 
file or directory
 ! ff680b3...b5dfe19 refs/pull/2301/merge -> origin/pr/2301/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2303/merge: No such 
file or directory
 ! 29d1858...ad6c0dc refs/pull/2303/merge -> origin/pr/2303/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2315/merge: No such 
file or directory
 ! e6d168f...67f684a refs/pull/2315/merge -> origin/pr/2315/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2316/merge: No such 
file or directory
 ! b2db5cb...fc0d1a7 refs/pull/2316/merge -> origin/pr/2316/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2326/merge: No such 
file or directory
 ! a5e2248...950055f refs/pull/2326/merge -> origin/pr/2326/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2352/merge: No such 
file or directory
 ! 585df37...11fa6a0 refs/pull/2352/merge -> origin/pr/2352/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/238/merge: No such 
file or directory
 ! 761dff3...3075dc1 refs/pull/238/merge -> origin/pr/238/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2401/head: No such 
file or