[jira] [Commented] (BEAM-1297) Update maven build plugins and improve maven performance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862667#comment-15862667
 ] 

ASF GitHub Bot commented on BEAM-1297:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/1985

[BEAM-1297] Update maven build plugins and improve maven performance

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1297-update-plugins

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1985.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1985


commit 9d69e60145adffed0ec6943a214ea637bd3bae54
Author: Ismaël Mejía 
Date:   2017-02-12T05:52:45Z

[BEAM-1297] Update maven shade plugin and fix typo

commit 6b832505393b10474893f0a59b66440f7f8d
Author: Ismaël Mejía 
Date:   2017-02-12T06:07:38Z

[BEAM-1297] Switch to JVM client mode JVM and disable extra JIT profiling
Extra JIT profiling is only for execution of each maven subtasks, this
improve maven startup time

commit 6ca672c7c718ce3266cbc193b9962cf68a463362
Author: Ismaël Mejía 
Date:   2017-02-12T06:18:16Z

[BEAM-1297] Enable parallel builds in travis (1C = 1 per core)




> Update maven build plugins and improve maven performance
> 
>
> Key: BEAM-1297
> URL: https://issues.apache.org/jira/browse/BEAM-1297
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>
> I was investigating ways to speed up the build / test execution. And I found 
> that a common source of delay comes from outdated maven plugins.
> Running this command
> mvn versions:display-plugin-updates
> I found that we have some outdated maven plugins, and additionally some minor 
> configuration issues so I fix them.
> In my local tests I got an improvement of 50s (over an average 11m run) for 
> the "mvn clean verify -Prelease" execution so I think this might be worth 
> checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1985: [BEAM-1297] Update maven build plugins and improve ...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/1985

[BEAM-1297] Update maven build plugins and improve maven performance

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1297-update-plugins

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1985.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1985


commit 9d69e60145adffed0ec6943a214ea637bd3bae54
Author: Ismaël Mejía 
Date:   2017-02-12T05:52:45Z

[BEAM-1297] Update maven shade plugin and fix typo

commit 6b832505393b10474893f0a59b66440f7f8d
Author: Ismaël Mejía 
Date:   2017-02-12T06:07:38Z

[BEAM-1297] Switch to JVM client mode JVM and disable extra JIT profiling
Extra JIT profiling is only for execution of each maven subtasks, this
improve maven startup time

commit 6ca672c7c718ce3266cbc193b9962cf68a463362
Author: Ismaël Mejía 
Date:   2017-02-12T06:18:16Z

[BEAM-1297] Enable parallel builds in travis (1C = 1 per core)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Flink #1614

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5fe11a2fc496dabf8450923e7aa232bdb7800747 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5fe11a2fc496dabf8450923e7aa232bdb7800747
 > git rev-list 5fe11a2fc496dabf8450923e7aa232bdb7800747 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 59653
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Flink] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 59653
ERROR: Failed to parse POMs
java.io.IOException: Cannot run program 
"/home/jenkins/tools/java/latest1.8/bin/java" (in directory 
":
 error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at hudson.Proc$LocalProc.(Proc.java:240)
at hudson.Proc$LocalProc.(Proc.java:212)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:815)
at hudson.Launcher$ProcStarter.start(Launcher.java:381)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1148)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to beam2(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:928)
at hudson.Launcher$ProcStarter.start(Launcher.java:381)
at 
hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:281)
at hudson.maven.ProcessCache.get(ProcessCache.java:236)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:778)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1741)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:531)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Dataflow #2270

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5fe11a2fc496dabf8450923e7aa232bdb7800747 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5fe11a2fc496dabf8450923e7aa232bdb7800747
 > git rev-list 5fe11a2fc496dabf8450923e7aa232bdb7800747 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 33739
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Dataflow] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 33739
ERROR: Failed to parse POMs
java.io.IOException: Cannot run program 
"/home/jenkins/tools/java/latest1.8/bin/java" (in directory 
":
 error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at hudson.Proc$LocalProc.(Proc.java:240)
at hudson.Proc$LocalProc.(Proc.java:212)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:815)
at hudson.Launcher$ProcStarter.start(Launcher.java:381)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1148)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to beam3(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:928)
at hudson.Launcher$ProcStarter.start(Launcher.java:381)
at 
hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:281)
at hudson.maven.ProcessCache.get(ProcessCache.java:236)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:778)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1741)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:531)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at 

[jira] [Commented] (BEAM-1467) Use well-known coder types for known window coders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862465#comment-15862465
 ] 

ASF GitHub Bot commented on BEAM-1467:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1984

[BEAM-1467] Add cross-SDK implementations and tests of IntervalWindowCoder

- Implement an `IntervalWindowCoder` for Python SDK and test it
- Add test data for `IntervalWindow` to `standard_coders.yaml` and test 
both SDKs
- Add utilities to Java SDK's `CommonCoderTest` to enable more easy 
generation of test data in future.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam interval-window-coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1984


commit 5ba5a95ce440a4a5babf3ae02df07ac32c7207ad
Author: Dan Halperin 
Date:   2017-02-10T19:56:00Z

Add cross-SDK implementations and tests of IntervalWindowCoder




> Use well-known coder types for known window coders
> --
>
> Key: BEAM-1467
> URL: https://issues.apache.org/jira/browse/BEAM-1467
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-fn-api, beam-model-runner-api
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> Known window types include:
> * GlobalWindow
> * IntervalWindow
> Standardizing the name and encodings of these windows will enable many more 
> pipelines to work across the Fn API with low overhead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1984: [BEAM-1467] Add cross-SDK implementations and tests...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1984

[BEAM-1467] Add cross-SDK implementations and tests of IntervalWindowCoder

- Implement an `IntervalWindowCoder` for Python SDK and test it
- Add test data for `IntervalWindow` to `standard_coders.yaml` and test 
both SDKs
- Add utilities to Java SDK's `CommonCoderTest` to enable more easy 
generation of test data in future.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam interval-window-coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1984


commit 5ba5a95ce440a4a5babf3ae02df07ac32c7207ad
Author: Dan Halperin 
Date:   2017-02-10T19:56:00Z

Add cross-SDK implementations and tests of IntervalWindowCoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1467) Use well-known coder types for known window coders

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1467:
-

 Summary: Use well-known coder types for known window coders
 Key: BEAM-1467
 URL: https://issues.apache.org/jira/browse/BEAM-1467
 Project: Beam
  Issue Type: Improvement
  Components: beam-model-fn-api, beam-model-runner-api
Reporter: Daniel Halperin
Assignee: Daniel Halperin
Priority: Minor


Known window types include:

* GlobalWindow
* IntervalWindow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1466) JSON utils extension

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862464#comment-15862464
 ] 

ASF GitHub Bot commented on BEAM-1466:
--

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/1983

[BEAM-1466] JSON utils extension

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam json-utils

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1983.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1983


commit 64f8d80387d1038488ff301b3f1954340892758e
Author: Aviem Zur 
Date:   2017-02-11T15:06:45Z

[BEAM-1466] JSON utils extension




> JSON utils extension
> 
>
> Key: BEAM-1466
> URL: https://issues.apache.org/jira/browse/BEAM-1466
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Create a JSON extension module which will contain transforms to aid with 
> handling JSONs.
> Suggested transforms:
> * Parse JSON strings to type OutputT.
> * Parse InputT to JSON strings.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1983: [BEAM-1466] JSON utils extension

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/1983

[BEAM-1466] JSON utils extension

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam json-utils

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1983.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1983


commit 64f8d80387d1038488ff301b3f1954340892758e
Author: Aviem Zur 
Date:   2017-02-11T15:06:45Z

[BEAM-1466] JSON utils extension




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-718) Migrate KinesisIO to use AutoValue to reduce boilerplate

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-718:


Assignee: (was: James Malone)

> Migrate KinesisIO to use AutoValue to reduce boilerplate
> 
>
> Key: BEAM-718
> URL: https://issues.apache.org/jira/browse/BEAM-718
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1231) Use well known coder types in Java for WindowedValue, GlobalWindow, and LengthPrefix

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1231.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Use well known coder types in Java for WindowedValue, GlobalWindow, and 
> LengthPrefix
> 
>
> Key: BEAM-1231
> URL: https://issues.apache.org/jira/browse/BEAM-1231
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: Not applicable
>
>
> Add support for using kind for well known coder types in Apache Beam for Java
> The following specific cloud object representations for the following types 
> should exist:
> kind:pair (TupleCoder with two components)
> kind:stream (IterableCoder with a single component)
> kind:global_window (GlobalWindowCoder)
> kind:length_prefix (A new type of coder which always encodes the length of 
> the value type as a prefix, has a single component)
> kind:windowed_value (A wrapper coder with two components (value coder and 
> window coder))
> These changes are towards having a common binary representation for certain 
> well known coders across multiple languages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-298) Make TestPipeline implement the TestRule interface

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862327#comment-15862327
 ] 

Stas Levin edited comment on BEAM-298 at 2/11/17 11:39 AM:
---

After taking a deeper look, {{beam-sdks-java-core}} declares {{junit}} as a 
{{provided}} dependency, which makes it availiable in compile and test scopes 
^1^ (since it has both compile and test scope needs for {{junit}}).

Modules that depend on {{beam-sdks-java-core}} in {{compile}} scope do not get 
{{junit}} due to maven's transitive dependency rules ^1^, so they need to 
explicitly declare their dependency on {{junit}} in either the {{provided}} or 
{{test}} scope, according to their needs.

If you do not have a compile time dependency on {{junit}} (which I assume is 
the case since your code probably compiles and the error you describe is a 
runtime one), I would expect the addition of a {{test}} scope dependency to fix 
the problem. Since it doesn't, I'm tempted to believe you're using the 
{{TestPipeline}} in a non {{test}} scope, where {{junit}} can indeed be missing 
from the classpath ^2^.

Could you please provide some more info regarding your scenario? 
Are you running {{TestPipeline}} as part of a unit test or production code on 
your end?

1. 
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
2. Or I could be completely wrong and the problem lies elsewhere.


was (Author: staslev):
After taking a deeper look, {{beam-sdks-java-core}} declares {{junit}} as a 
{{provided}} dependency, which makes it availiable in compile and test scopes 
^1^ (since it has both compile and test scope needs for {{junit}}).

Modules that depend on {{beam-sdks-java-core}} in {{compile}} scope do not get 
{{junit}} due to maven's transitive dependency rules ^1^, so they need to 
explicitly declare their dependency on {{junit}} in either the {{provided}} or 
{{test}} scope, according to their needs.

If you do not have a compile time dependency on {{junit}} (which I assume is 
the case since your code probably compiles and the error you describe is a 
runtime one), I would expect adding a {{test}} scope dependency to fix the 
problem. Since it doesn't, I'm tempted to believe you're using the 
{{TestPipeline}} in a non {{test}} scope, where {{junit}} can indeed be missing 
from the classpath ^2^.

Could you please provide some more info regarding your scenario? 
Are you running {{TestPipeline}} as part of a unit test or production code on 
your end?

1. 
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
2. Or I could be completely wrong and the problem lies elsewhere.

> Make TestPipeline implement the TestRule interface
> --
>
> Key: BEAM-298
> URL: https://issues.apache.org/jira/browse/BEAM-298
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.5.0
>
>
> https://github.com/junit-team/junit4/wiki/Rules
> A JUnit Rule allows a test to use a field annotated with @Rule to wrap 
> executing tests. Doing so allows the TestPipeline to, at the time the test 
> completes, assert that all applied transforms have been executed. This 
> ensures that all unit tests that utilize a TestPipeline rule either are 
> declared to explicitly not expect to execute or have executed the pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1393) Update Flink Runner to Flink 1.2.0

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862325#comment-15862325
 ] 

Jingsong Lee edited comment on BEAM-1393 at 2/11/17 9:08 AM:
-

BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not only 
round-robin. Let each element in ListState have the opportunity to select a 
KeyGroupIndex. {{CheckpointCoordinator}} use the KeyGroupIndex to redistribute 
state. 
I understood it. {{CheckpointCoordinator}} is run in JobManager. That is too 
heavy to read every elements.
Maybe Flink can abstract KeyGroup state, provide split(snapshot to several 
KeyGroups) and merge(restore by several KeyGroups) methods to manage state with 
KeyGroups.


was (Author: lzljs3620320):
BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not only 
round-robin. Let each element in ListState have the opportunity to select a 
KeyGroupIndex. {{CheckpointCoordinator}} use the KeyGroupIndex to redistribute 
state. 

> Update Flink Runner to Flink 1.2.0
> --
>
> Key: BEAM-1393
> URL: https://issues.apache.org/jira/browse/BEAM-1393
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
>
> When we update to 1.2.0 we can use the new internal Timer API that is 
> available to Flink operators: {{InternalTimerService}} and also use broadcast 
> state to store side-input data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1466) JSON utils extension

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1466:
---

Assignee: Aviem Zur  (was: Davor Bonaci)

> JSON utils extension
> 
>
> Key: BEAM-1466
> URL: https://issues.apache.org/jira/browse/BEAM-1466
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Create a JSON extension in Java extensions which will contain transforms to 
> aid with handling JSONs.
> Suggested transforms:
> * Parse JSON strings to type OutputT.
> * Parse InputT to JSON strings.
> * JSON format file based source.
> * JSON format file based sink.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-298) Make TestPipeline implement the TestRule interface

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862327#comment-15862327
 ] 

Stas Levin commented on BEAM-298:
-

Could you please provide some more info regarding your scenario and how I can 
reproduce it?
Are you running {{TestPipeline}} as part of a unit test or production code? 

> Make TestPipeline implement the TestRule interface
> --
>
> Key: BEAM-298
> URL: https://issues.apache.org/jira/browse/BEAM-298
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.5.0
>
>
> https://github.com/junit-team/junit4/wiki/Rules
> A JUnit Rule allows a test to use a field annotated with @Rule to wrap 
> executing tests. Doing so allows the TestPipeline to, at the time the test 
> completes, assert that all applied transforms have been executed. This 
> ensures that all unit tests that utilize a TestPipeline rule either are 
> declared to explicitly not expect to execute or have executed the pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862325#comment-15862325
 ] 

Jingsong Lee commented on BEAM-1393:


BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not only 
round-robin. Let each element in ListState have the opportunity to select a 
KeyGroupIndex. {{CheckpointCoordinator}} use the KeyGroupIndex to redistribute 
state. 

> Update Flink Runner to Flink 1.2.0
> --
>
> Key: BEAM-1393
> URL: https://issues.apache.org/jira/browse/BEAM-1393
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
>
> When we update to 1.2.0 we can use the new internal Timer API that is 
> available to Flink operators: {{InternalTimerService}} and also use broadcast 
> state to store side-input data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)