Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1533

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #99

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1771

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_PythonVerify #672

[Apache Jenkins Server]

See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/python-sdk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/python-sdk^{commit} # timeout=10
Checking out Revision aa603872c5cb79c6e9a4c12f0a49d68a942a6007 
(refs/remotes/origin/python-sdk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f aa603872c5cb79c6e9a4c12f0a49d68a942a6007
 > git rev-list aa603872c5cb79c6e9a4c12f0a49d68a942a6007 # timeout=10
[beam_PostCommit_PythonVerify] $ /bin/bash -xe /tmp/hudson4499646738822904656.sh
+ bash sdks/python/run_postcommit.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied (use --upgrade to upgrade): virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages
Cleaning up...

# INFRA does not install tox
pip install tox --user
Requirement already satisfied (use --upgrade to upgrade): tox in 
/home/jenkins/.local/lib/python2.7/site-packages
Cleaning up...

# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e py27 -c sdks/python/tox.ini
GLOB sdist-make: 

py27 inst-nodeps: 

py27 installed: You are using pip version 8.1.2, however version 9.0.0 is 
available.,You should consider upgrading via the 'pip install --upgrade pip' 
command.,apache-beam-sdk===0.3.0-incubating.dev,astroid==1.4.8,avro==1.8.1,backports.functools-lru-cache==1.2.1,configparser==3.5.0,dill==0.2.5,funcsigs==1.0.2,google-apitools==0.5.4,httplib2==0.9.2,isort==4.2.5,lazy-object-proxy==1.2.2,mccabe==0.5.2,mock==2.0.0,oauth2client==3.0.0,pbr==1.10.0,pep8==1.7.0,protorpc==0.11.1,pyasn1==0.1.9,pyasn1-modules==0.0.8,pylint==1.6.4,python-gflags==3.0.7,PyYAML==3.12,rsa==3.4.2,six==1.10.0,wrapt==1.10.8
py27 runtests: PYTHONHASHSEED='473989562'
py27 runtests: commands[0] | python --version
Python 2.7.6
py27 runtests: commands[1] | python 
apache_beam/examples/complete/autocomplete_test.py
Traceback (most recent call last):
  File "apache_beam/examples/complete/autocomplete_test.py", line 22, in 

import apache_beam as beam
  File 
"
 line 77, in 
from apache_beam import coders
  File 
"
 line 18, in 
from apache_beam.coders.coders import *
  File 
"
 line 22, in 
import google.protobuf
ImportError: No module named google.protobuf
ERROR: InvocationError: 
'
 apache_beam/examples/complete/autocomplete_test.py'
___ summary 
ERROR:   py27: commands failed
Build step 'Execute shell' marked build as failure

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1532

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1770

[Apache Jenkins Server]

See 

[1/3] incubator-beam-site git commit: Update beam_team Committer IDs

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site e841e7d85 -> 41e553944


Update beam_team Committer IDs

Add tgroh, jesseanderson


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b3c3c885
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b3c3c885
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b3c3c885

Branch: refs/heads/asf-site
Commit: b3c3c885526f430a4e17bac522a0c50d4af278bd
Parents: e841e7d
Author: Thomas Groh 
Authored: Tue Nov 8 15:24:17 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 8 15:58:09 2016 -0800

--
 src/_beam_team/team.md | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b3c3c885/src/_beam_team/team.md
--
diff --git a/src/_beam_team/team.md b/src/_beam_team/team.md
index 9368c37..12abad3 100644
--- a/src/_beam_team/team.md
+++ b/src/_beam_team/team.md
@@ -57,8 +57,8 @@ members:
 roles: champion, committer, PPMC
 time_zone: "+1"
   - name: Jesse Anderson
-apache_id:
-email: jesse [at] smokinghand [dot] com
+apache_id: jesseanderson
+email: jesseanderson [at] apache [dot] org
 organization: Smoking Hand
 roles: committer
 time_zone: "-8"
@@ -70,7 +70,7 @@ members:
 time_zone: "-8"
   - name: Kostas Tzoumas
 apache_id: ktzoumas
-email: kostas [at] data-artisans [dot] com
+email: ktzoumas [at] apache [dot] org
 organization: data Artisans
 roles: committer, PPMC
 time_zone: "+1"
@@ -94,13 +94,13 @@ members:
 time_zone: "+1"
   - name: Stephan Ewen
 apache_id: sewen
-email: stephan [at] data-artisans [dot] com
+email: sewen [at] apache [dot] org
 organization: data Artisans
 roles: committer, PPMC
 time_zone: "+1"
   - name: Thomas Groh
-apache_id:
-email:
+apache_id: tgroh
+email: tgroh [at] apache [dot] org
 organization: Google
 roles: committer
 time_zone: "-8"
@@ -112,7 +112,7 @@ members:
 time_zone: "-8"
   - name: Tom White
 apache_id: tomwhite
-email: tom [at] cloudera [dot] com
+email: tomwhite [at] apache [dot] org
 organization: Cloudera
 roles: committer, PPMC
 time_zone: "-8"

[3/3] incubator-beam-site git commit: This closes #73

[davor]

This closes #73


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/41e55394
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/41e55394
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/41e55394

Branch: refs/heads/asf-site
Commit: 41e5539442ffcfffecd9813dbe1ae095c0f6ad2f
Parents: e841e7d 3756225
Author: Davor Bonaci 
Authored: Tue Nov 8 15:58:25 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 8 15:58:25 2016 -0800

--
 content/contribute/team/index.html | 14 +++---
 src/_beam_team/team.md | 14 +++---
 2 files changed, 14 insertions(+), 14 deletions(-)
--

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/37562256
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/37562256
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/37562256

Branch: refs/heads/asf-site
Commit: 37562256e41a66a90171a471deb88a7e741872bb
Parents: b3c3c88
Author: Davor Bonaci 
Authored: Tue Nov 8 15:58:25 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 8 15:58:25 2016 -0800

--
 content/contribute/team/index.html | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/37562256/content/contribute/team/index.html
--
diff --git a/content/contribute/team/index.html 
b/content/contribute/team/index.html
index e5e420a..2c8fd04 100644
--- a/content/contribute/team/index.html
+++ b/content/contribute/team/index.html
@@ -244,8 +244,8 @@
   
 
   Jesse Anderson
-  
-  jesse [at] smokinghand [dot] com
+  jesseanderson
+  jesseanderson [at] apache [dot] org
   Smoking Hand
   committer
   -8
@@ -263,7 +263,7 @@
 
   Kostas Tzoumas
   ktzoumas
-  kostas [at] data-artisans [dot] com
+  ktzoumas [at] apache [dot] org
   data Artisans
   committer, PPMC
   +1
@@ -299,7 +299,7 @@
 
   Stephan Ewen
   sewen
-  stephan [at] data-artisans [dot] com
+  sewen [at] apache [dot] org
   data Artisans
   committer, PPMC
   +1
@@ -307,8 +307,8 @@
   
 
   Thomas Groh
-  
-  
+  tgroh
+  tgroh [at] apache [dot] org
   Google
   committer
   -8
@@ -326,7 +326,7 @@
 
   Tom White
   tomwhite
-  tom [at] cloudera [dot] com
+  tomwhite [at] apache [dot] org
   Cloudera
   committer, PPMC
   -8

[GitHub] incubator-beam-site pull request #73: Add tgroh Apache ID to beam_team

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/73


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam-site git commit: Remove parser since it's not on the executors.

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 81bb48952 -> e841e7d85


Remove parser since it's not on the executors.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0310c70e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0310c70e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0310c70e

Branch: refs/heads/asf-site
Commit: 0310c70ec3103edab1bff7592a8bcb97132704be
Parents: 81bb489
Author: Jason Kuster 
Authored: Tue Nov 8 15:52:22 2016 -0800
Committer: Jason Kuster 
Committed: Tue Nov 8 15:52:22 2016 -0800

--
 tools/append_index_html_to_internal_links.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0310c70e/tools/append_index_html_to_internal_links.py
--
diff --git a/tools/append_index_html_to_internal_links.py 
b/tools/append_index_html_to_internal_links.py
index da87f57..457e083 100644
--- a/tools/append_index_html_to_internal_links.py
+++ b/tools/append_index_html_to_internal_links.py
@@ -44,7 +44,7 @@ print 'Matches: ' + str(len(matches))
 for match in matches:
   print 'Fixing links in: ' + match
   mf = open(match)
-  soup = BeautifulSoup(mf, "lxml")
+  soup = BeautifulSoup(mf)
   # Iterates over every 
   for a in soup.findAll('a'):
 try:

[2/2] incubator-beam-site git commit: This closes #74

[davor]

This closes #74


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/e841e7d8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/e841e7d8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/e841e7d8

Branch: refs/heads/asf-site
Commit: e841e7d85c80b6393cb957646fe7b6548d26234c
Parents: 81bb489 0310c70
Author: Davor Bonaci 
Authored: Tue Nov 8 15:57:17 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 8 15:57:17 2016 -0800

--
 tools/append_index_html_to_internal_links.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] incubator-beam-site pull request #74: Remove parser since it's not on the ex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/74


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #74: Remove parser since it's not on the ex...

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/incubator-beam-site/pull/74

Remove parser since it's not on the executors.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/incubator-beam-site parser

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/74.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #74


commit 0310c70ec3103edab1bff7592a8bcb97132704be
Author: Jason Kuster 
Date:   2016-11-08T23:52:22Z

Remove parser since it's not on the executors.

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1769

[Apache Jenkins Server]

See 

[GitHub] incubator-beam-site pull request #72: Add tool to fix links in staged websit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/72


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam-site git commit: Add tool to fix links.

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 268cadca4 -> 81bb48952


Add tool to fix links.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/e5828ee4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/e5828ee4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/e5828ee4

Branch: refs/heads/asf-site
Commit: e5828ee4a886bf02dda0099c3c60e15ac429ece3
Parents: 268cadc
Author: Jason Kuster 
Authored: Tue Nov 8 14:52:06 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 8 15:43:58 2016 -0800

--
 tools/append_index_html_to_internal_links.py | 76 +++
 1 file changed, 76 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/e5828ee4/tools/append_index_html_to_internal_links.py
--
diff --git a/tools/append_index_html_to_internal_links.py 
b/tools/append_index_html_to_internal_links.py
new file mode 100644
index 000..da87f57
--- /dev/null
+++ b/tools/append_index_html_to_internal_links.py
@@ -0,0 +1,76 @@
+"""Script to fix the links in the staged website.
+Finds all internal links which do not have index.html at the end and appends
+index.html in the appropriate place (preserving anchors, etc).
+
+Usage:
+  From root directory, after running the jekyll build, execute
+  'python tools/append_index_html_to_internal_links.py'.
+
+Dependencies:
+  beautifulsoup4
+  Installable via pip as 'sudo pip install beautifulsoup4' or apt via
+  'sudo apt-get install python-beautifulsoup4'.
+
+"""
+
+import fnmatch
+import os
+import re
+from bs4 import BeautifulSoup
+
+# Original link match. Matches any string which starts with '/' and doesn't
+# have a file extension.
+linkMatch = r'^\/(.*\.(?!([^\/]+)$))?[^.]*$'
+
+# Regex which matches strings of type /internal/link/#anchor. Breaks into two
+# groups for ease of inserting 'index.html'.
+anchorMatch1 = r'(.+\/)(#[^\/]+$)'
+
+# Regex which matches strings of type /internal/link#anchor. Breaks into two
+# groups for ease of inserting 'index.html'.
+anchorMatch2 = r'(.+\/[a-zA-Z0-9]+)(#[^\/]+$)'
+
+
+matches = []
+# Recursively walk content directory and find all html files.
+for root, dirnames, filenames in os.walk('content'):
+  for filename in fnmatch.filter(filenames, '*.html'):
+# Javadoc does not have the index.html problem, so omit it.
+if 'javadoc' not in root:
+  matches.append(os.path.join(root, filename))
+
+print 'Matches: ' + str(len(matches))
+# Iterates over each matched file looking for link matches.
+for match in matches:
+  print 'Fixing links in: ' + match
+  mf = open(match)
+  soup = BeautifulSoup(mf, "lxml")
+  # Iterates over every 
+  for a in soup.findAll('a'):
+try:
+  hr = a['href']
+  if re.match(linkMatch, hr) is not None:
+if hr.endswith('/'):
+  # /internal/link/
+  a['href'] = hr + 'index.html'
+elif re.match(anchorMatch1, hr) is not None:
+  # /internal/link/#anchor
+  mat = re.match(anchorMatch1, hr)
+  a['href'] = mat.group(1) + 'index.html' + mat.group(2)
+elif re.match(anchorMatch2, hr) is not None:
+  # /internal/link#anchor
+  mat = re.match(anchorMatch2, hr)
+  a['href'] = mat.group(1) + '/index.html' + mat.group(2)
+else:
+  # /internal/link
+  a['href'] = hr + '/index.html'
+mf.close()
+
+html = soup.prettify("utf-8")
+# Write back to the file.
+with open(match, "wb") as f:
+  print 'Replacing ' + hr + ' with: ' + a['href']
+  f.write(html)
+except KeyError as e:
+  # Some  tags don't have an href.
+  continue

[2/2] incubator-beam-site git commit: This closes #72

[davor]

This closes #72


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/81bb4895
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/81bb4895
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/81bb4895

Branch: refs/heads/asf-site
Commit: 81bb48952c120a092d85905beaa5dca7635c9111
Parents: 268cadc e5828ee
Author: Davor Bonaci 
Authored: Tue Nov 8 15:44:09 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 8 15:44:09 2016 -0800

--
 tools/append_index_html_to_internal_links.py | 76 +++
 1 file changed, 76 insertions(+)
--

[GitHub] incubator-beam-site pull request #73: Add tgroh Apache ID to beam_team

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam-site/pull/73

Add tgroh Apache ID to beam_team



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam-site tgroh_email

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/73.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #73


commit 280ded34699a5e8494d55428f583964d50aef476
Author: Thomas Groh 
Date:   2016-11-08T23:24:17Z

Add tgroh Apache ID to beam_team




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #72: Add tool to fix links in staged websit...

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/incubator-beam-site/pull/72

Add tool to fix links in staged website.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/incubator-beam-site fix_links

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/72.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #72


commit b98ead91fe999b8decbcd3b9f8c52dda2fc96456
Author: Jason Kuster 
Date:   2016-11-08T22:52:06Z

Add tool to fix links.

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-922) runner-core does not pass under findbugs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649052#comment-15649052
 ] 

ASF GitHub Bot commented on BEAM-922:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1318

[BEAM-922] Fix findbugs errors and re-enable for runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @dhalperi 

The error in `UnboundedReadFromBoundedSource` was a real coding error; 
please review my small edit there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
runners-core-findbugs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1318.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1318


commit 2a4f6fffcdcf1804b1c173d6e4ab43ffc0431cb7
Author: Kenneth Knowles 
Date:   2016-11-08T22:49:09Z

Fix findbugs errors and re-enable for runners-core




> runner-core does not pass under findbugs
> 
>
> Key: BEAM-922
> URL: https://issues.apache.org/jira/browse/BEAM-922
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>
> @Kenn because mostly in state machines. Since Findbugs doesn't grok 
> reflection, probably just need suppressions.
> {code}
> INFO] Dead store to invoker rather than field with same name in new 
> org.apache.beam.runners.core.SplittableParDo$ProcessFn(DoFn, Coder, Coder, 
> Coder) [org.apache.beam.runners.core.SplittableParDo$ProcessFn] At 
> SplittableParDo.java:[line 242]
> [INFO] Possible null pointer dereference of 
> UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.reader
>  in 
> org.apache.beam.runners.core.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.advance()
>  
> [org.apache.beam.runners.core.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource,
>  
> org.apache.beam.runners.core.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource]
>  Dereferenced at UnboundedReadFromBoundedSource.java:[line 476]Known null at 
> UnboundedReadFromBoundedSource.java:[line 472]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterAll)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 130-135]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterEach)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 154-161]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterFirst)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 121-126]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterPane)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 139]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterProcessingTime)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 175]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterSynchronizedProcessingTime)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 117]
> [INFO] Private method 
> 

[GitHub] incubator-beam pull request #1318: [BEAM-922] Fix findbugs errors and re-ena...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1318

[BEAM-922] Fix findbugs errors and re-enable for runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @dhalperi 

The error in `UnboundedReadFromBoundedSource` was a real coding error; 
please review my small edit there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
runners-core-findbugs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1318.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1318


commit 2a4f6fffcdcf1804b1c173d6e4ab43ffc0431cb7
Author: Kenneth Knowles 
Date:   2016-11-08T22:49:09Z

Fix findbugs errors and re-enable for runners-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-907) Allow user to use a specific subclass of BoundedWindow in DoFn parameter lists

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-907.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Allow user to use a specific subclass of BoundedWindow in DoFn parameter lists
> --
>
> Key: BEAM-907
> URL: https://issues.apache.org/jira/browse/BEAM-907
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> Today, the user adds a {{BoundedWindow}} parameter to {{DoFn}} in order to 
> get access to the window of an element. In contrast, for 
> {{RestrictionTracker}} and {{State}} the user can place a concrete subclass. 
> We can add this feature pretty easily for the window, saving the user a cast.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-897) Datastore ITs have invalid PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649037#comment-15649037
 ] 

ASF GitHub Bot commented on BEAM-897:
-

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1317

[BEAM-897] Fix DatastoreIT PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

- Remove projectId from V1TestOptions and use the one from GcpOptions

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam ds_it

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1317.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1317


commit c2093c88bc18b03378c96605a298b0a1606f13d0
Author: Vikas Kedigehalli 
Date:   2016-11-08T22:41:25Z

Fix DatastoreIT test options




> Datastore ITs have invalid PipelineOptions
> --
>
> Key: BEAM-897
> URL: https://issues.apache.org/jira/browse/BEAM-897
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Vikas Kedigehalli
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> Datastore ITs currently do have multiple incompatible defaults, and this 
> should be rectified.
> cc [~pei...@gmail.com] [~lcwik]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1317: [BEAM-897] Fix DatastoreIT PipelineOption...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1317

[BEAM-897] Fix DatastoreIT PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

- Remove projectId from V1TestOptions and use the one from GcpOptions

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam ds_it

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1317.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1317


commit c2093c88bc18b03378c96605a298b0a1606f13d0
Author: Vikas Kedigehalli 
Date:   2016-11-08T22:41:25Z

Fix DatastoreIT test options




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1768

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1767

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648985#comment-15648985
 ] 

Amit Sela commented on BEAM-944:


Damn you weird JIRA markdown :-0

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   at 
> 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648968#comment-15648968
 ] 

Amit Sela edited comment on BEAM-944 at 11/8/16 10:28 PM:
--

Tested after the fix, but I actually fail to reproduce the failure you describe 
on master branch.
I'm running:{{mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount 
-Dexec.args="--inputFile=/tmp/kinglear.txt --output=/tmp/out-example-master 
--runner=DirectRunner"}} from {{examples/java}}.


was (Author: amitsela):
Tested after the fix. Didn't actually test before...

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> 

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648972#comment-15648972
 ] 

Pei He commented on BEAM-898:
-

One idea to support test arguments is to add "String getTestArg()" in 
TestPipelineOptions.
TestArg could be in the format of "--project=ABC --kind=DEF".
And, each individual IT tests could parse their own test arguemnts.

Any thoughts?

> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648968#comment-15648968
 ] 

Amit Sela commented on BEAM-944:


Tested after the fix. Didn't actually test before...

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648960#comment-15648960
 ] 

Davor Bonaci commented on BEAM-944:
---

Great! It has to be one of these three, right?
* After the fix in the pending PR, or
* spark runner is not on the classpath, or
* spark runner is on the classpath but spark-streaming is too.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #97

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-917) ExpectedLogs should clear saved records after each test.

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He closed BEAM-917.
---
Resolution: Fixed

> ExpectedLogs should clear saved records after each test.
> 
>
> Key: BEAM-917
> URL: https://issues.apache.org/jira/browse/BEAM-917
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> This affects unit tests that use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1766

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648918#comment-15648918
 ] 

ASF GitHub Bot commented on BEAM-664:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1315


> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1315: [BEAM-664] Revise 3 out of 4 wordcount ex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1315


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] incubator-beam git commit: This closes #1315

[kenn]

This closes #1315


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/084a5e8a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/084a5e8a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/084a5e8a

Branch: refs/heads/master
Commit: 084a5e8aedfdd53f3d76ad48b55a66a282881646
Parents: e35f571 9bc9c3f
Author: Kenneth Knowles 
Authored: Tue Nov 8 13:52:23 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:52:23 2016 -0800

--
 .../beam/examples/DebuggingWordCount.java   | 67 +---
 .../apache/beam/examples/MinimalWordCount.java  | 50 +++
 .../org/apache/beam/examples/WordCount.java | 62 +++---
 .../beam/examples/DebuggingWordCountTest.java   | 15 -
 4 files changed, 73 insertions(+), 121 deletions(-)
--

[2/4] incubator-beam git commit: Revise WordCount example to be better cross-runner example

[kenn]

Revise WordCount example to be better cross-runner example


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0b05a8c7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0b05a8c7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0b05a8c7

Branch: refs/heads/master
Commit: 0b05a8c7ff8e1f76516a6b13d504f776b5c9111e
Parents: c64cf36
Author: Kenneth Knowles 
Authored: Thu Nov 3 14:19:47 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:51:25 2016 -0800

--
 .../org/apache/beam/examples/WordCount.java | 62 
 1 file changed, 23 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0b05a8c7/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
index e7eab6e..5be0ddc 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
@@ -17,15 +17,13 @@
  */
 package org.apache.beam.examples;
 
-import com.google.common.base.Strings;
-import java.io.IOException;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.options.Default;
-import org.apache.beam.sdk.options.DefaultValueFactory;
 import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.Validation.Required;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -34,8 +32,6 @@ import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.transforms.Sum;
-import org.apache.beam.sdk.util.IOChannelFactory;
-import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
@@ -53,7 +49,7 @@ import org.apache.beam.sdk.values.PCollection;
  *   
  *
  * Basic concepts, also in the MinimalWordCount example:
- * Reading text files; counting a PCollection; writing to GCS.
+ * Reading text files; counting a PCollection; writing to text files
  *
  * New Concepts:
  * 
@@ -63,30 +59,31 @@ import org.apache.beam.sdk.values.PCollection;
  *   4. Defining your own pipeline options
  * 
  *
- * Concept #1: you can execute this pipeline either locally or using the 
selected runner.
+ * Concept #1: you can execute this pipeline either locally or using by 
selecting another runner.
  * These are now command-line options and not hard-coded as they were in the 
MinimalWordCount
  * example.
- * To execute this pipeline locally, specify a local output file or output 
prefix on GCS:
- * {@code
- *   --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PREFIX]
- * }
  *
  * To change the runner, specify:
  * {@code
  *   --runner=YOUR_SELECTED_RUNNER
  * }
  * 
- * See examples/java/README.md for instructions about how to configure 
different runners.
  *
- * The input file defaults to {@code 
gs://apache-beam-samples/shakespeare/kinglear.txt}
- * and can be overridden with {@code --inputFile}.
+ * To execute this pipeline, specify a local output file (if using the
+ * {@code DirectRunner}) or output prefix on a supported distributed file 
system.
+ * {@code
+ *   --output=[YOUR_LOCAL_FILE | YOUR_OUTPUT_PREFIX]
+ * }
+ *
+ * The input file defaults to a public data set containing the text of of 
King Lear,
+ * by William Shakespeare. You can override it and choose your own input with 
{@code --inputFile}.
  */
 public class WordCount {
 
   /**
-   * Concept #2: You can make your pipeline code less verbose by defining your 
DoFns statically out-
-   * of-line. This DoFn tokenizes lines of text into individual words; we pass 
it to a ParDo in the
-   * pipeline.
+   * Concept #2: You can make your pipeline assembly code less verbose by 
defining your DoFns
+   * statically out-of-line. This DoFn tokenizes lines of text into individual 
words; we pass it
+   * to a ParDo in the pipeline.
*/
   static class ExtractWordsFn extends DoFn {
 private final Aggregator emptyLines =
@@ -153,36 +150,23 @@ public class WordCount {
* Inherits standard configuration options.
*/
   public interface WordCountOptions extends PipelineOptions {
+
+/**
+ * By default, this example reads from a public dataset containing 

[1/4] incubator-beam git commit: Hardcode MinimalWordCount to the DirectRunner

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e35f571b0 -> 084a5e8ae


Hardcode MinimalWordCount to the DirectRunner

This makes it easy to immediately run, and removes various
non-portable instructions and others that aren't the easiest
for a "Getting Started" scenario.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c64cf367
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c64cf367
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c64cf367

Branch: refs/heads/master
Commit: c64cf367299b6fdbe25c62eec9840b02fbc9d518
Parents: e35f571
Author: Kenneth Knowles 
Authored: Thu Nov 3 14:18:43 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:51:23 2016 -0800

--
 .../apache/beam/examples/MinimalWordCount.java  | 50 +---
 1 file changed, 22 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c64cf367/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
index 6fc873e..6085539 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java
@@ -37,46 +37,33 @@ import org.apache.beam.sdk.values.KV;
  * argument processing, and focus on construction of the pipeline, which 
chains together the
  * application of core transforms.
  *
- * Next, see the {@link WordCount} pipeline, then the {@link 
DebuggingWordCount}, and finally
- * the {@link WindowedWordCount} pipeline, for more detailed examples that 
introduce additional
+ * Next, see the {@link WordCount} pipeline, then the {@link 
DebuggingWordCount}, and finally the
+ * {@link WindowedWordCount} pipeline, for more detailed examples that 
introduce additional
  * concepts.
  *
  * Concepts:
+ *
  * 
  *   1. Reading data from text files
  *   2. Specifying 'inline' transforms
- *   3. Counting a PCollection
- *   4. Writing data to Cloud Storage as text files
+ *   3. Counting items in a PCollection
+ *   4. Writing data to text files
  * 
  *
- * To execute this pipeline, first edit the code to set your project ID, 
the temp
- * location, and the output location. The specified GCS bucket(s) must already 
exist.
- *
- * Then, run the pipeline as described in the README. It will be deployed 
and run with the
- * selected runner. No args are required to run the pipeline. You can see the 
results in your
- * output bucket in the GCS browser.
+ * No arguments are required to run this pipeline. It will be executed with 
the DirectRunner. You
+ * can see the results in the output files in your current working directory, 
with names like
+ * "wordcounts-1-of-5. When running on a distributed service, you 
would use an appropriate
+ * file service.
  */
 public class MinimalWordCount {
 
   public static void main(String[] args) {
 // Create a PipelineOptions object. This object lets us set various 
execution
-// options for our pipeline, such as the associated Cloud Platform project 
and the location
-// in Google Cloud Storage to stage files.
+// options for our pipeline, such as the runner you wish to use. This 
example
+// will run with the DirectRunner by default, based on the class path 
configured
+// in its dependencies.
 PipelineOptions options = PipelineOptionsFactory.create();
 
-// In order to run your pipeline, you need to make following runner 
specific changes:
-//
-// CHANGE 1/3: Select a Beam runner, such as DataflowRunner or FlinkRunner.
-// CHANGE 2/3: Specify runner-required options.
-// For DataflowRunner, set project and temp location as follows:
-//   DataflowPipelineOptions dataflowOptions = 
options.as(DataflowPipelineOptions.class);
-//   dataflowOptions.setRunner(DataflowRunner.class);
-//   dataflowOptions.setProject("SET_YOUR_PROJECT_ID_HERE");
-//   
dataflowOptions.setTempLocation("gs://SET_YOUR_BUCKET_NAME_HERE/AND_TEMP_DIRECTORY");
-// For FlinkRunner, set the runner as follows. See {@code 
FlinkPipelineOptions}
-// for more details.
-//   options.setRunner(FlinkRunner.class);
-
 // Create the Pipeline object with the options we defined above.
 Pipeline p = Pipeline.create(options);
 
@@ -85,7 +72,10 @@ public class MinimalWordCount {
 // Concept #1: Apply a root transform to the pipeline; in this case, 
TextIO.Read to read a set
 // of input text files. TextIO.Read returns a PCollection where each 
element is 

[3/4] incubator-beam git commit: Revise DebuggingWordCount to be more portable

[kenn]

Revise DebuggingWordCount to be more portable


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9bc9c3f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9bc9c3f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9bc9c3f0

Branch: refs/heads/master
Commit: 9bc9c3f0fcab4571f60d4eb872df0904ee0eb99d
Parents: 0b05a8c
Author: Kenneth Knowles 
Authored: Thu Nov 3 14:50:02 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:51:25 2016 -0800

--
 .../beam/examples/DebuggingWordCount.java   | 67 +---
 .../beam/examples/DebuggingWordCountTest.java   | 15 -
 2 files changed, 28 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9bc9c3f0/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
index 1d2c83a..f7c537c 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java
@@ -50,10 +50,9 @@ import org.slf4j.LoggerFactory;
  *
  * New Concepts:
  * 
- *   1. Logging to Cloud Logging
- *   2. Controlling worker log levels
- *   3. Creating a custom aggregator
- *   4. Testing your Pipeline via PAssert
+ *   1. Logging using SLF4J, even in a distributed environment
+ *   2. Creating a custom aggregator (runners have varying levels of support)
+ *   3. Testing your Pipeline via PAssert
  * 
  *
  * To execute this pipeline locally, specify general pipeline configuration:
@@ -68,51 +67,20 @@ import org.slf4j.LoggerFactory;
  * }
  * 
  *
- * To use the additional logging discussed below, specify:
- * {@code
- *   --workerLogLevelOverrides={"org.apache.beam.examples":"DEBUG"}
- * }
- * 
- *
- * Note that when you run via mvn exec, you may need to escape
- * the quotations as appropriate for your shell. For example, in 
bash:
- * 
- * mvn compile exec:java ... \
- *   -Dexec.args="... \
- * 
--workerLogLevelOverrides={\\\"org.apache.beam.examples\\\":\\\"DEBUG\\\"}"
- * 
+ * The input file defaults to a public data set containing the text of of 
King Lear,
+ * by William Shakespeare. You can override it and choose your own input with 
{@code --inputFile}.
  *
- * Concept #2: Dataflow workers which execute user code are configured to 
log to Cloud
- * Logging by default at "INFO" log level and higher. One may override log 
levels for specific
- * logging namespaces by specifying:
- * 
- *   --workerLogLevelOverrides={"Name1":"Level1","Name2":"Level2",...}
- * 
- * For example, by specifying:
- * 
- *   --workerLogLevelOverrides={"org.apache.beam.examples":"DEBUG"}
- * 
- * when executing this pipeline using the Dataflow service, Cloud Logging 
would contain only
- * "DEBUG" or higher level logs for the {@code org.apache.beam.examples} 
package in
- * addition to the default "INFO" or higher level logs. In addition, the 
default Dataflow worker
- * logging configuration can be overridden by specifying
- * {@code --defaultWorkerLogLevel=}. 
For example,
- * by specifying {@code --defaultWorkerLogLevel=DEBUG} when executing this 
pipeline with
- * the Dataflow service, Cloud Logging would contain all "DEBUG" or higher 
level logs. Note
- * that changing the default worker log level to TRACE or DEBUG will 
significantly increase
- * the amount of logs output.
- *
- * The input file defaults to {@code 
gs://apache-beam-samples/shakespeare/kinglear.txt}
- * and can be overridden with {@code --inputFile}.
  */
 public class DebuggingWordCount {
   /** A DoFn that filters for a specific key based upon a regular expression. 
*/
   public static class FilterTextFn extends DoFn, KV> {
 /**
- * Concept #1: The logger below uses the fully qualified class name of 
FilterTextFn
- * as the logger. All log statements emitted by this logger will be 
referenced by this name
- * and will be visible in the Cloud Logging UI. Learn more at 
https://cloud.google.com/logging
- * about the Cloud Logging UI.
+ * Concept #1: The logger below uses the fully qualified class name of 
FilterTextFn as the
+ * logger. Depending on your SLF4J configuration, log statements will 
likely be qualified by
+ * this name.
+ *
+ * Note that this is entirely standard SLF4J usage. Some runners may 
provide a default SLF4J
+ * configuration that is most appropriate for their logging integration.
  */
 private static final Logger LOG = 

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648885#comment-15648885
 ] 

ASF GitHub Bot commented on BEAM-664:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1314


> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-937) The DirectRunner does not advance watermarks if reading from an empty reader

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-937.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> The DirectRunner does not advance watermarks if reading from an empty reader
> 
>
> Key: BEAM-937
> URL: https://issues.apache.org/jira/browse/BEAM-937
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam git commit: Add Spark and Flink runner version to root pom

[kenn]

Add Spark and Flink runner version to root pom


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/32b93815
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/32b93815
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/32b93815

Branch: refs/heads/master
Commit: 32b9381525bce5781a2d523175bdc34cd114299f
Parents: 09f6aa6
Author: Kenneth Knowles 
Authored: Tue Nov 8 11:53:22 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:39:06 2016 -0800

--
 pom.xml | 12 
 1 file changed, 12 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/32b93815/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bd6037e..749ca9c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -352,6 +352,18 @@
 
   
 org.apache.beam
+beam-runners-spark
+${project.version}
+  
+
+  
+org.apache.beam
+beam-runners-flink_2.10
+${project.version}
+  
+
+  
+org.apache.beam
 beam-examples-java
 ${project.version}
   

[GitHub] incubator-beam pull request #1314: [BEAM-664] Add runner-specific profiles t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1314


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-938) Break in direct.UnboundedReadEvaluatorFactoryTest.noElementsAvailableReaderIncludedInResidual

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648883#comment-15648883
 ] 

Thomas Groh commented on BEAM-938:
--

This was due to an incompletely updated test;

Specifically, the TestUnboundedSource was updated to provide a watermark that 
always advanced, but {{noElementsAvailableReaderIncludedInResidual}} used an 
{{UnboundedCountingSource}}

https://github.com/apache/incubator-beam/pull/1303 reverts the test, and 
https://github.com/apache/incubator-beam/pull/1313 is the complete fix.

> Break in 
> direct.UnboundedReadEvaluatorFactoryTest.noElementsAvailableReaderIncludedInResidual
> -
>
> Key: BEAM-938
> URL: https://issues.apache.org/jira/browse/BEAM-938
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Thomas Groh
> Fix For: 0.4.0-incubating
>
>
> Likely culprit:
> https://github.com/apache/incubator-beam/commit/ff7fe07be96de393b763e7b3d213734040aa3795



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam git commit: This closes #1314

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 09f6aa607 -> e35f571b0


This closes #1314


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e35f571b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e35f571b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e35f571b

Branch: refs/heads/master
Commit: e35f571b0df9e1b74013632c30743a1ddecd3e2a
Parents: 09f6aa6 8079284
Author: Kenneth Knowles 
Authored: Tue Nov 8 13:39:06 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:39:06 2016 -0800

--
 examples/java/pom.xml | 62 ++
 pom.xml   | 12 +
 2 files changed, 42 insertions(+), 32 deletions(-)
--

[2/3] incubator-beam git commit: Add runner-specific profiles to the examples pom.xml

[kenn]

Add runner-specific profiles to the examples pom.xml


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/80792848
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/80792848
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/80792848

Branch: refs/heads/master
Commit: 807928482706db1bd7904207487adc65c5f435b6
Parents: 32b9381
Author: Kenneth Knowles 
Authored: Thu Nov 3 16:17:18 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 8 13:39:06 2016 -0800

--
 examples/java/pom.xml | 62 ++
 1 file changed, 30 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/80792848/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 6d18a0f..f66fd36 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -39,22 +39,13 @@
   
 
   
+
 
 
-  include-runners
+  direct-runner
   
 true
   
@@ -62,41 +53,43 @@
 
   org.apache.beam
   beam-runners-direct-java
-  ${project.version}
   runtime
-  true
 
+  
+
 
+
+
+  flink-runner
+  
 
   org.apache.beam
   beam-runners-flink_2.10
-  ${project.version}
   runtime
-  true
 
+  
+
 
+
+
+  spark-runner
+  
 
   org.apache.beam
-  beam-runners-google-cloud-dataflow-java
-  ${project.version}
+  beam-runners-spark
   runtime
-  true
 
-
 
-  org.apache.beam
-  beam-runners-spark
-  ${project.version}
+  org.apache.spark
+  spark-streaming_2.10
+  ${spark.version}
   runtime
-  true
 
-
 
   org.apache.spark
   spark-core_2.10
   ${spark.version}
   runtime
-  true
   
 
   org.slf4j
@@ -104,16 +97,22 @@
 
   
 
+  
+
 
+
+
+  dataflow-runner
+  
 
-  org.apache.spark
-  spark-streaming_2.10
-  ${spark.version}
+  org.apache.beam
+  beam-runners-google-cloud-dataflow-java
   runtime
-  true
 
   
 
+
+
 
   jenkins-precommit
   
@@ -448,7 +447,6 @@
 
   org.apache.beam
   beam-runners-direct-java
-  ${project.version}
   test
 
 

[jira] [Resolved] (BEAM-938) Break in direct.UnboundedReadEvaluatorFactoryTest.noElementsAvailableReaderIncludedInResidual

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-938.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Break in 
> direct.UnboundedReadEvaluatorFactoryTest.noElementsAvailableReaderIncludedInResidual
> -
>
> Key: BEAM-938
> URL: https://issues.apache.org/jira/browse/BEAM-938
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Thomas Groh
> Fix For: 0.4.0-incubating
>
>
> Likely culprit:
> https://github.com/apache/incubator-beam/commit/ff7fe07be96de393b763e7b3d213734040aa3795



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648871#comment-15648871
 ] 

Amit Sela commented on BEAM-944:


Successfully executed WordCount example with direct runner.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>  

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648847#comment-15648847
 ] 

Davor Bonaci commented on BEAM-944:
---

`LeaderBoard` is in `examples/java8`, not `examples/java`. But, any other 
example from `examples/java` would do too.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> 

[jira] [Commented] (BEAM-891) Flake in Spark metrics library?

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648828#comment-15648828
 ] 

Amit Sela commented on BEAM-891:


OK, happened again here: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_SparkLocal/96/org.apache.beam$beam-runners-spark/testReport/junit/org.apache.beam.sdk.transforms/CreateTest/testCreate/

[~staslev] any ideas so far ?

> Flake in Spark metrics library?
> ---
>
> Key: BEAM-891
> URL: https://issues.apache.org/jira/browse/BEAM-891
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Stas Levin
>
> [~staslev] I think you implemented this functionality originally? Want to 
> take a look? CC [~amitsela]
> Run: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_SparkLocal/org.apache.beam$beam-runners-spark/43/testReport/junit/org.apache.beam.sdk.transforms/FilterTest/testFilterGreaterThan/
> Error:
> {code}
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: 5
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:77)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:53)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
>   at 
> org.apache.beam.sdk.transforms.FilterTest.testFilterGreaterThan(FilterTest.java:122)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: 5
>   at 
> scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:43)
>   at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:189)
>   at 
> scala.collection.mutable.ArrayBuffer.segmentLength(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.indexWhere(IndexedSeqOptimized.scala:198)
>   at scala.collection.mutable.ArrayBuffer.indexWhere(ArrayBuffer.scala:47)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:144)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:128)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at 
> scala.collection.mutable.BufferLike$class.$minus$eq(BufferLike.scala:126)
>   at scala.collection.mutable.AbstractBuffer.$minus$eq(Buffer.scala:48)
>   at 
> org.apache.spark.metrics.MetricsSystem.removeSource(MetricsSystem.scala:159)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.registerMetrics(SparkRuntimeContext.java:94)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.(SparkRuntimeContext.java:66)
>   at 
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:73)
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:146)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648820#comment-15648820
 ] 

Amit Sela commented on BEAM-944:


[~davor] I've tried running the same from {{example/java}} but I get:
{{java.lang.ClassNotFoundException: 
org.apache.beam.examples.complete.game.LeaderBoard}}
What am I missing ?

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648796#comment-15648796
 ] 

ASF GitHub Bot commented on BEAM-944:
-

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1316

[BEAM-944] Spark runner causes an exception when creating pipeline op…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…tions.

Create a SparkContextOptions for context-ware options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-944

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1316.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1316


commit 45823481d1356caad138ca5c8504276045425411
Author: Sela 
Date:   2016-11-08T21:05:13Z

[BEAM-944] Spark runner causes an exception when creating pipeline options.
Create a SparkContextOptions for context-ware options.




> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648794#comment-15648794
 ] 

Davor Bonaci commented on BEAM-944:
---

Not sure I understand. If we are moving these pipelineoptions and not 
registering some of them, nothing is changing conceptually. Everything is still 
available, just a little bit hidden.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> 

[GitHub] incubator-beam pull request #1316: [BEAM-944] Spark runner causes an excepti...

[amitsela]

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1316

[BEAM-944] Spark runner causes an exception when creating pipeline op…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…tions.

Create a SparkContextOptions for context-ware options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-944

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1316.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1316


commit 45823481d1356caad138ca5c8504276045425411
Author: Sela 
Date:   2016-11-08T21:05:13Z

[BEAM-944] Spark runner causes an exception when creating pipeline options.
Create a SparkContextOptions for context-ware options.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-704) KafkaIO should handle "latest offset" evenly, and persist it as part of the CheckpointMark.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-704.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> KafkaIO should handle "latest offset" evenly, and persist it as part of the 
> CheckpointMark.
> ---
>
> Key: BEAM-704
> URL: https://issues.apache.org/jira/browse/BEAM-704
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>Assignee: Raghu Angadi
> Fix For: 0.3.0-incubating
>
>
> Currently, the KafkaIO (when configured to "latest") will check the latest 
> offset on the worker. This means that each worker sees a "different" latest 
> for the time it checks for the partitions assigned to it.
> This also means that if a worker fails before starting to read, and new 
> messages were added in between, they would be missed.
> I think we should consider checking the offsets (could be the same for 
> "earliest") when running initialSplits (that's how Spark does that as well, 
> one call from the driver for all topic-partitions).
> I'd also suggest we persist the latest offset as part of the CheckpointMark 
> so that once latest is set, it is remembered until new messages arrive and it 
> doesn't need to be resolved again (and if there were new messages available 
> they won't be missed upon failure).
> For Spark this is even more important as state is passed in-between 
> micro-batches and sparse partitions may skip messages until a message finally 
> arrives within the read time-frame. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648620#comment-15648620
 ] 

ASF GitHub Bot commented on BEAM-664:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1315

[BEAM-664] Examples not windowed

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This PR breaks out all the non-`WindowedWordCount` examples from #1275 as 
their changes are much simpler and can go in immediately.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
examples-not-windowed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1315.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1315


commit ef6033e34152ef860f2195361b915164a67c0a72
Author: Kenneth Knowles 
Date:   2016-11-03T21:18:43Z

Hardcode MinimalWordCount to the DirectRunner

This makes it easy to immediately run, and removes various
non-portable instructions and others that aren't the easiest
for a "Getting Started" scenario.

commit 9fc73d069c35d5012b673d3932f3277dae3d1c1e
Author: Kenneth Knowles 
Date:   2016-11-03T21:19:47Z

Revise WordCount example to be better cross-runner example

commit 0996f90316478a205a220bebfb25c11a02da7873
Author: Kenneth Knowles 
Date:   2016-11-03T21:50:02Z

Revise DebuggingWordCount to be more portable

commit 64326c2e2229a8ae559715cf981ddcb89a98f80a
Author: Kenneth Knowles 
Date:   2016-11-03T23:17:18Z

Add runner-specific profiles to the examples pom.xml




> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1315: [BEAM-664] Examples not windowed

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1315

[BEAM-664] Examples not windowed

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This PR breaks out all the non-`WindowedWordCount` examples from #1275 as 
their changes are much simpler and can go in immediately.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
examples-not-windowed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1315.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1315


commit ef6033e34152ef860f2195361b915164a67c0a72
Author: Kenneth Knowles 
Date:   2016-11-03T21:18:43Z

Hardcode MinimalWordCount to the DirectRunner

This makes it easy to immediately run, and removes various
non-portable instructions and others that aren't the easiest
for a "Getting Started" scenario.

commit 9fc73d069c35d5012b673d3932f3277dae3d1c1e
Author: Kenneth Knowles 
Date:   2016-11-03T21:19:47Z

Revise WordCount example to be better cross-runner example

commit 0996f90316478a205a220bebfb25c11a02da7873
Author: Kenneth Knowles 
Date:   2016-11-03T21:50:02Z

Revise DebuggingWordCount to be more portable

commit 64326c2e2229a8ae559715cf981ddcb89a98f80a
Author: Kenneth Knowles 
Date:   2016-11-03T23:17:18Z

Add runner-specific profiles to the examples pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-945) BigQueryIO.parseTableSpec is package-private

[Andrew Martin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Martin updated BEAM-945:
---
Description: 
It was changed in this commit. 
https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6.

Is there a good reason for it to not be public? If not I will submit a PR to 
change it back.

  was:
It was changed in this commit. 
https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6.

Is there a good reason for it to not be public?


> BigQueryIO.parseTableSpec is package-private
> 
>
> Key: BEAM-945
> URL: https://issues.apache.org/jira/browse/BEAM-945
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.3.0-incubating
>Reporter: Andrew Martin
>Assignee: Daniel Halperin
>Priority: Blocker
>
> It was changed in this commit. 
> https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6.
> Is there a good reason for it to not be public? If not I will submit a PR to 
> change it back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-945) BigQueryIO.parseTableSpec is package-private

[Andrew Martin (JIRA)]

Andrew Martin created BEAM-945:
--

 Summary: BigQueryIO.parseTableSpec is package-private
 Key: BEAM-945
 URL: https://issues.apache.org/jira/browse/BEAM-945
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 0.3.0-incubating
Reporter: Andrew Martin
Assignee: Daniel Halperin
Priority: Blocker


It was changed in this commit. 
https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6.

Is there a good reason for it to not be public?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648605#comment-15648605
 ] 

ASF GitHub Bot commented on BEAM-664:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1314

[BEAM-664] Add runner-specific profiles to the examples pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @davorbonaci 

Breaking this commit from #1275 as it adds value immediately and that PR is 
likely to be delayed fixing the integration testing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam examples-profiles

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1314.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1314


commit 543835062797f986e1eb2afdbabb38281a580644
Author: Kenneth Knowles 
Date:   2016-11-03T23:17:18Z

Add runner-specific profiles to the examples pom.xml




> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1314: [BEAM-664] Add runner-specific profiles t...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1314

[BEAM-664] Add runner-specific profiles to the examples pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @davorbonaci 

Breaking this commit from #1275 as it adds value immediately and that PR is 
likely to be delayed fixing the integration testing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam examples-profiles

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1314.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1314


commit 543835062797f986e1eb2afdbabb38281a580644
Author: Kenneth Knowles 
Date:   2016-11-03T23:17:18Z

Add runner-specific profiles to the examples pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648595#comment-15648595
 ] 

Amit Sela commented on BEAM-944:


Seems fine. Give me 1-2 hours.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   at 
> 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1765

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648589#comment-15648589
 ] 

Kenneth Knowles commented on BEAM-944:
--

If you had {{SparkPipelineOptions}} registered and {{SparkContextOptions}} not 
registered, then instead of 
{{options.as(SparkPipelineOptions.class).setListeners(...)}} they would write 
something like {{options.as(SparkContextOptions.class).setListeners(...)}}.

It might actually make it clearer for users, so all the things they can easily 
toggle on the command line are in one place, and all the things that are more 
advanced code-only uses in another place.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648551#comment-15648551
 ] 

Amit Sela commented on BEAM-944:


It could be confusing to users though..

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   at 
> 

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #96

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648518#comment-15648518
 ] 

Amit Sela commented on BEAM-944:


Mmm... forgot about the fact that it will only try to load if registered, and 
those two options could be used only if explicitly, programmatically, defined 
(i.e., not via --args).
Would there be any other implications, except for having to use them only 
directly via the {{SparkPipelineOptions}} API ? is there something I'm missing 
?  

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> 

[GitHub] incubator-beam pull request #1302: Add an Enforcement enum

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1302


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add an Enforcement enum

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7855f29c5 -> 09f6aa607


Add an Enforcement enum

This tracks enabled enforcements independently of the PipelineOptions
object.

Move utility methods to construct enforcements and bundle factories
based around enabled enforcements to the Enforcement enum.

Only apply Immutability Enforcement if Immutability Enforcement is
applicable to the producing PTransform.

Remove EncodabilityEnforcement, as its responsibilities are handled in
CloningBundleFactory.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a3b80d1e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a3b80d1e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a3b80d1e

Branch: refs/heads/master
Commit: a3b80d1ea0919ec260b97fc8529b3625895d780e
Parents: 7855f29
Author: Thomas Groh 
Authored: Mon Nov 7 11:30:22 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 8 11:10:49 2016 -0800

--
 .../beam/runners/direct/DirectRunner.java   | 114 ---
 .../direct/EncodabilityEnforcementFactory.java  |  80 -
 .../ImmutabilityCheckingBundleFactory.java  |  11 +-
 .../direct/ImmutabilityEnforcementFactory.java  |   2 -
 .../direct/CloningBundleFactoryTest.java| 122 ++-
 .../beam/runners/direct/DirectRunnerTest.java   |  26 ++
 .../EncodabilityEnforcementFactoryTest.java | 323 ---
 7 files changed, 224 insertions(+), 454 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a3b80d1e/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 4d5a449..f4aeb3e 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -24,8 +24,11 @@ import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import java.io.IOException;
 import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
 import java.util.HashMap;
 import java.util.Map;
+import java.util.Set;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import javax.annotation.Nullable;
@@ -189,8 +192,72 @@ public class DirectRunner
 void add(Iterable values);
   }
 
+  /** The set of {@link PTransform PTransforms} that execute a UDF. Useful for 
some enforcements. */
+  private static final Set CONTAINS_UDF =
+  ImmutableSet.of(
+  Read.Bounded.class, Read.Unbounded.class, ParDo.Bound.class, 
ParDo.BoundMulti.class);
+
+  enum Enforcement {
+ENCODABILITY {
+  @Override
+  public boolean appliesTo(PTransform transform) {
+return true;
+  }
+},
+IMMUTABILITY {
+  @Override
+  public boolean appliesTo(PTransform transform) {
+return CONTAINS_UDF.contains(transform.getClass());
+  }
+};
+
+public abstract boolean appliesTo(PTransform transform);
+
+

+// Utilities for creating enforcements
+public static Set enabled(DirectOptions options) {
+  EnumSet enabled = EnumSet.noneOf(Enforcement.class);
+  if (options.isEnforceEncodability()) {
+enabled.add(ENCODABILITY);
+  }
+  if (options.isEnforceImmutability()) {
+enabled.add(IMMUTABILITY);
+  }
+  return Collections.unmodifiableSet(enabled);
+}
+
+public static BundleFactory bundleFactoryFor(Set 
enforcements) {
+  BundleFactory bundleFactory =
+  enforcements.contains(Enforcement.ENCODABILITY)
+  ? CloningBundleFactory.create()
+  : ImmutableListBundleFactory.create();
+  if (enforcements.contains(Enforcement.IMMUTABILITY)) {
+bundleFactory = 
ImmutabilityCheckingBundleFactory.create(bundleFactory);
+  }
+  return bundleFactory;
+}
+
+@SuppressWarnings("rawtypes")
+private static Map
+defaultModelEnforcements(Set enabledEnforcements) {
+  ImmutableMap.Builder
+  enforcements = ImmutableMap.builder();
+  ImmutableList.Builder enabledParDoEnforcements =
+  ImmutableList.builder();
+  if (enabledEnforcements.contains(Enforcement.IMMUTABILITY)) {
+enabledParDoEnforcements.add(ImmutabilityEnforcementFactory.create());
+  }
+  Collection 

[2/2] incubator-beam git commit: This closes #1302

[tgroh]

This closes #1302


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/09f6aa60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/09f6aa60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/09f6aa60

Branch: refs/heads/master
Commit: 09f6aa6074af35d1bafc913938bb18971d185767
Parents: 7855f29 a3b80d1
Author: Thomas Groh 
Authored: Tue Nov 8 11:10:50 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 8 11:10:50 2016 -0800

--
 .../beam/runners/direct/DirectRunner.java   | 114 ---
 .../direct/EncodabilityEnforcementFactory.java  |  80 -
 .../ImmutabilityCheckingBundleFactory.java  |  11 +-
 .../direct/ImmutabilityEnforcementFactory.java  |   2 -
 .../direct/CloningBundleFactoryTest.java| 122 ++-
 .../beam/runners/direct/DirectRunnerTest.java   |  26 ++
 .../EncodabilityEnforcementFactoryTest.java | 323 ---
 7 files changed, 224 insertions(+), 454 deletions(-)
--

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648493#comment-15648493
 ] 

Kenneth Knowles commented on BEAM-944:
--

Perhaps you can split it into two options classes, with the troublesome one not 
registered?

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1764

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648409#comment-15648409
 ] 

Amit Sela edited comment on BEAM-944 at 11/8/16 6:48 PM:
-

Wait, there is a {{runtime}} dependency in examples pom.. 
Both spark-core and spark-streaming are scoped {{runtime}}.

I that's the issue, we'll have the same problem and have to remove the provided 
context.

As for listeners, they're nice, useful for testing (to the best of my knowledge 
so far), but removing the provided context option will prevent the runner from 
running with the Spark Job Server, and intents to running Scio+SparkRunner in 
REPL. And maybe Zeppelin. All of which are examples of features users asked 
about..   


was (Author: amitsela):
Wait, there is a {{runtime}} dependency in examples pom.. 
Both spark-core and spark-streaming are scoped {{runtime}}.

I that's the issue, we'll have the same problem and have to remove the provided 
context.

As for listeners, they're nice, useful for testing (to the best of my knowledge 
so far), but the removing provided context option will prevent the runner from 
running with the Spark Job Server, and intents to running Scio+SparkRunner in 
REPL. And maybe Zeppelin. All of which are examples of features users asked 
about..   

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648409#comment-15648409
 ] 

Amit Sela edited comment on BEAM-944 at 11/8/16 6:45 PM:
-

Wait, there is a {{runtime}} dependency in examples pom.. 
Both spark-core and spark-streaming are scoped {{runtime}}.

I that's the issue, we'll have the same problem and have to remove the provided 
context.

As for listeners, they're nice, useful for testing (to the best of my knowledge 
so far), but the removing provided context option will prevent the runner from 
running with the Spark Job Server, and intents to running Scio+SparkRunner in 
REPL. And maybe Zeppelin. All of which are examples of features users asked 
about..   


was (Author: amitsela):
Wait, there is a {{runtime}} dependency in examples pom.. 

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648409#comment-15648409
 ] 

Amit Sela commented on BEAM-944:


Wait, there is a {{runtime}} dependency in examples pom.. 

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648401#comment-15648401
 ] 

Amit Sela commented on BEAM-944:


I can't think of anything immediate, but given there is no other way to pass 
things around, examples pom should add dependency on {{spark-streaming}}. Like 
I said, both spark-core and spark-streaming make-up Spark's compute engine, if 
we're OK with one of them, we should be fine with the other.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648394#comment-15648394
 ] 

Amit Sela edited comment on BEAM-944 at 11/8/16 6:38 PM:
-

Both the Listener and the provided context are fields that are not meant to be 
shipped to workers (@JsonIgnore) but this the only mechanism Beam provides at 
the moment to pass properties from the client to the driver.

Agree about avoiding to force class loading.

And eventually, if examples should depend on  spark, flink, or any other 
runner, consider spark as: spark-core + spark-streaming. Less convenient, but 
shouldn't make much difference.. 


was (Author: amitsela):
Both the Listener and the provided context are fields that are not meant to be 
shipped to workers (@JsonIgnore) but this the only mechanism Beam provides at 
the moment..

Agree about avoiding to force class loading.

And eventually, if examples should depend on  spark, flink, or any other 
runner, consider spark as: spark-core + spark-streaming. Less convenient, but 
shouldn't make much difference.. 

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648395#comment-15648395
 ] 

Davor Bonaci commented on BEAM-944:
---

I believe PipelineOptions will scan *registered* pipeline options only, plus 
those requested by the user when calling 
{code}something.as(SomeOptions.class){code}. It scans Spark ones because 
SparkRunner uses AutoService to register its options, 
https://github.com/apache/incubator-beam/blob/master/runners/spark/pom.xml#L165.
 I believe this behavior is roughly correct; but perhaps it could be less eager 
in some cases.

It would be good to move {code}JavaSparkContext{code} elsewhere, in addition to 
{code}JavaStreamingListener{code}. Basically, these aren't real pipeline 
options. They seem to be there because pipeline options make it easy to pass 
things around. To be honest, there are many such examples everywhere, this 
being just one of them: 
https://github.com/apache/incubator-beam/blob/465c7d1f1e4f77e3de218365005f06e8e4ec6167/sdks/java/core/src/main/java/org/apache/beam/sdk/options/GcsOptions.java#L46
 -- just the impact is smaller because it doesn't normally causes an exception.

Is there a better way to pass these around?

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648394#comment-15648394
 ] 

Amit Sela commented on BEAM-944:


Both the Listener and the provided context are fields that are not meant to be 
shipped to workers (@JsonIgnore) but this the only mechanism Beam provides at 
the moment..

Agree about avoiding to force class loading.

And eventually, if examples should depend on  spark, flink, or any other 
runner, consider spark as: spark-core + spark-streaming. Less convenient, but 
shouldn't make much difference.. 

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648377#comment-15648377
 ] 

Kenneth Knowles edited comment on BEAM-944 at 11/8/16 6:32 PM:
---

I think we should follow all three of the following paths:

# I agree that we should move away from putting miscellaneous objects in the 
{{PipelineOptions}} unless they are "just data". Basically things that could be 
command line options. For complex types, they need to be trivially convertible 
using Jackson's {{ObjectMapper}}. So I don't think that fits this case, where 
the listener embodies complex behavior. Am I wrong?
# Even if we follow that rule, I can imagine we may still have a 
{{PipelineOptions}} subclass that links to {{provided}} dependencies. So 
{{PipelineOptions}} also needs to validate later and not force class loading - 
it needs to respect the usual expectations of Java's linking. If it wouldn't 
cause a crash without {{PipelineOptions}}, it shouldn't cause a crash with it. 
This is not a quick fix for this issue but important long term.
# Last, we are moving towards always isolating each runner in its own profile 
in every example. This is an easy quick fix.


was (Author: kenn):
Thoughts:

# I agree that we should move away from putting miscellaneous objects in the 
{{PipelineOptions}} unless they are "just data". Basically things that could be 
command line options. For complex types, they need to be trivially convertible 
using Jackson's {{ObjectMapper}}. So I don't think that fits this case, where 
the listener embodies complex behavior. Am I wrong?
# Even if we follow that rule, I can imagine we may still have a 
{{PipelineOptions}} subclass that links to {{provided}} dependencies. So 
{{PipelineOptions}} also needs to validate later and not force class loading - 
it needs to respect the usual expectations of Java's linking. If it wouldn't 
cause a crash without {{PipelineOptions}}, it shouldn't cause a crash with it. 
This is not a quick fix for this issue but important long term.
# Last, we are moving towards always isolating each runner in its own profile 
in every example. This is an easy quick fix.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648377#comment-15648377
 ] 

Kenneth Knowles commented on BEAM-944:
--

Thoughts:

# I agree that we should move away from putting miscellaneous objects in the 
{{PipelineOptions}} unless they are "just data". Basically things that could be 
command line options. For complex types, they need to be trivially convertible 
using Jackson's {{ObjectMapper}}. So I don't think that fits this case, where 
the listener embodies complex behavior. Am I wrong?
# Even if we follow that rule, I can imagine we may still have a 
{{PipelineOptions}} subclass that links to {{provided}} dependencies. So 
{{PipelineOptions}} also needs to validate later and not force class loading - 
it needs to respect the usual expectations of Java's linking. If it wouldn't 
cause a crash without {{PipelineOptions}}, it shouldn't cause a crash with it. 
This is not a quick fix for this issue but important long term.
# Last, we are moving towards always isolating each runner in its own profile 
in every example. This is an easy quick fix.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648372#comment-15648372
 ] 

Amit Sela commented on BEAM-944:


As a side note, why aren't they really {{PipelineOptions}} ? is there another 
mechanism in Beam to pass along external properties ?

I feel like we're missing a runner-specific PipelineOptions.. maybe to be fused 
with the applicative PipelineOptions ? or have a separate mechanism for the 
runners. So that applicative PipelineOptions would be portable as well. WDYT ?

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648360#comment-15648360
 ] 

Amit Sela edited comment on BEAM-944 at 11/8/16 6:29 PM:
-

We either remove all of Spark-dependent PipelineOptions, or simply add 
dependency on {{spark-streaming}} as we did for {{spark-core}} in examples pom 
here 
:https://github.com/apache/incubator-beam/blob/master/examples/java/pom.xml#L95
Spark has a separate artifact for streaming, IMO it should be simply added.


was (Author: amitsela):
We either remove all of Spark-dependent PipelineOptions, or simply add 
dependency on {{spark-streaming}} as we did for {{spark-core}} in examples pom 
here 
:https://github.com/apache/incubator-beam/blob/master/examples/java/pom.xml#L95 

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1763

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648360#comment-15648360
 ] 

Amit Sela commented on BEAM-944:


We either remove all of Spark-dependent PipelineOptions, or simply add 
dependency on {{spark-streaming}} as we did for {{spark-core}} in examples pom 
here 
:https://github.com/apache/incubator-beam/blob/master/examples/java/pom.xml#L95 

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> 

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648341#comment-15648341
 ] 

Amit Sela commented on BEAM-944:


[~davor] is this because the reflective mechanism of {{PipelineOptions}} is 
scanning all implementations available ?

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> 

[jira] [Comment Edited] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648341#comment-15648341
 ] 

Amit Sela edited comment on BEAM-944 at 11/8/16 6:24 PM:
-

[~davor] is this because the reflective mechanism of {{PipelineOptions}} is 
scanning all implementations available ?

And what about {{JavaSparkContext}} for the ProvidedSparkContext option: 
https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L103
 ? should this be a part of PipelineOptions ?


was (Author: amitsela):
[~davor] is this because the reflective mechanism of {{PipelineOptions}} is 
scanning all implementations available ?

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> 

[jira] [Commented] (BEAM-937) The DirectRunner does not advance watermarks if reading from an empty reader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648334#comment-15648334
 ] 

ASF GitHub Bot commented on BEAM-937:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1313


> The DirectRunner does not advance watermarks if reading from an empty reader
> 
>
> Key: BEAM-937
> URL: https://issues.apache.org/jira/browse/BEAM-937
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1313: [BEAM-937][BEAM-938] Update Watermarks ev...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1313


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Update Watermarks even if a Reader is empty

[tgroh]

Update Watermarks even if a Reader is empty

This ensures that the pipeline will make progress even if a reader stops
producing elements.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/550978f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/550978f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/550978f6

Branch: refs/heads/master
Commit: 550978f630ee1e4424e981ddc5ff5e89aa8c797d
Parents: bfc527d
Author: Thomas Groh 
Authored: Mon Nov 7 12:59:06 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 8 10:17:13 2016 -0800

--
 .../direct/UnboundedReadEvaluatorFactory.java   |  6 ++-
 .../UnboundedReadEvaluatorFactoryTest.java  | 47 +---
 2 files changed, 35 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/550978f6/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index e529088..fb09b3e 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -35,6 +35,7 @@ import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.io.UnboundedSource.CheckpointMark;
 import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
+import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -143,12 +144,13 @@ class UnboundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
   // If the reader had no elements available, but the shard is not 
done, reuse it later
   resultBuilder.addUnprocessedElements(
   Collections.singleton(
-  element.withValue(
+  WindowedValue.timestampedValueInGlobalWindow(
   UnboundedSourceShard.of(
   shard.getSource(),
   shard.getDeduplicator(),
   reader,
-  shard.getCheckpoint();
+  shard.getCheckpoint()),
+  reader.getWatermark(;
 }
   } catch (IOException e) {
 if (reader != null) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/550978f6/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
index 9a7fec3..8d38275 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
@@ -49,7 +49,7 @@ import org.apache.beam.sdk.coders.AtomicCoder;
 import org.apache.beam.sdk.coders.BigEndianLongCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.io.CountingInput;
+import org.apache.beam.sdk.coders.VarLongCoder;
 import org.apache.beam.sdk.io.CountingSource;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.UnboundedSource;
@@ -63,11 +63,11 @@ import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.util.VarInt;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.PCollection;
 import org.hamcrest.Matchers;
 import org.joda.time.DateTime;
-import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.joda.time.ReadableInstant;
 import org.junit.Before;
@@ -230,7 +230,7 @@ public class UnboundedReadEvaluatorFactoryTest {
 TestPipeline p = TestPipeline.create();
 // Read with a very slow rate so by the second read there are no more 
elements
 PCollection pcollection =
-

[1/2] incubator-beam git commit: This closes #1313

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master bfc527d63 -> 7855f29c5


This closes #1313


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7855f29c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7855f29c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7855f29c

Branch: refs/heads/master
Commit: 7855f29c52faea3d9574004304252a71eb064cd8
Parents: bfc527d 550978f
Author: Thomas Groh 
Authored: Tue Nov 8 10:17:13 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 8 10:17:13 2016 -0800

--
 .../direct/UnboundedReadEvaluatorFactory.java   |  6 ++-
 .../UnboundedReadEvaluatorFactoryTest.java  | 47 +---
 2 files changed, 35 insertions(+), 18 deletions(-)
--

[jira] [Updated] (BEAM-835) add Intellij instructions to the contribution guide

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-835:
--
Assignee: Minudika Malshan

> add Intellij instructions to the contribution guide
> ---
>
> Key: BEAM-835
> URL: https://issues.apache.org/jira/browse/BEAM-835
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Minudika Malshan
>Priority: Minor
>  Labels: starter
>
> Add Intellij-specific instructions to the contribution guide, to go alongside 
> the Eclipse instructions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-944:
--
Description: 
Running any example with *any* runner throws an exception as soon as it tries 
to construct PipelineOptions as long as SparkRunner is on the class path:

{code}
mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
-Dexec.args="--runner=DirectRunner"
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Tutorial 0.0.1-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Tutorial 
---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/Users/fjp/dev/mobile/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
[INFO] No sources to compile
[INFO] 
[INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[WARNING] 
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at 
org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
... 6 more
Caused by: java.lang.TypeNotPresentException: Type 
org.apache.spark.streaming.api.java.JavaStreamingListener not present
at 
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
at 
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
at 
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at 
sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
at 
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
at 
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at 
sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
at 
java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
at 
java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
at 
java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
at 
java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:536)
... 7 more
Caused by: java.lang.ClassNotFoundException: 
org.apache.spark.streaming.api.java.JavaStreamingListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
... 25 more
{code}

It is caused by 

[jira] [Updated] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-944:
--
Description: 
Running any example with *any* runner throws an exception as soon as it tries 
to construct PipelineOptions as long as SparkRunner is on the class path:

{code}
mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
-Dexec.args="--runner=DirectRunner"
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Tutorial 0.0.1-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Tutorial 
---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/Users/fjp/dev/mobile/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
[INFO] No sources to compile
[INFO] 
[INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[WARNING] 
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at 
org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
... 6 more
Caused by: java.lang.TypeNotPresentException: Type 
org.apache.spark.streaming.api.java.JavaStreamingListener not present
at 
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
at 
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
at 
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at 
sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
at 
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
at 
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at 
sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
at 
java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
at 
java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
at 
java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
at 
java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:536)
... 7 more
Caused by: java.lang.ClassNotFoundException: 
org.apache.spark.streaming.api.java.JavaStreamingListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
... 25 more
{code}

It is introduced by 

[jira] [Updated] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-944:
--
Summary: Spark runner causes an exception when creating pipeline options  
(was: Spark runner: )

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with Spark runner throws an exception as soon as it tries 
> to construct PipelineOptions:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   at 
> 

[jira] [Created] (BEAM-944) Spark runner:

[Davor Bonaci (JIRA)]

Davor Bonaci created BEAM-944:
-

 Summary: Spark runner: 
 Key: BEAM-944
 URL: https://issues.apache.org/jira/browse/BEAM-944
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Affects Versions: 0.3.0-incubating
Reporter: Davor Bonaci
Assignee: Amit Sela
 Fix For: 0.4.0-incubating


Running any example with Spark runner throws an exception as soon as it tries 
to construct PipelineOptions:

{code}
mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
-Dexec.args="--runner=DirectRunner"
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Tutorial 0.0.1-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Tutorial 
---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/Users/fjp/dev/mobile/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
[INFO] No sources to compile
[INFO] 
[INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[WARNING] 
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at 
org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
... 6 more
Caused by: java.lang.TypeNotPresentException: Type 
org.apache.spark.streaming.api.java.JavaStreamingListener not present
at 
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
at 
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
at 
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at 
sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
at 
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
at 
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at 
sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
at 
java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
at 
java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
at 
java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
at 
java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:536)
... 7 more
Caused by: java.lang.ClassNotFoundException: 
org.apache.spark.streaming.api.java.JavaStreamingListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at 

[jira] [Commented] (BEAM-731) Replace DirectRunner with InProcessRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648235#comment-15648235
 ] 

ASF GitHub Bot commented on BEAM-731:
-

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/1240


> Replace DirectRunner with InProcessRunner
> -
>
> Key: BEAM-731
> URL: https://issues.apache.org/jira/browse/BEAM-731
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
>
> Remove the old DirectRunner and replace with the new InProcessRunner.
> There is an overhead for keeping both runners (testing/code maintenance etc.) 
> InProcessRunner has been available for a while, it is tested enough for the 
> being the default runner. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1240: [BEAM-731] Replace DirectRunner with InPr...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/1240


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---