[jira] [Comment Edited] (BEAM-8837) PCollectionVisualizationTest: possible bug
[
https://issues.apache.org/jira/browse/BEAM-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986465#comment-16986465
]
Ning Kang edited comment on BEAM-8837 at 12/7/19 12:10 AM:
---
The *@patch* in unit tests didn't work as expected and the code executes the
real logic to read cached PCollection when there is *None* cache manager in the
test.
I can reproduce the error trace by running the gradle task
{code:java}
14:20:16 test_display_plain_text_when_kernel_has_no_frontend
(apache_beam.runners.interactive.display.pcoll_visualization_test.PCollectionVisualizationTest)
... Exception in thread Thread-3982:
14:20:16 Traceback (most recent call last):
14:20:16 File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
14:20:16 self.run()
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/timeloop/job.py",
line 19, in run
14:20:16 self.execute(*self.args, **self.kwargs)
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 132, in continuous_update_display
14:20:16 updated_pv.display_facets(updating_pv=pv)
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 209, in display_facets
14:20:16 data = self._to_dataframe()
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 278, in _to_dataframe
14:20:16 for el in self._to_element_list():
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 266, in _to_element_list
14:20:16 if ie.current_env().cache_manager().exists('full',
self._cache_key):
14:20:16 AttributeError: 'NoneType' object has no attribute 'exists'
14:20:16
14:20:16 ok{code}
I've tried locally run with *unittest* and *nosetests*, both passed without
running into the issue. I'm pretty sure it's a gotcha in [where to
patch|https://docs.python.org/3/library/unittest.mock.html#where-to-patch].
When the test is ran by tox + nosetests, the patch doesn't work.
was (Author: ningk):
The *@patch* in unit tests didn't work as expected and the code executes the
real logic to read cached PCollection when there is *None* cache manager in the
test.
[~udim], do you have a link to the failed Jenkins test? I'll need to figure
out what kind of test invocation path failed this *@patch*.
I've tried locally run with *unittest* and *nosetests*, both passed without
running into the issue. I'm pretty sure it's a gotcha in [where to
patch|https://docs.python.org/3/library/unittest.mock.html#where-to-patch].
> PCollectionVisualizationTest: possible bug
> --
>
> Key: BEAM-8837
> URL: https://issues.apache.org/jira/browse/BEAM-8837
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Ning Kang
>Priority: Major
>
> This seems like a bug, even though the test passes:
> {code}
> test_display_plain_text_when_kernel_has_no_frontend
> (apache_beam.runners.interactive.display.pcoll_visualization_test.PCollectionVisualizationTest)
> ... Exception in thread Thread-4405:
> Traceback (most recent call last):
> File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
> self.run()
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/.eggs/timeloop-1.0.2-py3.7.egg/timeloop/job.py",
> line 19, in run
> self.execute(*self.args, **self.kwargs)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 132, in continuous_update_display
> updated_pv.display_facets(updating_pv=pv)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 209, in display_facets
> data = self._to_dataframe()
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
> line 278, in _to_dataframe
> for el in
[jira] [Comment Edited] (BEAM-8837) PCollectionVisualizationTest: possible bug
[
https://issues.apache.org/jira/browse/BEAM-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986465#comment-16986465
]
Ning Kang edited comment on BEAM-8837 at 12/7/19 12:10 AM:
---
The *@patch* in unit tests didn't work as expected and the code executes the
real logic to read cached PCollection when there is *None* cache manager in the
test.
I can reproduce the error trace by running the gradle task
{code:java}
14:20:16 test_display_plain_text_when_kernel_has_no_frontend
(apache_beam.runners.interactive.display.pcoll_visualization_test.PCollectionVisualizationTest)
... Exception in thread Thread-3982:
14:20:16 Traceback (most recent call last):
14:20:16 File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
14:20:16 self.run()
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/timeloop/job.py",
line 19, in run
14:20:16 self.execute(*self.args, **self.kwargs)
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 132, in continuous_update_display
14:20:16 updated_pv.display_facets(updating_pv=pv)
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 209, in display_facets
14:20:16 data = self._to_dataframe()
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 278, in _to_dataframe
14:20:16 for el in self._to_element_list():
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 266, in _to_element_list
14:20:16 if ie.current_env().cache_manager().exists('full',
self._cache_key):
14:20:16 AttributeError: 'NoneType' object has no attribute 'exists'
14:20:16
14:20:16 ok{code}
I've tried locally run with *unittest* and *nosetests*, both passed without
running into the issue. I'm pretty sure it's a gotcha in [where to
patch|https://docs.python.org/3/library/unittest.mock.html#where-to-patch].
When the test is run by tox + nosetests, the patch doesn't work.
was (Author: ningk):
The *@patch* in unit tests didn't work as expected and the code executes the
real logic to read cached PCollection when there is *None* cache manager in the
test.
I can reproduce the error trace by running the gradle task
{code:java}
14:20:16 test_display_plain_text_when_kernel_has_no_frontend
(apache_beam.runners.interactive.display.pcoll_visualization_test.PCollectionVisualizationTest)
... Exception in thread Thread-3982:
14:20:16 Traceback (most recent call last):
14:20:16 File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
14:20:16 self.run()
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/timeloop/job.py",
line 19, in run
14:20:16 self.execute(*self.args, **self.kwargs)
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 132, in continuous_update_display
14:20:16 updated_pv.display_facets(updating_pv=pv)
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 209, in display_facets
14:20:16 data = self._to_dataframe()
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 278, in _to_dataframe
14:20:16 for el in self._to_element_list():
14:20:16 File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py",
line 266, in _to_element_list
14:20:16 if ie.current_env().cache_manager().exists('full',
self._cache_key):
14:20:16 AttributeError: 'NoneType' object has no attribute 'exists'
14:20:16
14:20:16 ok{code}
I've tried locally run with *unittest* and *nosetests*, both passed
