[jira] [Commented] (BEAM-12794) PortableRunnerTestWithExternalEnv.test_pardo_timers flaky
[ https://issues.apache.org/jira/browse/BEAM-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404495#comment-17404495 ] yoshiki obata commented on BEAM-12794: -- [~ibzib] Yes, that's true. I'll fix this ASAP. > PortableRunnerTestWithExternalEnv.test_pardo_timers flaky > - > > Key: BEAM-12794 > URL: https://issues.apache.org/jira/browse/BEAM-12794 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness, test-failures >Reporter: Kyle Weaver >Assignee: yoshiki obata >Priority: P1 > > In python precommit. > Looks like there's also a bug in error handling. IIUC, this call is intended > to be a "cast" but really it's calling an arbitrary constructor, which may > take different arguments than a > BaseException. > https://github.com/apache/beam/blob/4559c75863d9d6c9dd9e48c2b4f12f2139410524/sdks/python/apache_beam/runners/worker/sdk_worker.py#L1105 > cc[~tvalentyn] [~yoshiki.obata] > > > [https://ci-beam.apache.org/job/beam_PreCommit_Python_Phrase/2755/testReport/junit/apache_beam.runners.portability.portable_runner_test/PortableRunnerTestWithExternalEnv/test_pardo_timers/] > > E RuntimeError: Pipeline job-721c3984-7f9c-4991-8473-c7d8c02f4d13 failed in > state FAILED: Error running pipeline. E Traceback (most recent call last): E > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/local_job_service.py", > line 274, in _run_job E self._pipeline_proto) E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", > line 206, in run_via_runner_api E return self.run_stages(stage_context, > stages) E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", > line 385, in run_stages E runner_execution_context, bundle_context_manager) > E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", > line 653, in _run_stage E bundle_manager)) E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", > line 770, in _run_bundle E data_input, data_output, input_timers, > expected_timer_output) E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", > line 1120, in process_bundle E raise RuntimeError(result.error) E > RuntimeError: Traceback (most recent call last): E File > "apache_beam/runners/common.py", line 1258, in > apache_beam.runners.common.DoFnRunner.process_user_timer E > self.do_fn_invoker.invoke_user_timer( E File "apache_beam/runners/common.py", > line 528, in apache_beam.runners.common.DoFnInvoker.invoke_user_timer E > self.output_processor.process_outputs( E File > "apache_beam/runners/common.py", line 1368, in > apache_beam.runners.common._OutputProcessor.process_outputs E for result in > results: E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py", > line 366, in process_timer E if state.read() == 0: E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py", > line 481, in read E return > self._combinefn.extract_output(self._read_accumulator()) E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py", > line 473, in _read_accumulator E self._underlying_bag_state.read()) E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/transforms/core.py", > line 1038, in merge_accumulators E return > [self._fn(_ReiterableChain(accumulators), *args, **kwargs)] E File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/transforms/core.py", > line 976, in __iter__ E for iterable in self.iterables: E File > "/home/jenkins/jenkins-slave/workspace/bea
[jira] [Commented] (BEAM-8954) tox: use isolated builds (PEP 517 and 518)
[ https://issues.apache.org/jira/browse/BEAM-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388045#comment-17388045 ] yoshiki obata commented on BEAM-8954: - [~udim] Would anyone work on this? I'll work on this if no one do. > tox: use isolated builds (PEP 517 and 518) > -- > > Key: BEAM-8954 > URL: https://issues.apache.org/jira/browse/BEAM-8954 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Udi Meiri >Priority: P3 > > See description in: > https://github.com/apache/beam/pull/10038 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10921) Interactive Runner Python unit tests are flaky on Windows
[
https://issues.apache.org/jira/browse/BEAM-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319455#comment-17319455
]
yoshiki obata commented on BEAM-10921:
--
Similar errors found on another tests under runners/interactive, but seem to be
not so frequently.
https://github.com/apache/beam/pull/14496/checks?check_run_id=2305880730
> Interactive Runner Python unit tests are flaky on Windows
> -
>
> Key: BEAM-10921
> URL: https://issues.apache.org/jira/browse/BEAM-10921
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Sam Rohde
>Priority: P1
> Labels: currently-failing, flake
> Fix For: Not applicable
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> Over the past few days python unit tests have been failing frequently. The
> errors always seem to occur when cleaning up the interactive environment:
> {code}
> ...
> [100%]
> == FAILURES
> ===
> _
> PipelineInstrumentTest.test_able_to_cache_intermediate_unbounded_source_pcollection
> _
> [gw2] win32 -- Python 3.5.4
> d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe
> self =
> testMethod=test_able_to_cache_intermediate_unbounded_source_pcollection>
> def setUp(self):
> > ie.new_env()
> apache_beam\runners\interactive\pipeline_instrument_test.py:46:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam\runners\interactive\interactive_environment.py:117: in new_env
> _interactive_beam_env.cleanup()
> apache_beam\runners\interactive\interactive_environment.py:273: in cleanup
> cache_manager.cleanup()
> apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
> shutil.rmtree(self._cache_dir)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:494: in rmtree
> return _rmtree_unsafe(path, onerror)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:384: in
> _rmtree_unsafe
> _rmtree_unsafe(fullname, onerror)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:389: in
> _rmtree_unsafe
> onerror(os.unlink, fullname, sys.exc_info())
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> path =
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full'
> onerror = .onerror at 0x01D6C3E5C7B8>
> def _rmtree_unsafe(path, onerror):
> try:
> if os.path.islink(path):
> # symlinks to directories are forbidden, see bug #1669
> raise OSError("Cannot call rmtree on a symbolic link")
> except OSError:
> onerror(os.path.islink, path, sys.exc_info())
> # can't continue even if onerror hook returns
> return
> names = []
> try:
> names = os.listdir(path)
> except OSError:
> onerror(os.listdir, path, sys.exc_info())
> for name in names:
> fullname = os.path.join(path, name)
> try:
> mode = os.lstat(fullname).st_mode
> except OSError:
> mode = 0
> if stat.S_ISDIR(mode):
> _rmtree_unsafe(fullname, onerror)
> else:
> try:
> > os.unlink(fullname)
> E PermissionError: [WinError 32] The process cannot access
> the file because it is being used by another process:
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:387: PermissionError
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10625) tests fails on windows - PortableRunnerTestWithSubprocesses causes grpc Deadline Exceeded
[
https://issues.apache.org/jira/browse/BEAM-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316318#comment-17316318
]
yoshiki obata commented on BEAM-10625:
--
[~kenn] Though it's not exactly the same as this, I encountered a similar error
when running on GitHub Actions with ubuntu-latest, python 3.7.
https://github.com/apache/beam/pull/1/checks?check_run_id=2278023928
Seems to be the same cause.
> tests fails on windows - PortableRunnerTestWithSubprocesses causes grpc
> Deadline Exceeded
> -
>
> Key: BEAM-10625
> URL: https://issues.apache.org/jira/browse/BEAM-10625
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Tobiasz Kedzierski
>Priority: P3
> Attachments: BEAM-10625.txt
>
>
> apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses
> causes grpc Deadline Exceeded on windows
> Link to the github workflow run with mentioned error:
> [https://github.com/TobKed/beam/runs/937577885?check_suite_focus=true]
> partial log:
> FsFException in thread wait_until_finish_read:
> Traceback (most recent call last):
> File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\threading.py", line
> 916, in _bootstrap_inner
> self.run()
> File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\threading.py", line
> 864, in run
> self._target(*self._args, **self._kwargs)
> File
> "D:\a\beam\beam\sdks\python\apache_beam\runners\portability\portable_runner.py",
> line 512, in read_messages
> for message in self._message_stream:
> File
> "d:\a\beam\beam\sdks\python\target\.tox\py36-win\lib\site-packages\grpc\_channel.py",
> line 416, in __next__
> return self._next()
> File
> "d:\a\beam\beam\sdks\python\target\.tox\py36-win\lib\site-packages\grpc\_channel.py",
> line 706, in _next
> raise self
> grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
> terminated with:
> status = StatusCode.DEADLINE_EXCEEDED
> details = "Deadline Exceeded"
> debug_error_string =
> "\{"created":"@1596375957.85900","description":"Error received from peer
> ipv6:[::1]:50984","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Deadline
> Exceeded","grpc_status":4}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8288) Cleanup Interactive Beam Python 2 support
[ https://issues.apache.org/jira/browse/BEAM-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299519#comment-17299519 ] yoshiki obata commented on BEAM-8288: - [~ningk] Since [PR 14172|https://github.com/apache/beam/pull/14172] was merged, can this issue be closed? > Cleanup Interactive Beam Python 2 support > - > > Key: BEAM-8288 > URL: https://issues.apache.org/jira/browse/BEAM-8288 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Ning Kang >Priority: P3 > Time Spent: 1.5h > Remaining Estimate: 0h > > As Beam is retiring Python 2, some special handle in Interactive Beam code > and tests will need to be cleaned up. > This Jira ticket tracks those changes to be cleaned up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-11461) Update tox to the latest for the simple and flexible configuration
[ https://issues.apache.org/jira/browse/BEAM-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-11461: - Resolution: Fixed Status: Resolved (was: Triage Needed) > Update tox to the latest for the simple and flexible configuration > -- > > Key: BEAM-11461 > URL: https://issues.apache.org/jira/browse/BEAM-11461 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P4 > Time Spent: 1h 50m > Remaining Estimate: 0h > > Currently, we use tox 3.11.1 for Python tests. > Updating it to the latest, we can use "generative section name" which allows > us not to define simple section definition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-11649) NotImplementedError raised because function InMemoryCache.clear is not implemented
[ https://issues.apache.org/jira/browse/BEAM-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-11649: - Resolution: Duplicate Status: Resolved (was: Triage Needed) > NotImplementedError raised because function InMemoryCache.clear is not > implemented > -- > > Key: BEAM-11649 > URL: https://issues.apache.org/jira/browse/BEAM-11649 > Project: Beam > Issue Type: Bug > Components: testing >Affects Versions: 2.27.0 >Reporter: yoshiki obata >Priority: P3 > > apache_beam.runners.interactive.testing.test_cache_manager.InMemoryCache does > not implement clear function though it inherits > apache_beam.runners.interactive.cache_manager.CacheManager. > Therefore, NotImplementedError is raised when InMemoryCache.clear is called. > This bug is found in this test; > https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/16819/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-11649) NotImplementedError raised because function InMemoryCache.clear is not implemented
yoshiki obata created BEAM-11649: Summary: NotImplementedError raised because function InMemoryCache.clear is not implemented Key: BEAM-11649 URL: https://issues.apache.org/jira/browse/BEAM-11649 Project: Beam Issue Type: Bug Components: testing Affects Versions: 2.27.0 Reporter: yoshiki obata apache_beam.runners.interactive.testing.test_cache_manager.InMemoryCache does not implement clear function though it inherits apache_beam.runners.interactive.cache_manager.CacheManager. Therefore, NotImplementedError is raised when InMemoryCache.clear is called. This bug is found in this test; https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/16819/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-11461) Update tox to the latest for the simple and flexible configuration
[ https://issues.apache.org/jira/browse/BEAM-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-11461: Assignee: yoshiki obata > Update tox to the latest for the simple and flexible configuration > -- > > Key: BEAM-11461 > URL: https://issues.apache.org/jira/browse/BEAM-11461 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P4 > > Currently, we use tox 3.11.1 for Python tests. > Updating it to the latest, we can use "generative section name" which allows > us not to define simple section definition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-11461) Update tox to the latest for the simple and flexible configuration
[ https://issues.apache.org/jira/browse/BEAM-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-11461: - Description: Currently, we use tox 3.11.1 for Python tests. Updating it to the latest, we can use "generative section name" which allows us not to define simple section definition. > Update tox to the latest for the simple and flexible configuration > -- > > Key: BEAM-11461 > URL: https://issues.apache.org/jira/browse/BEAM-11461 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: yoshiki obata >Priority: P4 > > Currently, we use tox 3.11.1 for Python tests. > Updating it to the latest, we can use "generative section name" which allows > us not to define simple section definition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-11461) Update tox to the latest for the simple and flexible configuration
yoshiki obata created BEAM-11461: Summary: Update tox to the latest for the simple and flexible configuration Key: BEAM-11461 URL: https://issues.apache.org/jira/browse/BEAM-11461 Project: Beam Issue Type: Improvement Components: sdk-py-core Reporter: yoshiki obata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Priority: P4 (was: P2) > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: P4 > Time Spent: 2h 20m > Remaining Estimate: 0h > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Labels: (was: stale-assigned) > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Time Spent: 2h 20m > Remaining Estimate: 0h > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204723#comment-17204723 ] yoshiki obata commented on BEAM-9944: - Currently, investigating how to change tests and writing a document about how to change test architecture including GitHub Actions. > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Labels: stale-assigned > Time Spent: 2h 20m > Remaining Estimate: 0h > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Labels: (was: stale-assigned) > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Time Spent: 2h 20m > Remaining Estimate: 0h > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10860) RuntimeError is potentially raised when BundleProcessorCache.shutdown called
[
https://issues.apache.org/jira/browse/BEAM-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata reassigned BEAM-10860:
Assignee: yoshiki obata
> RuntimeError is potentially raised when BundleProcessorCache.shutdown called
>
>
> Key: BEAM-10860
> URL: https://issues.apache.org/jira/browse/BEAM-10860
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: P2
>
> Following codes in BundleProcessorCache.shutdown should raise RuntimeError:
> dictionary changed size during iteration is raised.
> {code:python}
> for instruction_id in self.active_bundle_processors:
> self.active_bundle_processors[instruction_id][1].shutdown()
> del self.active_bundle_processors[instruction_id]{code}
> If active_bundle_processors is empty, these codes works fine.
> I found this in following GitHub Actions;
> [https://github.com/apache/beam/actions/runs/234476619]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-10860) RuntimeError is potentially raised when BundleProcessorCache.shutdown called
yoshiki obata created BEAM-10860:
Summary: RuntimeError is potentially raised when
BundleProcessorCache.shutdown called
Key: BEAM-10860
URL: https://issues.apache.org/jira/browse/BEAM-10860
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: yoshiki obata
Following codes in BundleProcessorCache.shutdown should raise RuntimeError:
dictionary changed size during iteration is raised.
{code:python}
for instruction_id in self.active_bundle_processors:
self.active_bundle_processors[instruction_id][1].shutdown()
del self.active_bundle_processors[instruction_id]{code}
If active_bundle_processors is empty, these codes works fine.
I found this in following GitHub Actions;
[https://github.com/apache/beam/actions/runs/234476619]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9944: --- Assignee: yoshiki obata > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Time Spent: 1h 10m > Remaining Estimate: 0h > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9943) Priotize Python versions for PreCommit/PostCommit tests
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161290#comment-17161290 ] yoshiki obata commented on BEAM-9943: - Update codes locally according to BEAM-9980 > Priotize Python versions for PreCommit/PostCommit tests > > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > - Define python version properties in the gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > version properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9943) Priotize Python versions for PreCommit/PostCommit tests
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9943: Labels: (was: stale-assigned) > Priotize Python versions for PreCommit/PostCommit tests > > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > - Define python version properties in the gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > version properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7672) Accept Py3 wheels in SDK harness container.
[ https://issues.apache.org/jira/browse/BEAM-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata resolved BEAM-7672. - Fix Version/s: Not applicable Resolution: Fixed > Accept Py3 wheels in SDK harness container. > --- > > Key: BEAM-7672 > URL: https://issues.apache.org/jira/browse/BEAM-7672 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: P3 > Fix For: Not applicable > > Time Spent: 6h > Remaining Estimate: 0h > > Currently we only accept Py27 wheels: > https://github.com/apache/beam/blob/05900a43de1f380f124c688cd8d804b5133bda64/sdks/python/container/boot.go#L39 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9980: Labels: (was: stale-assigned) > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Time Spent: 10m > Remaining Estimate: 0h > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Labels: (was: stale-assigned) > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156859#comment-17156859 ] yoshiki obata commented on BEAM-9980: - PR created. > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Labels: stale-assigned > Time Spent: 10m > Remaining Estimate: 0h > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9944 started by yoshiki obata. --- > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Labels: stale-assigned > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9829: --- Assignee: yoshiki obata > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9943) Priotize Python versions for PreCommit/PostCommit tests
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9943: --- Assignee: yoshiki obata > Priotize Python versions for PreCommit/PostCommit tests > > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > - Define python version properties in the gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > version properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9785) Add PostCommit suite for Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138593#comment-17138593 ] yoshiki obata commented on BEAM-9785: - [~epicfaace] This issue was made unassigned by Bot though PR is open. Feel free to self-assign if you are now working on this. > Add PostCommit suite for Python 3.8 > --- > > Key: BEAM-9785 > URL: https://issues.apache.org/jira/browse/BEAM-9785 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: P2 > Time Spent: 3.5h > Remaining Estimate: 0h > > Add PostCommit suites for Python 3.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Labels: (was: stale-assigned) > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7672) Accept Py3 wheels in SDK harness container.
[ https://issues.apache.org/jira/browse/BEAM-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-7672: --- Assignee: yoshiki obata > Accept Py3 wheels in SDK harness container. > --- > > Key: BEAM-7672 > URL: https://issues.apache.org/jira/browse/BEAM-7672 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: P3 > > Currently we only accept Py27 wheels: > https://github.com/apache/beam/blob/05900a43de1f380f124c688cd8d804b5133bda64/sdks/python/container/boot.go#L39 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7672) Accept Py3 wheels in SDK harness container.
[ https://issues.apache.org/jira/browse/BEAM-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128341#comment-17128341 ] yoshiki obata commented on BEAM-7672: - [~tvalentyn] I'll work on this if no one does. > Accept Py3 wheels in SDK harness container. > --- > > Key: BEAM-7672 > URL: https://issues.apache.org/jira/browse/BEAM-7672 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Valentyn Tymofieiev >Priority: P3 > > Currently we only accept Py27 wheels: > https://github.com/apache/beam/blob/05900a43de1f380f124c688cd8d804b5133bda64/sdks/python/container/boot.go#L39 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9829: Labels: (was: stale-assigned) > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9829 started by yoshiki obata. --- > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > Labels: stale-assigned > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9785) Add PostCommit suite for Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9785: --- Assignee: Ashwin Ramaswami > Add PostCommit suite for Python 3.8 > --- > > Key: BEAM-9785 > URL: https://issues.apache.org/jira/browse/BEAM-9785 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: Ashwin Ramaswami >Priority: P2 > Time Spent: 50m > Remaining Estimate: 0h > > Add PostCommit suites for Python 3.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17114125#comment-17114125 ] yoshiki obata edited comment on BEAM-9980 at 5/25/20, 3:18 PM: --- BEAM-9810 several tox tasks would be moved to py38/build.gradle was (Author: yoshiki.obata): several tox tasks would be moved to py38/build.gradle > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9944: --- Assignee: yoshiki obata > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17114125#comment-17114125 ] yoshiki obata commented on BEAM-9980: - several tox tasks would be moved to py38/build.gradle > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112459#comment-17112459 ] yoshiki obata edited comment on BEAM-9980 at 5/20/20, 5:22 PM: --- BEAM-7774 will add new task for dataflow test-suites. was (Author: yoshiki.obata): BEAM-7774 add new task for dataflow test-suites. > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112459#comment-17112459 ] yoshiki obata commented on BEAM-9980: - BEAM-7774 add new task for dataflow test-suites. > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9980 started by yoshiki obata. --- > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9980: --- Assignee: yoshiki obata > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9754) Dataflow runner currently does not support Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata reassigned BEAM-9754:
---
Assignee: (was: yoshiki obata)
> Dataflow runner currently does not support Python 3.8
> -
>
> Key: BEAM-9754
> URL: https://issues.apache.org/jira/browse/BEAM-9754
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Priority: Major
>
> Following tests fail due to the dataflow runner not supporting Python 3.8:
> - test_default_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_experiment_use_multiple_sdk_containers
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_blank
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_cost
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_speed
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_absent_in_released_sdks_with_runner_v2
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_absent_in_unreleased_sdk
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_custom_in_released_sdks
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_custom_in_released_sdks_with_runner_v2
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_default_in_released_sdks
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_uses_base_version_in_rc_releases
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_number_of_worker_harness_threads
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_pinned_worker_harness_image_tag_used_in_dev_sdk
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_pipeline_url
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_private_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_public_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_sdk_harness_container_images_get_set
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_set_network
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_set_subnetwork
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_image_tag_matches_base_sdk_version_of_an_rc
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_image_tag_matches_released_sdk_version
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_override_takes_precedence_over_sdk_defaults
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_biqquery_read_streaming_fail
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_dataflow_worker_jar_flag_adds_use_staged_worker_jar_experiment
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_dataflow_worker_jar_flag_non_fnapi_noop
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_min_cpu_platform_flag_is_propagated_to_experiments
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_remote_runner_translation
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_streaming_create_translation
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_streaming_engine_flag_adds_windmill_experiments
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_unsupported_fnapi_features
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_upload_graph_experiment
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_use_fastavro_experiment_is_added_on_py3_and_onwards
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_use_fastavro_experiment_is_not_added_when_use_avro_is_present
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_bad_path
> (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest)
> - test_full_completion
> (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest)
> Sample error message:
> {code}
> ==
> ERROR: test_default_ip_configuration
> (apache_beam.runners.datafl
[jira] [Assigned] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9829: --- Assignee: (was: yoshiki obata) > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9829: --- Assignee: yoshiki obata > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9754) Dataflow runner currently does not support Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107439#comment-17107439 ] yoshiki obata commented on BEAM-9754: - [~kamilwu] made dataflow tests skipped at python 3.8 in [PR 11707|https://github.com/apache/beam/pull/11707] > Dataflow runner currently does not support Python 3.8 > - > > Key: BEAM-9754 > URL: https://issues.apache.org/jira/browse/BEAM-9754 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > Following tests fail due to the dataflow runner not supporting Python 3.8: > - test_default_ip_configuration > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_experiment_use_multiple_sdk_containers > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_flexrs_blank > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_flexrs_cost > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_flexrs_speed > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_harness_override_absent_in_released_sdks_with_runner_v2 > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_harness_override_absent_in_unreleased_sdk > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_harness_override_custom_in_released_sdks > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_harness_override_custom_in_released_sdks_with_runner_v2 > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_harness_override_default_in_released_sdks > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_harness_override_uses_base_version_in_rc_releases > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_number_of_worker_harness_threads > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_pinned_worker_harness_image_tag_used_in_dev_sdk > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_pipeline_url > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_private_ip_configuration > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_public_ip_configuration > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_sdk_harness_container_images_get_set > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_set_network > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_set_subnetwork > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_worker_harness_image_tag_matches_base_sdk_version_of_an_rc > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_worker_harness_image_tag_matches_released_sdk_version > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_worker_harness_override_takes_precedence_over_sdk_defaults > (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest) > - test_biqquery_read_streaming_fail > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_dataflow_worker_jar_flag_adds_use_staged_worker_jar_experiment > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_dataflow_worker_jar_flag_non_fnapi_noop > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_min_cpu_platform_flag_is_propagated_to_experiments > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_remote_runner_display_data > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_remote_runner_translation > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_streaming_create_translation > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_streaming_engine_flag_adds_windmill_experiments > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_unsupported_fnapi_features > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_upload_graph_experiment > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_use_fastavro_experiment_is_added_on_py3_and_onwards > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_use_fastavro_experiment_is_not_added_when_use_avro_is_present > (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) > - test_bad_path > (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest) > - test_full_completion > (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest) > Sample error m
[jira] [Closed] (BEAM-9942) Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3*
[ https://issues.apache.org/jira/browse/BEAM-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata closed BEAM-9942. --- Fix Version/s: Not applicable Resolution: Resolved > Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3* > - > > Key: BEAM-9942 > URL: https://issues.apache.org/jira/browse/BEAM-9942 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8494) Python 3.8 Support
[ https://issues.apache.org/jira/browse/BEAM-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106315#comment-17106315 ] yoshiki obata commented on BEAM-8494: - [~piotr-szuberski] [~kamilwu] test-infra task are listed as sub-tasks of BEAM-9829 ( and I'm working on few of them) Feel free to self-assign to them and sub-tasks of this issue. > Python 3.8 Support > -- > > Key: BEAM-8494 > URL: https://issues.apache.org/jira/browse/BEAM-8494 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9943) Priotize Python versions for PreCommit/PostCommit tests
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9943: Summary: Priotize Python versions for PreCommit/PostCommit tests (was: Make it easy to switch Python versions for testing) > Priotize Python versions for PreCommit/PostCommit tests > > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > - Define python version properties in the gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > version properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9943) Make it easy to switch Python versions for testing
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9943: Description: - Define python version properties in the gradle.properties - PreCommit/PostCommit Tasks are to be defined dynamically according to the version properties. was: - Define python version properties in the gradle.properties - PreCommit/PostCommit Tasks are to be defined dynamically according to the version properties. - Version tied tests like FlinkValidatesRunner are also to be defined dynamically to be enabled to switch version easily. > Make it easy to switch Python versions for testing > -- > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > - Define python version properties in the gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > version properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9980: Status: Open (was: Triage Needed) > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9980: Description: We have some tests that we run only under specific Python versions, - FlinkValidatesRunner test runs using Python 3.5 - HDFS Python 3 tests are running only with Python 3.7 - Cross-language Py3 tests for Spark are running under Python 3.5 etc... Python versions to run them under need to be switchable. was: We have some tests that we run only under specific Python versions, - FlinkValidatesRunner test runs using Python 3.5 - HDFS Python 3 tests are running only with Python 3.7 - Cross-language Py3 tests for Spark are running under Python 3.5 Python versions to run them under need to be switchable. > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9980) Specific Python version tied tests are to be version-selective
yoshiki obata created BEAM-9980: --- Summary: Specific Python version tied tests are to be version-selective Key: BEAM-9980 URL: https://issues.apache.org/jira/browse/BEAM-9980 Project: Beam Issue Type: Sub-task Components: sdk-py-core Reporter: yoshiki obata We have some tests that we run only under specific Python versions, - FlinkValidatesRunner test runs using Python 3.5 - HDFS Python 3 tests are running only with Python 3.7 - Cross-language Py3 tests for Spark are running under Python 3.5 Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Description: To reduce resource usage for PreCommit tests, these needs are to be met. - For low-priority Python, tests only in the typehints package are run when PreCommit test called. And rest of them are moved to the PostCommit one. - For high-priority Python, PreCommit and PostCommit tests are AS IS. - Assured that we can add more tests easily later if there is a need. e.g. using filepath to identify which tests to run was: To reduce resource usage for PreCommit tests, these needs are to be met. - For low-priority Python, tests only in the typehints package are run when PreCommit test called. And rest of them are moved to the PostCommit one. - For high-priority Python, PreCommit and PostCommit tests are AS IS. - Assured that we can add more tests easily later if there is a need. > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. > e.g. using filepath to identify which tests to run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9943) Make it easy to switch Python versions for testing
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9943: Description: - Define python version properties in the gradle.properties - PreCommit/PostCommit Tasks are to be defined dynamically according to the version properties. - Version tied tests like FlinkValidatesRunner are also to be defined dynamically to be enabled to switch version easily. was: - Define high-priority/low-priority python version properties in the gradle.properties - PreCommit/PostCommit Tasks are to be defined dynamically according to the properties. - Version specified tests like FlinkValidatesRunner are also to be defined dynamically to be enabled to switch version easily. > Make it easy to switch Python versions for testing > -- > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > - Define python version properties in the gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > version properties. > - Version tied tests like FlinkValidatesRunner are also to be defined > dynamically to be enabled to switch version easily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Description: To reduce resource usage for PreCommit tests, these needs are to be met. - For low-priority Python, tests only in the typehints package are run when PreCommit test called. And rest of them are moved to the PostCommit one. - For high-priority Python, PreCommit and PostCommit tests are AS IS. - Assured that we can add more tests easily later if there is a need. was: To reduce resource usage for PreCommit tests, - Tests only in the typehints package are run when PreCommit test called. And rest of them are moved to the PostCommit one. - Assured that we can add more tests easily later if there is a need. > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > To reduce resource usage for PreCommit tests, these needs are to be met. > - For low-priority Python, tests only in the typehints package are run when > PreCommit test called. And rest of them are moved to the PostCommit one. > - For high-priority Python, PreCommit and PostCommit tests are AS IS. > - Assured that we can add more tests easily later if there is a need. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9943) Make it easy to switch Python versions for testing
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9943 started by yoshiki obata. --- > Make it easy to switch Python versions for testing > -- > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > - Define high-priority/low-priority python version properties in the > gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > properties. > - Version specified tests like FlinkValidatesRunner are also to be defined > dynamically to be enabled to switch version easily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Status: Open (was: Triage Needed) > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > To reduce resource usage for PreCommit tests, > - Tests only in the typehints package are run when PreCommit test called. And > rest of them are moved to the PostCommit one. > - Assured that we can add more tests easily later if there is a need. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9942) Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3*
[ https://issues.apache.org/jira/browse/BEAM-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9942 started by yoshiki obata. --- > Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3* > - > > Key: BEAM-9942 > URL: https://issues.apache.org/jira/browse/BEAM-9942 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9942) Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3*
[ https://issues.apache.org/jira/browse/BEAM-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9942: --- Assignee: yoshiki obata > Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3* > - > > Key: BEAM-9942 > URL: https://issues.apache.org/jira/browse/BEAM-9942 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9942) Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3*
[ https://issues.apache.org/jira/browse/BEAM-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9942: Status: Open (was: Triage Needed) > Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3* > - > > Key: BEAM-9942 > URL: https://issues.apache.org/jira/browse/BEAM-9942 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9943) Make it easy to switch Python versions for testing
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9943: --- Assignee: yoshiki obata > Make it easy to switch Python versions for testing > -- > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > - Define high-priority/low-priority python version properties in the > gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > properties. > - Version specified tests like FlinkValidatesRunner are also to be defined > dynamically to be enabled to switch version easily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9943) Make it easy to switch Python versions for testing
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9943: Status: Open (was: Triage Needed) > Make it easy to switch Python versions for testing > -- > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > - Define high-priority/low-priority python version properties in the > gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > properties. > - Version specified tests like FlinkValidatesRunner are also to be defined > dynamically to be enabled to switch version easily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9943) Make it easy to switch Python versions for testing
[ https://issues.apache.org/jira/browse/BEAM-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9943: Description: - Define high-priority/low-priority python version properties in the gradle.properties - PreCommit/PostCommit Tasks are to be defined dynamically according to the properties. - Version specified tests like FlinkValidatesRunner are also to be defined dynamically to be enabled to switch version easily. > Make it easy to switch Python versions for testing > -- > > Key: BEAM-9943 > URL: https://issues.apache.org/jira/browse/BEAM-9943 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > - Define high-priority/low-priority python version properties in the > gradle.properties > - PreCommit/PostCommit Tasks are to be defined dynamically according to the > properties. > - Version specified tests like FlinkValidatesRunner are also to be defined > dynamically to be enabled to switch version easily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
[ https://issues.apache.org/jira/browse/BEAM-9944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9944: Description: To reduce resource usage for PreCommit tests, - Tests only in the typehints package are run when PreCommit test called. And rest of them are moved to the PostCommit one. - Assured that we can add more tests easily later if there is a need. > Reduce PreCommit test to the minimum and move rest of them to PostCommit one > > > Key: BEAM-9944 > URL: https://issues.apache.org/jira/browse/BEAM-9944 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > To reduce resource usage for PreCommit tests, > - Tests only in the typehints package are run when PreCommit test called. And > rest of them are moved to the PostCommit one. > - Assured that we can add more tests easily later if there is a need. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9944) Reduce PreCommit test to the minimum and move rest of them to PostCommit one
yoshiki obata created BEAM-9944: --- Summary: Reduce PreCommit test to the minimum and move rest of them to PostCommit one Key: BEAM-9944 URL: https://issues.apache.org/jira/browse/BEAM-9944 Project: Beam Issue Type: Sub-task Components: sdk-py-core Reporter: yoshiki obata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9943) Make it easy to switch Python versions for testing
yoshiki obata created BEAM-9943: --- Summary: Make it easy to switch Python versions for testing Key: BEAM-9943 URL: https://issues.apache.org/jira/browse/BEAM-9943 Project: Beam Issue Type: Sub-task Components: sdk-py-core Reporter: yoshiki obata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9942) Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3*
yoshiki obata created BEAM-9942: --- Summary: Reduce the code duplication across beam/sdks/python/test-suites/$runner/py3* Key: BEAM-9942 URL: https://issues.apache.org/jira/browse/BEAM-9942 Project: Beam Issue Type: Sub-task Components: sdk-py-core Reporter: yoshiki obata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8494) Python 3.8 Support
[ https://issues.apache.org/jira/browse/BEAM-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102721#comment-17102721 ] yoshiki obata commented on BEAM-8494: - Thank you [~kamilwu]. I'll update gradle tasks in this weekend so that we can run py38 precommit tests via gradle. > Python 3.8 Support > -- > > Key: BEAM-8494 > URL: https://issues.apache.org/jira/browse/BEAM-8494 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8494) Python 3.8 Support
[ https://issues.apache.org/jira/browse/BEAM-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099659#comment-17099659 ] yoshiki obata commented on BEAM-8494: - Thank you for kind offer, [~kamilwu] ! Once working out the detail about how to update test infrastructure, the tasks could be divided and ask you and [~piotr-szuberski] for help. > Python 3.8 Support > -- > > Key: BEAM-8494 > URL: https://issues.apache.org/jira/browse/BEAM-8494 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9829: Description: We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions and tests to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] was: We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions and tests to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] And this issue is closely related to Python 3.8 support BEAM-8494 > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9829: --- Assignee: yoshiki obata > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] > And this issue is closely related to Python 3.8 support BEAM-8494 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9829: Description: We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions and tests to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] And this issue is closely related to Python 3.8 support BEAM-8494 was: We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E And this issue is closely related to Python 3.8 support BEAM-8494 > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions and tests to be tested frequently, and > vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > > [https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E] > And this issue is closely related to Python 3.8 support BEAM-8494 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9163) pydoc: Update sphinx_rtd_theme
[ https://issues.apache.org/jira/browse/BEAM-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata resolved BEAM-9163. - Fix Version/s: 2.19.0 Resolution: Fixed > pydoc: Update sphinx_rtd_theme > -- > > Key: BEAM-9163 > URL: https://issues.apache.org/jira/browse/BEAM-9163 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Udi Meiri >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.19.0 > > Time Spent: 50m > Remaining Estimate: 0h > > [~yoshiki.obata] > 0.4.3 is the latest: https://pypi.org/project/sphinx-rtd-theme/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9829: Description: We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E And this issue is closely related to Python 3.8 support BEAM-8494 was: We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E And this issue is closely related to Python 3.8 support > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions to be tested frequently, and vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E > And this issue is closely related to Python 3.8 support BEAM-8494 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9829) Restructure test infrastructure for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9829: Summary: Restructure test infrastructure for Python SDK (was: Restruct test infrastructure for Python SDK) > Restructure test infrastructure for Python SDK > -- > > Key: BEAM-9829 > URL: https://issues.apache.org/jira/browse/BEAM-9829 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: yoshiki obata >Priority: Major > > We need to support newer Python versions like 3.8, 3.9 continuously. > But maintaining test for all versions costs us much in point of development > and test resources. > So we should reconsider test infrastructure about following points; > - how to prioritize Python versions to be tested frequently, and vice versa > - how to switch easily which versions to have high priority > detail discussion is here: > https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E > And this issue is closely related to Python 3.8 support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9829) Restruct test infrastructure for Python SDK
yoshiki obata created BEAM-9829: --- Summary: Restruct test infrastructure for Python SDK Key: BEAM-9829 URL: https://issues.apache.org/jira/browse/BEAM-9829 Project: Beam Issue Type: Task Components: sdk-py-core, testing Reporter: yoshiki obata We need to support newer Python versions like 3.8, 3.9 continuously. But maintaining test for all versions costs us much in point of development and test resources. So we should reconsider test infrastructure about following points; - how to prioritize Python versions to be tested frequently, and vice versa - how to switch easily which versions to have high priority detail discussion is here: https://lists.apache.org/thread.html/rd070afcebff5c967ec3b25d1f7a77db5278992c1508082bf5f636acd%40%3Cdev.beam.apache.org%3E And this issue is closely related to Python 3.8 support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata resolved BEAM-9701. - Fix Version/s: Not applicable Resolution: Fixed > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > Fix For: Not applicable > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093591#comment-17093591 ] yoshiki obata commented on BEAM-9701: - PR changed: https://github.com/apache/beam/pull/11506 > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata resolved BEAM-9755.
-
Fix Version/s: Not applicable
Resolution: Fixed
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9717) python-future cause ImportError on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata resolved BEAM-9717. - Fix Version/s: Not applicable Resolution: Fixed > python-future cause ImportError on Python 3.8 > - > > Key: BEAM-9717 > URL: https://issues.apache.org/jira/browse/BEAM-9717 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > Fix For: Not applicable > > Time Spent: 0.5h > Remaining Estimate: 0h > > python-future 0.16.0 installed on apache-beam 2.19.0 cause ImportError when > used in Python 3.8 [1], that makes many tests FAIL. > This problem is already resolved in newer version, we must update it to the > latest. > [1] https://github.com/PythonCharmers/python-future/issues/447 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089844#comment-17089844 ] yoshiki obata commented on BEAM-9701: - This issue is ongoing here: https://github.com/apache/beam/pull/11444 > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9717) python-future cause ImportError on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9717 started by yoshiki obata. --- > python-future cause ImportError on Python 3.8 > - > > Key: BEAM-9717 > URL: https://issues.apache.org/jira/browse/BEAM-9717 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > python-future 0.16.0 installed on apache-beam 2.19.0 cause ImportError when > used in Python 3.8 [1], that makes many tests FAIL. > This problem is already resolved in newer version, we must update it to the > latest. > [1] https://github.com/PythonCharmers/python-future/issues/447 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9619) Install Python 3.8 on Jenkins workers
[ https://issues.apache.org/jira/browse/BEAM-9619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089843#comment-17089843 ] yoshiki obata commented on BEAM-9619: - [~tvalentyn] I heard that python 3.8 is installed on Jenkins from your message; https://lists.apache.org/thread.html/reea198effe021fa51d2574383c90eedd52b5fbcc2cc1d2f7ebb3efc0%40%3Cdev.beam.apache.org%3E Could this issue be closed? > Install Python 3.8 on Jenkins workers > - > > Key: BEAM-9619 > URL: https://issues.apache.org/jira/browse/BEAM-9619 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Valentyn Tymofieiev >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-9755 started by yoshiki obata.
---
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8494) Python 3.8 Support
[ https://issues.apache.org/jira/browse/BEAM-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089147#comment-17089147 ] yoshiki obata commented on BEAM-8494: - Thank you much [~piotr-szuberski]! Currently, opened subtasks are mostly done(not yet committed and pushed). > Python 3.8 Support > -- > > Key: BEAM-8494 > URL: https://issues.apache.org/jira/browse/BEAM-8494 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089129#comment-17089129
]
yoshiki obata edited comment on BEAM-9755 at 4/21/20, 11:44 PM:
Thank you [~tvalentyn] and [~udim]!
I found that cause of this seems to be MAP_ADD's stack order expectation
change(PEP 572) yesterday.
We can fix this with changing key and value pop order at
apache_beam.typehints.opcodes.mapp_add according to Python version.
was (Author: yoshiki.obata):
Thank you [~tvalentyn] and [~udim]!
I found that cause of this seems to be MAP_ADD's stack order expectation
change(PEP 572).
We can fix this with changing key and value pop order at
apache_beam.typehints.opcodes.mapp_add according to Python version.
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089129#comment-17089129
]
yoshiki obata commented on BEAM-9755:
-
Thank you [~tvalentyn] and [~udim]!
I found that cause of this seems to be MAP_ADD's stack order expectation
change(PEP 572).
We can fix this with changing key and value pop order at
apache_beam.typehints.opcodes.mapp_add according to Python version.
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9785) Add PostCommit suite for Python 3.8
yoshiki obata created BEAM-9785: --- Summary: Add PostCommit suite for Python 3.8 Key: BEAM-9785 URL: https://issues.apache.org/jira/browse/BEAM-9785 Project: Beam Issue Type: Sub-task Components: sdk-py-core Reporter: yoshiki obata Add PostCommit suites for Python 3.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9717) python-future cause ImportError on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9717: Status: Open (was: Triage Needed) > python-future cause ImportError on Python 3.8 > - > > Key: BEAM-9717 > URL: https://issues.apache.org/jira/browse/BEAM-9717 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > python-future 0.16.0 installed on apache-beam 2.19.0 cause ImportError when > used in Python 3.8 [1], that makes many tests FAIL. > This problem is already resolved in newer version, we must update it to the > latest. > [1] https://github.com/PythonCharmers/python-future/issues/447 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9701: Status: Open (was: Triage Needed) > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9717) python-future cause ImportError on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9717: --- Assignee: yoshiki obata > python-future cause ImportError on Python 3.8 > - > > Key: BEAM-9717 > URL: https://issues.apache.org/jira/browse/BEAM-9717 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > python-future 0.16.0 installed on apache-beam 2.19.0 cause ImportError when > used in Python 3.8 [1], that makes many tests FAIL. > This problem is already resolved in newer version, we must update it to the > latest. > [1] https://github.com/PythonCharmers/python-future/issues/447 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9701: --- Assignee: yoshiki obata > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: Major > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata updated BEAM-9755:
Status: Open (was: Triage Needed)
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9754) Dataflow runner currently does not support Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata updated BEAM-9754:
Status: Open (was: Triage Needed)
> Dataflow runner currently does not support Python 3.8
> -
>
> Key: BEAM-9754
> URL: https://issues.apache.org/jira/browse/BEAM-9754
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> Following tests fail due to the dataflow runner not supporting Python 3.8:
> - test_default_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_experiment_use_multiple_sdk_containers
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_blank
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_cost
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_speed
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_absent_in_released_sdks_with_runner_v2
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_absent_in_unreleased_sdk
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_custom_in_released_sdks
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_custom_in_released_sdks_with_runner_v2
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_default_in_released_sdks
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_uses_base_version_in_rc_releases
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_number_of_worker_harness_threads
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_pinned_worker_harness_image_tag_used_in_dev_sdk
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_pipeline_url
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_private_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_public_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_sdk_harness_container_images_get_set
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_set_network
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_set_subnetwork
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_image_tag_matches_base_sdk_version_of_an_rc
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_image_tag_matches_released_sdk_version
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_override_takes_precedence_over_sdk_defaults
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_biqquery_read_streaming_fail
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_dataflow_worker_jar_flag_adds_use_staged_worker_jar_experiment
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_dataflow_worker_jar_flag_non_fnapi_noop
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_min_cpu_platform_flag_is_propagated_to_experiments
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_remote_runner_translation
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_streaming_create_translation
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_streaming_engine_flag_adds_windmill_experiments
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_unsupported_fnapi_features
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_upload_graph_experiment
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_use_fastavro_experiment_is_added_on_py3_and_onwards
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_use_fastavro_experiment_is_not_added_when_use_avro_is_present
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_bad_path
> (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest)
> - test_full_completion
> (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest)
> Sample error message:
> {code}
> ==
> ERROR: test_default_ip_configuration
[jira] [Assigned] (BEAM-9754) Dataflow runner currently does not support Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata reassigned BEAM-9754:
---
Assignee: yoshiki obata
> Dataflow runner currently does not support Python 3.8
> -
>
> Key: BEAM-9754
> URL: https://issues.apache.org/jira/browse/BEAM-9754
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> Following tests fail due to the dataflow runner not supporting Python 3.8:
> - test_default_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_experiment_use_multiple_sdk_containers
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_blank
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_cost
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_flexrs_speed
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_absent_in_released_sdks_with_runner_v2
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_absent_in_unreleased_sdk
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_custom_in_released_sdks
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_custom_in_released_sdks_with_runner_v2
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_default_in_released_sdks
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_harness_override_uses_base_version_in_rc_releases
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_number_of_worker_harness_threads
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_pinned_worker_harness_image_tag_used_in_dev_sdk
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_pipeline_url
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_private_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_public_ip_configuration
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_sdk_harness_container_images_get_set
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_set_network
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_set_subnetwork
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_image_tag_matches_base_sdk_version_of_an_rc
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_image_tag_matches_released_sdk_version
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_worker_harness_override_takes_precedence_over_sdk_defaults
> (apache_beam.runners.dataflow.internal.apiclient_test.UtilTest)
> - test_biqquery_read_streaming_fail
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_dataflow_worker_jar_flag_adds_use_staged_worker_jar_experiment
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_dataflow_worker_jar_flag_non_fnapi_noop
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_min_cpu_platform_flag_is_propagated_to_experiments
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_remote_runner_translation
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_streaming_create_translation
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_streaming_engine_flag_adds_windmill_experiments
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_unsupported_fnapi_features
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_upload_graph_experiment
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_use_fastavro_experiment_is_added_on_py3_and_onwards
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_use_fastavro_experiment_is_not_added_when_use_avro_is_present
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest)
> - test_bad_path
> (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest)
> - test_full_completion
> (apache_beam.runners.dataflow.template_runner_test.TemplatingDataflowRunnerTest)
> Sample error message:
> {code}
> ==
> ERROR: test_default_ip_configuration
> (
[jira] [Assigned] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata reassigned BEAM-9755:
---
Assignee: yoshiki obata
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yoshiki obata updated BEAM-9755:
Description:
apache_beam.typehints.trivial_inference.infer_return_type does not return type
as expected in Python 3.8.
And this cause testDictComprehensionSimple
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
- Python 3.7.7
{code:python}
>> from apache_beam.typehints.trivial_inference import infer_return_type
>> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
>> depth=5)
Dict[str, int]{code}
- Python 3.8.2
{code:python}
>> from apache_beam.typehints.trivial_inference import infer_return_type
>> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
>> depth=5)
Dict[int, str]{code}
CPython bytecode changes[1] may affect.
[1] https://docs.python.org/3/whatsnew/3.8.html
was:
apache_beam.typehints.trivial_inference.infer_return_type does not return type
as expected in Python 3.8.
And this cause testDictComprehensionSimple
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
- Python 3.7.7
{code:Python}
>> from apache_beam.typehints.trivial_inference import infer_return_type
>> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
>> depth=5)
Dict[str, int]{code}
- Python 3.8.2
{code:Python}
>> from apache_beam.typehints.trivial_inference import infer_return_type
>> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
>> depth=5)
Dict[int, str]{code}
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086939#comment-17086939 ] yoshiki obata commented on BEAM-9701: - pyarrow dependent version is updated on 2.20.0, so we have to update fastavro's one. > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9701) Enable installing on Python 3.8
[ https://issues.apache.org/jira/browse/BEAM-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata updated BEAM-9701: Description: We cannot *pip install apache-beam* on Python 3.8 because building wheels of fastavro -and pyarrow- are failed on it. Updating versions of them at setup.py and confirm there is no effect for other Python versions to do so are needed. was: We cannot `pip install apache-beam` on Python 3.8 because building wheels of fastavro and pyarrow are failed on it. Updating versions of them at setup.py and confirm there is no effect for other Python versions to do so are needed. > Enable installing on Python 3.8 > --- > > Key: BEAM-9701 > URL: https://issues.apache.org/jira/browse/BEAM-9701 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Priority: Major > > We cannot *pip install apache-beam* on Python 3.8 because building wheels of > fastavro -and pyarrow- are failed on it. > Updating versions of them at setup.py and confirm there is no effect for > other Python versions to do so are needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
yoshiki obata created BEAM-9755:
---
Summary: infer_return_type does not infer as expected in Python 3.8
Key: BEAM-9755
URL: https://issues.apache.org/jira/browse/BEAM-9755
Project: Beam
Issue Type: Sub-task
Components: sdk-py-core
Reporter: yoshiki obata
apache_beam.typehints.trivial_inference.infer_return_type does not return type
as expected in Python 3.8.
And this cause testDictComprehensionSimple
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
- Python 3.7.7
{code:Python}
>> from apache_beam.typehints.trivial_inference import infer_return_type
>> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
>> depth=5)
Dict[str, int]{code}
- Python 3.8.2
{code:Python}
>> from apache_beam.typehints.trivial_inference import infer_return_type
>> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
>> depth=5)
Dict[int, str]{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
