[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163101#comment-17163101 ] Brian Hulette commented on BEAM-10010: -- [~heejong] started automated tests that fulfill this in https://github.com/apache/beam/pull/12060 / BEAM-10208. Thank you! > Test Python SqlTransform on fn_api_runner > - > > Key: BEAM-10010 > URL: https://issues.apache.org/jira/browse/BEAM-10010 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.24.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to run with the fn_api_runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155803#comment-17155803 ] Brian Hulette commented on BEAM-10010: -- Looks like this may just work now. The bug may have been due to BEAM-10308 > Test Python SqlTransform on fn_api_runner > - > > Key: BEAM-10010 > URL: https://issues.apache.org/jira/browse/BEAM-10010 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > > It should be possible to run with the fn_api_runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151752#comment-17151752 ] Beam JIRA Bot commented on BEAM-10010: -- This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Test Python SqlTransform on fn_api_runner > - > > Key: BEAM-10010 > URL: https://issues.apache.org/jira/browse/BEAM-10010 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Priority: P2 > > It should be possible to run with the fn_api_runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147522#comment-17147522 ] Beam JIRA Bot commented on BEAM-10010: -- This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Test Python SqlTransform on fn_api_runner > - > > Key: BEAM-10010 > URL: https://issues.apache.org/jira/browse/BEAM-10010 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Labels: stale-assigned > > It should be possible to run with the fn_api_runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112674#comment-17112674 ] Brian Hulette commented on BEAM-10010: -- cc: [~robertwb], [~chamikara] Could you help me debug these errors? They only occur when I try to run xlang SqlTransform with fn_runner. Flink and Spark don't have any issues. The test_generate_data (Java sees a Python coder) error seems to occur about 25% of the time. The test_tagged_join (Python sees a Java coder) error always occurs. > Test Python SqlTransform on fn_api_runner > - > > Key: BEAM-10010 > URL: https://issues.apache.org/jira/browse/BEAM-10010 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > > It should be possible to run with the fn_api_runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[
https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112672#comment-17112672
]
Brian Hulette commented on BEAM-10010:
--
test_tagged_join reliably produces the following error from Python at execution
time:
{code}
apache_beam/runners/portability/fn_api_runner/fn_runner.py:859: in
process_bundle
output.transform_id).append(output.data)
apache_beam/runners/portability/fn_api_runner/execution.py:591: in get_buffer
coder_impl=self.get_input_coder_impl(transform_id))
apache_beam/runners/portability/fn_api_runner/execution.py:555: in
get_input_coder_impl
return self.get_coder_impl(coder_id)
apache_beam/runners/portability/fn_api_runner/execution.py:574: in
get_coder_impl
return self.execution_context.pipeline_context.coders[coder_id].get_impl()
apache_beam/runners/pipeline_context.py:137: in __getitem__
return self.get_by_id(id)
apache_beam/runners/pipeline_context.py:103: in get_by_id
self._id_to_proto[id], self._pipeline_context)
apache_beam/coders/coders.py:373: in from_runner_api
for c in coder_proto.component_coder_ids
apache_beam/coders/coders.py:373: in
for c in coder_proto.component_coder_ids
apache_beam/runners/pipeline_context.py:103: in get_by_id
self._id_to_proto[id], self._pipeline_context)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[
https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112669#comment-17112669
]
Brian Hulette commented on BEAM-10010:
--
test_generate_data sometimes yields the following error from Java at execution
time:
{code}
E RuntimeError:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: Unknown Coder URN
beam:coder:pickled_python:v1. Known URNs: [beam:coder:avro:generic:v1,
beam:coder:bytes:v1, beam:coder:bool:v1, beam:coder:string_utf8:v1,
beam:coder:kv:v1, beam:coder:varint:v1, beam:coder:interval_window:v1,
beam:coder:iterable:v1, beam:coder:timer:v1, beam:coder:length_prefix:v1,
beam:coder:global_window:v1, beam:coder:windowed_value:v1,
beam:coder:param_windowed_value:v1, beam:coder:double:v1, beam:coder:row:v1]
E at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
E at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)
E at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
E at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
E at
org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:164)
E at
org.apache.beam.runners.core.construction.CoderTranslation.fromKnownCoder(CoderTranslation.java:132)
E at
org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:124)
E at
org.apache.beam.fn.harness.BeamFnDataWriteRunner.(BeamFnDataWriteRunner.java:139)
E at
org.apache.beam.fn.harness.BeamFnDataWriteRunner$Factory.createRunnerForPTransform(BeamFnDataWriteRunner.java:100)
E at
org.apache.beam.fn.harness.BeamFnDataWriteRunner$Factory.createRunnerForPTransform(BeamFnDataWriteRunner.java:77)
E at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:236)
E at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:198)
E at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:491)
E at
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:275)
E at
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:552)
E at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:270)
[jira] [Commented] (BEAM-10010) Test Python SqlTransform on fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108545#comment-17108545 ] Brian Hulette commented on BEAM-10010: -- Tried this out locally, but I'm running into some test failures. test_generate_date is flaky, test_tagged_join is just failing. Described the issue on dev@: https://lists.apache.org/thread.html/r180bf5a0183ca1216c0f575cc30f7335c1494989a1b7bf90c9fef209%40%3Cdev.beam.apache.org%3E > Test Python SqlTransform on fn_api_runner > - > > Key: BEAM-10010 > URL: https://issues.apache.org/jira/browse/BEAM-10010 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > > It should be possible to run with the fn_api_runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
