hm, interesting, are we using pybind anywhere? I didn't see any references
to it. I can give it a try on python 3.8 too though.
On Wed, Dec 8, 2021 at 9:19 AM Brian Hulette <bhule...@google.com> wrote:
> A google search for "__import__ not found" turned up an issue filed with
> pybind [1]. I can't deduce a root cause from the discussion there, but it
> looks like they didn't experience the issue in Python 3.8 - it could be
> interesting to see if your problem goes away there.
>
> It looks like +Charles Chen <c...@google.com> added the __import__('re')
> workaround in [2], maybe he remembers what was going on?
>
> [1] https://github.com/pybind/pybind11/issues/2557
> [2] https://github.com/apache/beam/pull/5071
>
> On Wed, Dec 8, 2021 at 5:30 AM Steve Niemitz <sniem...@apache.org> wrote:
>
>> Yeah, I can't imagine this is a "normal" problem.
>>
>> I'm on linux w/ py 3.7. My script does have a __name__ == '__main__'
>> block.
>>
>> On Wed, Dec 8, 2021 at 12:38 AM Ning Kang <ni...@google.com> wrote:
>>
>>> I tried a pipeline:
>>>
>>> p = beam.Pipeline(DataflowRunner(), options=options)
>>> text = p | beam.Create(['Hello World, Hello You'])
>>>
>>>
>>> def tokenize(x):
>>> import re
>>> return re.findall('Hello', x)
>>>
>>>
>>> flatten = text | 'Split' >>
>>> (beam.FlatMap(tokenize).with_output_types(str))
>>> pipeline_result = p.run()
>>>
>>>
>>> Didn't run into the issue.
>>>
>>> What OS and Python version are you using? Does your script come with a
>>> `if __name__ == '__main__': `?
>>>
>>> On Tue, Dec 7, 2021 at 6:58 PM Steve Niemitz <sniem...@apache.org>
>>> wrote:
>>>
>>>> I have a fairly simple python word count job (although the packaging is
>>>> a little more complicated) that I'm trying to run. (note: I'm explicitly
>>>> NOT using save_main_session.)
>>>>
>>>> In it is a method to tokenize the incoming text to words, and I used
>>>> something similar to how the wordcount example worked.
>>>>
>>>> def tokenize(row):
>>>> import re
>>>> return re.findall(r'[A-Za-z\']+', row.text)
>>>>
>>>> which is then used as the function for a FlatMap:
>>>> | 'Split' >> (
>>>> beam.FlatMap(tokenize).with_output_types(str))
>>>>
>>>> However, if I run this job on dataflow (2.33), the python runner fails
>>>> with a bizarre error:
>>>> INFO:apache_beam.runners.dataflow.dataflow_runner:2021-12-07T20:59:59.704Z:
>>>> JOB_MESSAGE_ERROR: Traceback (most recent call last):
>>>> File "apache_beam/runners/common.py", line 1232, in
>>>> apache_beam.runners.common.DoFnRunner.process
>>>> File "apache_beam/runners/common.py", line 572, in
>>>> apache_beam.runners.common.SimpleInvoker.invoke_process
>>>> File "/tmp/tmpq_8l154y/wordcount_test.py", line 75, in tokenize
>>>> ImportError: __import__ not found
>>>>
>>>> I was able to find an example in the streaming wordcount snippet that
>>>> did something similar, but very strange [1]:
>>>> | 'ExtractWords' >>
>>>> beam.FlatMap(lambda x: __import__('re').findall(r'[A-Za-z\']+',
>>>> x))
>>>>
>>>> For whatever reason this actually fixed the issue in my job as well. I
>>>> can't for the life of me understand why this works, or why the normal
>>>> import fails. Someone else must have run into this same issue though for
>>>> that streaming wordcount example to be like that. Any ideas what's going
>>>> on here?
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py#L692
>>>>
>>>
