Found this stackoverflow question. https://stackoverflow.com/questions/59831397/run-setup-function-from-setuptools-only-if-name-main
>From what I understand, normally its not needed since setup.py is generally not imported. But yes, a main guard would prevent code from running if the setup.py is imported. I think it would be best to first understand why setup.py being imported in the first place. Which is a bit odd. Does one of our dependencies or tools do this? Is that tool making an assumption that we have main guarded our setup.py? It seems like run_integration_test.sh <https://github.com/apache/beam/blob/1f48ac6610f760e7856ea4f0cf3d5c1e9cce71c3/sdks/python/scripts/run_integration_test.sh> runs setup.py itself, via nose. Is this an intended behaviour of nose? Is it doing some setup first then running setup.py? On Wed, Oct 28, 2020 at 1:38 PM Heejong Lee <[email protected]> wrote: > I've encountered the following error while I was testing Python > integration tests via `run_integration_test.sh` on MacOS: > > Traceback (most recent call last): > File "<string>", line 1, in <module> > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", > line 116, in spawn_main > exitcode = _main(fd, parent_sentinel) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", > line 125, in _main > prepare(preparation_data) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", > line 236, in prepare > _fixup_main_from_path(data['init_main_from_path']) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", > line 287, in _fixup_main_from_path > main_content = runpy.run_path(main_path, > File "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line > 265, in run_path > return _run_module_code(code, init_globals, run_name, > File "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line > 97, in _run_module_code > _run_code(code, mod_globals, init_globals, > File "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line > 87, in _run_code > exec(code, run_globals) > File "/Users/heejong/Work/beam/sdks/python/setup.py", line 259, in > <module> > setuptools.setup( > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/setuptools/__init__.py", > line 153, in setup > return distutils.core.setup(**attrs) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/distutils/core.py", > line 148, in setup > dist.run_commands() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/distutils/dist.py", > line 966, in run_commands > self.run_command(cmd) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/distutils/dist.py", > line 985, in run_command > cmd_obj.run() > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/commands.py", > line 158, in run > TestProgram(argv=argv, config=self.__config) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/core.py", > line 118, in __init__ > unittest.TestProgram.__init__( > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/unittest/main.py", line > 100, in __init__ > self.parseArgs(argv) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/core.py", > line 145, in parseArgs > self.config.configure(argv, doc=self.usage()) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/config.py", > line 346, in configure > self.plugins.configure(options, self) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/plugins/manager.py", > line 284, in configure > cfg(options, config) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/plugins/manager.py", > line 99, in __call__ > return self.call(*arg, **kw) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/plugins/manager.py", > line 167, in simple > result = meth(*arg, **kw) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose_xunitmp.py", > line 42, in configure > manager = multiprocessing.Manager() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", > line 57, in Manager > m.start() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/managers.py", > line 579, in start > self._process.start() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", > line 121, in start > self._popen = self._Popen(self) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", > line 284, in _Popen > return Popen(process_obj) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/popen_spawn_posix.py", > line 32, in __init__ > super().__init__(process_obj) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/popen_fork.py", > line 19, in __init__ > self._launch(process_obj) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/popen_spawn_posix.py", > line 42, in _launch > prep_data = spawn.get_preparation_data(process_obj._name) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", > line 154, in get_preparation_data > _check_not_importing_main() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", > line 134, in _check_not_importing_main > raise RuntimeError(''' > RuntimeError: > An attempt has been made to start a new process before the > current process has finished its bootstrapping phase. > > This probably means that you are not using fork to start your > child processes and you have forgotten to use the proper idiom > in the main module: > > if __name__ == '__main__': > freeze_support() > ... > > The "freeze_support()" line can be omitted if the program > is not going to be frozen to produce an executable. > Traceback (most recent call last): > File "setup.py", line 259, in <module> > setuptools.setup( > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/setuptools/__init__.py", > line 153, in setup > return distutils.core.setup(**attrs) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/distutils/core.py", > line 148, in setup > dist.run_commands() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/distutils/dist.py", > line 966, in run_commands > self.run_command(cmd) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/distutils/dist.py", > line 985, in run_command > cmd_obj.run() > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/commands.py", > line 158, in run > TestProgram(argv=argv, config=self.__config) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/core.py", > line 118, in __init__ > unittest.TestProgram.__init__( > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/unittest/main.py", line > 100, in __init__ > self.parseArgs(argv) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/core.py", > line 145, in parseArgs > self.config.configure(argv, doc=self.usage()) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/config.py", > line 346, in configure > self.plugins.configure(options, self) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/plugins/manager.py", > line 284, in configure > cfg(options, config) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/plugins/manager.py", > line 99, in __call__ > return self.call(*arg, **kw) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose/plugins/manager.py", > line 167, in simple > result = meth(*arg, **kw) > File > "/Users/heejong/Work/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/nose_xunitmp.py", > line 42, in configure > manager = multiprocessing.Manager() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", > line 57, in Manager > m.start() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/managers.py", > line 583, in start > self._address = reader.recv() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/connection.py", > line 250, in recv > buf = self._recv_bytes() > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/connection.py", > line 414, in _recv_bytes > buf = self._recv(4) > File > "/Users/heejong/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/connection.py", > line 383, in _recv > raise EOFError > EOFError > > It was okay with Python 3.7 so the issue is probably specific to Python > 3.8 on MacOS. > > The solution would be putting __main__ guard around the setup function in > `setup.py` file but I'm not sure whether it's a kind of anti-pattern or > not. Any comments from Python experts? > > >
