Well that worked! Thank you. I only had to adjust some dependencies. The execution (not the installation) of miso depends on _sqlite3 which wasn't checked before, I still need to solve this issue. Related to this I get a strange easybuild MPI error. When I load miso and execute the binary, it complains that it can't find sqlite. If I execute easybuild then, it gives me this strange error message (attached file). I wonder where the relation is?
Commands: $ module load misopy $ miso # here I get the missing sqlite error $ eb -S sqlite # here the error occurs I will contribute when I got this issue solved. Oliver ________________________________ From: Kenneth Hoste [kenneth.ho...@ugent.be] Sent: Tuesday, June 23, 2015 8:42 PM To: easybuild@lists.ugent.be Cc: Stolpe, Oliver Subject: Re: [easybuild] Binary of a python package Hi Oliver, (you're not subscribed to this mailing list with the email address you used, but I did accept your mail since you're subscribed with another mail address and this is clearly a useful post; I added that in CC just to make sure you get this) On 23/06/15 17:29, Stolpe, Oliver wrote: Hello list, I wrote an easyconfig file to install a python package. The installation is successful besides the fact that easybuild throws away the binary that is generated (or main entrance point for the standalone). When I install the software on my machine with python setup.py build/install --user, it puts the binary to ~/.local/bin, but I don't know where easybuild puts this file. When I look into the easybuild/software/MYSOFTWARE directory, there is no bin folder. Please help me save the binary :-) Welcome to the tarpit that is Python packaging... There's good news (twice) and bad news. The 1st good news is that I fixed your problem. The bad news is that I had to patch something in the misopy 'sources' (well, the setup.py script). The 2nd good news is that the patch is trivial. :-) The problem is that misopy is using something called 'entry points', but not doing it 100% correctly. See https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/ . The key phrase on that page is "You must use setuptools, otherwise this won’t work." The setup.py of misopy has this though: from distutils.core import setup, Extension This should actually be: from setuptools import setup, Extension EasyBuild installs Python packages using "python setup.py build" and "python setup.py install". In other words, it trusts that the setup.py script is doing what it should be doing. And in this case, it's not. If you use pip to install misopy, you're unknowingly dancing around this problem, since pip apparently 'fixes' this by itself by providing something different than setuptools when importing from distutils.core . Ah, the joys of having 100 different ways of installing Python packages... Anyway, the problem is fixed using the slightly modified easyconfig file and trivial patch file in attachment. Note that I also made EasyBuild check that the 'bin/miso' script is actually being installed. If you would comment out the patch, you'll notice that EasyBuild complains that 'bin/miso' is missing, and thus that the installation is incomplete. (behind the scenes, it's already making sure that "python -c 'import misopy' " works too) I've reported this issue to the MISO developers, see https://github.com/yarden/MISO/pull/78 . Please consider contributing back your working easyconfig file, see https://github.com/hpcugent/easybuild/wiki/Contributing-back . If you're not up for that, let me know if it's OK if I do this for you instead. regards, Kenneth Thanks, Oliver The easyconfig file: easyblock = 'PythonPackage' name = 'misopy' version = '0.5.3' homepage = 'http://genes.mit.edu/burgelab/miso/index.html' description = """MISO (Mixture of Isoforms) is a probabilistic framework that quantitates the expression level of alternatively spliced genes from RNA-Seq data, and identifies differentially regulated isoforms or exons across samples. By modeling the generative process by which reads are produced from isoforms in RNA-Seq, the MISO model uses Bayesian inference to compute the probability that a read originated from a particular isoform.""" toolchain = {'name': 'goolf', 'version': '1.4.10'} source_urls = ['http://pypi.python.org/packages/source/m/misopy'] sources = ['%(namelower)s-%(version)s.tar.gz'] python = 'Python' pythonver = '2.7.6' versionsuffix = '-%s-%s' % (python, pythonver) dependencies = [ ('matplotlib', '1.3.1', versionsuffix), (python, pythonver), ] moduleclass = 'bio'
ImportError: No module named _sqlite3 $ eb -S sqlite [f967b29c4cb7:00121] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file ess_hnp_module.c at line 170 -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_plm_base_select failed --> Returned value Not found (-13) instead of ORTE_SUCCESS -------------------------------------------------------------------------- [f967b29c4cb7:00121] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file runtime/orte_init.c at line 128 -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_ess_set_name failed --> Returned value Not found (-13) instead of ORTE_SUCCESS -------------------------------------------------------------------------- [f967b29c4cb7:00121] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file orted/orted_main.c at line 353 [f967b29c4cb7:00120] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon on the local node in file ess_singleton_module.c at line 343 [f967b29c4cb7:00120] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon on the local node in file ess_singleton_module.c at line 140 [f967b29c4cb7:00120] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon on the local node in file runtime/orte_init.c at line 128 -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_ess_set_name failed --> Returned value Unable to start a daemon on the local node (-128) instead of ORTE_SUCCESS -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): ompi_mpi_init: orte_init failed --> Returned "Unable to start a daemon on the local node" (-128) instead of "Success" (0) -------------------------------------------------------------------------- [f967b29c4cb7:120] *** An error occurred in MPI_Init_thread [f967b29c4cb7:120] *** on a NULL communicator [f967b29c4cb7:120] *** Unknown error [f967b29c4cb7:120] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort -------------------------------------------------------------------------- An MPI process is aborting at a time when it cannot guarantee that all of its peer processes in the job will be killed properly. You should double check that everything has shut down cleanly. Reason: Before MPI_INIT completed Local host: f967b29c4cb7 PID: 120 --------------------------------------------------------------------------
