Re: Should python-build-system packages have native-inputs?
Hartmut Goebel writes: > Am 28.04.2018 um 12:11 schrieb Chris Marusich: > >> Because the python-build-system never cross-compiles, > > This is an implementation detail which might might change. And if we > remove all inputs now, we need to add again them later. This is a lot of > work, I know since I've cleaned this up for all Python modules. IMHO > it's not a good idea for drop this knowledge from the code. I agree. Mark
Re: Should python-build-system packages have native-inputs?
Fis Trivial writes: > I'm confused, some native-inputs are for testing. They shouldn't be > installed in normal case. native-inputs are not only for testing. More generally, when cross-compiling, any programs that must be run on the build machine must be 'native-inputs'. Any programs that must be run on the target machine must be 'inputs'. Mark
Re: Should python-build-system packages have native-inputs?
Hartmut Goebel writes: >> the python-build-system does not cross-compile. > > In any case, this is a current limitation only :-) I see. If we actually do plan on implementing some kind of cross-compilation support for the python-build-system, then I can understand why it makes sense to proactively put into the native-inputs those things which might possibly need to be native-inputs when that time comes. Can we at least mention in the manual that the python-build-system doesn't currently cross-compile, so native-inputs will be treated the same as inputs for now, but we still recommend putting the "build-only dependencies" in native-inputs in order to future-proof our package definitions? That alone would have helped clarify things for me when I was starting out. After reading about Python extension modules a little more, it seems that they can in fact be cross-compiled. I didn't look into Python 2, and I don't know what it would take to enable such cross-compilation in the python-build-system. For those following along, here are some related links. An explanation of Python extension modules: https://docs.python.org/3/extending/building.html#distributing-your-extension-modules Some Python open bugs that mention cross-compilation: https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus&%40sort=-activity&%40group=priority&%40filter=status&status=-1%2C1%2C3&%40search_text=cross-compile&submit=search+in+open+issues -- Chris signature.asc Description: PGP signature
Re: Should python-build-system packages have native-inputs?
Am 28.04.2018 um 12:11 schrieb Chris Marusich: I understand your concerns, and I understand why this is hard to get for a Pythonista. But this is exactly why we added this section to the manual. > Because the python-build-system never cross-compiles, This is an implementation detail which might might change. And if we remove all inputs now, we need to add again them later. This is a lot of work, I know since I've cleaned this up for all Python modules. IMHO it's not a good idea for drop this knowledge from the code. > If the > python-build-system actually did support cross-compilation, then this > might be a different story. Maybe this is going to change somewhen :-) We should aim to the top, not the status quo :-) > My understanding is that the concept of "native-inputs" for a package > only makes sense when that package uses a build system that can > cross-compile, This is my understanding, too. But the python-build-system might be able to cross-compile somewhen and then this information is essential. >> And for extension modules it would allow compiling on a faster >> environment (e.g. x86 vs. ARMv4). >> >> (I was not aware of python packages are not cross-compiled, thus I can >> only guess the reason why this is not possible: Python distutils may not >> be able to *cross*-compile extension modules. Maybe we could work on this.) > I am curious about extension modules. I understand they are tied > closely to the underlying architecture, but I have little experience > with them, so I'm not sure how they relate to cross compilation. Extension modules are simply modules or libraries written in C/C++ or other languages. Even modules written in Cython would be counted in here, since they are translated to C and then compiled into platform dependent code. > In any > case, it doesn't change the fact that today, the python-build-system > does not cross-compile. In any case, this is a current limitation only :-) -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | signature.asc Description: OpenPGP digital signature
Re: Should python-build-system packages have native-inputs?
Hi Fis and Hartmut, Thank you for the quick response! Hartmut Goebel writes: > As Fis already wrote: These native-inputs are for testing and shouldn't > be installed in normal case. It's true that for some of the packages that use the python-build-system, we have been putting the dependencies required for testing (such as python-pytest) into the package's native-inputs. However, whether such dependencies are inputs or native-inputs does not matter. Because the python-build-system never cross-compiles, all of the inputs, propagated-inputs, and native-inputs will be included in the single list that gets passed to each of the build phases via the #:inputs keyword argument. You can verify this yourself by inserting debug statements in the build phases. In other words, it doesn't matter if we put python-pytest in a package's inputs or its native-inputs. The end result is the same. If the python-build-system actually did support cross-compilation, then this might be a different story. However, the python-build-system doesn't cross-compile. As a result, native-inputs and inputs are treated the same in all of the phases defined in guix/build/python-build-system.scm. > Please see "Python Modules" in the manual: > > Python packages required only at build time---e.g., those listed with > the @code{setup_requires} keyword in @file{setup.py}---or only for > testing---e.g., those in @code{tests_require}---go into > @code{native-inputs}. The rationale is that (1) they do not need to be > propagated because they are not needed at run time, and (2) in a > cross-compilation context, it's the ``native'' input that we'd want. Thank you for mentioning the manual; I had forgotten that we include explicit guidance for Python modules. I've just reviewed the "Python Modules" section. I think we should not be advising people to use native-inputs in packages that use the python-build-system. There is no meaningful difference between "native-inputs" and "inputs" in this case, so asking people to contemplate the difference is like asking them a kōan. It's just going to cause confusion. This is confusing. And that is precisely why I think we should stop declaring native-inputs for packages that use the python-build-system. My understanding is that the concept of "native-inputs" for a package only makes sense when that package uses a build system that can cross-compile, such as the gnu-build-system. Because the python-build-system never cross-compiles, it doesn't make sense to declare native-inputs for a package that uses the python-build-system. Instead, those dependencies should just be declared as inputs. >> * Are there any circumstances under which it actually WOULD make sense >> to cross-compile a Python package? > > Of course: Pure-python packages should be able to be cross-compiled > without any problems, sicne the bytes-code is the same for all > platforms. I'm not sure that's the same thing as cross compilation. When cross compiling a program for a different architecture, the output of the build is different for each architecture. If Python's bytecode is the same for all platforms, then it sounds like no cross-compilation is necessary, which suggests that the notion of "cross compilation" does not make sense for Python code. Did I misinterpret what you meant? > And for extension modules it would allow compiling on a faster > environment (e.g. x86 vs. ARMv4). > > (I was not aware of python packages are not cross-compiled, thus I can > only guess the reason why this is not possible: Python distutils may not > be able to *cross*-compile extension modules. Maybe we could work on this.) I am curious about extension modules. I understand they are tied closely to the underlying architecture, but I have little experience with them, so I'm not sure how they relate to cross compilation. In any case, it doesn't change the fact that today, the python-build-system does not cross-compile. -- Chris signature.asc Description: PGP signature
Re: Should python-build-system packages have native-inputs?
Am 28.04.2018 um 08:50 schrieb Chris Marusich: > * Should we change these native-inputs to inputs to prevent confusion? > I can personally vouch for the fact that the presence of native-inputs > in python-build-system packages confused the heck out of me at first! As Fis already wrote: These native-inputs are for testing and shouldn't be installed in normal case. Please see "Python Modules" in the manual: Python packages required only at build time---e.g., those listed with the @code{setup_requires} keyword in @file{setup.py}---or only for testing---e.g., those in @code{tests_require}---go into @code{native-inputs}. The rationale is that (1) they do not need to be propagated because they are not needed at run time, and (2) in a cross-compilation context, it's the ``native'' input that we'd want. > * Are there any circumstances under which it actually WOULD make sense > to cross-compile a Python package? Of course: Pure-python packages should be able to be cross-compiled without any problems, sicne the bytes-code is the same for all platforms. And for extension modules it would allow compiling on a faster environment (e.g. x86 vs. ARMv4). (I was not aware of python packages are not cross-compiled, thus I can only guess the reason why this is not possible: Python distutils may not be able to *cross*-compile extension modules. Maybe we could work on this.) -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: Should python-build-system packages have native-inputs?
Chris Marusich writes: > Hi Guix, > > I've noticed that a fair number of packages in gnu/packages/python.scm > using the python-build-system declare native-inputs. I suspect that in > every case, these should actually just be inputs. I also suspect that > this is benign, except perhaps for the fact that it may confuse > Pythonistas who (like myself) initially started out by looking at these > packages as examples of how to get started defining packages in Guix. > > The python-build-system's "lower" procedure (in > guix/build-system/python.scm) explicitly forbids cross-compilation: > > --8<---cut here---start->8--- > (define* (lower name > #:key source inputs native-inputs outputs system target > (python (default-python)) > #:allow-other-keys > #:rest arguments) > "Return a bag for NAME." > (define private-keywords > '(#:source #:target #:python #:inputs #:native-inputs)) > > (and (not target) ;XXX: no cross-compilation >(bag > (name name) > (system system) > (host-inputs `(,@(if source > `(("source" ,source)) > '()) > ,@inputs > > ;; Keep the standard inputs of 'gnu-build-system'. > ,@(standard-packages))) > (build-inputs `(("python" ,python) > ,@native-inputs)) > (outputs outputs) > (build python-build) > (arguments (strip-keyword-arguments private-keywords arguments) > --8<---cut here---end--->8--- > > As for the native-inputs, they get stored in the bag's build-inputs, > which eventually find their way to the "inputs" keyword argument used on > the build side by the various build phases. In fact, the inputs, > propagated-inputs, and native-inputs of any package that uses the > python-build-system are all put into this "inputs" keyword argument. > > With this in mind, I have two questions: > > * Should we change these native-inputs to inputs to prevent confusion? > I can personally vouch for the fact that the presence of native-inputs > in python-build-system packages confused the heck out of me at first! > > * Are there any circumstances under which it actually WOULD make sense > to cross-compile a Python package? > > For now, I think the answers to these questions are "sure" and "probably > not", respectively. I'm very curious to hear your thoughts about the > second question, in particular! I'm confused, some native-inputs are for testing. They shouldn't be installed in normal case.
Should python-build-system packages have native-inputs?
Hi Guix, I've noticed that a fair number of packages in gnu/packages/python.scm using the python-build-system declare native-inputs. I suspect that in every case, these should actually just be inputs. I also suspect that this is benign, except perhaps for the fact that it may confuse Pythonistas who (like myself) initially started out by looking at these packages as examples of how to get started defining packages in Guix. The python-build-system's "lower" procedure (in guix/build-system/python.scm) explicitly forbids cross-compilation: --8<---cut here---start->8--- (define* (lower name #:key source inputs native-inputs outputs system target (python (default-python)) #:allow-other-keys #:rest arguments) "Return a bag for NAME." (define private-keywords '(#:source #:target #:python #:inputs #:native-inputs)) (and (not target) ;XXX: no cross-compilation (bag (name name) (system system) (host-inputs `(,@(if source `(("source" ,source)) '()) ,@inputs ;; Keep the standard inputs of 'gnu-build-system'. ,@(standard-packages))) (build-inputs `(("python" ,python) ,@native-inputs)) (outputs outputs) (build python-build) (arguments (strip-keyword-arguments private-keywords arguments) --8<---cut here---end--->8--- As for the native-inputs, they get stored in the bag's build-inputs, which eventually find their way to the "inputs" keyword argument used on the build side by the various build phases. In fact, the inputs, propagated-inputs, and native-inputs of any package that uses the python-build-system are all put into this "inputs" keyword argument. With this in mind, I have two questions: * Should we change these native-inputs to inputs to prevent confusion? I can personally vouch for the fact that the presence of native-inputs in python-build-system packages confused the heck out of me at first! * Are there any circumstances under which it actually WOULD make sense to cross-compile a Python package? For now, I think the answers to these questions are "sure" and "probably not", respectively. I'm very curious to hear your thoughts about the second question, in particular! -- Chris signature.asc Description: PGP signature