Re: Recommended way of installing system-wide python application and libraries

[Gregor Riepl]

As far as how to do this within an existing cmake project, 
unfortunately, there doesn't seem to be a clear/easy way.  The only 
cmake example I can think off of the top of my head is cvc5.  It still 
uses setup.py though, so not a great future-looking example (and I had 
to patch it to build the Python bindings in Debian): 
https://github.com/cvc5/cvc5/blob/main/src/api/python/CMakeLists.txt


We've been using cmake + pybuild very successfully for packaging the 
various Cura components:


https://salsa.debian.org/3dprinting-team/cura/-/blob/master/debian/rules
https://salsa.debian.org/3dprinting-team/cura-engine/-/blob/master/debian/rules
https://salsa.debian.org/3dprinting-team/uranium/-/blob/master/debian/rules
etc.

Cura is a mix of Python and C++, so a pure Python tools approach 
wouldn't have worked well. Upstream used cmake exclusively in the past, 
and that worked very well together with pybuild. Unfortunately, they 
recently introduced a new build environment based on conan, which is why 
we haven't managed to package new Cura versions yet.


The best solution right now is probably: 
https://scikit-build-core.readthedocs.io

It's specifically designed for combining CMake + Python.
I don't know how well this is supported in Debian so far.

Re: Recommended way of installing system-wide python application and libraries

[Ivan Perez]

Thanks!! I'll answer to the last comment but I appreciate everyone's input!

Hello again Ivan (different neighborhood :)),

:)))

The "modern" way of building Python libraries/applications seems to be to
> use a pyproject.toml.  The pyproject.toml specifies a backend build system
> that knows how to build the library/application and produce a wheel.


Thanks for this pointer. I didn't know about this.

As far as how to do this within an existing cmake project, unfortunately,
> there doesn't seem to be a clear/easy way.


For now, we ended up switching to venvs. I just published a release
candidate.

https://github.com/NASA-SW-VnV/ikos/discussions/253

Following @salvo's comment, I'm going to try and begin the process of
packaging this for debian. I suspect it may be a bit complicated because
ikos is two things (a library in C++ and a set of tools in Python) under a
trench coat.

This is by far not my domain. If anyone is willing to lend a hand, I could
definitely use any help that I can get, but I understand that folks are
busy and everybody wants help.

Cheers,

Ivan



On Thu, 7 Dec 2023 at 16:12, Scott Talbert  wrote:

> On Tue, 5 Dec 2023, Ivan Perez wrote:
>
> > Hi everyone!
> >
> > I'm currently trying to bring a tool we have at NASA Ames up to speed:
> > https://github.com/NASA-SW-VnV/ikos
> >
> > IKOS is a static analyzer for C. I'm really hoping that IKOS can be
> included
> > in Debian in the near future.
> >
> > IKOS is implemented as a  C++ library, and a number of python
> > tools/wrappers. The tools call mains in modules defined in a python
> library
> > `ikos`.
> >
> > As of right now, our CMakeFiles attempt to install everything (by
> default)
> > under /opt/ikos/.
> >
> > I'm having lots of issues getting the python portions installed
> system-wide.
> > I initially upgraded distutils to setuptools, but a recent update is now
> > asking that I use a venv. More details and a link to a dockerfile can be
> > found here: https://github.com/NASA-SW-VnV/ikos/discussions/241.
> >
> > While I can hack a solution that "works" (either by making a venv under
> the
> > target dir or by means of break-system-packages), I'd prefer to use
> > recommended practices, and also conform to the way that things are done
> in
> > Debian/Ubuntu.
> >
> > What would be the recommended way of installing IKOS system-wide in
> Debian?
>
> Hello again Ivan (different neighborhood :)),
>
> The "modern" way of building Python libraries/applications seems to be to
> use a pyproject.toml.  The pyproject.toml specifies a backend build system
> that knows how to build the library/application and produce a wheel.
>
> As far as how to do this within an existing cmake project, unfortunately,
> there doesn't seem to be a clear/easy way.  The only cmake example I can
> think off of the top of my head is cvc5.  It still uses setup.py though,
> so not a great future-looking example (and I had to patch it to build the
> Python bindings in Debian):
> https://github.com/cvc5/cvc5/blob/main/src/api/python/CMakeLists.txt
>
> If you're open to other options for building python libraries, meson might
> be something to look into.  I haven't used it extensively, though.  It
> does seem to have better support for building Python though.
>
> Regards,
> Scott

Re: Recommended way of installing system-wide python application and libraries

[Scott Talbert]

On Tue, 5 Dec 2023, Ivan Perez wrote:


Hi everyone!

I'm currently trying to bring a tool we have at NASA Ames up to speed:
https://github.com/NASA-SW-VnV/ikos

IKOS is a static analyzer for C. I'm really hoping that IKOS can be included
in Debian in the near future.

IKOS is implemented as a  C++ library, and a number of python
tools/wrappers. The tools call mains in modules defined in a python library
`ikos`.

As of right now, our CMakeFiles attempt to install everything (by default)
under /opt/ikos/.

I'm having lots of issues getting the python portions installed system-wide.
I initially upgraded distutils to setuptools, but a recent update is now
asking that I use a venv. More details and a link to a dockerfile can be
found here: https://github.com/NASA-SW-VnV/ikos/discussions/241.

While I can hack a solution that "works" (either by making a venv under the
target dir or by means of break-system-packages), I'd prefer to use
recommended practices, and also conform to the way that things are done in
Debian/Ubuntu.

What would be the recommended way of installing IKOS system-wide in Debian?


Hello again Ivan (different neighborhood :)),

The "modern" way of building Python libraries/applications seems to be to 
use a pyproject.toml.  The pyproject.toml specifies a backend build system 
that knows how to build the library/application and produce a wheel.


As far as how to do this within an existing cmake project, unfortunately, 
there doesn't seem to be a clear/easy way.  The only cmake example I can 
think off of the top of my head is cvc5.  It still uses setup.py though, 
so not a great future-looking example (and I had to patch it to build the 
Python bindings in Debian): 
https://github.com/cvc5/cvc5/blob/main/src/api/python/CMakeLists.txt


If you're open to other options for building python libraries, meson might 
be something to look into.  I haven't used it extensively, though.  It 
does seem to have better support for building Python though.


Regards,
Scott

Re: Recommended way of installing system-wide python application and libraries

[Salvo Tomaselli]

Hello,

I guess you plan on maintaining it yourself.

You should do an "Intention to package" bug, you do that running:

reportbug wnpp

There is a tool called "lintian" that you will probably find useful to check 
your packaging, as you get it into shape.

I'd recommend the debian-mentors IRC channel on oftc and the same named 
mailing list.

Good luck

-- 
Salvo Tomaselli

"Io non mi sento obbligato a credere che lo stesso Dio che ci ha dotato di
senso, ragione ed intelletto intendesse che noi ne facessimo a meno."
-- Galileo Galilei

https://ltworf.codeberg.page/

signature.asc
Description: This is a digitally signed message part.

Re: Recommended way of installing system-wide python application and libraries

[c . buhtz]

Hello Ivan,

I am not an expert in Python Packaging and also not in Debian-specific 
Packaging of Python stuff. So please take my advice with care. But I do 
have some experience and did a lot of experiments.


When it comes to pure Python Packaging I would suggest my demo 
repository. It is not finished yet but it explains some packaging issues 
and use cases with examples.




When this repo is "finished" and well reviewed by the Python community I 
will start a similar repo with demos about Debian Python Packaing. That 
is the plan.


The background of all that is that I have to migrate a quit old project 
("Back In Time") from a make-based build-system to modern state of the 
art Python build-system.



update is now asking that I use a venv. More details and a link to a


That is a a "new thing" with Debian 12 following PEP 668 
().

Currently myself I do struggle with this PEP in my development process.
Im the end I do use "--break-system-packages" or "export 
PIP_BREAK_SYSTEM_PACKAGES=1".
It is not a solution but a workaround until I find a better way to 
integrate PEP 668 into my workflow.


I am not sure about it but to my understanding the recommended way to 
install packages via "pip" (e.g. from PyPi) is to use "pipx" instead of 
"pip". "pipx" will handle virtuel environments in the back. But this 
will cause some other issues I haven't solved yet.


Kind
Christian

Re: Recommended way of installing system-wide python application and libraries

[Andrey Rakhmatullin]

On Tue, Dec 05, 2023 at 02:10:01AM -0800, Ivan Perez wrote:
> Hi everyone!
> 
> I'm currently trying to bring a tool we have at NASA Ames up to speed:
> https://github.com/NASA-SW-VnV/ikos
> 
> IKOS is a static analyzer for C. I'm really hoping that IKOS can be
> included in Debian in the near future.
> 
> IKOS is implemented as a  C++ library, and a number of python
> tools/wrappers. The tools call mains in modules defined in a python library
> `ikos`.
> 
> As of right now, our CMakeFiles attempt to install everything (by default)
> under /opt/ikos/.
> 
> I'm having lots of issues getting the python portions installed
> system-wide. I initially upgraded distutils to setuptools, but a recent
> update is now asking that I use a venv. More details and a link to a
> dockerfile can be found here:
> https://github.com/NASA-SW-VnV/ikos/discussions/241.
> 
> While I can hack a solution that "works" (either by making a venv under the
> target dir or by means of break-system-packages), I'd prefer to use
> recommended practices, and also conform to the way that things are done in
> Debian/Ubuntu.
> 
> What would be the recommended way of installing IKOS system-wide in Debian?
This answer may not be useful for you but the only recommended way of
installing Python modules system-wide is by making a proper Debian package
for them using proper Debian Python packaging helpers, and the only other
recommended way of installing Python modules is venvs.
If I needed to install this I would use either of these two, depending on
whether I actually need to have it system-wide and whether I want an
official package uploaded to Debian.
Alternatively, as you mentioned /opt, if the software supports installing
*everything* in /opt (maybe it adds that to sys.path when it runs), then
doing that is also fine (as it doesn't touch things outside /opt and
cannot break those).

Recommended way of installing system-wide python application and libraries

[Ivan Perez]

Hi everyone!

I'm currently trying to bring a tool we have at NASA Ames up to speed:
https://github.com/NASA-SW-VnV/ikos

IKOS is a static analyzer for C. I'm really hoping that IKOS can be
included in Debian in the near future.

IKOS is implemented as a  C++ library, and a number of python
tools/wrappers. The tools call mains in modules defined in a python library
`ikos`.

As of right now, our CMakeFiles attempt to install everything (by default)
under /opt/ikos/.

I'm having lots of issues getting the python portions installed
system-wide. I initially upgraded distutils to setuptools, but a recent
update is now asking that I use a venv. More details and a link to a
dockerfile can be found here:
https://github.com/NASA-SW-VnV/ikos/discussions/241.

While I can hack a solution that "works" (either by making a venv under the
target dir or by means of break-system-packages), I'd prefer to use
recommended practices, and also conform to the way that things are done in
Debian/Ubuntu.

What would be the recommended way of installing IKOS system-wide in Debian?

Thanks,

Ivan

PS. Apologies if this is considered off-topic for this list.