Re: Ubuntu Focal update of broken Calibre package

[Dmitry Shachnev]

Hi James!

On Tue, Dec 22, 2020 at 06:11:15PM -0700, James Cuzella wrote:
> Yes!  Qt5 is indeed a very large project too!  I'm sure there is a Lord of
> the Rings meme that applies here like: "One does not simply backport Qt5" 

This is true :)

> Simply using the backportpackage tool on qtwebengine-opensource-src started
> a build dependency hunt that resulted in finding a massive set of Qt5
> packages that needed to be backported (without relaxing version
> constraints).
> A lot of these eventually resolved into being blocked on circular
> dependency chains.
>
> I was able to spend a few hours on this to visually represent what I found
> (See Gist linked below):
>
> https://gist.github.com/b005fa0ef6e600c6a6e0dfd22dd3e604
>
> I'm not sure how the Debian or Ubuntu teams build and backport Qt5 packages
> when so many build dependencies resolve circularly.
> I suppose they must bootstrap this dependency chain somehow, but it seems
> like more work than I have time to figure out at the moment.

Yes, in Ubuntu we bootstrap the packages manually for every new Qt release.

If you build packages locally, you can follow the instruction in qtbase's
debian/README.source (I have just pushed a minor update for it):

https://salsa.debian.org/qt-kde-team/qt/qtbase/-/blob/master/debian/README.source#L16

If you are pushing packages to a PPA, you can use the following procedure:

- Push qtbase and qtdeclarative where the -doc and -doc-html packages, and
also all Build-Depends-Indep are removed.

- Push qttools with the same change, and additionally remove libqt5webkit5-dev
build-dependency and make qwebview_archs empty in debian/rules.

- Wait until qttools builds on amd64, then push the normal versions of all
packages.

In Debian this bootstrapping happens automatically because Debian does
separate builds for "amd64" (arch-dependent) and "all" (arch-indep)
architectures.

Also keep in mind that if you update Qt, you will also need to do no-change
rebuilds of packages that depend on qtbase-abi-5-x-y virtual package or on
qtdeclarative-abi-5-x-y (usually these are packages that use private ABI).
Otherwise they can stop working.

--
Dmitry Shachnev


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[James Cuzella]

Thanks Dmitry for pointing me in the right direction!

I tried to backport the set of Qt5 and qtwebengine packages and found out
what large set of build dependencies it was!

So what I said is true: this requires simultaneous changes in many packages,
> even if it's just a rebuild for some of them.
>
>
Yes!  Qt5 is indeed a very large project too!  I'm sure there is a Lord of
the Rings meme that applies here like: "One does not simply backport Qt5" 

Simply using the backportpackage tool on qtwebengine-opensource-src started
a build dependency hunt that resulted in finding a massive set of Qt5
packages that needed to be backported (without relaxing version
constraints).
A lot of these eventually resolved into being blocked on circular
dependency chains.

I was able to spend a few hours on this to visually represent what I found
(See Gist linked below):

https://gist.github.com/b005fa0ef6e600c6a6e0dfd22dd3e604


I'm not sure how the Debian or Ubuntu teams build and backport Qt5 packages
when so many build dependencies resolve circularly.
I suppose they must bootstrap this dependency chain somehow, but it seems
like more work than I have time to figure out at the moment.

Any ideas for how to simplify this?

Thanks,
- James Cuzella

On Sun, Dec 13, 2020 at 10:16 AM Dmitry Shachnev  wrote:

> Hi James!
>
> On Fri, Dec 11, 2020 at 08:43:16PM -0700, James Cuzella wrote:
> > I would appreciate any help with figuring out how to get that
> > last python3-pyqt5.qtwebengine package to be generated from the pyqt5
> > source package.  I'm assuming this is where it comes from, given that all
> > the other similarly named ones were generated from that one.  Also, the
> > Debian package site shows this as the source package:
> > https://packages.debian.org/stretch/python3-pyqt5.qtwebengine
>
> It used to be built from pyqt5 source, but since then it got a new source
> package, pyqt5webengine:
>
> https://launchpad.net/ubuntu/+source/pyqt5webengine
>
> https://packages.debian.org/sid/source/pyqt5webengine
>
> --
> Dmitry Shachnev
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Dmitry Shachnev]

Hi James!

On Fri, Dec 11, 2020 at 08:43:16PM -0700, James Cuzella wrote:
> I would appreciate any help with figuring out how to get that
> last python3-pyqt5.qtwebengine package to be generated from the pyqt5
> source package.  I'm assuming this is where it comes from, given that all
> the other similarly named ones were generated from that one.  Also, the
> Debian package site shows this as the source package:
> https://packages.debian.org/stretch/python3-pyqt5.qtwebengine

It used to be built from pyqt5 source, but since then it got a new source
package, pyqt5webengine:

https://launchpad.net/ubuntu/+source/pyqt5webengine

https://packages.debian.org/sid/source/pyqt5webengine

--
Dmitry Shachnev


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[James Cuzella]

Hello,

I recently wanted to test out Calibre with a newer Kindle Fire tablet. It
turns out that I ran into many of the issues mentioned with the
broken 4.99.4+dfsg+really4.12.0-1build1 package version.  I was able to get
this old version of Calibre working to identify the tablet on Focal with an
updated version of libmtp.  This is in my PPA here:
https://launchpad.net/~trinitronx/+archive/ubuntu/calibre

So, knowing some packaging tricks for Ubuntu & Debian, I decided to try and
create an updated Calibre package in a PPA for Focal / 20.04.1.  I ran into
the issue of first needing to backport all the build dependencies.  I
started out by using the same version of Calibre ( 5.6.0+dfsg ) from the
source package in Ubuntu groovy, and then after reading this thread trying
to backport the versions from Debian bullseye & sid.  So far I've been able
to backport the following:


   - pyqt-builder  1.4.0+dfsg-1 and 1.6.0+dfsg-1
   - sip4  4.19.24+dfsg-1
   - sip5  5.3.0+dfsg-2  and 5.5.0+dfsg-1
   - pyqt5-sip 12.8.1-1

Note: I also backported debhelper 13.2 in that PPA, but ended up not using
it because strict build-depends on "debhelper (=12)" is on most of the
other packages in Focal, and Soyuz / pbuilder will fail on that build
dependency.  Instead, I just switched back to "debhelper (=12)" in the
source package's debian/control files.
These are in this PPA:
https://launchpad.net/~trinitronx/+archive/ubuntu/focal-backport-buildeps


> Most packages I mentioned will need a rebuild and a patch to use a different
> location for PyQt5 *.sip files.
>
>
This explains exactly the issues I ran into for the main Calibre package! 
So far I have tried two routes to resolve the build dependency:


   - sip4 4.19.24+dfsg-1 + pyqt-builder 1.4.0+dfsg-1
  - Patched Calibre's setup.py to use Python3
  - Patched Calibre's setup/build.py to use sip-module = "sip"
  - Ran into issues with python3-sipbuilder
 - It could not find the package path, as you described:
 SIPing 3 files...
 /usr/bin/python3.8 -c import os;
 
os.chdir('/media/terabyte/src/pub/calibre-debian/calibre/build/pyqt/pictureflow');
 from sipbuild.tools.build import main; main(); --verbose
--no-make --qmake
 /usr/bin/qmake
 Querying qmake about your Qt installation...
 /usr/bin/qmake -query
 These bindings will be built: pictureflow.
 Generating the pictureflow bindings...
 -c: Unable to find file "QtWidgets/QtWidgetsmod.sip"

 - There was a problem with how
 python3-sipbuilder's "get_package_dir" function was working.
 The "sip_parts" result of the split only had 1 array item: ["sip"]
 The code discards the last array item, assuming it's a filename
 extension.
 Therefore, it creates an empty array that gets passed to:
 os.path.join()
  - sip5 5.3.0+dfsg-2  + pyqt-builder 1.4.0+dfsg-1 + pyqt5-dev
   5.14.1+dfsg-3build1
   - This option required just patching Calibre's setup.py to use Python3
  - Tried downgrading Build-Depends to pyqt5-dev (>=
  5.14.1+dfsg-3build1)
  - I became blocked at first on backporting the build dependencies for
  the pyqt5-sip / PyQt5.sip and many other pyqt5 packages.  Not
sure what the
  issue was (I've never seen or used PyQt5 or SIP before this...
seemed like
  a large rabbit hole)
   -  sip5 5.5.0+dfsg-1 + pyqt-builder / python3-pyqtbuild 1.6.0+dfsg-1
+ pyqt5-sip / python3-pyqt5.sip
12.8.1-1 + pyqt5 5.15.2+dfsg-1 (all backported)
   - Again, just added the patch Calibre's setup.py to use Python3
  - This time it's missing a package: python3-pyqt5.qtwebengine

  dpkg-checkbuilddeps: error: Unmet build dependencies:
  python3-pyqt5.qtwebengine (>= 5.12.1-4+b1)


You can see this in Debian: after I uploaded new PyQt5 (built with SIP 5) to
> unstable, immediately some packages started to fail to build from source:
>
> - https://bugs.debian.org/971173 (krita)
> - https://bugs.debian.org/971217 (python-poppler-qt5)
> - https://bugs.debian.org/971172 (veusz)
>
> And packages that were not rebuilt got runtime errors:
>
> - https://bugs.debian.org/971538 (qgis)
> - https://bugs.debian.org/970921 (calibre)
>
> So what I said is true: this requires simultaneous changes in many packages,
> even if it's just a rebuild for some of them.
>
>

I think if I or someone else here can figure out how to get
python3-pyqt5.qtwebengine to build, then Calibre may also be able to build
with this set of dependencies.  Note, I'm not an "official" Debian or
Ubuntu developer and have not gone through the onboarding / mentorship
process to become one.  So for now, these efforts are for a Focal targeted
PPA only.  However, with some help this could be the first step towards
getting an official set into focal-backports.

I would appreciate any help with figuring out how to get that
last python3-pyqt5.qtwebengine package to be generated from the pyqt5
source package. 

Re: Ubuntu Focal update of broken Calibre package

[Norbert Preining]

Hi Julian,

thanks for the guidance, but I am not sure anymore what should be done
and what Ubuntu devs prefer. There was the suggestion of only
backporting the groovy version to focal instead of jumpint to v5.

I guess most Ubuntu devs are busy with the release, but if at some point
you have a bit of air to comment on your preferred option, please let me
know, because I don't want to waste cycles (of both sides) with
preparing a version that will not make it.

Best

Norbert

On Tue, 20 Oct 2020, Julian Andres Klode wrote:
> On Fri, Oct 16, 2020 at 01:18:49AM +0900, Norbert Preining wrote:
> > Hi Lukasz,
> > 
> > so, here are a few more answers to your questions:
> > 
> > On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
> > >  * How badly is calibre broken on focal right now? Is it really
> > > unusable in its current state? Examples of how broken things are so
> > > that we can understand the situation better
> > 
> > Yes. I just installed a vm, updated to the latest version of the
> > packages of focal, installed calibre, started calibre, and got
> > 
> > norbert@ubuntu2004-vm:~$ calibre
> > Traceback (most recent call last):
> >   File "/usr/bin/calibre", line 20, in 
> > sys.exit(calibre())
> >   File "/usr/lib/calibre/calibre/gui_launch.py", line 73, in calibre
> > main(args)
> >   File "/usr/lib/calibre/calibre/gui2/main.py", line 543, in main
> > listener = create_listener()
> >   File "/usr/lib/calibre/calibre/gui2/main.py", line 514, in create_listener
> > return Listener(address=gui_socket_address())
> >   File "/usr/lib/calibre/calibre/utils/ipc/server.py", line 110, in __init__
> > self._listener._unlink.cancel()
> > AttributeError: 'NoneType' object has no attribute 'cancel'
> > norbert@ubuntu2004-vm:~$ vim
> > 
> > So well, it is completely useless.
> > 
> > This can also be seen by the list of upstream bugs reported
> > https://bugs.launchpad.net/bugs/1899700
> > https://bugs.launchpad.net/bugs/1899674
> > https://bugs.launchpad.net/bugs/1899355
> > https://bugs.launchpad.net/bugs/1899035
> > https://bugs.launchpad.net/bugs/1899029
> > https://bugs.launchpad.net/bugs/1898940
> > https://bugs.launchpad.net/bugs/1898904
> > 
> > 
> > >  * How does the automated test coverage on calibre look like? Do all
> > > new features come with unit testing? What about autopkgtests (I don't
> > > think I see any?)?
> > 
> > There are not autopkgtests, but there is an extensive test suite built
> > into calibre.
> > 
> > >  * What would be the acceptance criteria for the new version? What
> > > testing should be performed to make sure the new version works as
> > > expected and doesn't regress any existing users (assuming calibre in
> > > focal right now is at least usable to some extent)
> > 
> > Since it does not even start, I guess there is no regression for
> > focal users, only for those upgrading from a previous release.
> > 
> > 
> > Together with YOKOTA Hiroshi (in Cc), who has done most of the work on
> > recent packaging, I have prepared a version for focal (SIP4, debhelper
> > 12), built it on my focal machine, and successfully run it. Source and
> > amd64 packages are available here:
> > 
> > deb http://www.preining.info/debian focal main
> > deb-src http://www.preining.info/debian focal main
> > 
> > (signed with my gpg key https://www.preining.info/rsa.asc)
> > 
> > What are the next steps you are expecting from me?
> > 
> > - prepare a package for groovy and separately for focal?
> 
> Yes.
> 
> > - what are the version numbers you want to see?
> 
> 
> If it's based on e.g.  5.3.0+dfsg-1, you want
> 
> 5.3.0+dfsg-1ubuntu0.20.04.1 for focal
> 5.3.0+dfsg-1ubuntu0.20.10.1 for groovy
> 
> or
> 
> 5.3.0+dfsg-1~ubuntu<20.04.1/20.10.1> would work
> too, but I think the first version is better.
> 
> > - how should we proceed?
> 
> Open - or repurpose an existing bug - with the SRU template:
> 
> https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template
> 
> e.g.
> 
>   [Impact]
>   calibre does not start, is an unstable pre-release with a lots of bugs
>   [Test case]
>   calibre should start and be functional
>   [Regression potential]
>   It did not start before, so it can hardly regress
>   [Other info]
>   RT members agreed in ... that we can update to the
>   latest release.
> 
>   Attached are debdiffs against the version  in Debian.
> 
> with a bit nicer wording and more details :D
> 
> And close it in the changelog with LP: #thatbugnumber.
> 
> Then attach a debdiff against the debian version to the bug
> report for each focal and groovy, and subscribe ~ubuntu-sponsors.
> 
> -- 
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer  i speak de, en
> 

--
PREINING Norbert  https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

-- 
ubuntu-devel mailing list

Re: Ubuntu Focal update of broken Calibre package

[Julian Andres Klode]

On Fri, Oct 16, 2020 at 01:18:49AM +0900, Norbert Preining wrote:
> Hi Lukasz,
> 
> so, here are a few more answers to your questions:
> 
> On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
> >  * How badly is calibre broken on focal right now? Is it really
> > unusable in its current state? Examples of how broken things are so
> > that we can understand the situation better
> 
> Yes. I just installed a vm, updated to the latest version of the
> packages of focal, installed calibre, started calibre, and got
> 
> norbert@ubuntu2004-vm:~$ calibre
> Traceback (most recent call last):
>   File "/usr/bin/calibre", line 20, in 
> sys.exit(calibre())
>   File "/usr/lib/calibre/calibre/gui_launch.py", line 73, in calibre
> main(args)
>   File "/usr/lib/calibre/calibre/gui2/main.py", line 543, in main
> listener = create_listener()
>   File "/usr/lib/calibre/calibre/gui2/main.py", line 514, in create_listener
> return Listener(address=gui_socket_address())
>   File "/usr/lib/calibre/calibre/utils/ipc/server.py", line 110, in __init__
> self._listener._unlink.cancel()
> AttributeError: 'NoneType' object has no attribute 'cancel'
> norbert@ubuntu2004-vm:~$ vim
> 
> So well, it is completely useless.
> 
> This can also be seen by the list of upstream bugs reported
> https://bugs.launchpad.net/bugs/1899700
> https://bugs.launchpad.net/bugs/1899674
> https://bugs.launchpad.net/bugs/1899355
> https://bugs.launchpad.net/bugs/1899035
> https://bugs.launchpad.net/bugs/1899029
> https://bugs.launchpad.net/bugs/1898940
> https://bugs.launchpad.net/bugs/1898904
> 
> 
> >  * How does the automated test coverage on calibre look like? Do all
> > new features come with unit testing? What about autopkgtests (I don't
> > think I see any?)?
> 
> There are not autopkgtests, but there is an extensive test suite built
> into calibre.
> 
> >  * What would be the acceptance criteria for the new version? What
> > testing should be performed to make sure the new version works as
> > expected and doesn't regress any existing users (assuming calibre in
> > focal right now is at least usable to some extent)
> 
> Since it does not even start, I guess there is no regression for
> focal users, only for those upgrading from a previous release.
> 
> 
> Together with YOKOTA Hiroshi (in Cc), who has done most of the work on
> recent packaging, I have prepared a version for focal (SIP4, debhelper
> 12), built it on my focal machine, and successfully run it. Source and
> amd64 packages are available here:
> 
>   deb http://www.preining.info/debian focal main
>   deb-src http://www.preining.info/debian focal main
> 
> (signed with my gpg key https://www.preining.info/rsa.asc)
> 
> What are the next steps you are expecting from me?
> 
> - prepare a package for groovy and separately for focal?

Yes.

> - what are the version numbers you want to see?


If it's based on e.g.  5.3.0+dfsg-1, you want

5.3.0+dfsg-1ubuntu0.20.04.1 for focal
5.3.0+dfsg-1ubuntu0.20.10.1 for groovy

or

5.3.0+dfsg-1~ubuntu<20.04.1/20.10.1> would work
too, but I think the first version is better.

> - how should we proceed?

Open - or repurpose an existing bug - with the SRU template:

https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

e.g.

  [Impact]
  calibre does not start, is an unstable pre-release with a lots of bugs
  [Test case]
  calibre should start and be functional
  [Regression potential]
  It did not start before, so it can hardly regress
  [Other info]
  RT members agreed in ... that we can update to the
  latest release.

  Attached are debdiffs against the version  in Debian.

with a bit nicer wording and more details :D

And close it in the changelog with LP: #thatbugnumber.

Then attach a debdiff against the debian version to the bug
report for each focal and groovy, and subscribe ~ubuntu-sponsors.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Norbert Preining]

Hi Lukasz,

so, here are a few more answers to your questions:

On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
>  * How badly is calibre broken on focal right now? Is it really
> unusable in its current state? Examples of how broken things are so
> that we can understand the situation better

Yes. I just installed a vm, updated to the latest version of the
packages of focal, installed calibre, started calibre, and got

norbert@ubuntu2004-vm:~$ calibre
Traceback (most recent call last):
  File "/usr/bin/calibre", line 20, in 
sys.exit(calibre())
  File "/usr/lib/calibre/calibre/gui_launch.py", line 73, in calibre
main(args)
  File "/usr/lib/calibre/calibre/gui2/main.py", line 543, in main
listener = create_listener()
  File "/usr/lib/calibre/calibre/gui2/main.py", line 514, in create_listener
return Listener(address=gui_socket_address())
  File "/usr/lib/calibre/calibre/utils/ipc/server.py", line 110, in __init__
self._listener._unlink.cancel()
AttributeError: 'NoneType' object has no attribute 'cancel'
norbert@ubuntu2004-vm:~$ vim

So well, it is completely useless.

This can also be seen by the list of upstream bugs reported
https://bugs.launchpad.net/bugs/1899700
https://bugs.launchpad.net/bugs/1899674
https://bugs.launchpad.net/bugs/1899355
https://bugs.launchpad.net/bugs/1899035
https://bugs.launchpad.net/bugs/1899029
https://bugs.launchpad.net/bugs/1898940
https://bugs.launchpad.net/bugs/1898904


>  * How does the automated test coverage on calibre look like? Do all
> new features come with unit testing? What about autopkgtests (I don't
> think I see any?)?

There are not autopkgtests, but there is an extensive test suite built
into calibre.

>  * What would be the acceptance criteria for the new version? What
> testing should be performed to make sure the new version works as
> expected and doesn't regress any existing users (assuming calibre in
> focal right now is at least usable to some extent)

Since it does not even start, I guess there is no regression for
focal users, only for those upgrading from a previous release.


Together with YOKOTA Hiroshi (in Cc), who has done most of the work on
recent packaging, I have prepared a version for focal (SIP4, debhelper
12), built it on my focal machine, and successfully run it. Source and
amd64 packages are available here:

deb http://www.preining.info/debian focal main
deb-src http://www.preining.info/debian focal main

(signed with my gpg key https://www.preining.info/rsa.asc)

What are the next steps you are expecting from me?

- prepare a package for groovy and separately for focal?
- what are the version numbers you want to see?
- how should we proceed?

Best

Norbert

--
PREINING Norbert  https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Eli Schwartz]

On 10/14/20 7:29 AM, Dmitry Shachnev wrote:
> Hi Norbert, Lukasz and all!
> 
> Unfortunately, Calibre 5.x requires SIP 5 and PyQt5 that is built 
> against SIP 5. Moving PyQt5 to the new SIP is a major transition, 
> that happened in Debian recently (two weeks ago) and did not happen 
> in Ubuntu yet because of freeze.
> 
> This requires changes in many packages simultaneously: at least 
> pyqt5, pyqt5charts, pyqt5webengine, qscintilla2, calibre, 
> python-poppler-qt5, veusz, krita and qgis.
> 
> I am planning to land this change early in Groovy+1 cycle.

Just for the record, this is untrue.

Arch Linux has built python3 PyQt5 using Sip 5 via /usr/bin/sip-build,
since Dec 13 18:01:34 2019 while continuing to build python2 PyQt5 using
Sip 4 via python2 configure.py

This worked well enough that even though on Nov 24 20:33:38 2019 I began
shipping multiple repository packages for calibre -- one "calibre"
package built with python2 and one "calibre-python3" package built with
python3 -- both using Sip 4, they worked fine. The calibre-python3
package was, as expected, buggy due to being beta quality, but it never
failed due to pyqt5/sip itself.

There are still various packages in our distro archives building with
Sip 4 but successfully using the pyqt5 bindings built using Sip 5.

Old versions of some of those packages (at least krita, qgis) did need
patches to change the location of the sip dir.
qscintilla2 did need to be rebuilt with no changes, then a week later
the 2.11.4 update moved to sip-build.

It's plainly possible to mix them at least a little. From memory, we did
not even need to rebuild (most of) the packages.

YMMV, but it should definitely be feasible to update pyqt5/sip5, test
everything that build-depends on them, and leave many of them alone if
they're not ready to move.

>> Is the above (update to 5.2.0) possible in Ubuntu Focal, and if 
>> yes, what kind if steps are necessary?
> 
> So for Focal and Groovy we need a version of Calibre that still uses 
> SIP 4. Last such version in Debian was 4.99.12+dfsg+really4.23.0-1, 
> Groovy already has that. If you know some specific fixes, maybe they 
> can be applied on top of what Focal or Groovy has.

Even if you could solve the immediate, critical crashes, calibre +
python3 is a combination that is only beta quality until 5.0.0, as many
issues were found during the final stages of the beta.

I strongly advise Ubuntu Focal to NOT ship beta-quality code that
upstream never released and refuses to sign off on, for the next 5 years
as an LTS. It is a recipe for heartbreak and bad relationships with
upstream.
It was never intended to be present in a stable, frozen distro release
anywhere.

In recent times, the Debian package for calibre has gained a lot of
polish, and gotten rid of legacy baggage from the "bad old days" where
upstream simply told all users that Debian was the main reason users
should never use distro packages. I'm kind of hoping that Ubuntu doesn't
become a new reason for upstream to begrudge distros.

...

Possible options going forward:

very much not ideal:

- reinstate a stable legacy python2 build for Ubuntu (needs various
  python2 deps re-added)
- dropping calibre from the archives
- doing nothing: keep a broken package that crashes on startup and
  drives upstream nuts
- cherry-picking some specific fix: keep a partially broken beta package
  that crashes at odd, unpredictable times, and drives upstream
  occasionally nuts

ideal:

- getting a Sip 5 collection in place which still supports some packages
  stuck on Sip 4, and package calibre 5.2.0

not great, but workable:

- Revert code in calibre to make it build with Sip 4 again, and package
  calibre 5.2.0:

https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025


...

I suspect that last option would be the easiest resolution. It should work.

-- 
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User



signature.asc
Description: OpenPGP digital signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Norbert Preining]

Hi Lukasz, Dmitry, Eli, all

thanks for your messages and interest and support. Sorry for the late
replies, time zones (I am Japan based).

On Wed, 14 Oct 2020, Lukasz Zemczak wrote:
> With my SRU team hat on, after reading what you said about the status
> of current calibre, I would say it is possible to update the package
> version in focal to 5.2.0. But we would need to know a bit more, and
> there is some (important) paperwork to do.

Sounds like a good idea.

On Wed, 14 Oct 2020, Dmitry Shachnev wrote:
> Tomorrow is final freeze, so it is really bad timing for this.

Indeed.

> > - Revert code in calibre to make it build with Sip 4 again, and package
> >   calibre 5.2.0:
> >
> > https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025
> >
> > I suspect that last option would be the easiest resolution. It should work.
> 
> That would be the easiest option, yes.
> 
> I don't volunteer to work on this (no time, sorry), but I can sponsor an
> upload if someone gets a feature freeze exception for this and prepares/tests
> the upload.

I think I can prepare a package based on the Debian calibre 5.2.0
version in sid/testing, that reverts the Sip5 changes.

I will install a focal vm and see if building works out of the box.

If that is done, I will contact you again for further steps.

Is this ok a plan for you or do you want/need anything else in this
area right now?

All the best

Norbert

--
PREINING Norbert  https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Dmitry Shachnev]

Hi Eli!

On Wed, Oct 14, 2020 at 11:52:44AM -0400, Eli Schwartz wrote:
> > This requires changes in many packages simultaneously: at least 
> > pyqt5, pyqt5charts, pyqt5webengine, qscintilla2, calibre, 
> > python-poppler-qt5, veusz, krita and qgis.
> >
> > I am planning to land this change early in Groovy+1 cycle.
>
> Just for the record, this is untrue.
>
> Arch Linux has built python3 PyQt5 using Sip 5 via /usr/bin/sip-build,
> since Dec 13 18:01:34 2019 while continuing to build python2 PyQt5 using
> Sip 4 via python2 configure.py
>
> This worked well enough that even though on Nov 24 20:33:38 2019 I began
> shipping multiple repository packages for calibre -- one "calibre"
> package built with python2 and one "calibre-python3" package built with
> python3 -- both using Sip 4, they worked fine. The calibre-python3
> package was, as expected, buggy due to being beta quality, but it never
> failed due to pyqt5/sip itself.
>
> There are still various packages in our distro archives building with
> Sip 4 but successfully using the pyqt5 bindings built using Sip 5.
>
> Old versions of some of those packages (at least krita, qgis) did need
> patches to change the location of the sip dir.
> qscintilla2 did need to be rebuilt with no changes, then a week later
> the 2.11.4 update moved to sip-build.
>
> It's plainly possible to mix them at least a little. From memory, we did
> not even need to rebuild (most of) the packages.

Most packages I mentioned will need a rebuild and a patch to use a different
location for PyQt5 *.sip files.

You can see this in Debian: after I uploaded new PyQt5 (built with SIP 5) to
unstable, immediately some packages started to fail to build from source:

- https://bugs.debian.org/971173 (krita)
- https://bugs.debian.org/971217 (python-poppler-qt5)
- https://bugs.debian.org/971172 (veusz)

And packages that were not rebuilt got runtime errors:

- https://bugs.debian.org/971538 (qgis)
- https://bugs.debian.org/970921 (calibre)

So what I said is true: this requires simultaneous changes in many packages,
even if it's just a rebuild for some of them.

> YMMV, but it should definitely be feasible to update pyqt5/sip5, test
> everything that build-depends on them, and leave many of them alone if
> they're not ready to move.

Tomorrow is final freeze, so it is really bad timing for this.

> not great, but workable:
>
> - Revert code in calibre to make it build with Sip 4 again, and package
>   calibre 5.2.0:
>
> https://github.com/kovidgoyal/calibre/commit/7a4b3f61ff24f8c39c8d5cf86c54da9de9267025
>
> I suspect that last option would be the easiest resolution. It should work.

That would be the easiest option, yes.

I don't volunteer to work on this (no time, sorry), but I can sponsor an
upload if someone gets a feature freeze exception for this and prepares/tests
the upload.

--
Dmitry Shachnev

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Dmitry Shachnev]

Hi Norbert, Lukasz and all!

On Wed, Oct 14, 2020 at 09:06:35AM +0900, Norbert Preining wrote:
> Dear all,
>
> (please Cc)
>
> I am the Debian maintainer of Calibre, and unfortunately it seems that
> for Focal Ubuntu has pulled a preliminary version of Calibre, which is
> **seriously** broken and unusable, not even starting in most cases.
>
> We were forced by the Python3 transition to temporarily ship pre-release
> versions of Calibre. In particular, Ubuntu Focal ships
>   4.99.4+dfsg+really4.12.0-1build1
> which is version 4.12 with experimental Python3 patches on top of it.
> This worked for a short time being until Calibre 5 was released with
> proper Python3 support.
>
> Due to this unfortunate squeeze in release timing, Ubuntu Focal users
> now have a seriously broken Calibre, and upstream is swamped with bug
> reports.
>
> I would strongly suggest and support, and help preparing, an update to
> Focal based on the current version in Debian/testing, 5.2.0+dfsg-1,
> which has been out since quite some time and field-tested with Python3
> in various environments, due to upstream having switched to Py3, too.

Unfortunately, Calibre 5.x requires SIP 5 and PyQt5 that is built against
SIP 5. Moving PyQt5 to the new SIP is a major transition, that happened
in Debian recently (two weeks ago) and did not happen in Ubuntu yet because
of freeze.

This requires changes in many packages simultaneously: at least pyqt5,
pyqt5charts, pyqt5webengine, qscintilla2, calibre, python-poppler-qt5,
veusz, krita and qgis.

I am planning to land this change early in Groovy+1 cycle.

> Is the above (update to 5.2.0) possible in Ubuntu Focal, and if yes,
> what kind if steps are necessary?

So for Focal and Groovy we need a version of Calibre that still uses SIP 4.
Last such version in Debian was 4.99.12+dfsg+really4.23.0-1, Groovy already
has that. If you know some specific fixes, maybe they can be applied on top
of what Focal or Groovy has.

--
Dmitry Shachnev


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal update of broken Calibre package

[Lukasz Zemczak]

Hello Norbert,

With my SRU team hat on, after reading what you said about the status
of current calibre, I would say it is possible to update the package
version in focal to 5.2.0. But we would need to know a bit more, and
there is some (important) paperwork to do.

First thing to remember is that for a new version to be releasable to
a stable series it also has to be present in all the newer series as
well - so groovy upwards. We are long past debian import freeze so
groovy is still on 4.99.12+dfsg+really4.23.0-1 - and syncing 5.2.0
now, so late in the cycle, requires an approved Feature Freeze
Exception [1], since we're also past feature freeze. It's a very very
unfortunate time, since Final Freeze for groovy is tomorrow, and we're
a bit reluctant with accepting risky pieces. The 'good' news is,
calibre is only seeded in our Ubuntu Studio flavor, so the impact
should be manageable. I would like the Ubuntu Studio flavor
representatives to chip in and (maybe) help out with the paperwork
there.

Once we have it in groovy+ we can look into backporting it to focal.
This requires filling in some SRU paperwork as per the policy [2].
What makes it a bit problematic from the SRU perspective is that the
debdiff between 4.99.4+dfsg+really4.12.0-1build1 (in focal) and 5.2.0
is 7600433 lines long. Looking at the diff itself, most of it are
translation changes, but still... it's quite a change nevertheless.
The things that need to be thought of before performing the backport:
 * How badly is calibre broken on focal right now? Is it really
unusable in its current state? Examples of how broken things are so
that we can understand the situation better
 * How does the automated test coverage on calibre look like? Do all
new features come with unit testing? What about autopkgtests (I don't
think I see any?)?
 * What would be the acceptance criteria for the new version? What
testing should be performed to make sure the new version works as
expected and doesn't regress any existing users (assuming calibre in
focal right now is at least usable to some extent)

If needed, we can help a bit with some of the SRU bits.

Cheers,

[1] 
https://wiki.ubuntu.com/FreezeExceptionProcess#FeatureFreeze_for_new_upstream_versions
[2] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

On Wed, 14 Oct 2020 at 10:20, Norbert Preining  wrote:
>
> Dear all,
>
> (please Cc)
>
> I am the Debian maintainer of Calibre, and unfortunately it seems that
> for Focal Ubuntu has pulled a preliminary version of Calibre, which is
> **seriously** broken and unusable, not even starting in most cases.
>
> We were forced by the Python3 transition to temporarily ship pre-release
> versions of Calibre. In particular, Ubuntu Focal ships
> 4.99.4+dfsg+really4.12.0-1build1
> which is version 4.12 with experimental Python3 patches on top of it.
> This worked for a short time being until Calibre 5 was released with
> proper Python3 support.
>
> Due to this unfortunate squeeze in release timing, Ubuntu Focal users
> now have a seriously broken Calibre, and upstream is swamped with bug
> reports.
>
> I would strongly suggest and support, and help preparing, an update to
> Focal based on the current version in Debian/testing, 5.2.0+dfsg-1,
> which has been out since quite some time and field-tested with Python3
> in various environments, due to upstream having switched to Py3, too.
>
> Is the above (update to 5.2.0) possible in Ubuntu Focal, and if yes,
> what kind if steps are necessary?
>
> Note that I am not an Ubuntu developers, but Debian developer and
> maintainer of Calibre.
>
> Thanks and all the best
>
> Norbert
>
> --
> PREINING Norbert  https://www.preining.info
> Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
Łukasz 'sil2100' Zemczak
 Foundations Team
 lukasz.zemc...@canonical.com
 www.canonical.com

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel