Re: Packaging new version of ZODB (Zope Object Database)

[Julien Muchembled]

Updated status of ZODB packaging.

I didn't write it in my previous mail because there was still a lot time before 
deadlines but now, it's becoming short. I'd like to see ZODB updated for 
Stretch, at least that's what I had in mind when I started months ago. Having a 
very old version of ZODB until 2019, without even support for python 3, would 
be frustrating.

I am actually looking for sponsors who could upload them in time. They all 
should be in good state. Maybe Arnaud will be available again for the last 
uploads.

Here is the dependency graph of packages that need(ed) to be updated (I forgot 
to mention a few of them in my previous package):

zdaemon -> python-zc.customdoctests, zconfig
zodb -> python-persistent, transaction, zodbpickle, python-btrees, zconfig, 
zc.lockfile (bin-new)
zeo -> zodb, zdaemon
(+ zodb recommending zeo)

Packages would be uploaded in 3 rounds:
(1) python-zc.customdoctests (new), zconfig (bin-new)
(2) python-persistent, transaction, zodbpickle (new), python-btrees (new), 
zc.lockfile, zdaemon (bin-new)
(3) zodb (bin-new), zeo (new)

(1) is done already.

(2) 3 are on mentors.debian.net
  https://mentors.debian.net/package/python-btrees (RFS #847609)
  https://mentors.debian.net/package/python-persistent
  https://mentors.debian.net/package/zodbpickle (RFS #847608)
  
transaction, zc.lockfile and zdaemon are also finished on my side, but I guess 
the priority is the new packages.

They all should be in good state.

(3) So that zodb/zeo are uploaded and accepted before the 26th


About VCS, it's too late to think about it now. I'm interested in this topic so 
I'll be happy to work on this once ZODB is updated. For the moment:
- I followed blindly the policy for the new 
python-zc.customdoctests/zodbpickle/python-btrees packages
- I stay on svn://anonscm.debian.org/pkg-zope/ for the packages of the Zope 
team, and for the new 'zeo'.


Regards,
Julien

gitignore and import of upstream tarballs (was Re: Packaging new version of ZODB (Zope Object Database))

[Julien Muchembled]

Le 12/05/16 à 20:21, Julien Muchembled a écrit :
> So I plan to create only 3 new packages in DPMT:
> - python-btrees
> - python-zc.customdoctests
> - zodbpickle

I am also preparing an update for python-persistent.

But for all that, I followed https://wiki.debian.org/Python/GitPackaging and 
there's no indication about gitignore'd files and the *.egg-info directory.

1. Both upstream repositories have a .gitignore file at the root (ignoring 
*.egg-info among other files) but on PyPI:
- BTrees-4.3.1.tar.gz contains it
- persistent-4.2.2.tar.gz does not contain it

I also have a ~/.gitignore file that ignores *.egg-info/ anyway.

2. The last release of zc.customdoctests is quite old and the tarball contain a 
egg-info folder that differs slightly to what the new setuptools generates, 
which is the source of bugs like #825921

Having generated files in Git is not great.


1 thing at a time so just simple questions:
- Should the 'Creating a new package' paragraph on the wiki says 'git add -f .' 
instead of 'git add .' ?
- It's quite obvious that ~/.gitignore should be ignored, but is it also the 
case for the one that may be in the tarball ?
- Do we always want *.egg-info directories in Git ?

Meanwhile, the wiki should also recommend to add the following line to 
debian/source/options:
  extend-diff-ignore="\.egg-info/"

Julien



signature.asc
Description: OpenPGP digital signature

Re: Packaging new version of ZODB (Zope Object Database)

[Julien Muchembled]

Le 11/04/16 à 16:15, Scott Kitterman a écrit :
> On Friday, November 04, 2016 10:47:32 AM Barry Warsaw wrote:
>> On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:
>>> Not sure if all python-modules repositories are like persistent, but for
>>> me, mixing Debian work with imported tarballs in the same branch is
>>> terrible. When possible, I prefer to fork the upstream repository,
>>> otherwise no upstream source at all.
>>
>> You're not alone, but I think that's still a minority opinion in the team.
>> Our packages are so tightly integrated with PyPI, and source tarballs are
>> such an ingrained aspect of that service, that a pristine-tar based
>> approach for team packages still makes sense, IMHO.
> 
> You can integrate a new upstream directly from an upstream git repository 
> with 
> git-dpm if that's what makes sense for a particular upstream.

That's tempting and I was about to ask how, but I doubt the team would accept. 
The Zope foundation already releases everything on PyPI and the policy says: « 
Complete upstream git history should be avoided in the upstream branch. [...] 
When you must (not want to) deviate, »

So I plan to create only 3 new packages in DPMT:
- python-btrees
- python-zc.customdoctests
- zodbpickle


Packages maintained by the Debian/Ubuntu Zope Team, like zodb, will stay where 
there are (since I prefer git-svn+source-less), and I'll also create 'zeo' 
there (because it depends on zodb).


https://wiki.debian.org/Python/GitPackaging:
> Q: Source-full or source-less branches? A: Source-full branches please! There 
> are lots of good reasons for this, including the ability to easily diff 
> between upstream versions, [...]

Such source-full tree is unusable for me. I need git-blame too and 
auto-generated files are annoying.


Julien

Re: DPMT Membership request​ (was Packaging new version of ZODB (Zope Object Database))

[Piotr Ożarowski]

[Julien Muchembled, 2016-11-30]
> I see that many people request membership on the ML. I did it on the
> Alioth interface and it looks like my request was lost.

yeah, that's the process described in our policy. I don't know how to
disable Alioth/FusionForge join button (I even tried to set project's
visibility to false for all "Any user logged in" roles... it didn't help)

> There's already this whole thread explaining what I'm doing.
> 
> My Alioth login is jmuchemb-guest.
> I have read https://python-modules.alioth.debian.org/policy.html and accept 
> it.

I added you to the team, welcome :)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

DPMT Membership request​ (was Packaging new version of ZODB (Zope Object Database))

[Julien Muchembled]

Hello,

Le 11/01/16 à 20:10, Julien Muchembled a écrit :
> I am co-maintaining the 'zodb' package and I am also an upstream developer.
>   http://www.zodb.org/ - a native object database for Python
> 
> Debian currently packages branch 3.x of ZODB, which is getting quite old, 
> with only support for Python 2. It's been a while that I am working locally 
> in packaging a newer version, and I have almost finished.
> 
> With branch 4.x, ZODB has been splitted into 4 packages (-> means Depends):
>  ZEO -> ZODB -> (Btrees, persistent)
> The split of persistent was already backported.
> 
> And in addition to that:
> - new dependencies that are not packaged yet
> - some existing dependencies needs to be updated as well
> 
> I write to debian-python, because some of the involved packages are not 
> specific to Zope. Actually, I even think that ZODB itself is not specific to 
> Zope, but well, changing section of existing packages can be another topic. 
> I've just requested to join the Python Modules Packaging Team.

I see that many people request membership on the ML. I did it on the Alioth 
interface and it looks like my request was lost.

There's already this whole thread explaining what I'm doing.

My Alioth login is jmuchemb-guest.
I have read https://python-modules.alioth.debian.org/policy.html and accept it.

Greetings,
Julien

Re: Packaging new version of ZODB (Zope Object Database)

[Scott Kitterman]

On Friday, November 04, 2016 10:47:32 AM Barry Warsaw wrote:
> On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:
> >I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not
> >like?)
> 
> git-dpm is usually pretty easy, but it's really only used in a few cases,
> such as importing a new upstream, managing the patch stack, and tagging. 
> We document the team's use cases pretty well so you don't even really have
> to remember much:
> 
> https://wiki.debian.org/Python/GitPackaging#New_upstream_release
> 
> For a lot of other package management tasks (e.g. building source packages,
> cloning, pulling, etc.), gbp works just fine.
> 
> That said, we know git-dpm has not been developed in a very long time, and
> is for all intents and purposes, abandonware.  It works, so I don't think
> there's a huge urgency to get rid of it (obviously, since we haven't ;) but
> it should be in our long-term team goals to move away from it.
> 
> >Not sure if all python-modules repositories are like persistent, but for
> >me, mixing Debian work with imported tarballs in the same branch is
> >terrible. When possible, I prefer to fork the upstream repository,
> >otherwise no upstream source at all.
> 
> You're not alone, but I think that's still a minority opinion in the team.
> Our packages are so tightly integrated with PyPI, and source tarballs are
> such an ingrained aspect of that service, that a pristine-tar based
> approach for team packages still makes sense, IMHO.

You can integrate a new upstream directly from an upstream git repository with 
git-dpm if that's what makes sense for a particular upstream.

Scott K

Re: Packaging new version of ZODB (Zope Object Database)

[Barry Warsaw]

On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:

>I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not
>like?)

git-dpm is usually pretty easy, but it's really only used in a few cases, such
as importing a new upstream, managing the patch stack, and tagging.  We
document the team's use cases pretty well so you don't even really have to
remember much:

https://wiki.debian.org/Python/GitPackaging#New_upstream_release

For a lot of other package management tasks (e.g. building source packages,
cloning, pulling, etc.), gbp works just fine.

That said, we know git-dpm has not been developed in a very long time, and is
for all intents and purposes, abandonware.  It works, so I don't think there's
a huge urgency to get rid of it (obviously, since we haven't ;) but it should
be in our long-term team goals to move away from it.

>Not sure if all python-modules repositories are like persistent, but for me,
>mixing Debian work with imported tarballs in the same branch is
>terrible. When possible, I prefer to fork the upstream repository, otherwise
>no upstream source at all.

You're not alone, but I think that's still a minority opinion in the team.
Our packages are so tightly integrated with PyPI, and source tarballs are such
an ingrained aspect of that service, that a pristine-tar based approach for
team packages still makes sense, IMHO.

Cheers,
-Barry

Re: Packaging new version of ZODB (Zope Object Database)

[Julien Muchembled]

Le 11/02/16 à 23:51, Barry Warsaw a écrit :
> On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:
>> This has  already been discussed  but all  the packages in  pkg-zope SVN
>> repository will have to be  moved to python-{modules, apps} repositories
>> (because there  is almost no activity  on pkg-zope and most  modules are
>> used independently  of Zope anyway)  and we should use  debian-python ML
>> for the  same reason,  so yes,  please use  debian-python ML  and commit
>> everything to python-{modules, apps} repositories.
> 
> +1.  I do still touch some of the ztk packages and would dearly love to ditch
> svn, but just haven't had the time to think about a proper migration.  Should
> we just admit defeat and do on-demand conversions, preserving history if
> possible but not worrying about it too much?
> 
> And then what about just using gbp and ignoring git-dpm?  The latter still
> kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is that
> a useful option?

I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not 
like?)

Not sure if all python-modules repositories are like persistent, but for me, 
mixing Debian work with imported tarballs in the same branch is terrible. When 
possible, I prefer to fork the upstream repository, otherwise no upstream 
source at all.

Actually, I already use git-svn+gbp locally for zodb (I have a debian symlink 
between the upstream and debian repos; that even works with patches), so it's 
not urgent to convert for me.

Julien

Re: Packaging new version of ZODB (Zope Object Database)

[Julien Muchembled]

Le 11/02/16 à 04:41, Paul Wise a écrit :
> On Wed, Nov 2, 2016 at 8:51 AM, Julien Muchembled wrote:
> 
>> Bug#842870: ITP: zodbpickle -- Fork of pickle module, for ZODB
> 
> Please ensure the security team are informed about this fork, via
> their embedded-code-copies file:
> 
> https://wiki.debian.org/EmbeddedCodeCopies
> 
Thanks, I didn't know. I'll do it.

BTW, about the fact that it's a fork, I'm not sure how to write debian/copyright
The upstream license is somewhat a concatenation of:
- the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 (license of the original 
code)
- and the Zope Public License (ZPL) Version 2.1 (contributions of the Zope 
Foundation)

 https://github.com/zopefoundation/zodbpickle/blob/master/LICENSE.txt

As starting point, I have the attached file, which is similar to what I have to 
python-btrees and zc.customdoctests

Julien
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: zodbpickle
Upstream-Contact: Zope Foundation and Contributors 
Source: http://pypi.python.org/pypi/zodbpickle

Files: *
Copyright: (c) 2013-2016 Zope Foundation and Contributors.
License: Zope-2.1
 Zope Public License (ZPL) Version 2.1
 .
 A copyright notice accompanies this license document that identifies the
 copyright holders.
 .
 This license has been certified as open source. It has also been designated as
 GPL compatible by the Free Software Foundation (FSF).
 .
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 .
 1. Redistributions in source code must retain the accompanying copyright
 notice, this list of conditions, and the following disclaimer.
 .
 2. Redistributions in binary form must reproduce the accompanying copyright
 notice, this list of conditions, and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 .
 3. Names of the copyright holders must not be used to endorse or promote
 products derived from this software without prior written permission from the
 copyright holders.
 .
 4. The right to distribute this software or to use it for any purpose does not
 give you the right to use Servicemarks (sm) or Trademarks (tm) of the
 copyright
 holders. Use of them is covered by separate agreement with the copyright
 holders.
 .
 5. If any files are modified, you must cause the modified files to carry
 prominent notices stating that you changed the files and the date of any
 change.
 .
 Disclaimer
 .
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED
 OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
 OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
 EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT,
 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
 EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


signature.asc
Description: OpenPGP digital signature

Re: Packaging new version of ZODB (Zope Object Database)

[Arnaud Fontaine]

Hi,

Barry Warsaw  writes:

> +1.  I do still  touch some of the ztk packages  and would dearly love
> to ditch svn,  but just haven't had  the time to think  about a proper
> migration.  Should we just admit  defeat and do on-demand conversions,
> preserving history if possible but not worrying about it too much?

I haven't  had time to  work on  the migration. On-demand  conversion is
fine with me. Is there any problem in preserving history while migrating
From SVN to Git?

Cheers,
-- 
Arnaud Fontaine


signature.asc
Description: PGP signature

Re: Packaging new version of ZODB (Zope Object Database)

[Scott Kitterman]

On November 2, 2016 6:51:56 PM EDT, Barry Warsaw  wrote:
>On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:
>
>>> I write  to debian-python, because  some of the involved  packages
>are
>>> not specific to  Zope. Actually, I even think that  ZODB itself is
>not
>>> specific to Zope, but well,  changing section of existing packages
>can
>>> be another topic.  
>>
>>This has  already been discussed  but all  the packages in  pkg-zope
>SVN
>>repository will have to be  moved to python-{modules, apps}
>repositories
>>(because there  is almost no activity  on pkg-zope and most  modules
>are
>>used independently  of Zope anyway)  and we should use  debian-python
>ML
>>for the  same reason,  so yes,  please use  debian-python ML  and
>commit
>>everything to python-{modules, apps} repositories.
>
>+1.  I do still touch some of the ztk packages and would dearly love to
>ditch
>svn, but just haven't had the time to think about a proper migration. 
>Should
>we just admit defeat and do on-demand conversions, preserving history
>if
>possible but not worrying about it too much?
>
>And then what about just using gbp and ignoring git-dpm?  The latter
>still
>kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is
>that
>a useful option?

Dgit and git-dpm are orthogonal.  I'd suggest converting the same way we did 
DPMT for now and then they'll be included in whatever we migrate to next.

Scott K

Re: Packaging new version of ZODB (Zope Object Database)

[Barry Warsaw]

On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:

>> I write  to debian-python, because  some of the involved  packages are
>> not specific to  Zope. Actually, I even think that  ZODB itself is not
>> specific to Zope, but well,  changing section of existing packages can
>> be another topic.  
>
>This has  already been discussed  but all  the packages in  pkg-zope SVN
>repository will have to be  moved to python-{modules, apps} repositories
>(because there  is almost no activity  on pkg-zope and most  modules are
>used independently  of Zope anyway)  and we should use  debian-python ML
>for the  same reason,  so yes,  please use  debian-python ML  and commit
>everything to python-{modules, apps} repositories.

+1.  I do still touch some of the ztk packages and would dearly love to ditch
svn, but just haven't had the time to think about a proper migration.  Should
we just admit defeat and do on-demand conversions, preserving history if
possible but not worrying about it too much?

And then what about just using gbp and ignoring git-dpm?  The latter still
kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is that
a useful option?

can-of-worms-ly y'rs,
-Barry


pgpgflcDONUfX.pgp
Description: OpenPGP digital signature

Re: Packaging new version of ZODB (Zope Object Database)

[Paul Wise]

On Wed, Nov 2, 2016 at 8:51 AM, Julien Muchembled wrote:

> Bug#842870: ITP: zodbpickle -- Fork of pickle module, for ZODB

Please ensure the security team are informed about this fork, via
their embedded-code-copies file:

https://wiki.debian.org/EmbeddedCodeCopies

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Packaging new version of ZODB (Zope Object Database)

[Arnaud Fontaine]

Hi,

> I write  to debian-python, because  some of the involved  packages are
> not specific to  Zope. Actually, I even think that  ZODB itself is not
> specific to Zope, but well,  changing section of existing packages can
> be another topic.

This has  already been discussed  but all  the packages in  pkg-zope SVN
repository will have to be  moved to python-{modules, apps} repositories
(because there  is almost no activity  on pkg-zope and most  modules are
used independently  of Zope anyway)  and we should use  debian-python ML
for the  same reason,  so yes,  please use  debian-python ML  and commit
everything to python-{modules, apps} repositories.

Also, let  me know  when you  want to  upload as  I can  sponsor without
problem.

Cheers,
-- 
Arnaud Fontaine


signature.asc
Description: PGP signature

Re: Packaging new version of ZODB (Zope Object Database)

[Julien Muchembled]

Le 11/01/16 à 20:10, Julien Muchembled a écrit :
> * python-btrees & python-zodbpickle (done)
> https://github.com/zopefoundation/BTrees
> https://github.com/zopefoundation/zodbpickle
> 
> - new packages
> - Debian Python Modules Team
> 
> [...]
> 
> * python-zeo (not started)
> https://github.com/zopefoundation/ZEO
> 
> [...]
> 
> * zc.customdoctests (done)
> https://github.com/zopefoundation/zc.customdoctests
> 
> - new package
> - Debian Python Modules Team
> 
> I haven't done any ITP yet.

Bug#842870: ITP: zodbpickle -- Fork of pickle module, for ZODB
Bug#842874: ITP: python-btrees -- scalable persistent object containers for 
Python
Bug#842876: ITP: zc.customdoctests -- Use Python doctest with other languages

I'll do ZEO later.

Julien

Packaging new version of ZODB (Zope Object Database)

[Julien Muchembled]

Hello,

I am co-maintaining the 'zodb' package and I am also an upstream developer.
  http://www.zodb.org/ - a native object database for Python

Debian currently packages branch 3.x of ZODB, which is getting quite old, with 
only support for Python 2. It's been a while that I am working locally in 
packaging a newer version, and I have almost finished.

With branch 4.x, ZODB has been splitted into 4 packages (-> means Depends):
 ZEO -> ZODB -> (Btrees, persistent)
The split of persistent was already backported.

And in addition to that:
- new dependencies that are not packaged yet
- some existing dependencies needs to be updated as well

I write to debian-python, because some of the involved packages are not 
specific to Zope. Actually, I even think that ZODB itself is not specific to 
Zope, but well, changing section of existing packages can be another topic. 
I've just requested to join the Python Modules Packaging Team.

So here is the status of my work:

* Generally, I add support for Python 3 everywhere it is missing. I only added 
PyPy support when it's trivial (e.g. zodbpickle), because that would need work 
on even more existing packages.

* python-zodb (almost finished)

- changelog entries to be written
- decide an upgrade plan for the split of ZEO (some users may have installed 
zodb for it): maybe a Recommends: ?
- to give a better overview, here's the new Depends: for Python2:
  python-btrees (>= 4.2.0), python-persistent (>= 4.2.0), python-six, 
python-transaction (>= 1.5.0), python-zc.lockfile, python-zconfig, 
python-zodbpickle (>= 0.6.0), python-zope.interface, python:any (<< 2.8), 
python:any (>= 2.7.5-5~)

* python-persistent (done)

- trivial update to 4.2.1
- I'd like to commit to python-persistent.git (python-modules), or someone does 
it for me

* python-btrees & python-zodbpickle (done)
https://github.com/zopefoundation/BTrees
https://github.com/zopefoundation/zodbpickle

- new packages
- Debian Python Modules Team

* python-transaction (done)

- update to version 1.6.1
- svn repos is late compared to the packaged version

* python-zeo (not started)
https://github.com/zopefoundation/ZEO

* python-zdaemon (done)

- needs to be updated for ZEO
- new deps: mock, zc.customdoctests

* zc.customdoctests (done)
https://github.com/zopefoundation/zc.customdoctests

- new package
- Debian Python Modules Team

I haven't done any ITP yet.

Julien