Re: [Twisted-Python] Deprecating subproject packaging

[HawkOwl]

> On 23 Apr 2015, at 08:01, Glyph  wrote:
> 
> One immediate thing I think we should decouple, though, are our native 
> dependencies.  IOCPReactor should remain in Twisted, but iocpsupport should 
> be a separate, optional dependency, available as an extra.  Similarly, 
> _initgroups, sendmsg, portmap, and raiser should all be optional 
> dependencies, so that Twisted itself can be a pure wheel, and you can opt in 
> to enhanced functionality by installing extras.
> 

Also worth noting, pulling twisted.python.constants out into a separate package 
has a PR open so that part of this can become reality: 
https://github.com/twisted/constantly/pull/1

- Hawkie



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Glyph]

> On Apr 22, 2015, at 09:28, Adi Roiban  wrote:
> 
 So, I think we should:
 
 - Deprecate the subproject setup.pys starting with 15.2 and stop packaging 
 subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in 
 September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 
 major release notice).
 - Remove the support code for subprojects in twisted.python._release + the 
 release scripts in the 16.0 release.
> 
> I am ok with deprecating this ... but I hope that this will not
> discourage the effort to decouple the Twisted subprojects.

There is not really any such effort underway.  However, if anything, this 
change would make such an effort easier.  If we want to release the projects 
separately, we should maintain them separately; the lesson of this "subproject" 
split is that what users want, to perceive that projects are more 
"lightweight", is a different "front door" to each project - different 
websites, different documentation indexes, different bug trackers - and not 
necessarily just a smaller download.  I'm pretty sure that if we split 
everything up but the actual releases we'd have better luck convincing people 
twisted is not hard to understand.

One immediate thing I think we should decouple, though, are our native 
dependencies.  IOCPReactor should remain in Twisted, but iocpsupport should be 
a separate, optional dependency, available as an extra.  Similarly, 
_initgroups, sendmsg, portmap, and raiser should all be optional dependencies, 
so that Twisted itself can be a pure wheel, and you can opt in to enhanced 
functionality by installing extras.

> I would like to see the subproject developed as independent
> projects... this should help enforcing the decoupling and would speed
> up the test suite.

Speeding up the test suite should be accomplished by actually fixing and 
optimizing the tests, not by chopping big sections of it off.  The amount of 
tests we have right now can run very quickly; but there are two immediate 
problems:

1. we are not leveraging twistd -j, and since we have no buildbots that run it, 
we keep making changes that break it for our test suite.  We should introduce a 
builder that runs the tests in parallel.
2. many of the tests still return Deferreds when they should not, spin the 
reactor, wait for real timeouts, and so on.  we should endeavor to reduce the 
number of real-IO tests as much as possible.

> Right now if you make a change in twisted.positioning, buildbot will
> run all tests :(

Maybe we could speed up twisted.positioning's tests without sacrificing 
quality, but we would need to continue to run twisted.positioning's tests for 
every change to twisted core regardless, even if they were separate projects, 
or we would be reducing the amount of integration coverage received by twisted 
core.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[HawkOwl]

> On 23 Apr 2015, at 01:37, Kevin Horn  wrote:
> 
> Agreed that this is a good idea, though I'd still like to see NEWS broken out 
> by "subproject" (or maybe we should say "topic" now?).  I don't care whether 
> it's in the same file or not, but when a new Twisted release comes out I find 
> reviewing changes much easier if I can think about one "topic" at a time.

Do you mean keeping the NEWS file as-is currently? I think that's a good idea. 
It's a nice, logical grouping.

I personally think that it should pruned down a little -- Twisted Runner & Pair 
don't really need their own sections, and Twisted News & Conch could probably 
be rolled into the core changelist as well. Not sure, I guess we'll come to 
that when there's tickets in review for it. :)

- Hawkie


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Kevin Horn]

Agreed that this is a good idea, though I'd still like to see NEWS broken
out by "subproject" (or maybe we should say "topic" now?).  I don't care
whether it's in the same file or not, but when a new Twisted release comes
out I find reviewing changes much easier if I can think about one "topic"
at a time.

Otherwise +100.  I'm annoyed every time I have to install a downstream
Twisted package and it isn't _all_ of Twisted.

On Wed, Apr 22, 2015 at 8:28 AM, Adi Roiban  wrote:

> >>> So, I think we should:
> >>>
> >>> - Deprecate the subproject setup.pys starting with 15.2 and stop
> packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July,
> 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases
> + 1 major release notice).
> >>> - Remove the support code for subprojects in twisted.python._release +
> the release scripts in the 16.0 release.
>
> I am ok with deprecating this ... but I hope that this will not
> discourage the effort to decouple the Twisted subprojects.
>
> I would like to see the subproject developed as independent
> projects... this should help enforcing the decoupling and would speed
> up the test suite.
>
> Right now if you make a change in twisted.positioning, buildbot will
> run all tests :(
>
> --
> Adi Roiban
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
--
Kevin Horn
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Adi Roiban]

>>> So, I think we should:
>>>
>>> - Deprecate the subproject setup.pys starting with 15.2 and stop packaging 
>>> subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in 
>>> September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 
>>> major release notice).
>>> - Remove the support code for subprojects in twisted.python._release + the 
>>> release scripts in the 16.0 release.

I am ok with deprecating this ... but I hope that this will not
discourage the effort to decouple the Twisted subprojects.

I would like to see the subproject developed as independent
projects... this should help enforcing the decoupling and would speed
up the test suite.

Right now if you make a change in twisted.positioning, buildbot will
run all tests :(

-- 
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Notice for downstream package maintainers: Subproject packages are going away

[HawkOwl]

Hi,

As discussed in the ML thread 
(http://twistedmatrix.com/pipermail/twisted-python/2015-April/029386.html), and 
with general "yes"ness, I think I should make it official -- Twisted subproject 
packages are going away.

Downstream packagers that package Twisted as multiple subprojects 
(core+[mail,conch,names,etc]) should replace them with one Twisted package, 
from the source tarball released under 
`http://twistedmatrix.com/Releases/Twisted/{VERSION}/Twisted-{VERSION}.tar.bz2`.

The roadmap for this is to completely drop the subproject packages at 16.0 at 
the latest, to give downstream packagers time to update. If the major 
distributions that package Twisted by using the subprojects[1] (FreeBSD, 
Debian, Gentoo) migrate to the one Twisted package before then, there is no use 
releasing the subproject packages and they will be dropped that release.

- Hawkie

[1] Fedora and Arch already just have the one python-twisted package.


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[HawkOwl]

> On 22 Apr 2015, at 19:46, Glyph  wrote:
> 
> 
>> On Apr 22, 2015, at 04:04, HawkOwl  wrote:
> 
>> So, I think we should:
>> 
>> - Deprecate the subproject setup.pys starting with 15.2 and stop packaging 
>> subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in 
>> September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 
>> major release notice).
>> - Remove the support code for subprojects in twisted.python._release + the 
>> release scripts in the 16.0 release.
> 
> If anything, this seems like too generous a timeline to me.  We have the 
> deprecation process to preserve API stability.  Do the subproject setup.py's 
> expose any API surface?  "from twisted.web import setup" already appears to 
> be an ImportError to me.  I don't think that the subproject release process 
> has helped anyone in a long time.  Is there any reason why we can't just drop 
> it from the next release, other than "the change might not land in time"?
> 
> One word of warning - do try to reach out and contact the package maintainers 
> directly, as they may not follow this list closely, and for some reason they 
> seem keen on separated release tarballs.  They might have a compelling 
> use-case (though I doubt it), but at least they should be notified so that 
> unified packages show up promptly after the release where we first issue them.
> 
> -glyph

The timeline generosity is simply so that downstream maintainers can get their 
stuff sorted before we drop it. I'm not sure how fast they move, and if all the 
major ones do it before that, I guess we can drop it early.

Since there doesn't seem to be any "omg no Hawkie what are you doing" from 
either you or JP, and otherwise "yes" from people I've asked/on this list so 
far, I'm guessing that I'll take the initiative and reach out to the downstream 
maintainers with some degree of "this is what we're doing now unless there's a 
very good reason".

- Hawkie


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[exarkun]

On 08:04 am, hawk...@atleastfornow.net wrote:

Hi,

I think that Twisted should deprecate subproject packaging


You probably won't hear much if any disagreement with this proposal on 
this list.


The sub-projects were never for people who like and are into Twisted. 
They were a tool to help convince people who don't use Twisted yet that 
it's not an unapproachable monolith.


We could discuss whether they have been effective in this regard but if 
you want to hear from people who *might* actually benefit from the 
split, you probably need to talk to people who aren't using Twisted.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Glyph]

> On Apr 22, 2015, at 04:04, HawkOwl  > wrote:

> So, I think we should:
> 
> - Deprecate the subproject setup.pys starting with 15.2 and stop packaging 
> subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in 
> September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major 
> release notice).
> - Remove the support code for subprojects in twisted.python._release + the 
> release scripts in the 16.0 release.

If anything, this seems like too generous a timeline to me.  We have the 
deprecation process to preserve API stability.  Do the subproject setup.py's 
expose any API surface?  "from twisted.web import setup" already appears to be 
an ImportError to me.  I don't think that the subproject release process has 
helped anyone in a long time.  Is there any reason why we can't just drop it 
from the next release, other than "the change might not land in time"?

One word of warning - do try to reach out and contact the package maintainers 
directly, as they may not follow this list closely, and for some reason they 
seem keen on separated release tarballs.  They might have a compelling use-case 
(though I doubt it), but at least they should be notified so that unified 
packages show up promptly after the release where we first issue them.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Phil Mayers]

On 22/04/15 09:04, HawkOwl wrote:

Hi,

I think that Twisted should deprecate subproject packaging -- they


Agreed. It's largely pointless.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[Hynek Schlawack]

> I think this is a good idea and am willing to put the work in to implement 
> this. Does anyone object to this? Is there a use case for keeping the 
> subprojects packagable that I don't know about?

+10

1. The split is due to Twisted not fitting on actual diskettes in ancient times.
2. It adds a frustrating amount of complexity.

—h
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Deprecating subproject packaging

[HawkOwl]

Hi,

I think that Twisted should deprecate subproject packaging -- they aren't 
uploaded to pip, don't support Python 3, I don't think we have docs for using 
them anywhere, and when included in downstream they usually only end up in 
confusion as various imports fail but some succeed due to only some subprojects 
being installed. Eventuall removing them means we have less code (and only one 
setup.py + setup3.py), less packages, and simplifies things.

It does mean that you can't *just* get Twisted Web, but Twisted Core is so big 
(internet, trial, amp, python, logger, cred, protocols -- even positioning is 
in there) that the size difference is marginal. For example, uncompressed 
Twisted (without the docs) is 14MB, while Core alone is 7MB. Assuming that you 
want to use Twisted Web, you'll need Twisted-Web and Twisted-Names (because not 
using Twisted Names in web.http.Request is deprecated), which is ~9MB. The 
difference in tarball size is even less -- Twisted without docs being 2.2MB and 
Twisted Core + Web + Names being 1.4MB.

Plus, I am not sure that there is "subprojects" really, anymore -- it's all 
just Twisted.

So, I think we should:

- Deprecate the subproject setup.pys starting with 15.2 and stop packaging 
subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in 
September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major 
release notice).
- Remove the support code for subprojects in twisted.python._release + the 
release scripts in the 16.0 release.
- Migrate Twisted's source dist to being done by "python setup.py sdist" 
instead of manually assembling the tarball. This means we can generate the 
requisite PKG-INFO metadata automatically. This should be done for the main 
tarball (which is uploaded to PyPI) in maybe 15.2, but not the subproject 
tarballs (Twisted Core + rest) which will remain the same until they are 
removed in 16.0.
- Replace manually uploading these distributions to PyPI and use twine to 
automate it for 15.2, when Twisted's tarball has the needed PKG-INFO (from 
being generated by sdist). Twine doesn't support uploading .msis, but as we now 
have Python wheels on PyPI, I think we can do without uploading MSIs, and just 
have them on the website.

Keeping the projects topfiles as they are I think is good -- although I am not 
sure if having a NEWS in each subproject is useful. Maybe we should just have 
the one NEWS file once 16.0 lands, since the other ones wouldn't be packaged, 
and are therefore less useful.

As for downstream:
- Apple includes all of Twisted, so I don't think this will affect them.
- Ubuntu and Debian package the subprojects separately. I guess they would have 
to make dummy packages that depend on the monolithic python-twisted instead.
- Fedora (so Red Hat too?) have obsoleted the subprojects and point them at 
python-twisted instead, so they don't package the subprojects.
- FreeBSD packages the Twisted subprojects separately, so they would have to do 
the same as Debian to migrate.

Overall, I don't think this adversely affects downstream in the long term -- it 
does what Red Hat did as of Fedora 21, and it'll end up being less packages 
overall.

I think this is a good idea and am willing to put the work in to implement 
this. Does anyone object to this? Is there a use case for keeping the 
subprojects packagable that I don't know about?

- Hawkie


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python