Re: [Twisted-Python] tubes now a separate project

[Glyph]

> On Dec 29, 2014, at 12:46 PM, Glyph Lefkowitz  wrote:
> 
> 
>> On Dec 27, 2014, at 9:46 AM, exar...@twistedmatrix.com 
>>  wrote:
>> 
>> On 12:13 am, gl...@twistedmatrix.com  wrote:
>>> Apropos of recent discussion, I have moved tubes >> > to github >> >.
>>> 
>>> I've filed some issues there and plan to file more.  Please start sending 
>>> pull requests if you'd like to work on it with me.
>> 
>> Cool!  Thank you very much for this.  I'm looking forward to doing some 
>> serious playing around with this.
> 
> Thanks for your encouragement and interest.  I look forward to hearing more 
> from you about it :).
> 
> I've also gotten control of the PyPI package and uploaded some packages:
> 
> https://warehouse.python.org/project/Tubes/ 
> 
> 
> So you can "pip install tubes" and something will happen.  Not the right 
> thing, mind you, 0.0.0 is the version number for a reason :).  But something.

Tubes is now on readthedocs:

>

and travis:

>

and coveralls:

>

and all the badges for these various things are green:

>

including 100% twistedchecker-clean and 100% pydoctor-clean.

So the code is in a good position to start accepting pull requests now.

> -glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] tubes now a separate project

[Glyph Lefkowitz]

> On Dec 27, 2014, at 9:46 AM, exar...@twistedmatrix.com wrote:
> 
> On 12:13 am, gl...@twistedmatrix.com wrote:
>> Apropos of recent discussion, I have moved tubes  to 
>> github .
>> 
>> I've filed some issues there and plan to file more.  Please start sending 
>> pull requests if you'd like to work on it with me.
> 
> Cool!  Thank you very much for this.  I'm looking forward to doing some 
> serious playing around with this.

Thanks for your encouragement and interest.  I look forward to hearing more 
from you about it :).

I've also gotten control of the PyPI package and uploaded some packages:

https://warehouse.python.org/project/Tubes/ 


So you can "pip install tubes" and something will happen.  Not the right thing, 
mind you, 0.0.0 is the version number for a reason :).  But something.

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] tubes now a separate project

[exarkun]

On 12:13 am, gl...@twistedmatrix.com wrote:
Apropos of recent discussion, I have moved tubes  
to github .


I've filed some issues there and plan to file more.  Please start 
sending pull requests if you'd like to work on it with me.


Cool!  Thank you very much for this.  I'm looking forward to doing some 
serious playing around with this.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] tubes now a separate project

[Glyph Lefkowitz]

> On Dec 26, 2014, at 4:13 PM, Glyph Lefkowitz  wrote:
> 
> Apropos of recent discussion, I have moved tubes  to 
> github .
> 
> I've filed some issues there and plan to file more.  Please start sending 
> pull requests if you'd like to work on it with me.

Also possibly of interest: you can now try out tubes via

pip install git+https://github.com/twisted/tubes

I am going to be putting something on PyPI soon; I've gotten in touch with the 
person who registered  and then 
apparently did nothing with it via their bitbucket account but have not heard 
back yet.  If anyone has a more direct contact method I'd be happy if you could 
put me in touch :-).

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] tubes now a separate project

[Glyph Lefkowitz]

Apropos of recent discussion, I have moved tubes  to github 
.

I've filed some issues there and plan to file more.  Please start sending pull 
requests if you'd like to work on it with me.

-glyph
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Daniel Sank]

A helix is twisted but isn't necessarily a tube. Macaroni on the other
hand...

On Thu, Dec 11, 2014 at 4:09 PM, Christopher Armstrong <
ra...@twistedmatrix.com> wrote:
>
>
> > On Dec 11, 2014, at 6:00 PM, Donald Stufft  wrote:
> >
> >
> >> On Dec 11, 2014, at 6:55 PM, Itamar Turner-Trauring <
> ita...@itamarst.org> wrote:
> >>
> >> I would suggest including the Twisted support inside tubes as a first
> pass, restricted to a module (e.g. tubes.helical). Less overhead and as
> long as it's optional it's not preventing anyone from using tubes with
> other frameworks.
> >>
> >> Otherwise, +100.
> >
> > If you do this, I would just do something like
> tubes.integrations.{twisted,tornado,asyncio} etc.
> >
>
>
> I think I prefer integration modules to be separate, though I myself have
> broken this rule in the Effect library (which has an effect.twisted
> module). I think I may separate it out to another library if I ever release
> Effect as a non-alpha. Keeping integration libraries separate keeps your
> dependencies nice and simple. And we live in a world where installing
> dependencies is trivial now, right? :)
>
> But yeah, +100. I’m really happy you came to this conclusion.
>
> -radix
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>


-- 
Daniel Sank
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Christopher Armstrong]

> On Dec 11, 2014, at 6:00 PM, Donald Stufft  wrote:
> 
> 
>> On Dec 11, 2014, at 6:55 PM, Itamar Turner-Trauring  
>> wrote:
>> 
>> I would suggest including the Twisted support inside tubes as a first pass, 
>> restricted to a module (e.g. tubes.helical). Less overhead and as long as 
>> it's optional it's not preventing anyone from using tubes with other 
>> frameworks.
>> 
>> Otherwise, +100.
> 
> If you do this, I would just do something like 
> tubes.integrations.{twisted,tornado,asyncio} etc.
> 


I think I prefer integration modules to be separate, though I myself have 
broken this rule in the Effect library (which has an effect.twisted module). I 
think I may separate it out to another library if I ever release Effect as a 
non-alpha. Keeping integration libraries separate keeps your dependencies nice 
and simple. And we live in a world where installing dependencies is trivial 
now, right? :)

But yeah, +100. I’m really happy you came to this conclusion.

-radix
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Donald Stufft]

> On Dec 11, 2014, at 6:55 PM, Itamar Turner-Trauring  
> wrote:
> 
> I would suggest including the Twisted support inside tubes as a first pass, 
> restricted to a module (e.g. tubes.helical). Less overhead and as long as 
> it's optional it's not preventing anyone from using tubes with other 
> frameworks.
> 
> Otherwise, +100.

If you do this, I would just do something like 
tubes.integrations.{twisted,tornado,asyncio} etc.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Itamar Turner-Trauring]

I would suggest including the Twisted support inside tubes as a first 
pass, restricted to a module (e.g. tubes.helical). Less overhead and as 
long as it's optional it's not preventing anyone from using tubes with 
other frameworks.


Otherwise, +100.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Glyph]

> On Jul 10, 2014, at 21:32, exar...@twistedmatrix.com wrote:
> 
> Tubes have been largely Glyph's effort (though a lot of people have 
> contributed in one way or another).  And a large effort it's been. 
> Development is proceeding in a Twisted branch and comes to over three 
> thousand lines of additions so far.
> 
> Given the large size of the implementation and the long time that this effort 
> has been underway (I remember the Twisted meetup at the Rackspace offices 
> that *I* attended when I was visiting SF... a year and a half ago... at which 
> point tubes wasn't exactly a brand new project) I'd like to re-raise the idea 
> that the best next step for the project is to see some distribution in its 
> *current* state.

When Jean-Paul originally wrote this message, I strongly disagreed.

In the intervening six months, I've been working on Tubes on and off.  In the 
last couple of weeks, I've had a bit of time to make a concerted effort to 
improve the branch, and it is starting to approach something like a finished 
state: there is still a lot of polish necessary, but as I've tested them 
against more and more use-cases, the core abstractions seem to be holding up 
rather well, and several rounds of increasingly elaborate use-cases did not 
result in any big rewrites or mystifying un-debuggable re-entrancy scenarios in 
the implementation.

One change Thomas Hervé assisted me in making today was to remove Deferred 
support from the core of "tube" itself, and instead factor it out into a 
separate module.  Interested readers may find that module here:



It's not finished (mostly it needs testing for errbacks and the addition of 
yielding non-Deferred values).

This change radically simplifies the conceptual model around consuming 
asynchronous results within the implementation. If you want to deal with 
results instead of Deferreds, you just create this tube that turns Deferreds 
into their results - the only special-casing inside tube itself is a special 
value that says "when you yield this, don't actually emit an output to the next 
drain in the series".

Another thing that this change does is illustrate the fact that literally 
nothing in the core part of Tubes has come to depend on Twisted - and indeed, 
as the factoring has improved over time, the coupling between Twisted and the 
core Tubes abstraction has become looser and looser.  This is a very good thing 
architecturally.  Examining the few remaining points of coupling, there are 
pretty much only two: twisted's logging serves as the global error-handler hook 
(which can be easily changed to a simple bootstrapping API) and the use of 
Failure as the "reason" parameter to the .flowStopped and .stopped methods.  
This was copied from IProtocol, of course, but I think that upon reflection it 
is a bad idea.  Failure, even at its best, is simply an abstraction over 
exception-as-raised.  For Deferreds, this makes sense; asynchronous errors are 
similar to, and may be directly caused by, code raising an unexpected 
exception.  However, although Tubes are analogous, a raised exception is an 
anomalous way for a flow to terminate, and I think the right thing to do there 
is actually to unconditionally log the exception, passing only a token value 
saying "application error" into the flowStopped method itself; the main 
use-case for the "reason" parameter is to distinguish between subtly different 
protocol conditions which may cause a connection termination; it's not an 
appropriate place to start handling unexpected exceptions.  (When I have 
written code that does this using Twisted the inevitable result is an exception 
whose traceback disappears mysteriously and makes debugging a pain.)

To sum up, the more this abstraction decouples from other things within 
Twisted, the more it becomes depended upon rather than itself doing the 
depending, the cleaner, simpler, and easier to use it gets.

So now I think I was wrong.  I now agree with Jean-Paul, and in fact the 
pendulum of my opinion has swung a bit farther in the other direction.  Not 
only should Tubes be released independently, I think they are a lower-level 
library that Twisted should eventually depend on.  This library could just as 
easily have adapters for other loops, like tornado and asyncio, as well as 
providing potentially interesting support for doing things like eventually 
replacing Deferreds - see for example .

So rather than have a single "tubes" package, I think I will break the branch 
into two pieces:

The "tubes" package itself, which will have no external dependencies and will 
pretty much just be buffer-management, fan-in, fan-out, buffering, throttling, 
filtering, routing, and framing-protocol code, intended to remain separate for 
the long term.  This will probably mean some small amount of code duplication 
between LineReceiver et. al. and tubes

Re: [Twisted-Python] Tubes!

[Thomas Hervé]

On Thu, Jul 10, 2014 at 10:37 PM, Glyph Lefkowitz
 wrote:
> While I currently believe that Tubes's API has firmed up and its current API
> is suitable for general purpose use, I have believed that at various points
> in the past as well when it was completely wrong.  This sentiment is very
> much of the "this time for sure!" variety, and I will not have confidence
> that it's actually complete until we have made it all the way through the
> documentation, examples, and testing of a full proof of concept - at which
> point I believe it would be suitable to include in a Twisted release anyway.

I think this is a wrong approach. When people say "You never get it
right the first time", I don't think they mean the first time they
write it, but rather the first time somebody else uses it. You'll
never be able to release something perfect.

> So I am keenly interested in ways to address this problem rather than to
> work around it.  If we are going to try to develop new big Twisted features
> outside of Twisted,  maybe that's a good idea, but then we need a modified
> code-review policy for accepting those projects back in where they going to
> be subjected to code-review standards during development rather than in one
> giant burst at the end.

Now you're talking about the real issue. It's almost impossible to
write anything non-trivial in Twisted nowadays. One part is the
compatibility policy, with the whole "Who knows who may subclass that
thing. Don't ever change an attribute!". Overall we have a development
process averses to change. That's why tubes isn't released yet: you
know that if we put it out there we won't be able to fix it. For the
same reasons we haven't been able to fix Web or Conch.

If the answer is to put more stuff outside, I think it's rather sad.
I'd hope it would be to make the development process more lightweight.

--
Thomas

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Tom Prince]

Glyph Lefkowitz  writes:

While I currently believe that Tubes's API has firmed up and its 
current API is suitable for general purpose use, I have believed 
that at various points in the past as well when it was 
completely wrong. This sentiment is very much of the "this time 
for sure!" variety, and I will 
[...]
In this case, I actually want the twisted compatibility policy 
to apply.


These two points seem to be somewhat conflicting.

So I am keenly interested in ways to address this problem rather 
than to work around it.  If we are going to try to develop new 
big Twisted features outside of Twisted,  maybe that's a good 
idea, but then we need a modified code-review policy for 
accepting those projects back in where they going to be 
subjected to code-review standards during development rather 
than in one giant burst at the end.


This seems like a reasonable thing to do. If all the code has been 
subject to review according to twisteds procedure, then it would 
seem to be reasonable to require that the changes needed to move 
the code in to twisted would need to be reviewed (so moving files 
around and adjusting imports and the like).


Even if my opinion were inverted on all of the points above, I 
really want to avoid doing this - I especially want to avoid the 
part where it somehow doesn't work on Twisted's heterogenous CI 
system when we try to reintegrate it after apparently working on 
some other one-version-of-linux CI infrastructure for some time 
:).


It would certainly be possible to hook twisted's CI system to 
additional projects.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Tobias Oberstein]

> Specifically, I think it would be beneficial to set up a tubes project on 
> Github
> under the Twisted organization and try for a release in the very near future.
> 
> I think this has several advantages over the status quo:

+1 for splitting into a separate project.

even if Tubes would be considered experimental / mostly unusable right now.

the best thing that could happen: people actually start playing around with it. 
which provides real-world feedback.

"finishing" it in the backroom and only later release something "ready" without 
gathering real-world feedback: that's a risk IMO, rather than good practice.

I guess I am a proponent of the "release early release often" mantra.

better release something unfinished and gain feedback, than release something 
"finished" not meeting user expectations.

just my 2cts,

cheers,
/Tobias 

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes! (package metadata and version constraints)

[Glyph]

On Jul 10, 2014, at 5:23 PM, exar...@twistedmatrix.com wrote:

> On 10 Jul, 08:37 pm, gl...@twistedmatrix.com wrote:
>> 
>> On Jul 10, 2014, at 12:32 PM, exar...@twistedmatrix.com wrote:
>>> 2) As a separate Python package, the logistics of actually using tubes are 
>>> simpler (just consider how you might declare a dependency on a branch of 
>>> Twisted - keeping in mind you may want to use tubes in a project that 
>>> already depends on some version of Twisted).  It may not make sense to say 
>>> that it is the same quality as Twisted proper right off the bat (on the 
>>> other hand, it may well - I suspect tubes in its current form actually is a 
>>> lot higher quality than large sections of Twisted) but that doesn't mean 
>>> people (not to mention the tubes project) can't benefit from being able to 
>>> experiment with it.
>> 
>> I would love it if there were a way to release a package in an actually 
>> experimental state, and not just have the release of a package implicitly 
>> tell people that it's time to put it into production and demand long-term 
>> support for it.  Quick sanity check: go run 'pip freeze' in a production 
>> virtualenv you're running - what percentage of the version numbers that come 
>> back start with a zero?  I will bet a significant amount of money that it's 
>> not 0% :-).
>> 
>> As it stands, if you're not willing to use a random outdated branch of 
>> Twisted with unknown bugs that may change without warning, you're probably 
>> not willing to adopt Tubes yet.
> 
> I think you missed part of my point here.
> 
> I want to try tubes and see if they make my application simpler or better or 
> faster or whatever.  My application depends on Twisted >= 14.0.0.  It doesn't 
> matter how or why: that's what the metadata says and I can't use an older 
> version of Twisted unless I do a bunch of stupid package/distribution related 
> hacking that I don't want to do.  *So* much of the Python tooling has now 
> moved to requiring and respecting explicit dependency declarations that 
> trying to side-step these now is a significant hassle.  Separating tubes from 
> Twisted solves this problem.
> 
> It's not at all a question of whether the code is stable or production ready 
> or even works at all, it's a matter of packaging constraints.

If you want to set up a development environment to work on this and try it out, 
I think you can do it right now.  I believe this simple and intuitive 
incantation will work:

pip install -e 
'git+...@github.com:/twisted/twisted@3c10ad3323f86d7194a3a4c9fd4e2e1bcd441fec#egg=Twisted-tubes-1956-7'

That's a post-14.0.0 package with the appropriate metadata (and I didn't cheat 
and merge forward first or anything, this is the most recent branch as of 
several weeks now).  Doing exactly that in a mktmpenv and then doing 'pip 
install twisted==14.0.0' results in a 'requirement already satisfied' for me, 
and everything imports as expected.

I think there's still probably stuff to talk about regarding the rest of your 
message but if this can get you past your immediate hurdle that would be great 
:).

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes! (package metadata and version constraints)

[exarkun]

On 10 Jul, 08:37 pm, gl...@twistedmatrix.com wrote:


On Jul 10, 2014, at 12:32 PM, exar...@twistedmatrix.com wrote:
2) As a separate Python package, the logistics of actually using tubes 
are simpler (just consider how you might declare a dependency on a 
branch of Twisted - keeping in mind you may want to use tubes in a 
project that already depends on some version of Twisted).  It may not 
make sense to say that it is the same quality as Twisted proper right 
off the bat (on the other hand, it may well - I suspect tubes in its 
current form actually is a lot higher quality than large sections of 
Twisted) but that doesn't mean people (not to mention the tubes 
project) can't benefit from being able to experiment with it.


I would love it if there were a way to release a package in an actually 
experimental state, and not just have the release of a package 
implicitly tell people that it's time to put it into production and 
demand long-term support for it.  Quick sanity check: go run 'pip 
freeze' in a production virtualenv you're running - what percentage of 
the version numbers that come back start with a zero?  I will bet a 
significant amount of money that it's not 0% :-).


As it stands, if you're not willing to use a random outdated branch of 
Twisted with unknown bugs that may change without warning, you're 
probably not willing to adopt Tubes yet.


I think you missed part of my point here.

I want to try tubes and see if they make my application simpler or 
better or faster or whatever.  My application depends on Twisted >= 
14.0.0.  It doesn't matter how or why: that's what the metadata says and 
I can't use an older version of Twisted unless I do a bunch of stupid 
package/distribution related hacking that I don't want to do.  *So* much 
of the Python tooling has now moved to requiring and respecting explicit 
dependency declarations that trying to side-step these now is a 
significant hassle.  Separating tubes from Twisted solves this problem.


It's not at all a question of whether the code is stable or production 
ready or even works at all, it's a matter of packaging constraints.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Donald Stufft]

Hmm, perhaps —pre shouldn’t be allowed in config files :V

On Jul 10, 2014, at 5:19 PM, Glyph Lefkowitz  wrote:

> 
> On Jul 10, 2014, at 1:47 PM, Christopher Armstrong  
> wrote:
> 
>> For what it’s worth, if you add an “a%n” to the end of a version, pip won’t 
>> install it unless you specify the version exactly. e.g., “tubes” version 
>> “0.1a1” won’t be found if you type “pip install tubes”, only “pip install 
>> tubes==0.1a1”.
> 
> 
> Also "pip install --pre tubes", I believe.  Nevertheless, people put that 
> into a config file somewhere and forget about it forever ;).
> 
> -glyph
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Glyph Lefkowitz]

On Jul 10, 2014, at 1:47 PM, Christopher Armstrong  
wrote:

> For what it’s worth, if you add an “a%n” to the end of a version, pip won’t 
> install it unless you specify the version exactly. e.g., “tubes” version 
> “0.1a1” won’t be found if you type “pip install tubes”, only “pip install 
> tubes==0.1a1”.


Also "pip install --pre tubes", I believe.  Nevertheless, people put that into 
a config file somewhere and forget about it forever ;).

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Christopher Armstrong]

On July 10, 2014 at 3:40:26 PM, Glyph Lefkowitz (gl...@twistedmatrix.com) wrote:


On Jul 10, 2014, at 12:32 PM, exar...@twistedmatrix.com wrote:


2) As a separate Python package, the logistics of actually using tubes are 
simpler (just consider how you might declare a dependency on a branch of 
Twisted - keeping in mind you may want to use tubes in a project that already 
depends on some version of Twisted).  It may not make sense to say that it is 
the same quality as Twisted proper right off the bat (on the other hand, it may 
well - I suspect tubes in its current form actually is a lot higher quality 
than large sections of Twisted) but that doesn't mean people (not to mention 
the tubes project) can't benefit from being able to experiment with it.

I would love it if there were a way to release a package in an actually 
experimental state, and not just have the release of a package implicitly tell 
people that it's time to put it into production and demand long-term support 
for it.  Quick sanity check: go run 'pip freeze' in a production virtualenv 
you're running - what percentage of the version numbers that come back start 
with a zero?  I will bet a significant amount of money that it's not 0% :-).

As it stands, if you're not willing to use a random outdated branch of Twisted 
with unknown bugs that may change without warning, you're probably not willing 
to adopt Tubes yet.


For what it’s worth, if you add an “a%n” to the end of a version, pip won’t 
install it unless you specify the version exactly. e.g., “tubes” version 
“0.1a1” won’t be found if you type “pip install tubes”, only “pip install 
tubes==0.1a1”.



-- 
Christopher Armstrong
http://twitter.com/radix
http://wordeology.com/

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes!

[Glyph Lefkowitz]

On Jul 10, 2014, at 12:32 PM, exar...@twistedmatrix.com wrote:

> Hello all,
> 
> Some of you may have heard rumors of some work in progress on a replacement 
> for Twisted's IConsumer/IProducer interfaces.
> 
> Tubes have been largely Glyph's effort (though a lot of people have 
> contributed in one way or another).  And a large effort it's been. 
> Development is proceeding in a Twisted branch and comes to over three 
> thousand lines of additions so far.
> 
> Given the large size of the implementation and the long time that this effort 
> has been underway (I remember the Twisted meetup at the Rackspace offices 
> that *I* attended when I was visiting SF... a year and a half ago... at which 
> point tubes wasn't exactly a brand new project) I'd like to re-raise the idea 
> that the best next step for the project is to see some distribution in its 
> *current* state.

Unfortunately, while I can see a lot of hypothetical benefit to what you're 
describing, I don't think this is appropriate in this specific case.  With 
other, superficially similar projects (large new features within Twisted), this 
might be the right thing to do, with some caveats about how we do new-feature 
integration that I discuss below.  Some of the aspects of tubes make it extra 
hard to split out though, despite the fact that it has few immediate 
dependencies.

The Tubes package implements a new primitive, which means that everything that 
uses it is going to be very tightly coupled to its precise semantics, and it 
has very little wiggle-room in terms of evolution once it's been released.

In its current state, Tubes is basically a research project.  Every focused 
burst of activity on it has resulted in a complete, 100% break in backwards 
compatibility, on the level of its API, its terminology, and its semantics.  
Having a separate release might not imply that there will be a compatibility 
policy on par with the strictness of Twisted's current one, but it does usually 
imply some level of support or continued development.  If anyone had written an 
application against a previous revision of the Tubes branch, it would probably 
have been broken in the first place and it would definitely not still be 
working today (nor would it really be possible to port it or evolve it to use 
the new version without a complete rewrite).

While I currently believe that Tubes's API has firmed up and its current API is 
suitable for general purpose use, I have believed that at various points in the 
past as well when it was completely wrong.  This sentiment is very much of the 
"this time for sure!" variety, and I will not have confidence that it's 
actually complete until we have made it all the way through the documentation, 
examples, and testing of a full proof of concept - at which point I believe it 
would be suitable to include in a Twisted release anyway.

> Specifically, I think it would be beneficial to set up a tubes project on 
> Github under the Twisted organization and try for a release in the very near 
> future.
> 
> I think this has several advantages over the status quo:
> 
> 1) As an independent project, tubes will attract more attention than it 
> presently gets as a relatively unknown ticket & branch of Twisted.

I would appreciate attention in the form of code review, commentary, and 
experimentation.  I would not appreciate attention in the form of actual users, 
though.  At least, not right now.  Maybe quite soon, though, depending on how 
the next few development sessions go.

> 2) As a separate Python package, the logistics of actually using tubes are 
> simpler (just consider how you might declare a dependency on a branch of 
> Twisted - keeping in mind you may want to use tubes in a project that already 
> depends on some version of Twisted).  It may not make sense to say that it is 
> the same quality as Twisted proper right off the bat (on the other hand, it 
> may well - I suspect tubes in its current form actually is a lot higher 
> quality than large sections of Twisted) but that doesn't mean people (not to 
> mention the tubes project) can't benefit from being able to experiment with 
> it.

I would love it if there were a way to release a package in an actually 
experimental state, and not just have the release of a package implicitly tell 
people that it's time to put it into production and demand long-term support 
for it.  Quick sanity check: go run 'pip freeze' in a production virtualenv 
you're running - what percentage of the version numbers that come back start 
with a zero?  I will bet a significant amount of money that it's not 0% :-).

As it stands, if you're not willing to use a random outdated branch of Twisted 
with unknown bugs that may change without warning, you're probably not willing 
to adopt Tubes yet.

> 3) Decoupling tubes from Twisted frees tubes from certain of Twisted's 
> policies which are more challenging to follow for the kind of non- trivial, 
> brand new code base that tubes is

[Twisted-Python] Tubes!

[exarkun]

Hello all,

Some of you may have heard rumors of some work in progress on a 
replacement for Twisted's IConsumer/IProducer interfaces.


Tubes have been largely Glyph's effort (though a lot of people have 
contributed in one way or another).  And a large effort it's been. 
Development is proceeding in a Twisted branch and comes to over three 
thousand lines of additions so far.


Given the large size of the implementation and the long time that this 
effort has been underway (I remember the Twisted meetup at the Rackspace 
offices that *I* attended when I was visiting SF... a year and a half 
ago... at which point tubes wasn't exactly a brand new project) I'd like 
to re-raise the idea that the best next step for the project is to see 
some distribution in its *current* state.


Specifically, I think it would be beneficial to set up a tubes project 
on Github under the Twisted organization and try for a release in the 
very near future.


I think this has several advantages over the status quo:

 1) As an independent project, tubes will attract more attention than it 
presently gets as a relatively unknown ticket & branch of Twisted.


 2) As a separate Python package, the logistics of actually using tubes 
are simpler (just consider how you might declare a dependency on a 
branch of Twisted - keeping in mind you may want to use tubes in a 
project that already depends on some version of Twisted).  It may not 
make sense to say that it is the same quality as Twisted proper right 
off the bat (on the other hand, it may well - I suspect tubes in its 
current form actually is a lot higher quality than large sections of 
Twisted) but that doesn't mean people (not to mention the tubes project) 
can't benefit from being able to experiment with it.


 3) Decoupling tubes from Twisted frees tubes from certain of Twisted's 
policies which are more challenging to follow for the kind of non- 
trivial, brand new code base that tubes is.  Technically we could just 
say that these policies don't apply to a tubes package *in* Twisted but 
this kind of subtle distinction is often lost on users (ie application 
developers).


   a) Twisted's compatibility policy need not apply.  It could either be 
sped up or abandoned more thoroughly.  I'm generally a fan of being 
backwards compatible even when you have few users because it actually 
makes development easier, but loosening the policy to say things might 
break if it's just really inconvenient to keep them working (whereas 
Twisted goes to the inconvenience to keep them working) seems 
reasonable.


   b) tubes can undergo a faster release cycle to benefit more from user 
feedback.


 4) At this point, a normal review of the tubes branch is going to be a 
problem.  We do not have good tools or mechanisms for dealing with 
branches this large.  The code in the current tubes branch can just 
become master of a new project.  Development going forward from this 
point should continue to follow the feature-branch, small-changes, pre- 
commit-peer-review process.  But those 3k lines are written already. 
Short of an extremely expensive effort to break the work up into 
smaller, self-contained pieces there's simply never going to be a *good* 
review in the typical style.


Additionally, it may turn out that tubes can remain independent 
indefinitely.  Someday perhaps Twisted would come to depend on it to 
allow the various protocols and applications implemented in Twisted to 
benefit from the superior abstractions it provides.  Or maybe once it 
has undergone a few iterations it will make sense to bring it back to 
Twisted.  I don't think this needs to be decided now.


There are downsides, of course.  All of the boring maintenance involved 
with having a separate project - setting up CI, actually doing the 
releases, etc.  Perhaps we could find some volunteers to help out with 
these tasks, though, in exchange for getting some great code out there?


I'm curious what the folks out there who develop applications using 
Twisted would find to be the easiest path forward.  I'm also curious to 
hear what Glyph thinks about all this. ;)


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python