On Mon, Dec 21, 2015 at 6:05 AM, Samuel Bishop <lucract...@gmail.com> wrote:
> Hi Andrew, > > Thanks for the detailed response. I'm glad to hear you have tried a few > different ways to implement this, and having read through your reply, and > the channels docs again, I have more questions :-) > > Your point about asynchronous code introducing needless complexity is very > well founded, its one of the reasons I like Actor Model Concurrency over > other concurrency models, it scales well computationally, and *mentally*, > as a programmer, I am trying break down my real world goals into discrete > steps of computations, comparisons, iterations and by combining these > steps, I construct my program. Actor Model Concurrency, is no more > complicated than saying "This step can be done without affecting anything > else thats going on", which makes the 'actor/task' much easier to introduce > to existing synchronous code than most other concurrency methods. > I agree - I'm a big fan of actors in general and it's usually what I turn to to write complex simulations when I'm mucking around with game backends. > > One thing that sticks with me as I think about the current channels > design, relates to the idea I've raised in another discussion ( > https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ). > If I put my architect hat on and think about the future a bit, a 'more' > actor based design to channels might simplify later additional async work. > Since all the websocket stuff is new, I'll stick to the existing HTTP cycle > in this example. In version 1 a 'vanilla' Django task/actor would be a full > HTTP request response cycle, like how Django and WSGI currently work. Our > Django task is effectively our WSGI task, much like Pulsar Pulse. If all of > Django is contained inside our new task/worker/supervisor Actor layer, then > it becomes fairly trivial to do things like create separate request > handler, database query, and template rendering task/actors and chain their > calls. Since in the previously linked to discussion thread, I'm talking > about decoupling the ORM from the 'upper' layers of Django, its hard to not > also think what opportunities that would open up for async efforts. How do > you see the architecture of Django channels accommodating such improvements > in future? At the moment your focused on keeping things compatible and > avoiding the need for any of the existing synchronous Django code to > change, is this introducing any design tradeoffs that you are aware of with > regards to hypothetical future developments? > So, what you propose obviously isn't feasible in the short term - I'm doing as much as I can to decouple Channels from "rewrite all of Django" - but notably, channels is a subset of the actor model. In particular, you could see consumers as stateless, single-action actors - but, this means that any future abstraction would be able to build on that. > > Django Channels feels like it contains a possible asynchronous replacement > for signals but is 'holding back', many cases where I can see myself using > a channel, I'm using them with a signal handler, can you elaborate a bit on > your thoughts here? I assume its a compatibility decision, but is there > more to it than just that? > It's entirely a compatibility decision - Channels needs to ship in a way where code written for 1.9 will work the same, barring a few edge cases where people dip down to raw WSGI. You can entirely replace signals with custom channels, and I want to ship example code that does so (it's two lines of code, pretty much), but they work in a different way - for example, right now the pre_save signal can actually prevent a save from occurring, whereas if it were on a channel we'd have to then link the save consumer up to be run as a result of pre_save, which starts down a path of compatibility nightmares. Channels should be flexible enough that if you want to pursue the actor pattern you can - the underlying library basically just has send() and receive_many([channel_name, ...]) methods, so you can see how that could be used to make different task-oriented processes rather than the generic worker model that's going in now - it's just a case of keeping a major addition like this as simple and backwards-compatible as possible. Rewriting everything from scratch would be lovely, but unfortunately part of maintaining a framework is not doing that. Part of me, though, would be very interested in such a project being undertaken to see how far it could go, though I'm not sure you could then call the end result Django - at least, not without getting there incrementally over many releases. Andrew > On Thursday, 17 December 2015 19:08:09 UTC+8, Andrew Godwin wrote: >> >> Hi Samuel, >> >> You'll notice if you look carefully that I avoid saying "async" almost >> anywhere in the Channels announcements or documentation, and when I do, >> it's qualified very carefully. Channels makes Django event-driven, but it >> does not provide full "in-process" async like Twisted, asyncio, or >> something like Pulsar. >> >> Channels is actually kind of related to the actor model in some ways, but >> it's simpler; imagine it as a pool of identical actors that are all >> stateless, apart from the interface servers, which have the incoming socket >> and thus the state. >> >> The key thing is that Channels does nothing to make Django asynchronous. >> All the Django code will remain synchronous (the ORM, templating, etc.), >> and run in synchronous processes that just process one message after the >> next. The only part of the system that will require an async framework is >> the interface servers, as to be anywhere near efficient they need to serve >> more than one socket at once; right now we're using Twisted and asyncio for >> those, purely as they're incredibly mature and well-supported by the >> community, but you could swap out anything there that achieved the same >> goal (greenlets, for example, though I personally dislike their >> implicitness). >> >> Channels essentially takes the worker-prefork model that's been used in >> webservers for years and applies it to a lower-level message concept rather >> than HTTP requests (http requests are just one kind of message). This not >> only has the advantage of a lot of previous work and research into >> architecting and scaling these kinds of systems, it also means the actual >> amount of work in core Django isn't crazy; it's more about providing nice >> places to wrap up the worker loop and nice utilities around authentication >> and session management. >> >> If you read through the docs there's a bit more about this, but >> basically, the fundamental design goal here is that Channels be no harder >> to use than normal Django; indeed, it's almost exactly the same view >> concept, but instead of a request you get a message, and the newer >> developers I've interviewed about it seem to be comfortable with the >> extension of the current abstraction straight away. I don't ever want >> people to have to write asynchronous code if they don't want to; it's >> currently so easy to shoot yourself in the foot in almost any Python async >> solution that I don't think that makes for a good developer experience. >> >> The "complexity" of the message backends you describe is because they're >> network-transparent; there is indeed a simple, in-process one that uses >> threads too, but the whole point of this design is that you run Django over >> multiple processes on many machines, and in order to get something like >> that you need a place to funnel messages through (another potential backend >> design could use direct connection and process discovery, perhaps, or an >> Erlang-like process spawn-and-track design with an overarching controller >> process with access to many pre-selected machines). >> >> The idea of making an asynchronous Django is tangential to making >> Channels; this is more about changing the design abstraction of Django so >> we can fit the concept in, and then providing a neat solution that drops >> into the gap. There's nothing stopping someone making the worker processes >> for Channels async-aware as well, so they can run multiple message >> consumers at once, but that's such a complex and risk-fraught task that I >> don't think it's worth it when we can get 90% of the benefits with a much >> simpler and easier-to-design-and-understand system. If you want to see an >> example of actual async Django, see Amber's talk from Django Under the Hood >> this year, where she managed to get it working decently well under Twisted. >> >> I expect Channels to smooth out Django's performance mostly as a >> side-effect of the worker model providing demand-based load balancing by >> design, but I doubt it will be a huge increase until people start using the >> primitives it provides to offload tasks to other workers. >> >> I hope that somewhat long-winded explanation helps - if not, let me know >> what's still unclear and I'll try to tackle it. Clearing up what we're >> trying to do and communicating it is important to me; I want to make sure >> everyone knows so we can get good feedback on this stuff. For what it's >> worth, I did consider the actor model along with several other kinds of >> concurrency approaches (I've been prototyping bits of this for years); this >> approach ended up being the nicest design and the one I have the most >> confidence in that we can scale and shard to large work volumes. >> >> Andrew >> >> On Thu, Dec 17, 2015 at 10:30 AM, Samuel Bishop <lucra...@gmail.com> >> wrote: >> >>> I'm uncertain how popular the suggestion will be but ... are "channels" >>> the right solution to our async/concurrency problems? ( I'm all for >>> django-channels the project and the work to solve these issues) >>> All the talk of channels, workers, queues and back ends in the >>> explanations I'm reading as I follow along with the progress of 'Django >>> Channels", just feels "less good" than solving this problem via Actor Model >>> Concurrency (championed by Erlang, Elixir, Stackless Python, etc). >>> >>> There is an excellent actor model concurrency library for python that >>> already supports 'async django'. "Pulsar" ( >>> http://pythonhosted.org/pulsar/index.html) and the documentation here >>> http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways >>> to use actor model concurrency to attack the issue of using Django with >>> asynchronous web stuff like Websockets, etc. >>> Some of what's being described in the explanations of Django channels >>> sounds very similar to what they provide to work around the limitations of >>> django's blocking request processing cycle other parts sound much more >>> complicated, for instance the need for various back ends >>> in-memory/redis/etc. >>> >>> I know Pulsar is currently Python 3.4 and newer only, but in the past it >>> supported 2.7, so anything we can learn from their concurrency approach can >>> work on Python 2.7 to maintain our backwards compatibility going forward. >>> You can see an example of Pulsar using python 2 if you look on the >>> benchmark page for their python powered 'redis clone' - >>> https://gist.github.com/lsbardel/8068579 - One of the benchmarks is >>> using pypy so the code definitely worked in a python 2.7 environment. >>> >>> Would the work to integrate channels as part of Django improve the >>> ability to run Django in a completely async mode and enhance its >>> performance when run under Pulsar? or would it just be something that >>> needed to be turned off if we wanted to use something like Pulsar to power >>> our "async django". Is there anything to be gained from using a more actor >>> driven concurrency model or adopting any of the other methods used in their >>> 'pulse' django app? >>> I suppose I'm just trying to work out how to compare using >>> Django-Channels with using Pulse from the Pulsar project. >>> >>> On Wednesday, 16 December 2015 19:23:56 UTC+8, Andrew Godwin wrote: >>> >>>> >>>> On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann < >>>> in...@markusholtermann.eu> wrote: >>>> >>>>> >>>>> >If I get it right -- Curtis' description is spot-on; some tasks will >>>>> >still >>>>> >need Celery, Channels will take care of many others. >>>>> >>>>> For me it's more a question of "is a client directly involved". >>>>> >>>> >>>> I don't agree; I think it's more "do you have enough workers to handle >>>> this". Channels makes all workers serve all requests, so if you're doing >>>> e.g. video encoding you can't segregate it from request handling. However, >>>> you could have multiple channel backends and thus multiple worker pools to >>>> achieve this. >>>> >>>> Andrew >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@googlegroups.com. >>> To post to this group, send email to django-d...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com >>> <https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uoBd6wC%2BcgXjQv%2BLZtGVqGRZLO2P6jeCyOokkC94XhjSQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
