Re: A proposal for a new auto-reloader in Django

[Aymeric Augustin]

Hello David,

This reasoning makes sense.

You say that watchman "keeps triggering the command once an event occurs". This 
is correct; as a consequence "the command" must stop the currently running 
instance of the development server and start a new one. This requires 
overhauling significantly the current implementation of the development server.

You need to understand the two-process structure of the current autoreloader 
and the workarounds for allowing runserver to start with invalid settings to 
get a good grasp on the requirements for a good developer experience. I hope we 
can do away with these workarounds with a better architecture.

Best regards,

-- 
Aymeric.



> On 3 Apr 2017, at 03:36, qingnian...@gmail.com wrote:
> 
> Hi Aymeric,
>   I feel like using Watchman might be the easiest solution. It keeps 
> triggering the command once an event occurs, so autoreload.py does not need 
> to restart the server in case of a syntax error. Pywatchman does not work on 
> Python 3 and has some dependency problems, so I'm planning to use the command 
> line interface of the Watchman library. Watchdog only triggers the given 
> command once, so it cannot survive a syntax error and requires the 
> auto-reloader to deal with multi-threading, the most challenging part in this 
> project (likely to enter the same problem described in #2330 
> .) Therefore, I'd start with 
> Watchman and see how to implement this with Watchdog later. How do you feel 
> about this plan? 
> 
> Cheers,
> David Ma
> 
> On Sunday, April 2, 2017 at 6:33:38 AM UTC-7, Aymeric Augustin wrote:
> Hello,
> 
> On 1 Apr 2017, at 05:52, qingn...@gmail.com  wrote:
> 
>> For the pure-Python solution, I might implement a standalone autoreloader 
>> based on Carl's work and replace the current one. Does this look good to you?
> 
> 
> Perhaps.
> 
> It depends on the details (features, installation requirements, cross 
> platform support, etc.).
> 
> It's up to you to argument in your proposal why this would be a good solution.
> 
> Best regards,
> 
> -- 
> Aymeric.
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/3c284301-15e9-44d9-9b7c-2d866b7f7d13%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/E4AB822D-B4FF-44CD-972C-DBB3CCD6EC22%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[Carl Meyer]

Hi David,

I wouldn't bother with the wsgiwatcher repo; it's proof-of-concept code
from one sprint's worth of hacking, not used by anyone. Look at hupper
instead, it is based on wsgiwatcher, but much changed, and actually used
in production.

Carl

On 04/02/2017 04:27 PM, qingnian...@gmail.com wrote:
> Hi Carl,
>   I don't quite understand why get_module_paths() in your wsgiwatcher
> project is returning a list of python module paths. I thought it would
> return the directory that needs to be monitored. Could you please tell
> me how this part works? Thanks.
> 
> David Ma
> 
> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
> 
> Anyone working on this project should at least be aware of
> https://github.com/Pylons/hupper 
> (based on work David Glick and I
> originally did in https://github.com/carljm/wsgiwatcher
> ), which aims to
> be a framework-agnostic solution to this problem for any Python web
> project. Docs at
> http://docs.pylonsproject.org/projects/hupper/en/latest/
> 
> 
> 
> 
> Carl
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/99fbed1c-3ac4-64b9-5030-3d5e2533cef4%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Aymeric,
  I feel like using Watchman might be the easiest solution. It keeps 
triggering the command once an event occurs, so autoreload.py does not need 
to restart the server in case of a syntax error. Pywatchman does not work 
on Python 3 and has some dependency problems, so I'm planning to use the 
command line interface of the Watchman library. Watchdog only triggers the 
given command once, so it cannot survive a syntax error and requires the 
auto-reloader to deal with multi-threading, the most challenging part in 
this project (likely to enter the same problem described in #2330 
.) Therefore, I'd start with 
Watchman and see how to implement this with Watchdog later. How do you feel 
about this plan? 

Cheers,
David Ma

On Sunday, April 2, 2017 at 6:33:38 AM UTC-7, Aymeric Augustin wrote:
>
> Hello,
>
> On 1 Apr 2017, at 05:52, qingn...@gmail.com  wrote:
>
> For the pure-Python solution, I might implement a standalone autoreloader 
> based on Carl's work and replace the current one. Does this look good to 
> you?
>
>
> Perhaps.
>
> It depends on the details (features, installation requirements, cross 
> platform support, etc.).
>
> It's up to you to argument in your proposal why this would be a good 
> solution.
>
> Best regards,
>
> -- 
> Aymeric.
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3c284301-15e9-44d9-9b7c-2d866b7f7d13%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Carl,
  I don't quite understand why get_module_paths() in your wsgiwatcher 
project is returning a list of python module paths. I thought it would 
return the directory that needs to be monitored. Could you please tell me 
how this part works? Thanks.

David Ma

On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
>
> Anyone working on this project should at least be aware of 
> https://github.com/Pylons/hupper (based on work David Glick and I 
> originally did in https://github.com/carljm/wsgiwatcher), which aims to 
> be a framework-agnostic solution to this problem for any Python web 
> project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/ 
> 
>  
>
> Carl 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[Aymeric Augustin]

Hello,

On 1 Apr 2017, at 05:52, qingnian...@gmail.com wrote:

> For the pure-Python solution, I might implement a standalone autoreloader 
> based on Carl's work and replace the current one. Does this look good to you?


Perhaps.

It depends on the details (features, installation requirements, cross platform 
support, etc.).

It's up to you to argument in your proposal why this would be a good solution.

Best regards,

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/C906C8DA-32D9-48A7-B1BB-6693B0F0C672%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Aymeric,

  Thanks for this detailed and informative response! I will try to figure 
out the difficulties of integrating Watchman during the weekend. For the 
pure-Python solution, I might implement a standalone autoreloader based on 
Carl's work and replace the current one. Does this look good to you?

David Ma

On Thursday, March 30, 2017 at 6:39:42 AM UTC-7, Aymeric Augustin wrote:
>
> Hello, 
>
> > On 29 Mar 2017, at 01:05, qingn...@gmail.com  wrote: 
> > 
> > the best library to replace Watchdog is Watchman, a library that 
> supports both of these three platforms (for Windows it's still in the alpha 
> stage.) 
>
> Django currently doesn't use Watchdog. Watchdog is a cross-platform Python 
> library that can be used for implementing an autoreloader. It's a plausible 
> alternative to Watchman, likely easier to integrate, but Watchman has some 
> fancy features that I'd love to have available -- like waiting for git to 
> finish its operations before notifying a change. 
>
> > It runs as a standalone process and has a python wrapper library that is 
> incompatible with Python 3. Therefore, one of the goals is to figure out 
> how to interact with this library. 
>
> In order to be selected for this project, you have to convince us that you 
> can solve this problem. At a minimum you must prove that you understand 
> where the difficulties lie and suggest some possible approaches to tackle 
> them. 
>
> > So my goals would be: 
> > 
> > • Replace the old auto-reloader with the Watchman version 
>
> Since installing watchman isn't trivial, you need to keep a pure-Python 
> alternative so that beginners get an acceptable experience out of the box. 
> This alternative may be the current "poll every second approach" or could 
> be backed by watchdog, assuming it's pip-installable without complicated 
> dependencies (e.g. a C compiler and development headers for a bunch of 
> libraries) on all operating systems. 
>
> > • Make sure that the new auto-reloader can work with Python3 and 
> on other platforms 
>
> Did you mean "work with Python 2" ? In any case, this isn't necessary, as 
> you'd be working off the master branch which no longer supports Python 2. 
>
> > • Make it compatible with other Python libraries 
> > • Allow Django users to choose the files that should be watched 
> > • Integrate the Watchman package and its Python wrapper into 
> Django 
>
> Watchman should be installed separately by developers who wish to use it. 
> Django should use it when it's available (and there should be a flag to 
> disable that if for any reason someone has watchman installed systemwide 
> but doesn't want to use it). 
>
> I hope this helps! 
>
> -- 
> Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b8f5740c-a154-4b9b-90f1-9a79db712f0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[Melvyn Sopacua]

On Thursday 30 March 2017 22:01:00 qingnian...@gmail.com wrote:
> Hi Carl,
>   Thanks for mentioning this awesome project! I saw it in one of the
> discussions but did not take a close look. I'll definitely check this
> out and try to integrate wsgiwatcher/watcher.py into Django.
> 
> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
> > Anyone working on this project should at least be aware of
> > https://github.com/Pylons/hupper
> >  > pper&sa=D&sntz=1&usg=AFQjCNGVwtqvdo53UFfK80kaQ1qxL7ST8Q> (based 
on
> > work David Glick and I
> > originally did in https://github.com/carljm/wsgiwatcher
> >  > giwatcher&sa=D&sntz=1&usg=AFQjCNGqLfsmxsp37Lng7_d_DaVzja9c6Q>),
> > which aims to
> > be a framework-agnostic solution to this problem for any Python web
> > project. Docs at
> > http://docs.pylonsproject.org/projects/hupper/en/latest/
> > 
> > Carl
I was going to suggest the same, because you already get watchdog support for 
free 
and can probably build on that integration to pick up watchman. Also, it's used 
in 
devpi[1], so it gets some scruteny from there (and devpi is awesome to 
distribute 
internal or augmented Django apps amongst projects).
-- 
Melvyn Sopacua


[1] http://doc.devpi.net/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1871865.SgAvvE3WaE%40devstation.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Brice,
  Thanks for the suggestion! I'm not sure whether if I can successfully 
implement a library like you've mentioned, so for now I might stick with 
the Watch* libraries available. I really love the goals you've listed and 
would add these to my proposal.

David Ma

On Thursday, March 30, 2017 at 5:24:55 AM UTC-7, contact wrote:
>
> Hi David, 
> I think it's a good idea to improve what we have now, mostly for the 
> problems you described. 
> Don't you think it would be a better idea to make a reloader which 
> doesn't depend on Django at all? You could make a library working with a 
> settings file, and create a base settings file for Django projects. 
> You could define whatever should be called to start and restart the 
> server, and be able to define file types and folders that should be seen 
> as changes that require a restart or not. It could then be used easily 
> with any kind of Python project, and even allow specific behaviours, 
> like launching a server reload when a new plugin is uploaded to a 
> specific folder, or when one of them is updated. 
> Every Python framework could then embed their own base settings file, 
> which would allow this project to touch a wider audience. 
>
> - Brice 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f1194326-ed76-490f-999a-93cf6fc49890%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Carl,
  Thanks for mentioning this awesome project! I saw it in one of the 
discussions but did not take a close look. I'll definitely check this out 
and try to integrate wsgiwatcher/watcher.py into Django.

On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
>
> Anyone working on this project should at least be aware of 
> https://github.com/Pylons/hupper 
> 
>  
> (based on work David Glick and I 
> originally did in https://github.com/carljm/wsgiwatcher 
> ),
>  
> which aims to 
> be a framework-agnostic solution to this problem for any Python web 
> project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/ 
>
> Carl 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/59d28e31-49ea-41a3-b814-6025e155b256%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A proposal for a new auto-reloader in Django

[Carl Meyer]

Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper (based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher), which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/af0d4ea6-e97e-522e-947a-83db87c6ebd0%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: A proposal for a new auto-reloader in Django

[Aymeric Augustin]

Hello,

> On 29 Mar 2017, at 01:05, qingnian...@gmail.com wrote:
> 
> the best library to replace Watchdog is Watchman, a library that supports 
> both of these three platforms (for Windows it's still in the alpha stage.)

Django currently doesn't use Watchdog. Watchdog is a cross-platform Python 
library that can be used for implementing an autoreloader. It's a plausible 
alternative to Watchman, likely easier to integrate, but Watchman has some 
fancy features that I'd love to have available -- like waiting for git to 
finish its operations before notifying a change.

> It runs as a standalone process and has a python wrapper library that is 
> incompatible with Python 3. Therefore, one of the goals is to figure out how 
> to interact with this library.

In order to be selected for this project, you have to convince us that you can 
solve this problem. At a minimum you must prove that you understand where the 
difficulties lie and suggest some possible approaches to tackle them.

> So my goals would be:
> 
>   • Replace the old auto-reloader with the Watchman version

Since installing watchman isn't trivial, you need to keep a pure-Python 
alternative so that beginners get an acceptable experience out of the box. This 
alternative may be the current "poll every second approach" or could be backed 
by watchdog, assuming it's pip-installable without complicated dependencies 
(e.g. a C compiler and development headers for a bunch of libraries) on all 
operating systems.

>   • Make sure that the new auto-reloader can work with Python3 and on 
> other platforms

Did you mean "work with Python 2" ? In any case, this isn't necessary, as you'd 
be working off the master branch which no longer supports Python 2.

>   • Make it compatible with other Python libraries
>   • Allow Django users to choose the files that should be watched
>   • Integrate the Watchman package and its Python wrapper into Django

Watchman should be installed separately by developers who wish to use it. 
Django should use it when it's available (and there should be a flag to disable 
that if for any reason someone has watchman installed systemwide but doesn't 
want to use it).

I hope this helps!

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/F3334210-CCA2-4300-A368-6B48D596694B%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.