Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Aymeric,

  Thanks for this detailed and informative response! I will try to figure 
out the difficulties of integrating Watchman during the weekend. For the 
pure-Python solution, I might implement a standalone autoreloader based on 
Carl's work and replace the current one. Does this look good to you?

David Ma

On Thursday, March 30, 2017 at 6:39:42 AM UTC-7, Aymeric Augustin wrote:
>
> Hello, 
>
> > On 29 Mar 2017, at 01:05, qingn...@gmail.com  wrote: 
> > 
> > the best library to replace Watchdog is Watchman, a library that 
> supports both of these three platforms (for Windows it's still in the alpha 
> stage.) 
>
> Django currently doesn't use Watchdog. Watchdog is a cross-platform Python 
> library that can be used for implementing an autoreloader. It's a plausible 
> alternative to Watchman, likely easier to integrate, but Watchman has some 
> fancy features that I'd love to have available -- like waiting for git to 
> finish its operations before notifying a change. 
>
> > It runs as a standalone process and has a python wrapper library that is 
> incompatible with Python 3. Therefore, one of the goals is to figure out 
> how to interact with this library. 
>
> In order to be selected for this project, you have to convince us that you 
> can solve this problem. At a minimum you must prove that you understand 
> where the difficulties lie and suggest some possible approaches to tackle 
> them. 
>
> > So my goals would be: 
> > 
> > • Replace the old auto-reloader with the Watchman version 
>
> Since installing watchman isn't trivial, you need to keep a pure-Python 
> alternative so that beginners get an acceptable experience out of the box. 
> This alternative may be the current "poll every second approach" or could 
> be backed by watchdog, assuming it's pip-installable without complicated 
> dependencies (e.g. a C compiler and development headers for a bunch of 
> libraries) on all operating systems. 
>
> > • Make sure that the new auto-reloader can work with Python3 and 
> on other platforms 
>
> Did you mean "work with Python 2" ? In any case, this isn't necessary, as 
> you'd be working off the master branch which no longer supports Python 2. 
>
> > • Make it compatible with other Python libraries 
> > • Allow Django users to choose the files that should be watched 
> > • Integrate the Watchman package and its Python wrapper into 
> Django 
>
> Watchman should be installed separately by developers who wish to use it. 
> Django should use it when it's available (and there should be a flag to 
> disable that if for any reason someone has watchman installed systemwide 
> but doesn't want to use it). 
>
> I hope this helps! 
>
> -- 
> Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b8f5740c-a154-4b9b-90f1-9a79db712f0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding signals to bulk update/create operations

[Todor Velichkov]

@Tim, sorry about that, I did a search before I posted, but it looks like 
it slipped away somehow.

@Anssi, thank you for your response.
I really haven't think about threat safety, but maybe its because in our 
specific usage it's close to impossible to happen.

What do you think about this:
1) Put everything into a transaction
2) Before update, count the queryset.
3) Fire the pre_update with the queryset, and the count.
4) Do the actual update -> get the updated rows
5) Assert the updated rows is equal to the queryset.count().
6) Revert the transaction if there is a difference.

No pk fetching, this is left to the be implemented by the listener if he 
needs it.

On Friday, March 31, 2017 at 8:01:45 AM UTC+3, Anssi Kääriäinen wrote:
>
> The problem with passing the queryset is that it's possible that some 
> object is added to or removed from the queryset between the pre_update and 
> actual update execution. To avoid this the execution should go somewhere 
> along the lines of:
>1) if there is pre_update or post_update do stages 2-5, if not, update 
> as with current code
>2) fetch primary keys for models to be updated to a set, with 
> .select_for_update() applied
>3) fire pre_update, give the primary key set as argument
>4) do the update against a queryset with .filter(pk__in=pk_set)
>5) fire post_update with primary key set as argument
>
> This way the pre and post update signals will execute against a fixed set 
> of instances. The bad part is that this can significantly slow down the 
> .update() call, but to get actually safe signals, I don't see a way around 
> that.
>
>  - Anssi
>
> On Friday, March 31, 2017 at 3:51:34 AM UTC+3, Tim Graham wrote:
>>
>> There's an accepted ticket about adding pre_update and post_update 
>> signals: https://code.djangoproject.com/ticket/21461. From a quick 
>> glance, I think this is what you're proposing.
>>
>> On Thursday, March 30, 2017 at 4:28:00 PM UTC-4, Todor Velichkov wrote:
>>>
>>> Consider the following piece of code:
>>>
>>> @receiver(pre_save, sender=MyModel)
>>> def my_handler(sender, **kwargs):
>>> instance = kwargs['instance']
>>> if instance.verified:
>>> do_something(instance)
>>> else:
>>> do_else(instance)
>>>
>>>
>>>
>>> Its good, because it keeps `MyModel` decoupled from `do_something`
>>>
>>> But there is one flaw. If we do:
>>>
>>> MyModel.objects.filter(verified=False).update(verified=True)
>>>
>>> we are screwed, `do_something` is not executed, and our models get 
>>> out-of-sync.
>>>
>>> If we try to get smart and manually fire the pre_save signal for each 
>>> instance, we are gonna have a hard time.
>>> Its gonna be slow.
>>> And its gonna be memory inefficient.
>>>
>>> We already experienced it in our app.
>>>
>>> So our new approach is like this:
>>>
>>> pre_bulk_update = Signal(providing_args=["queryset", "update_kwargs"
>>> ])
>>> post_bulk_update = Signal(providing_args=["update_kwargs",])
>>>
>>> pre_bulk_create = Signal(providing_args=["objs", "batch_size"])
>>> post_bulk_create = Signal(providing_args=["objs", "batch_size"])
>>>
>>>
>>> class MyModelQuerySet(models.QuerySet):
>>> def update(self, **kwargs):
>>> pre_bulk_update.send(sender=self.model, queryset=self, 
>>> update_kwargs=kwargs)
>>> res = super(MyModelQuerySet, self).update(**kwargs)
>>> # The queryset will be altered after the update call
>>> # so no reason to send it.
>>> post_bulk_update.send(sender=self.model, update_kwargs=
>>> kwargs)
>>> return res
>>>
>>> def bulk_create(self, objs, batch_size=None):
>>> pre_bulk_create.send(sender=self.model, objs=objs, 
>>> batch_size=batch_size)
>>> res = super(MyModelQuerySet, self).bulk_create(objs, 
>>> batch_size)
>>> post_bulk_create.send(sender=self.model, objs=objs, 
>>> batch_size=batch_size)
>>> return res
>>>
>>>
>>> class MyModel(models.Model):
>>> #...
>>> objects = MyModelQuerySet.as_manager()
>>>
>>>
>>>
>>> This gives us a nice interface to handle all kind of changes regarding 
>>> `MyModel`
>>> Our example usage looks like this:
>>>
>>>
>>> @receiver(pre_save, sender=MyModel)
>>> def my_handler(sender, **kwargs):
>>> instance = kwargs['instance']
>>> if instance.verified:
>>> do_something(instance)
>>> else:
>>> do_else(instance)
>>>
>>>
>>> @receiver(pre_bulk_update, sender=MyModel)
>>> def my_bulk_update_handler(sender, **kwargs):
>>> update_kwargs = kwargs['update_kwargs']
>>> if 'verified' not in update_kwargs:
>>> # no change im interested in
>>> # no need to take any action
>>> return
>>>
>>> queryset = kwargs['queryset']
>>> pks_to_be_updated = queryset.values_list('pk', flat=True)
>>> if update_kwargs['verified']:
>>> do_something_bulk_update_implementation(pks_to_be_updated)
>>> else:
>>>

Re: A proposal for a new auto-reloader in Django

[Melvyn Sopacua]

On Thursday 30 March 2017 22:01:00 qingnian...@gmail.com wrote:
> Hi Carl,
>   Thanks for mentioning this awesome project! I saw it in one of the
> discussions but did not take a close look. I'll definitely check this
> out and try to integrate wsgiwatcher/watcher.py into Django.
> 
> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
> > Anyone working on this project should at least be aware of
> > https://github.com/Pylons/hupper
> >  > pper=D=1=AFQjCNGVwtqvdo53UFfK80kaQ1qxL7ST8Q> (based 
on
> > work David Glick and I
> > originally did in https://github.com/carljm/wsgiwatcher
> >  > giwatcher=D=1=AFQjCNGqLfsmxsp37Lng7_d_DaVzja9c6Q>),
> > which aims to
> > be a framework-agnostic solution to this problem for any Python web
> > project. Docs at
> > http://docs.pylonsproject.org/projects/hupper/en/latest/
> > 
> > Carl
I was going to suggest the same, because you already get watchdog support for 
free 
and can probably build on that integration to pick up watchman. Also, it's used 
in 
devpi[1], so it gets some scruteny from there (and devpi is awesome to 
distribute 
internal or augmented Django apps amongst projects).
-- 
Melvyn Sopacua


[1] http://doc.devpi.net/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1871865.SgAvvE3WaE%40devstation.
For more options, visit https://groups.google.com/d/optout.