Increasing iteration count for the PBKDF2 password hasher

[Shaheed Haque]

Hi,

As happens from time-to-time, I see the 5.1 alpha recently announced has
increased the iteration count for the PBKDF2 password hasher (from 720k to
870k), and the putative release notes for 5.2 mention a further increase
(to 1M).

I assume this iteration count has something to do with the noticeable time
it takes to run User.set_password()? Is there something that can be done to
mitigate any further increase in the execution time of .set_password(), or
am I barking up the wrong tree?

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcETxAtMbHfnD1GQFVgWwR8ABOAy%3DjaRuhRW7mQhnOxeQ%40mail.gmail.com.

Streamlining Model validation of ForeignKeys

[Shaheed Haque]

Hi,

I have two models Source and Input, where Input has an FK to Source like
this:

class Source(db_models.Model):
scheme = db_models.CharField(...)
path = db_models.CharField(...)
display_as = db_models.CharField(...)

class Input(db_models.Model):
name = db_models.CharField(...)
description = db_models.CharField(...)
type = db_models.CharField(...)
source = db_models.ForeignKey(Source, on_delete=db_models.CASCADE)
reuse = db_models.CharField(...)
value = db_models.CharField(...)

The instances of Source are statically created, and number 20 or so in
total. The instances of Input number in the millions, and may be loaded
from external files/network endpoints. When Input instances are created, we
perform normal Django validation processing, involving
form.is_valid()/form.errors. As we know, Django 5.x does something like:

   1. Form field validation using Form.clean_xxx().
   2. Form cross-field validation using Form.clean().
   3. Model validation, including validating the FK.

like this:

...
File "/.../django/forms/forms.py", line 197, in is_valid return
self.is_bound and not self.errors File "/.../django/forms/forms.py", line
192, in errors self.full_clean() File "/.../django/forms/forms.py", line
329, in full_clean self._post_clean() File "/.../django/forms/models.py",
line 496, in _post_clean self.instance.full_clean(exclude=exclude,
validate_unique=False) File "/.../django/db/models/base.py", line 1520, in
full_clean self.clean_fields(exclude=exclude) File
"/.../django/db/models/base.py", line 1572, in clean_fields setattr(self,
f.attname, f.clean(raw_value, self)) File
"/.../django/db/models/fields/__init__.py", line 830, in clean
self.validate(value, model_instance) File
"/.../django/db/models/fields/related.py", line 1093, in validate if not
qs.exists():
...

In one experiment, I observe that this results in 143k queries, taking a
total of 43s. Is there a way to short circuit this Model-level validation?
Since I know the ~20 instances of Source, even a cache of
Source.objects.all() would be a very cheap, but I cannot see any way to
inject that into the code for line 1093:

   def validate(self, value, model_instance):
   if self.remote_field.parent_link:
   return
   super().validate(value, model_instance)
   if value is None:
   return

   using = router.db_for_read(self.remote_field.model,
instance=model_instance)
   qs = self.remote_field.model._base_manager.using(using).filter(
 queryset created here
   **{self.remote_field.field_name: value}
   )
   qs = qs.complex_filter(self.get_limit_choices_to())
   if not qs.exists():
 queryset evaluated here, line 1093
   raise exceptions.ValidationError(...)

We actually have several versions of this problem so any ideas are welcome.

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfzag7v6t7_pvbRnyJ-chMFHyjmkXsDkRouTg7JOteRfQ%40mail.gmail.com.

Raising StopIteration causes Django to become completely unresponsive

[Shaheed Haque]

Hi,

I'm running Django 4.2.7 with Python 3.11 on Ubuntu on my development
setup (so no gunicorn, nginx etc, just standalone Django). I just ran
into the following issue which, once it occurs, causes Django to
become completely unresponsive:

=
2023-12-19 16:07:02,763 [ERROR] asyncio: Exception in callback
_chain_future.._set_state(,
) at /usr/lib/python3.11/async
io/futures.py:381
handle: ._set_state(, ) at
/usr/lib/python3.11/asyncio/futures.py:381>
Traceback (most recent call last):
 File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run
   self._context.run(self._callback, *self._args)
 File "/usr/lib/python3.11/asyncio/futures.py", line 383, in _set_state
   _copy_future_state(other, future)
 File "/usr/lib/python3.11/asyncio/futures.py", line 359, in _copy_future_state
   dest.set_exception(_convert_future_exc(exception))
TypeError: StopIteration interacts badly with generators and cannot be
raised into a Future
=

Once this happens, Django becomes unresponsive (which is what I guess
the dire wording of the exception is hinting at). A trivial reproducer
looks like this:

=
def post(self, request: HttpRequest, *args, **kwargs):
x = next(iter({}.items()))
...
=

Obviously, with an empty dictionary, the next() on the iter()
immediately raises StopIteration, which in turn causes the traceback
above.

In the particular case at hand, the empty dictionary arises for
artificial reasons, and so is not a big concern. However, my code does
have scattered references to next() and my concern is that Django
itself becomes unresponsive.

Am I doing something silly? Is it expected that Django will actually
become unresponsive in such cases?

Advice appreciated.

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jemn9R%3Ds-%2BC06-5oCoZkEyQ_%2BQUyjY%3D%2BW1%3DOzxK%2BxXuTQ%40mail.gmail.com.

Re: Performance profiling Django Channels async consumers

[Shaheed Haque]

On Tue, 28 Nov 2023, 15:26 Filbert,  wrote:

> crickets...sigh...
>

Is your question is more about profiling Python (async) code than anything
specifically to do with Django?

If so, then obviously Google is likely a good place to start.

If not, and your issue is more about support for async from tools like the
Django Debug Toolbar, it might be worth being explicit about that.

(FWIW, we faced a similar issue with trying to analyse db usage from
Celery, and ended up brewing some custom tooling loosely based on the
Django Debug Toolbar output).




On Monday, November 27, 2023 at 11:29:37 AM UTC-5 Filbert wrote:
>
>> We are heavily using Django Channels async consumers and haven't found a
>> way instrument for performance profiling.
>>
>> We've tried NewRelic, but per their developers say they don't have
>> support for Django Channels.
>>
>> We run certain websocket connections through Gunicorn, Uvicorn,
>> Channels-Async and are looking for anyway to instrument the code to get a
>> better analysis of what code is hogging the event loop.
>>
>> Advice appreciated.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/9e05ae10-ca38-4853-b875-d4594bda394en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdNB8xh7Rk8q0uLq%3DfEuoyvA38ZgutMhAdEn6KBrdez%3DA%40mail.gmail.com.

Running migrations with multiple databases

[Shaheed Haque]

Hi,

I'm on Django 4.2 atop Postgres. In my project settings.py, I have a
main/default database connection, and a second set up like this:

==
DATABASES = {
'default': {
...
'NAME': 'foo', # Postgres DATABASE NAME
...
},
'archive_restore': {
...
'NAME': 'archive_restore',   # Postgres DATABASE NAME
...
},
# Other entries for specialised purposes such as custom Postgres
Foreign Data Wrappers.
===

As you can see, the Postgres database names are "foo" and "archive_restore"
respectively. For all normal Django purposes, we want to use "default", aka
"foo". The "archive_restore" connection/database is used in conjunction
with a bunch of psql commands to create a subset of the main Django ORM
data. Once populated via pg_restore, I need to run the DJango migrations on
them for eventual use under Django. I had assumed that a command like this:

./manage.py migrate --database archive_restore

would modify archive_restore/archive_restore. However, what seems to happen
is that the migrations are:

   - Actually run in default/foo (the migrations include code generated by
   makemigrations and custom RunPython stuff).
   - But recorded in the django_migrations table in
   archive_restore/archive_restore.

whereas I was expecting that they would be both run and recorded in the
latter. Have I overlooked some setting/restriction, or is this a bug?

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2je%2B%3D67tZBeGBMUO8Dmqmq9SaCRuopLxqrECuXb1C8YriQ%40mail.gmail.com.

Re: Slow Performance of Django Web App

[Shaheed Haque]

Have you tried using https://django-debug-toolbar.readthedocs.io/en/latest/?
It can be a little fiddling about with, but it's time based profiling
support has been invaluable in tracking down a few critical queries.

On Sun, 11 Jun 2023, 20:00 Kunal Mittal,  wrote:

> I've a django application serving 10k+ users daily, that application has
> everything to be on the top of performance metrics but still some API's
> have high latencies, application serves lots of pages with nice css and
> various images, on most the pages they are being served by s3 while on some
> it's via django static files. We have implemented caching on akamai for
> every static file that application serves, whenever that cache disappears
> latency of each and every API spikes exponentially, at that time even
> loading metrics becomes a pain, server does not pick those requests. Can
> anyone suggest what's going on here.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/d32868ba-a2df-41e3-8f41-3ecc3c1b3471n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2je1B-m24UtOCjfK_VwQj1TYrdQH8czBS-bLotPsx1hGyw%40mail.gmail.com.

Re: Django channels scalling

[Shaheed Haque]

Generally, as you may know, it is not very easy to get memory usage stats
on Unix/Linux without specialised tooling.

However, in my experience, with default TCP settings, the kernel overhead
plus whatever "typical" app overhead will not often exceed say 100 kB per
connection.

Now, we had originally used an even smaller 1GB server than yours, and we
did run out of memory, with attendant catastrophic failures. We were able
to trace the failures to certain ORM queries on tables with large JSONB
columns (in the MB range). Once we addressed that, we were fine. At no
point, even with our usage  pattern of connect per message (!!!) , did
sockets usage even warrant a second thought.


On Thu, 22 Dec 2022, 00:01 Divit Rao,  wrote:

> Hi Team,
> currently I have deployed a Django project on EC2 server which has a RAM
> of about 2GB using Gunicorn and Ngnix, I'm using Django channels on same
> server, so as of now we have around 200 active users. currently each user
> will have 5 active WebSocket connections/consumers (for chat, notification,
> etc.) open so currently around 1000 open connections are present at a time.
> As of now WebSocket's are working fine but in future we are expecting a
> growth in user so as the active WebSocket connections will grow, and at
> some point, due to excessive load, the server will start throwing issues
> regarding socket connections my question is how to scale django channels.
> Like should i just increase server size or is there any other solutions?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/3e759081-5b3e-4a74-9668-e54e2e4e9d56n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfmotxJB3MPtKp8wU7iSpmYAupmyj%3DJtGC5ZU2KxN6aaw%40mail.gmail.com.

Re: getting unique id

[Shaheed Haque]

Are all the threads in the same Python process? Or the same machine? Do
they have to persist across process (or machine) restarts?

On Wed, 23 Nov 2022, 16:57 Larry Martell,  wrote:

> I have an app that needs to get a unique ID. Many threads run at the
> same time that need one. I would like the IDs to be sequential. When I
> need a unique ID I do this:
>
> with transaction.atomic():
> max_batch_id =
>
> JobStatus.objects.select_for_update(nowait=False).aggregate(Max('batch_id'))
> json_dict['batch_id'] = max_batch_id['batch_id__max'] + 1
> status_row = JobStatus(**json_dict)
> status_row.save()
>
> But multiple jobs are getting the same ID. Why does the code not work
> as I expect? What is a better way to accomplish what I need?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CACwCsY54vF6ZK7yy--15hy1u5vxJ%2Bq9Zp_ODGfAX973gPYpecw%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jczhpfeAy3khqQiW-LQfKekidQbZs6GqvjDXemi1S8KcQ%40mail.gmail.com.

Re: Multi-tenant SSO that supports OpenID and MS Azure AD

[Shaheed Haque]

Hi,

On Tue, 8 Nov 2022 at 17:54, Filbert  wrote:

> Thanks. By multiple IDPs, do you men multiple IDPs for multiple tenants?
> Since we are multi-tenant, some tenants will not have IDP providers, others
> may use social auths, and most will use their own IDP like
> LDAP/AD/Azure/etc.
>

I'll explain what I needed/have implemented, and you can decide if that
maps to your problem domain. We have a cloud application accessed by
multiple Clients. Each Client has multiple administrators and multiple
non-admin users. Client A's data/users must be fully separated from Client
B's data/users (except that there are globally unique login identifiers).
All this is built on standard Django password-based authentication, with a
custom multi-tenancy implementation.

   - In addition, individual non-admin users can use django-social-auth to
   login via one of several Social Networks.
   - In addition, a Client's admins can configure multiple SAML IdPs for
   SSO. This works out of the box using django-social-auth's SAML backend.
   - In addition, a Client's admins can configure multiple OIDC Identity
   Providers for SSO. Since the django-social-auth OIDC backend only supports
   one Identity Provider, I wrapped it with a layer more-or-less inspired by
   the SAML backend to allow for multiple Identity Providers.

I model both the SAML IdPs and OIDC Identity Providers using a single
Django model. In principle, this layer is extensible to accomodate any
behaviour differences between different implementations. That said, to this
point, everything works with completely standard protocol handling.

Now, I'm really just a noob as far as all this is concerned, however, I
added the following features which seemed essential to me:

   - Social Network authentication can be disabled by a Client's admins
   using a hook into django-social-auth's pipeline.
   - Both the SAML and OIDC SSO implementations are "sandboxed" to the
   given Client.The idea being that a breach of Client A's authentication
   setup must not be allowed to compromise Client B.

I'm sure there are clever folk out there who could prove that these extras
were not needed, but hey ho.

Hth, Shaheed


> On Monday, November 7, 2022 at 2:49:18 PM UTC-5 shahee...@gmail.com wrote:
>
>> Tim,
>>
>>
>> On Mon, 7 Nov 2022, 19:28 Tim Nelson,  wrote:
>>
>>> I am looking for a Django SSO package that supports OpenId and Azure AD
>>> with multi-tenant support. My sense is I am going to have to roll my own by
>>> forking the best of breed SSO's and adding multi-tenant support to them.
>>>
>>> Is my assessment correct?
>>>
>>
>> If I understand your question correctly, yes.
>>
>> For context, I needed to add generic support for both SAML and OIDC based
>> SSO. The SAML support in django-social-auth supports multiple IDPs
>> directly, but the OIDC support does not. I had to brew up a wrapper which
>> gave me the effect of multiple Identity Providers.
>>
>> Thanks.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/CAK09zooyoOOKO7Dyp5LAbqWDphLwfQLBKX2rRWhXZHPqNNjEtQ%40mail.gmail.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/afd7fa61-d2a1-4a18-8e95-8a976cd5a5ban%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jd6PepmON2D3xLSGetTMtGxnt8xBbvaBtb70Q2PdRGpKQ%40mail.gmail.com.

Re: Multi-tenant SSO that supports OpenID and MS Azure AD

[Shaheed Haque]

Tim,

On Mon, 7 Nov 2022, 19:28 Tim Nelson,  wrote:

> I am looking for a Django SSO package that supports OpenId and Azure AD
> with multi-tenant support. My sense is I am going to have to roll my own by
> forking the best of breed SSO's and adding multi-tenant support to them.
>
> Is my assessment correct?
>

If I understand your question correctly, yes.

For context, I needed to add generic support for both SAML and OIDC based
SSO. The SAML support in django-social-auth supports multiple IDPs
directly, but the OIDC support does not. I had to brew up a wrapper which
gave me the effect of multiple Identity Providers.

Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAK09zooyoOOKO7Dyp5LAbqWDphLwfQLBKX2rRWhXZHPqNNjEtQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jffhzb8b1nbdmJde0KJrC4s4j5L13%2BjVsX5LAvC9UwLYA%40mail.gmail.com.

Re: Is there a way to pass arbitrary argument from serializer.save to model.save method

[Shaheed Haque]

Add an attribute to instance?

On Fri, 23 Sep 2022, 19:21 Sencer Hamarat,  wrote:

> Hi,
>
> I need to pass an arbitrary argument from the serializer.save() method to
> model.save() method?
>
> Such as:
>
> serialized_data = AModelSerializer(instance, data=adata_dict)
> if serialized_data.is_valid():
> serialized_data.save(an_arg="value")
>
> class Amodel(models.Model)
> def save(*args, **kwargs):
>  extra_arg = kwargs["an_arg"]
>
>
> The method demonstrated above is not working due to nature of serializer.
> Bu I can't figure out any other way to do that.
>
> Right now, I'm open to suggestions.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CACp8TZjF3_eQ6KSQKnatVHQGQsh%3Dz6Kr4pdVEM151VbnSq5N1g%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcbEjKR%2Bpkww3r8sB3Kh1%3D3JfhYQA9niMBHhiZ9AUgE8w%40mail.gmail.com.

Re: Need help on reducing cognitive complexity

[Shaheed Haque]

The first step is to ensure you have a precise and accurate understanding
of the code before optimization. Right now, I am suspicious of at least 2
things in the code.

First, the function returns any of {None, -1, True}. While this is
certainly allowed, is this correct? If not, please correct the code. If so,
then can you explain the rule that governs the return type (assuming it is
not literally what is written).

Second, as your previous respond ant hinted, the first 2 lines look like an
optimisation attempt, except for the return type issue. Can you clarify if
the purpose of these lines is optimisation, or not?




On Thu, 21 Jul 2022, 06:53 Sencer Hamarat,  wrote:

> Hi everyone,
>
> I have a code block with high cognitive complexity below:
>
> @staticmethod
> def in_circle_check(obj_type, item, uuid, item_is_array=False):
>
> if obj_type not in ['trees', 'flowers']:
> return None
>
> plant = get_plant_with_circles(uuid)
> if not plant:
> return None
>
> if obj_type == 'trees':
> if plant['in_all_tree_types']:
> return True
> circle_objects = plant['tree_circle']['items']
> else:
> if plant['in_all_flower_types']:
> return True
> circle_objects = plant['flower_circle']['items']
>
> if isinstance(circle_objects, list):
> if item_is_array:
> for item in item:
> if item not in circle_objects:
> return -1
> return True
> else:
> if item in circle_objects:
> return True
> else:
> return -1
>
> return -1
>
>
> I try to move the first 3 if statements on the beginning of the block into
> new methods but that maneuver ends up with raising complexity.
>
> Can anybody help me to learn how to reduce cognitive complexity in code
> blocks like this?
>
> Kind regards,
> Sencer HAMARAT
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CACp8TZhBa9EbekcT1ApmzdDRVk2vCb64%3DvvXHrSawO2RJSySpQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jewMaP41PJZN8OVFqZ2hw41A9r8c8DDjRJc0VG%3DqP9X5g%40mail.gmail.com.

Re: Django could not parse the remainder

[Shaheed Haque]

>From memory, At least in Jinja templates, you can explicitly use the
dictionary form {{ key["hypen-ated"] }}... Have you tried that?

On Thu, 23 Jun 2022, 14:44 Ryan Nowakowski,  wrote:

>
> https://stackoverflow.com/questions/8252387/how-do-i-access-dictionary-keys-that-contain-hyphens-from-within-a-django-templa
>
> On Mon, Jun 20, 2022 at 07:52:46AM -0700, Koushik Romel wrote:
> > When trying to get a value from restapi in for loop to display it takes
> > hyphen(-) as minus(-). And there is no way to change the api structure
> > because it is fixed.
> > Im struck in this anyone help me..
> > The exact error and the expected json is given below is down below
> >
> > when i try to print {{key}} the whole for loop data it displays like this
> > {'_id': ObjectId('xxx'), 'date': 'jun/20/2022', 'dst-active':
> > 'false', 'gmt-offset': '+05:30', 'time': '20:19:04',
> > 'time-zone-autodetect': 'true', 'time-zone-name': 'Asia/Kolkata'}
> >
> > when i specify {{key.dst-active}} it throws this error
> > TemplateSyntaxError at /store/Could not parse the remainder: '-active'
> from
> > 'key.dst-active'
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/20220623134418.GE18918%40fattuba.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdi0tCsM1ypeUir9NRmw4LjOgUuR6VEJyrpXt4np9gt5A%40mail.gmail.com.

Re: String Concatenate

[Shaheed Haque]

On Wed, 20 Apr 2022, 11:05 Kasper Laudrup,  wrote:

> On 20/04/2022 11.33, Ankit Chaurasia wrote:
> > from django.db import models
> > class Customer(models.Model):
> >  name = models.CharField(max_length=20)
> >  phone = models.IntegerField()
> > I want to concatenate name + last digit of phone number.
> >
>
> It makes no sense to store a phone number as an integer. A phone number
> is not something you can increment, decrement or do similar arithmetic
> operations with.
>

Not only that, but it is all too easy to lose the leading zeroes that are
often part of a phone number. Kasper is right: don't go there. (I won't
even mention about leading plus signs...)


> Just use a CharField or probably even better, a field dedicated to phone
> numbers:
>
> https://pypi.org/project/django-phonenumber-field/
>
> Using a more proper field type will make it simple to do the
> concatenation using standard Python.
>
> Kind regards,
>
> Kasper Laudrup
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/789404f7-e665-d102-1dee-406cbfcfe4cd%40stacktrace.dk
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdzq8qvP47BfxgDbRJ25qAM4cqBvqPhmFZWvvZM3-rqzw%40mail.gmail.com.

Re: How to create dynamic models in django??

[Shaheed Haque]

Check out the api mentioned here
https://stackoverflow.com/questions/34768732/temporary-models-in-django#48042468

On Sun, 27 Feb 2022, 03:42 Prashanth Patelc, 
wrote:

> How to create dynamic models in django rest framework?
> Is there any chance to create dynamic models with APIs
>
> Any examples please send me thanks in advance..
>
> 1) Requirement is need create table name and fields in frontend
> 2) we are  getting the data and store in to the db create db structure
> 3) get the table name and fields create table in backend &postgresql store
> to
> 4)this code don't update or add into the models
> 5)store the data into the tables
> 4)get the data into the tables using orm or any raw queries
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAMCU6Coop%3DVU67%2B1Ui%2BswvA2xijUYcre%2B%3D-Z_CzLnf-1pRRXUw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jeQh8xK7jzvpk202%3DW0hi89LDoFOKAD_eG_syJbmQG-zg%40mail.gmail.com.

Re: What to treat as 'source'

[Shaheed Haque]

On Fri, 18 Feb 2022 at 00:16, Michael Powell  wrote:
>
> Hello,
>
> Perhaps a general CMS related topic... What (necessarily, 'how') do we treat 
> as source code? If we have a Django or an OrchardCore/CMS, for example. The 
> infrastructure, obviously, i.e. the framework, intermediate derivations, etc. 
> What about any authoring efforts, milestones, versioning, etc? Similarly, 
> where/when are the authoring efforts usually done? As part of 'dev' inner 
> loop? Or might there also be an author inner/outer loop? Then with whatever 
> corresponding database backups, migrations, etc.

I think of it like this:

- Starting with "If I were to try to do a release of my product, what
files would I need?"
- Subtract, or rather separate out, any credentials since live
credentials won't be in checked-in source.
- Add in any files and artefacts needed to run tests.
- Add in anything needed to deploy the product.
- Add in any source for documentation, either customer facing or
internal specs/procedures.
- Some special consideration may be needed for large binary objects.

All that belongs in the source control system, meaning that it is
backed up along with development history/comments. Note that this is
essentially independent of whether the product is software: it could
just as easily be the design for an ASIC, or a marketing website.

End-user data is NOT part of the source (it should be backed up of
course, but not in the source control system).

Does that help?

Thanks, Shaheed

> Thank you...
>
> Best regards,
>
> Michael W. Powell
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAM%3DaUXoVdL4RgHbS6dk7mw-dB6n7SBcx3g5Q0RiJhRG_GTuTeQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcntpXpWGzku%3DjV2-Key-Uz9bSH4epsCbVbNyB1Vf4N5g%40mail.gmail.com.

Re: Idea

[Shaheed Haque]

Oh and it's never one language at the front end. Start with HTML(5), throw
in CSS and JS(ES6?).

And that's before jQuery(old hat), Vue, React, Angular(N). All of which
look suspiciously like domain specific languages in their mutual
incompatibility despite being JS.

The reality is not of language choices, but that multiple domains have been
found to exist, and trying to span them with one syntax is very hard.

On Sun, 31 Oct 2021, 23:32 Peter of the Norse,  wrote:

> Most of the people that think switching to node.js only know PHP.  That’s
> why it looks like a good idea.
>
> On Sep 20, 2021, at 2:46 AM, David Nugent  wrote:
>
> This is almost the same case that nodejs users have put forward since it
> was invented - why learn 2 languages when 1 will do?  node backend and js
> front end, one language.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/380EFF57-ED6C-493B-AD3E-921FC1F1AB98%40gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdXW7EFWs0RFftd3%2BABce5HK656p4Faqmdcu9BaTdiFnw%40mail.gmail.com.

Re: DJNAGO Many to Many field MultiSelect Into DropdownList

[Shaheed Haque]

On Mon, 18 Oct 2021 at 15:41, Sebastian Jung 
wrote:

> When u use SelectMultiple then you get a select widget where you can
> select many options Your code are wrong in forms.py in widgets=
>
> Am Mo., 18. Okt. 2021 um 16:22 Uhr schrieb 'Maryam Yousaf' via Django
> users :
>
>> It is already selectmultiple . I need dropdown where I can select
>> multiple values .
>> currently, it is coming like this:
>>
>
Apologies if this is stating the obvious, but to select more than one item,
are you using "shift+click" or "ctrl+click" or just "click"?

You need to use the modifier key with "click", not just "click".


>
>> On Mon, 18 Oct 2021 at 16:07, Sebastian Jung 
>> wrote:
>>
>>> Hello,
>>>
>>> then change in widgets Select() to SelectMultiple().
>>>
>>> https://docs.djangoproject.com/en/3.2/ref/forms/widgets/#selectmultiple
>>>
>>> Regards
>>>
>>> Am Mo., 18. Okt. 2021 um 15:51 Uhr schrieb 'Maryam Yousaf' via Django
>>> users :
>>>
 Hi,

 I have one manytomany field in one of my model which is currently
 coming as Multiselect but I need a dropdown where user can select multiple
 values as I have huge data to show.

 I am trying this in forms.py but it is not showing dropdown field.

 Kindly help me out.

 *Model.py:*

 class Chain(models.Model):
 chain_id = models.AutoField(primary_key=True)
 chain_name = models.CharField(max_length=255, unique=True)
 chain_type = models.ForeignKey(ChainType, on_delete=models.CASCADE)
 history = HistoricalRecords()

 def __str__(self):
 return self.chain_name

 class Meta:
 ordering = ('chain_name',)


 class Brg(models.Model):
 brg_campaign_id = models.AutoField(primary_key=True)
 campaign_tracking = models.ForeignKey(CampaignTracking,
 on_delete=models.CASCADE)
 brg_name = models.ForeignKey(Chain, on_delete=models.CASCADE,
 related_name="primary_brg",
 help_text='Add Brgs/chain names for these above campaign has run')
 brg_benchmark = models.ManyToManyField(Chain,
 related_name="competitor_brg", null=True,
 blank=True, help_text='Add max.5 other benchmarks brgs to check overall
 impact')
 history = HistoricalRecords()

 def __str__(self):
 return "Brg names list"

 class Meta:
 verbose_name = 'Brg Campaign Tracking'

 *forms.py:*
 class ChainForm(forms.ModelForm):
 class Meta:
 model: Chain
 # fields = ('campaign_tracking', 'brg_name', 'brg_benchmark',)
 widgets = {
 'chain_name': Select(),
 }

 Regards,
 maryam


 --
 This email and any files transmitted with it contain confidential
 information and/or privileged or personal advice. This email is intended
 for the addressee(s) stated above only. If you are not the addressee of the
 email please do not copy or forward it or otherwise use it or any part of
 it in any form whatsoever. If you have received this email in error please
 notify the sender and remove the e-mail from your system. Thank you.

 This is an email from the company Just Eat Takeaway.com N.V., a public
 limited liability company with corporate seat in Amsterdam, the
 Netherlands, and address at Oosterdoksstraat 80, 1011 DK Amsterdam,
 registered with the Dutch Chamber of Commerce with number 08142836 and
 where the context requires, includes its subsidiaries and associated
 undertakings.

 --
 You received this message because you are subscribed to the Google
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-users/ac59ba2a-5baa-4302-88c9-0dd17606fdaen%40googlegroups.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/CAKGT9mwXNTc2638qv-vScbHZW2uRq3utmu%2BbfM5Mz06WERmt2g%40mail.gmail.com
>>> 
>>> .
>>>
>>
>>
>> --
>> This email and any files transmitted with it contain confidential
>> information and/or privileged or personal advice. This email is intended
>> for the addressee(s) stated above only. If you are not the addressee of the
>> email please do not copy or forward it or otherwise use it or any par

How to update part of a JSONField with a computed value?

[Shaheed Haque]

Hi,

I'm using Django 3.2 on Postgres12, and I have a model with a JSONField
which contains a simple dict (actually, a Django formset :-)). I would like
to update just one value in the dict. I can get this to work when the new
value is a hardcoded numeric 333 (quoted, as seems to be needed) like this:

pay_items.update(inputs=JSONBSet('inputs', ['inputs-0-value'],
Value("333"), True))

where JSONBSet is a Func, as indicated below. However, I'd like to actually
compute the value. I've tried a variety of things, without success. Here is
one attempt:

pay_items.update(inputs=JSONBSet('inputs', ['inputs-0-value'],
 Cast(F('inputs__inputs-0-value'),
FloatField()) + Value("3"),
 True))


This fails like this:

ERROR:  function jsonb_set(jsonb, unknown, double precision, boolean) does
not exist at character 42
HINT:  No function matches the given name and argument types. You might
need to add explicit type casts.
STATEMENT:  UPDATE "paiyroll_payitem" SET "inputs" =
JSONB_SET("paiyroll_payitem"."inputs", '{inputs-0-value}',
(CAST(("paiyroll_payitem"."inputs" -> 'inputs-0-value') AS double
precision) + '3'), true) WHERE "paiyroll_payitem"."id" IN (...))

I've tried all sorts of variants with Cast, ExpressionWrapper and so on to
no avail, so I'd be grateful for any shove in the right direction!

Thanks, Shaheed

Encls: implementation of JSONBSet derived from
https://code.djangoproject.com/ticket/32519 and related threads:

class JSONBSet(Func):
function = 'JSONB_SET'
arity = 4
output_field = CharField()

def __init__(self, field, path, value, create: bool = True):
path = Value('{{{0}}}'.format(','.join(path)))
create = Value(create)
super().__init__(field, path, value, create)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcT_k36yc3atyJHj7rHiMf0PnvsEdm6bbT7L%3DhgDpZbyQ%40mail.gmail.com.

Re: Filtering OR-combined queries

[Shaheed Haque]

Just to close the loop on this, the defect in
https://code.djangoproject.com/ticket/32717 is on its way to being
resolved, but my original reason for using OR'd queries rather than Q()
expressions was that I had not quite grokked the use of ~Q() to implement
.exclude(). Simon kindly pointed out the code could have been written like
this:

def jurisdiction_qs(for_jurisdiction):
filter_ = Q(jurisdiction=for_jurisdiction)
if for_jurisdiction != 'Universal':
filter_ |= Q(jurisdiction='Universal') & ~Q(
name__in=Buss.objects.filter(filter_).values_list('name', flat=True)
)
if for_jurisdiction != 'Company':
   filter_ |= Q(jurisdiction='Company') & ~Q(
name__in=Buss.objects.filter(filter_).values_list('name', flat=True)
)
return Buss.objects.filter(filter_)


Notice the use of "& ~Q()"...

Thanks, Shaheed

On Wed, 5 May 2021 at 17:01, Shaheed Haque  wrote:

> After testing with 3.2.1, I filed
> https://code.djangoproject.com/ticket/32717.
>
> On Tue, 4 May 2021 at 13:23, Shaheed Haque  wrote:
>
>> Simon,
>>
>> Thanks for the heads up. At first glance, the release notes don't *quite*
>> seem to match what I see but as you suggest, I will certainly check the new
>> release before filing an issue.
>>
>> TTFN, Shaheed
>>
>> On Mon, 3 May 2021 at 16:10, Simon Charette  wrote:
>>
>>> Hello Shaheed,
>>>
>>> I didn't look at your issue into details but since it involves exclude,
>>> subqueries, an queryset combination and only manifests itself on Django 3.2
>>> I wouldn't be surprised if it was related to some regressions in this area
>>> that are fixed in Django 3.2.1 which is meant to be released tomorrow
>>>
>>> https://docs.djangoproject.com/en/3.2/releases/3.2.1/
>>>
>>> Cheers,
>>> Simon
>>> Le vendredi 30 avril 2021 à 09:58:53 UTC-4, shahee...@gmail.com a
>>> écrit :
>>>
>>>> Hi,
>>>>
>>>> On Fri, 30 Apr 2021 at 11:52, Sebastian Jung 
>>>> wrote:
>>>>
>>>>> Take q for or Filterung:
>>>>>
>>>>>
>>>>> https://stackoverflow.com/questions/6567831/how-to-perform-or-condition-in-django-queryset
>>>>>
>>>>
>>>> Thanks for the response, but the OR is not the problem: it works just
>>>> fine. (FWIW, I used the "|" form rather than the Q() form because it is not
>>>> clear to me how one can express a .exclude() when using Q()). The problem
>>>> here is that the result of the OR cannot be reliably .filter()d.
>>>>
>>>> As I tried to explain, I can see that in some cases, a .filter() of the
>>>> OR'd result does indeed add extra SQL to the query, whereas in others it
>>>> does not:
>>>>
>>>> qs_or_result.filter(a='b')
>>>>
>>>> This works for some values of "a" but not others, I can use this to
>>>> achieve the same effect:
>>>>
>>>> Buss.objects.filter(a='b').intersection(qs_or_result)
>>>>
>>>> though it is rather clumsy!!! FWIW, this is with Django 3.2. I'm
>>>> inclined to think this is a bug in the ORM, though I have not dived into
>>>> the code to track it down.
>>>>
>>>> Thanks, Shaheed
>>>>
>>>>
>>>>>
>>>>>
>>>>> Shaheed Haque  schrieb am Fr., 30. Apr. 2021,
>>>>> 02:43:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a query which ORs some selects on a single table together like
>>>>>> this:
>>>>>>
>>>>>> jurisdiction = 'aaa'
>>>>>> qs = Buss.objects.filter(jurisdiction=jurisdiction)
>>>>>> qs = qs | Buss.objects.filter(jurisdiction='xxx').exclude(name__in
>>>>>> =qs.values_list('name', flat=True))
>>>>>> qs = qs | Buss.objects.filter(jurisdiction='yyy').exclude(name__in
>>>>>> =qs.values_list('name', flat=True))
>>>>>>
>>>>>> This seems to work just fine (and the raw SQL looks suitably
>>>>>> complicated):
>>>>>>
>>>>>> SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
>>>>>> "paiyroll_buss"."name

Re: Filtering OR-combined queries

[Shaheed Haque]

After testing with 3.2.1, I filed
https://code.djangoproject.com/ticket/32717.

On Tue, 4 May 2021 at 13:23, Shaheed Haque  wrote:

> Simon,
>
> Thanks for the heads up. At first glance, the release notes don't *quite*
> seem to match what I see but as you suggest, I will certainly check the new
> release before filing an issue.
>
> TTFN, Shaheed
>
> On Mon, 3 May 2021 at 16:10, Simon Charette  wrote:
>
>> Hello Shaheed,
>>
>> I didn't look at your issue into details but since it involves exclude,
>> subqueries, an queryset combination and only manifests itself on Django 3.2
>> I wouldn't be surprised if it was related to some regressions in this area
>> that are fixed in Django 3.2.1 which is meant to be released tomorrow
>>
>> https://docs.djangoproject.com/en/3.2/releases/3.2.1/
>>
>> Cheers,
>> Simon
>> Le vendredi 30 avril 2021 à 09:58:53 UTC-4, shahee...@gmail.com a écrit :
>>
>>> Hi,
>>>
>>> On Fri, 30 Apr 2021 at 11:52, Sebastian Jung 
>>> wrote:
>>>
>>>> Take q for or Filterung:
>>>>
>>>>
>>>> https://stackoverflow.com/questions/6567831/how-to-perform-or-condition-in-django-queryset
>>>>
>>>
>>> Thanks for the response, but the OR is not the problem: it works just
>>> fine. (FWIW, I used the "|" form rather than the Q() form because it is not
>>> clear to me how one can express a .exclude() when using Q()). The problem
>>> here is that the result of the OR cannot be reliably .filter()d.
>>>
>>> As I tried to explain, I can see that in some cases, a .filter() of the
>>> OR'd result does indeed add extra SQL to the query, whereas in others it
>>> does not:
>>>
>>> qs_or_result.filter(a='b')
>>>
>>> This works for some values of "a" but not others, I can use this to
>>> achieve the same effect:
>>>
>>> Buss.objects.filter(a='b').intersection(qs_or_result)
>>>
>>> though it is rather clumsy!!! FWIW, this is with Django 3.2. I'm
>>> inclined to think this is a bug in the ORM, though I have not dived into
>>> the code to track it down.
>>>
>>> Thanks, Shaheed
>>>
>>>
>>>>
>>>>
>>>> Shaheed Haque  schrieb am Fr., 30. Apr. 2021,
>>>> 02:43:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a query which ORs some selects on a single table together like
>>>>> this:
>>>>>
>>>>> jurisdiction = 'aaa'
>>>>> qs = Buss.objects.filter(jurisdiction=jurisdiction)
>>>>> qs = qs | Buss.objects.filter(jurisdiction='xxx').exclude(name__in
>>>>> =qs.values_list('name', flat=True))
>>>>> qs = qs | Buss.objects.filter(jurisdiction='yyy').exclude(name__in
>>>>> =qs.values_list('name', flat=True))
>>>>>
>>>>> This seems to work just fine (and the raw SQL looks suitably
>>>>> complicated):
>>>>>
>>>>> SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
>>>>> "paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"
>>>>> WHERE ("paiyroll_buss"."jurisdiction" = aaa OR
>>>>> ("paiyroll_buss"."jurisdiction" = xxx AND NOT ("paiyroll_buss"."name" IN
>>>>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa)))
>>>>> OR ("paiyroll_buss"."jurisdiction" = yyy AND NOT ("paiyroll_buss"."name" 
>>>>> IN
>>>>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE (U0."jurisdiction" = aaa 
>>>>> OR
>>>>> (U0."jurisdiction" = xxx AND NOT (U0."name" IN (SELECT U0."name" FROM
>>>>> "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa
>>>>>
>>>>> Now, if I post-filter qs using something other than "aaa" (the first
>>>>> term above) like this:
>>>>>
>>>>>qs.filter('xxx)
>>>>>
>>>>> then the resulting SQL has an 'AND "paiyroll_buss"."jurisdiction" =
>>>>> xxx' as one might

Re: Filtering OR-combined queries

[Shaheed Haque]

Simon,

Thanks for the heads up. At first glance, the release notes don't *quite*
seem to match what I see but as you suggest, I will certainly check the new
release before filing an issue.

TTFN, Shaheed

On Mon, 3 May 2021 at 16:10, Simon Charette  wrote:

> Hello Shaheed,
>
> I didn't look at your issue into details but since it involves exclude,
> subqueries, an queryset combination and only manifests itself on Django 3.2
> I wouldn't be surprised if it was related to some regressions in this area
> that are fixed in Django 3.2.1 which is meant to be released tomorrow
>
> https://docs.djangoproject.com/en/3.2/releases/3.2.1/
>
> Cheers,
> Simon
> Le vendredi 30 avril 2021 à 09:58:53 UTC-4, shahee...@gmail.com a écrit :
>
>> Hi,
>>
>> On Fri, 30 Apr 2021 at 11:52, Sebastian Jung 
>> wrote:
>>
>>> Take q for or Filterung:
>>>
>>>
>>> https://stackoverflow.com/questions/6567831/how-to-perform-or-condition-in-django-queryset
>>>
>>
>> Thanks for the response, but the OR is not the problem: it works just
>> fine. (FWIW, I used the "|" form rather than the Q() form because it is not
>> clear to me how one can express a .exclude() when using Q()). The problem
>> here is that the result of the OR cannot be reliably .filter()d.
>>
>> As I tried to explain, I can see that in some cases, a .filter() of the
>> OR'd result does indeed add extra SQL to the query, whereas in others it
>> does not:
>>
>> qs_or_result.filter(a='b')
>>
>> This works for some values of "a" but not others, I can use this to
>> achieve the same effect:
>>
>> Buss.objects.filter(a='b').intersection(qs_or_result)
>>
>> though it is rather clumsy!!! FWIW, this is with Django 3.2. I'm inclined
>> to think this is a bug in the ORM, though I have not dived into the code to
>> track it down.
>>
>> Thanks, Shaheed
>>
>>
>>>
>>>
>>> Shaheed Haque  schrieb am Fr., 30. Apr. 2021,
>>> 02:43:
>>>
>>>> Hi,
>>>>
>>>> I have a query which ORs some selects on a single table together like
>>>> this:
>>>>
>>>> jurisdiction = 'aaa'
>>>> qs = Buss.objects.filter(jurisdiction=jurisdiction)
>>>> qs = qs | Buss.objects.filter(jurisdiction='xxx').exclude(name__in
>>>> =qs.values_list('name', flat=True))
>>>> qs = qs | Buss.objects.filter(jurisdiction='yyy').exclude(name__in
>>>> =qs.values_list('name', flat=True))
>>>>
>>>> This seems to work just fine (and the raw SQL looks suitably
>>>> complicated):
>>>>
>>>> SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
>>>> "paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"
>>>> WHERE ("paiyroll_buss"."jurisdiction" = aaa OR
>>>> ("paiyroll_buss"."jurisdiction" = xxx AND NOT ("paiyroll_buss"."name" IN
>>>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa)))
>>>> OR ("paiyroll_buss"."jurisdiction" = yyy AND NOT ("paiyroll_buss"."name" IN
>>>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE (U0."jurisdiction" = aaa OR
>>>> (U0."jurisdiction" = xxx AND NOT (U0."name" IN (SELECT U0."name" FROM
>>>> "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa
>>>>
>>>> Now, if I post-filter qs using something other than "aaa" (the first
>>>> term above) like this:
>>>>
>>>>qs.filter('xxx)
>>>>
>>>> then the resulting SQL has an 'AND "paiyroll_buss"."jurisdiction" =
>>>> xxx' as one might expect. However, if I try to post-filter qs like this:
>>>>
>>>> qs.filter('aaa') # 'aaa' was the first term in the original trio of
>>>> clauses
>>>>
>>>> Then the formed SQL looks like this:
>>>>
>>>> SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
>>>> "paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"

Re: Filtering OR-combined queries

[Shaheed Haque]

Hi,

On Fri, 30 Apr 2021 at 11:52, Sebastian Jung 
wrote:

> Take q for or Filterung:
>
>
> https://stackoverflow.com/questions/6567831/how-to-perform-or-condition-in-django-queryset
>

Thanks for the response, but the OR is not the problem: it works just fine.
(FWIW, I used the "|" form rather than the Q() form because it is not clear
to me how one can express a .exclude() when using Q()). The problem here is
that the result of the OR cannot be reliably .filter()d.

As I tried to explain, I can see that in some cases, a .filter() of the
OR'd result does indeed add extra SQL to the query, whereas in others it
does not:

qs_or_result.filter(a='b')

This works for some values of "a" but not others, I can use this to achieve
the same effect:

Buss.objects.filter(a='b').intersection(qs_or_result)

though it is rather clumsy!!! FWIW, this is with Django 3.2. I'm inclined
to think this is a bug in the ORM, though I have not dived into the code to
track it down.

Thanks, Shaheed


>
>
> Shaheed Haque  schrieb am Fr., 30. Apr. 2021,
> 02:43:
>
>> Hi,
>>
>> I have a query which ORs some selects on a single table together like
>> this:
>>
>> jurisdiction = 'aaa'
>> qs = Buss.objects.filter(jurisdiction=jurisdiction)
>> qs = qs | Buss.objects.filter(jurisdiction='xxx').exclude(name__in
>> =qs.values_list('name', flat=True))
>> qs = qs | Buss.objects.filter(jurisdiction='yyy').exclude(name__in
>> =qs.values_list('name', flat=True))
>>
>> This seems to work just fine (and the raw SQL looks suitably complicated):
>>
>> SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
>> "paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"
>> WHERE ("paiyroll_buss"."jurisdiction" = aaa OR
>> ("paiyroll_buss"."jurisdiction" = xxx AND NOT ("paiyroll_buss"."name" IN
>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa)))
>> OR ("paiyroll_buss"."jurisdiction" = yyy AND NOT ("paiyroll_buss"."name" IN
>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE (U0."jurisdiction" = aaa OR
>> (U0."jurisdiction" = xxx AND NOT (U0."name" IN (SELECT U0."name" FROM
>> "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa
>>
>> Now, if I post-filter qs using something other than "aaa" (the first term
>> above) like this:
>>
>>qs.filter('xxx)
>>
>> then the resulting SQL has an 'AND "paiyroll_buss"."jurisdiction" = xxx'
>> as one might expect. However, if I try to post-filter qs like this:
>>
>> qs.filter('aaa') # 'aaa' was the first term in the original trio of
>> clauses
>>
>> Then the formed SQL looks like this:
>>
>> SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
>> "paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"
>> WHERE ("paiyroll_buss"."jurisdiction" = aaa OR
>> ("paiyroll_buss"."jurisdiction" = xxx AND NOT ("paiyroll_buss"."name" IN
>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa)))
>> OR ("paiyroll_buss"."jurisdiction" = yyy AND NOT ("paiyroll_buss"."name" IN
>> (SELECT U0."name" FROM "paiyroll_buss" U0 WHERE (U0."jurisdiction" = aaa OR
>> (U0."jurisdiction" = xxx AND NOT (U0."name" IN (SELECT U0."name" FROM
>> "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa
>>
>> i.e. just like the original 3-clause query. (I realise that in this case,
>> the query degenerates to the first term). What am I missing?
>>
>> Any clues appreciated.
>>
>> Thanks, Shaheed
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAHAc2jcLVcFFiXRZtX_iGTJ2YJ0CNsObRFDyw1Ss6i3nV-X-vw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-users/CAHAc2jcLVcFFiXRZtX_iGTJ2YJ0CNsObRFDyw1Ss6i3nV-X-vw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAKGT9mwaiSjyrrMJfrvoXMShfnqjxhonkHTbyOVdCGqJwqK85w%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-users/CAKGT9mwaiSjyrrMJfrvoXMShfnqjxhonkHTbyOVdCGqJwqK85w%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdWB-9FLJWZbcjuEp53-M9jGrv7YUN_hecWa4w4Nzkh%2BQ%40mail.gmail.com.

Filtering OR-combined queries

[Shaheed Haque]

Hi,

I have a query which ORs some selects on a single table together like this:

jurisdiction = 'aaa'
qs = Buss.objects.filter(jurisdiction=jurisdiction)
qs = qs | Buss.objects.filter(jurisdiction='xxx').exclude(name__in
=qs.values_list('name', flat=True))
qs = qs | Buss.objects.filter(jurisdiction='yyy').exclude(name__in
=qs.values_list('name', flat=True))

This seems to work just fine (and the raw SQL looks suitably complicated):

SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
"paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"
WHERE ("paiyroll_buss"."jurisdiction" = aaa OR
("paiyroll_buss"."jurisdiction" = xxx AND NOT ("paiyroll_buss"."name" IN
(SELECT U0."name" FROM "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa)))
OR ("paiyroll_buss"."jurisdiction" = yyy AND NOT ("paiyroll_buss"."name" IN
(SELECT U0."name" FROM "paiyroll_buss" U0 WHERE (U0."jurisdiction" = aaa OR
(U0."jurisdiction" = xxx AND NOT (U0."name" IN (SELECT U0."name" FROM
"paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa

Now, if I post-filter qs using something other than "aaa" (the first term
above) like this:

   qs.filter('xxx)

then the resulting SQL has an 'AND "paiyroll_buss"."jurisdiction" = xxx' as
one might expect. However, if I try to post-filter qs like this:

qs.filter('aaa') # 'aaa' was the first term in the original trio of
clauses

Then the formed SQL looks like this:

SELECT "paiyroll_buss"."id", "paiyroll_buss"."jurisdiction",
"paiyroll_buss"."name", "paiyroll_buss"."description" FROM "paiyroll_buss"
WHERE ("paiyroll_buss"."jurisdiction" = aaa OR
("paiyroll_buss"."jurisdiction" = xxx AND NOT ("paiyroll_buss"."name" IN
(SELECT U0."name" FROM "paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa)))
OR ("paiyroll_buss"."jurisdiction" = yyy AND NOT ("paiyroll_buss"."name" IN
(SELECT U0."name" FROM "paiyroll_buss" U0 WHERE (U0."jurisdiction" = aaa OR
(U0."jurisdiction" = xxx AND NOT (U0."name" IN (SELECT U0."name" FROM
"paiyroll_buss" U0 WHERE U0."jurisdiction" = aaa

i.e. just like the original 3-clause query. (I realise that in this case,
the query degenerates to the first term). What am I missing?

Any clues appreciated.

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcLVcFFiXRZtX_iGTJ2YJ0CNsObRFDyw1Ss6i3nV-X-vw%40mail.gmail.com.

Re: Mixin add field to fields in forms

[Shaheed Haque]

Actually, looking again at the source, it might be generally belter to use
self.base_fields. (Yes, my previous answer is not an exact copy of my code,
so I got it wrong).

On Sun, 18 Apr 2021 at 17:16, Shaheed Haque  wrote:

>
>
> On Sun, 18 Apr 2021 at 14:38, sebasti...@gmail.com <
> sebastian.ju...@gmail.com> wrote:
>
>> Hello,
>>
>> Thanks for your fast response. But i don't know how i can
>> use  self.declared_fields and why this would help me...
>>
>
> The "how" is easy. Here is a fragment from my code where the problem is
> that every instance of FilesForm has content dependent on external factors
> (the "..." below):
>
>   class FilesForm(forms.Form):
>   def __init__(self, *args, **kwargs):
>self.declared_fields = OrderedDict()
>for name, field in items():
>self.declared_fields[name] = field
>super().__init__(*args, **kwargs)
>
> The "why" is a deeper question, but in short, that's what the source code
> says the DeclarativeFieldsMetaClass for forms uses. Technically, I believe
> the code is a little ick because self.declared_fileds is a class member,
> not an instance member and so needs the funny-looking assignment so the
> instance has a value it can hack. You may need to copy the original dict as
> in "self.declared_fields = OrderedDict(self.declared_fields)" depending on
> your use case.
>
> (Generally, the Django docs are amazing, but this is one area where I
> needed the source.)
>
> Shaheed
>
>
>
>> Regards
>>
>> shahee...@gmail.com schrieb am Sonntag, 18. April 2021 um 15:14:03 UTC+2:
>>
>>> Try updating self.declared_fields instead.
>>>
>>> On Sun, 18 Apr 2021, 13:47 sebasti...@gmail.com, 
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>>
>>>> *forms.py:*
>>>>
>>>> class Newsletterform(StandardMixin):
>>>> class Meta:
>>>> model = Newsletter
>>>> fields = ['newslettername', 'from_link', 'to_list',
>>>> 'email_subject', 'text_message', 'html_message' ]
>>>>
>>>> *Mixins.py:*
>>>> class StandardMixin(forms.ModelForm):
>>>> class Meta:
>>>> abstract = True
>>>>
>>>> def __init__(self, *args, **kwargs):
>>>>
>>>> self.Meta.fields.append('owner')
>>>> super(StandardMixin, self).__init__(*args, **kwargs)
>>>>
>>>>
>>>> *i would append in Meta.Fields owner and after this super. But in
>>>> Template this field are not shown. *
>>>>
>>>> *Why?*
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Django users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to django-users...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/django-users/bb10dfa0-ed1e-459c-8e06-46af264a3b7en%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/django-users/bb10dfa0-ed1e-459c-8e06-46af264a3b7en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/f16d610f-acce-4eef-8940-8a80ff1d36a3n%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-users/f16d610f-acce-4eef-8940-8a80ff1d36a3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcQqpz0DaHHpo9OVp0Gt%3DETTMpq5hk69jb2%2BqmAf6%3Dq2w%40mail.gmail.com.

Re: Mixin add field to fields in forms

[Shaheed Haque]

On Sun, 18 Apr 2021 at 14:38, sebasti...@gmail.com <
sebastian.ju...@gmail.com> wrote:

> Hello,
>
> Thanks for your fast response. But i don't know how i can
> use  self.declared_fields and why this would help me...
>

The "how" is easy. Here is a fragment from my code where the problem is
that every instance of FilesForm has content dependent on external factors
(the "..." below):

  class FilesForm(forms.Form):
  def __init__(self, *args, **kwargs):
   self.declared_fields = OrderedDict()
   for name, field in items():
   self.declared_fields[name] = field
   super().__init__(*args, **kwargs)

The "why" is a deeper question, but in short, that's what the source code
says the DeclarativeFieldsMetaClass for forms uses. Technically, I believe
the code is a little ick because self.declared_fileds is a class member,
not an instance member and so needs the funny-looking assignment so the
instance has a value it can hack. You may need to copy the original dict as
in "self.declared_fields = OrderedDict(self.declared_fields)" depending on
your use case.

(Generally, the Django docs are amazing, but this is one area where I
needed the source.)

Shaheed



> Regards
>
> shahee...@gmail.com schrieb am Sonntag, 18. April 2021 um 15:14:03 UTC+2:
>
>> Try updating self.declared_fields instead.
>>
>> On Sun, 18 Apr 2021, 13:47 sebasti...@gmail.com, 
>> wrote:
>>
>>> Hello,
>>>
>>>
>>> *forms.py:*
>>>
>>> class Newsletterform(StandardMixin):
>>> class Meta:
>>> model = Newsletter
>>> fields = ['newslettername', 'from_link', 'to_list',
>>> 'email_subject', 'text_message', 'html_message' ]
>>>
>>> *Mixins.py:*
>>> class StandardMixin(forms.ModelForm):
>>> class Meta:
>>> abstract = True
>>>
>>> def __init__(self, *args, **kwargs):
>>>
>>> self.Meta.fields.append('owner')
>>> super(StandardMixin, self).__init__(*args, **kwargs)
>>>
>>>
>>> *i would append in Meta.Fields owner and after this super. But in
>>> Template this field are not shown. *
>>>
>>> *Why?*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/bb10dfa0-ed1e-459c-8e06-46af264a3b7en%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/f16d610f-acce-4eef-8940-8a80ff1d36a3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfi_R%3DyybuWeix1g1TTo5QkrvWB6bJ%3DFnLFBf1hMXsNBQ%40mail.gmail.com.

Re: Mixin add field to fields in forms

[Shaheed Haque]

Try updating self.declared_fields instead.

On Sun, 18 Apr 2021, 13:47 sebasti...@gmail.com, 
wrote:

> Hello,
>
>
> *forms.py:*
>
> class Newsletterform(StandardMixin):
> class Meta:
> model = Newsletter
> fields = ['newslettername', 'from_link', 'to_list',
> 'email_subject', 'text_message', 'html_message' ]
>
> *Mixins.py:*
> class StandardMixin(forms.ModelForm):
> class Meta:
> abstract = True
>
> def __init__(self, *args, **kwargs):
>
> self.Meta.fields.append('owner')
> super(StandardMixin, self).__init__(*args, **kwargs)
>
>
> *i would append in Meta.Fields owner and after this super. But in Template
> this field are not shown. *
>
> *Why?*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/bb10dfa0-ed1e-459c-8e06-46af264a3b7en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jff8TzHzJ7td91AOH7xgmqvD%2BK7i7R5OPWDZS8oRxs7pQ%40mail.gmail.com.

Django 3.1/asgiref 3.3.1 support for StreamingHttpResponse

[Shaheed Haque]

Hi,

I am attempting to upgrade from Django 3.0 to 3.1 and asgiref 2.x to 3.x,
and noticed a breakage in a view which I suspect might be related to its
usage of StreamingHttpResponse. The code in question now raises the
following exception:

2020-12-08 17:50:05,159 [ERROR] daphne.server: Exception inside
application: You cannot call this from an async context - use a thread or
sync_to_async.
Traceback (most recent call last):
 File "/usr/local/lib/python3.8/dist-packages/channels/staticfiles.py",
line 44, in __call__
   return await self.application(scope, receive, send)
 File "/usr/local/lib/python3.8/dist-packages/channels/routing.py", line
71, in __call__
   return await application(scope, receive, send)
 File
"/usr/local/lib/python3.8/dist-packages/django/core/handlers/asgi.py", line
168, in __call__
   await self.send_response(response, send)
 File
"/usr/local/lib/python3.8/dist-packages/django/core/handlers/asgi.py", line
242, in send_response
   for part in response:
 File "/main/srhaque/Innovatieltd/source/paiyroll/utils/time_machine.py",
line 202, in fill
   for files in filler:
 File
"/main/srhaque/Innovatieltd/source/paiyroll/views/employee/mobile_views.py",
line 636, in regenerate_documents
   pay_run_db = models.PayRun.objects.get(id=pay_run['id'])
 File "/usr/local/lib/python3.8/dist-packages/django/db/models/manager.py",
line 85, in manager_method
   return getattr(self.get_queryset(), name)(*args, **kwargs)
 File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 425, in get
   num = len(clone)
 File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 269, in __len__
   self._fetch_all()
 File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 1308, in _fetch_all
   self._result_cache = list(self._iterable_class(self))
 File
"/usr/local/lib/python3.8/dist-packages/django_viewflow-1.6.1-py3.8.egg/viewflow/managers.py",
line 79, in __iter__
   for process in base_iterator:
 File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py",
line 53, in __iter__
   results = compiler.execute_sql(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size)
 File
"/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py",
line 1154, in execute_sql
   cursor = self.connection.cursor()
 File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py",
line 24, in inner
   raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from
an async context - use a thread or sync_to_async.

(There is a subsequent exception, appended below, which I assume that is
caused by this exception). I see that the immediate cause of the problem is
the ORM query in my mobile_views.py, but I cannot seem to find the correct
way to wrap the query in sync_to_async() or database_sync_to_async(). Any
hints or pointers would be most welcome.

Thanks, Shaheed

Attachment: second exception for reference...

2020-12-08 17:50:05,168 [ERROR] asyncio: Exception in callback
AsyncioSelectorReactor.callLater..run() at
/usr/local/lib/python3.8/dist-packages/twisted/internet/asyncioreactor.py:287

handle: .run() at
/usr/local/lib/python3.8/dist-packages/twisted/internet/asyncioreactor.py:287>

Traceback (most recent call last):
 File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
   self._context.run(self._callback, *self._args)
 File
"/usr/local/lib/python3.8/dist-packages/twisted/internet/asyncioreactor.py",
line 290, in run
   f(*args, **kwargs)
 File "/usr/local/lib/python3.8/dist-packages/daphne/server.py", line 295,
in application_checker
   protocol.handle_exception(exception)
 File "/usr/local/lib/python3.8/dist-packages/daphne/http_protocol.py",
line 301, in handle_exception
   self.basic_error(500, b"Internal Server Error", "Exception inside
application.")
 File "/usr/local/lib/python3.8/dist-packages/daphne/http_protocol.py",
line 342, in basic_error
   self.handle_reply(
 File "/usr/local/lib/python3.8/dist-packages/daphne/http_protocol.py",
line 242, in handle_reply
   raise ValueError("HTTP response has already been started")
ValueError: HTTP response has already been started

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfX9cb5jfLmtD0vDWQtSoYfiZ_oJpi%2BNdpsX6PJ_e13Vw%40mail.gmail.com.

Re: controlling stdout

[Shaheed Haque]

On Sat, 5 Dec 2020, 21:57 Tom Sgouros,  wrote:

> Thank you. Now I guess I'll work on figuring out *where* it was redirected.
>

Sounds good.

But the "dunder" version of sys.stdout is an excellent clue -- I did not
> know about that, and bonus gratitude points for giving me a new word to use.
>

Lol. Good luck...


>  -Tom
>
> On Sat, Dec 5, 2020 at 9:06 AM Shaheed Haque 
> wrote:
>
>> I would hazard a guess that Django has already redirected stderr and or
>> stdout before your code gets to run.
>>
>> I recommend that you stop your code in the debugger just *before* you try
>> your redirection and look at the state of sys.stderr and sys.stdout. Also,
>> I'm not sure if you are aware of the dunder versions of these (see
>> https://docs.python.org/3/library/sys.html)? Comparing sys.stdout with
>> sys.__stdout__ might reveal interesting results for example.
>>
>> Once you know what Django has already done, I expect you will be able to
>> progress.
>>
>> On Sat, 5 Dec 2020, 13:18 Tom Sgouros,  wrote:
>>
>>> Matlab is the native language of the group for which I am building this
>>> interface. It is for access to ocean sensor data, and it's being done this
>>> way so the scientists whose sensors they are can use a significant body of
>>> legacy code in their analyses. (Though I was planning to sneak in an option
>>> to use python scripts because that's what I prefer.)
>>>
>>> But the fact remains that I have no trouble capturing stdout when I run
>>> this python on the command line and it doesn't work when I run it as part
>>> of a Django views.py and I don't where to begin trying to understand why
>>> not.
>>>
>>>  -Tom
>>>
>>> On Fri, Dec 4, 2020 at 3:33 PM Kris A. Stern 
>>> wrote:
>>>
>>>> Why not try something Python based instead which is also free. What
>>>> kind of functionality are you looking for? Are you planning to plot, if so
>>>> you could try numpy, scipy, and matplotlib.
>>>>
>>>> On Sat, 5 Dec 2020 at 02:02, Tom Sgouros  wrote:
>>>>
>>>>> Anybody have any random thoughts about this? I have run out of ideas
>>>>> to test.
>>>>>
>>>>> Thank you,
>>>>>
>>>>>  -Tom
>>>>>
>>>>> On Wed, Dec 2, 2020 at 10:32 PM Tom Sgouros 
>>>>> wrote:
>>>>>
>>>>>> Hello all:
>>>>>>
>>>>>> I have a Django application that dispatches processing to a matlab
>>>>>> script using exec. How can I capture the stdout and stderr of the 
>>>>>> script's
>>>>>> execution?
>>>>>>
>>>>>> Outside of Django, this works for a dumb matlab script called tt that
>>>>>> just prints a message:
>>>>>>
>>>>>> >>> import matlab.engine, StringIO
>>>>>> >>> eng = matlab.engine.start_matlab()
>>>>>> >>> out = StringIO.StringIO()
>>>>>> >>> err = StringIO.StringIO()
>>>>>> >>> exec('eng.tt(25, stdout=out, stderr=err)')
>>>>>> >>> out.getvalue()
>>>>>> "This is a disp command.\n\nret =\n\n '25\n\n"
>>>>>>
>>>>>> Inside a Django method, the exec seems to function properly, but the
>>>>>> out and err variables remain empty. Where's my output? Did the logging
>>>>>> system snatch it? I assume I'm missing something obvious here but any
>>>>>> pointers would be welcome.
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>>  -Tom
>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Django users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to django-users+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/django-users/35b264b3-f793-4131-a6e4-700e51aebbfan%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/django-users/35b264b3-f793-4131-a6e4-700e51aebbfan%40googlegroups.com?utm_medium=email&utm_source=footer>
>>

Re: controlling stdout

[Shaheed Haque]

I would hazard a guess that Django has already redirected stderr and or
stdout before your code gets to run.

I recommend that you stop your code in the debugger just *before* you try
your redirection and look at the state of sys.stderr and sys.stdout. Also,
I'm not sure if you are aware of the dunder versions of these (see
https://docs.python.org/3/library/sys.html)? Comparing sys.stdout with
sys.__stdout__ might reveal interesting results for example.

Once you know what Django has already done, I expect you will be able to
progress.

On Sat, 5 Dec 2020, 13:18 Tom Sgouros,  wrote:

> Matlab is the native language of the group for which I am building this
> interface. It is for access to ocean sensor data, and it's being done this
> way so the scientists whose sensors they are can use a significant body of
> legacy code in their analyses. (Though I was planning to sneak in an option
> to use python scripts because that's what I prefer.)
>
> But the fact remains that I have no trouble capturing stdout when I run
> this python on the command line and it doesn't work when I run it as part
> of a Django views.py and I don't where to begin trying to understand why
> not.
>
>  -Tom
>
> On Fri, Dec 4, 2020 at 3:33 PM Kris A. Stern  wrote:
>
>> Why not try something Python based instead which is also free. What kind
>> of functionality are you looking for? Are you planning to plot, if so you
>> could try numpy, scipy, and matplotlib.
>>
>> On Sat, 5 Dec 2020 at 02:02, Tom Sgouros  wrote:
>>
>>> Anybody have any random thoughts about this? I have run out of ideas to
>>> test.
>>>
>>> Thank you,
>>>
>>>  -Tom
>>>
>>> On Wed, Dec 2, 2020 at 10:32 PM Tom Sgouros  wrote:
>>>
 Hello all:

 I have a Django application that dispatches processing to a matlab
 script using exec. How can I capture the stdout and stderr of the script's
 execution?

 Outside of Django, this works for a dumb matlab script called tt that
 just prints a message:

 >>> import matlab.engine, StringIO
 >>> eng = matlab.engine.start_matlab()
 >>> out = StringIO.StringIO()
 >>> err = StringIO.StringIO()
 >>> exec('eng.tt(25, stdout=out, stderr=err)')
 >>> out.getvalue()
 "This is a disp command.\n\nret =\n\n '25\n\n"

 Inside a Django method, the exec seems to function properly, but the
 out and err variables remain empty. Where's my output? Did the logging
 system snatch it? I assume I'm missing something obvious here but any
 pointers would be welcome.

 Thank you,

  -Tom


 --
 You received this message because you are subscribed to the Google
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-users/35b264b3-f793-4131-a6e4-700e51aebbfan%40googlegroups.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/CAFUviqs6XXxfzOqQ4uCnVBEo2cN%2BbMMPHPEO-wtbxPVa94PtHw%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAMxZ8Sm7avV%3Df664b%3D8hYQWgAUwvpL%2BLVZV%3D31Hy21B2xeSDtA%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAFUviqvvOZUorVSmT1DSe_GO53_UEntvshiEzLxtUCAg%2BRFcPw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this

Re: Process Data during server initialization

[Shaheed Haque]

Well, it's a bit of a blunt weapon, but Python is going to run any code it
finds at module scope as the process is initialised. So you could cause
your heavy stuff to be run there? Or at least spawned from there?

On Thu, 22 Oct 2020, 22:59 Okware Aldo,  wrote:

> Hi Lois,
>
> To expand on what Scott is saying.
> - model DB tables to store the data you need to render the graphs
> - build a service to preprocess the data - this service can run
> periodically depending on how up to date you need the graphs to be (a
> microservice)
> - build a single endpoint to simple fetch the data the graphs need
> (Django) if this is a monolith the normal Django views should do the trick
>
>
>
>
>
>
> On Fri, Oct 23, 2020 at 12:01 AM Lois Greene-Hernandez 
> wrote:
>
>> Well it's data that I need to populate the pages.  It's very processor
>> intensive and I'll need to pass it back to the views once it's processed.
>> Can I pass data back to the views once I've run a task?  I'm looking into
>> caching alternatives.
>>
>> Thanks
>> Lois
>>
>> On Thu, Oct 22, 2020 at 4:55 PM Scott Sawyer  wrote:
>>
>>> Could you run a cron job on the system to analyze the data periodically?
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 22, 2020, at 1:30 PM, Lois Greene-Hernandez 
>>> wrote:
>>>
>>> Hello,
>>>
>>> I'm working on a data and calculation intensive django application.  It
>>> uses pandas and plotly.py to create many charts and graphs.  I have some
>>> control over the design of the application but not total control.  One
>>> decision that I stuck with is that all charts and graphs are preprocessed
>>> on the first page that loads.  If it were up to me, I'd process what is
>>> needed at the time but its not up to me.
>>>
>>> Since I need to load and process a lot of data prior to page load, I was
>>> wondering if it would be possible to do this during server start up.  I'd
>>> also need to pass a log of data from whatever startup process I ran to the
>>> django views.
>>>
>>> Any information that you could shed on this problem would be greatly
>>> appreciated.
>>>
>>> Thanks
>>>
>>> Lois J. Greene-Hernandez
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com
>>> 
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Django users" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/django-users/X1joo7QEbnc/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> django-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/F90610CE-721D-4FEB-8992-FAF63A6C6368%40gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAAGmcw0tk9fd%3DhjvpdZTpqqf9-gDDKgtW7YYfvJxVcCo1DvD4g%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAMEZma-P3Z6yoAJJGCWbDpd8YEWkc_kV8uS4FKDNvsR-LF-4WA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdi4d_Rz4X4Kue-CbB%2B88-EQyD5dcXqxSysKvBB9hFXFA%40mail.gmail.com.

Re: Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

[Shaheed Haque]

Indeed. And to close out the discussion I can confirm that my FDW
implementations were using JSON columns, and the issue was resolved by
switching to JSONB instead.

On Sun, 18 Oct 2020, 14:40 Jason,  wrote:

> looks like you got a response back, and there's more context in the linked
> ticket too.  tl;dr json is not jsonb, and django only works with the latter
>
> On Thursday, October 15, 2020 at 10:02:30 AM UTC-4 shahee...@gmail.com
> wrote:
>
>> Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.
>>
>> On Thursday, 15 October 2020 at 12:51:22 UTC+1 Jason wrote:
>>
>>> this is pretty interesting, and may be something to report to the devs
>>> at https://groups.google.com/g/django-developers, or open up a ticket
>>> at https://code.djangoproject.com/  I don't see any tickets with the
>>> query *jsonfield foreign data* that are similar with your experience,
>>> so you may have found an edge case.
>>>
>>> On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 shahee...@gmail.com
>>> wrote:
>>>

 Hi,

 I have a Django model working fine under Django 3.0 (i.e. "Django<3.1")
 which looks like this:

 ===
 class Job(models.Model):
 id = models.CharField(max_length=36, primary_key=True)
 queue = models.CharField(max_length=40)
 args = JSONField()
 kwargs = JSONField()
 type = models.CharField(max_length=80)
 ...

 class Meta:
 managed = False  # <--   The table is implemented as a
 Postgres FDW wrapper.
 db_table = 'jobs'
 ===

 I am testing the update to Django 3.1.2 and hit an error in executing
 this line:

 jobs = list(models.Job.objects.filter(queue='celery',
 state='scheduled'))

 The error is as follows from pytest (i.e. stack trace with local
 variables too):

 ==
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 _ _ _ _
 /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287:
 in __iter__
self._fetch_all()
self   = 
 /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308:
 in _fetch_all
self._result_cache = list(self._iterable_class(self))
self   = 
 /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in
 __iter__
for row in compiler.results_iter(results):
annotation_col_map = {}
compiler   = >>> at 0x7f8685e49160>
db = 'fdw'
init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state',
 ...]
klass_info = {'model': ,
 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
known_related_objects = []
model_cls  = 
model_fields_end = 9
model_fields_start = 0
queryset   = 
related_populators = []
results= [[('8f4ab6b0-914f-4a75-972d-febbe55011fc',
 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}],
 {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
select = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []),
 None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None...,
 paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs,
 paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
select_fields = [0, 1, 2, 3, 4, 5, ...]
self   = >>> 0x7f86836f3040>
 /usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100:
 in apply_converters
value = converter(value, expression, connection)
connection = >>> object at 0x7f869a321670>
converter  = >>> >
converters = [(2, ([>>> >], Col(jobs,
 pa...NField.from_db_value of
 >], Col(jobs,
 paiyroll.Job.details)))]
convs  = [>>> >]
expression = Col(jobs, paiyroll.Job.args)
pos= 2
row= ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
 ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
 'paiyroll.tasks.function_run', 'scheduled', ...]
rows   = 
self   = >>> at 0x7f8685e49160>
value  = ['paiyroll.tasks', 'call_to_string',
 'call_to_string', [], {}]





 */usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74:
 in from_db_valuereturn json.loads(value, cls=self.decoder)
connection = >>> object at 0x7f869a321670>expression = Col(jobs, paiyroll.Job.args)
self   = 
value  = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
 [], {}] *
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 _ _ _ _

 s = ['paiyroll.tasks', 'call_to_

Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

[Shaheed Haque]

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1")
which looks like this:

===
class Job(models.Model):
id = models.CharField(max_length=36, primary_key=True)
queue = models.CharField(max_length=40)
args = JSONField()
kwargs = JSONField()
type = models.CharField(max_length=80)
...

class Meta:
managed = False  # <--   The table is implemented as a
Postgres FDW wrapper.
db_table = 'jobs'
===

I am testing the update to Django 3.1.2 and hit an error in executing this
line:

jobs = list(models.Job.objects.filter(queue='celery',
state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables
too):

==
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in
__iter__
   self._fetch_all()
   self   = 
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in
_fetch_all
   self._result_cache = list(self._iterable_class(self))
   self   = 
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in
__iter__
   for row in compiler.results_iter(results):
   annotation_col_map = {}
   compiler   = 
   db = 'fdw'
   init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
   klass_info = {'model': ,
'select_fields': [0, 1, 2, 3, 4, 5, ...]}
   known_related_objects = []
   model_cls  = 
   model_fields_end = 9
   model_fields_start = 0
   queryset   = 
   related_populators = []
   results= [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
'paiyroll.tasks.function_run', 'scheduled', ...)]]
   select = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []),
None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None...,
paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs,
paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
   select_fields = [0, 1, 2, 3, 4, 5, ...]
   self   = 
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100:
in apply_converters
   value = converter(value, expression, connection)
   connection = 
   converter  = >
   converters = [(2, ([>], Col(jobs,
pa...NField.from_db_value of
>], Col(jobs,
paiyroll.Job.details)))]
   convs  = [>]
   expression = Col(jobs, paiyroll.Job.args)
   pos= 2
   row= ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
'paiyroll.tasks.function_run', 'scheduled', ...]
   rows   = 
   self   = 
   value  = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
[], {}]





*/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74:
in from_db_valuereturn json.loads(value, cls=self.decoder)
   connection = expression = Col(jobs, paiyroll.Job.args)
   self   = 
   value  = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
[], {}] *
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls =
None
object_hook = None, parse_float = None, parse_int = None, parse_constant =
None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
   parse_int=None, parse_constant=None, object_pairs_hook=None,
**kw):
   """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
   containing a JSON document) to a Python object.

   ``object_hook`` is an optional function that will be called with the
   result of any object literal decode (a ``dict``). The return value
of
   ``object_hook`` will be used instead of the ``dict``. This feature
   can be used to implement custom decoders (e.g. JSON-RPC class
hinting).

   ``object_pairs_hook`` is an optional function that will be called
with the
   result of any object literal decoded with an ordered list of pairs.
The
   return value of ``object_pairs_hook`` will be used instead of the
``dict``.
   This feature can be used to implement custom decoders.  If
``object_hook``
   is also defined, the ``object_pairs_hook`` takes priority.

   ``parse_float``, if specified, will be called with the string
   of every JSON float to be decoded. By default this is equivalent to
   float(num_str). This can be used to use another datatype or parser
   for JSON floats (e.g. decimal.Decimal).

   ``parse_int``, if specified, will be called with the string
   of every JSON int to be decoded. By default this is equivalent to
   int(num_str). This can be used to use another datatype or parser
   for JSON integers (e.g

Re: Django ORM Model meta internal API to current API update

[Shaheed Haque]

On Tue, 8 Sep 2020 at 03:54, Asif Saif Uddin  wrote:

> any help? I can share the full function
>

I've poked around some of these dark corners, but only since the late 2.x
series. After a bit of searching this
find-the-field-that-connects-a-model-to-a-through-model

and the linked

suggest these are/were private APIs and AFAIK, the best you can do is try
to figure out what the old code was doing, and rework them using the likes
of:

   - self._meta.related_objects
   - self._meta.fields_map
   - self._meta.local_many_to_man

and the other similarly named fields. One thing to note: if a field x
points to a *set* of objects, you won't find a "self.x", but rather a
"self.x_set" which must be followed using a query such as
"self.x_set.all()" rather than a simple field dereference.

HTH, Shaheed

On Wednesday, August 19, 2020 at 3:42:09 AM UTC+6 vickkymb...@gmail.com
> wrote:
>
>> I don't bro i just started new in Django
>>
>> Victor
>>
>> On Tue, Aug 18, 2020, 3:46 PM Asif Saif Uddin  wrote:
>>
>>>
>>> Hi,
>>> I have got an old project of django 1.8 which use codes like
>>>
>>> links = [rel.get_accessor_name() for rel in
>>> self._meta.get_all_related_objects()]
>>>
>>> How would I upgrade this code snippets to django 2.2 LTS?
>>>
>>> TIA
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/c408d424-7c94-43de-beae-4071d424e2ecn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/e83b7021-3158-41b2-b48f-33d9f45270a6n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfeKWgiD97bYYePPAv0zMm2zKXqnirLsM5fUm%3DEC2xRfA%40mail.gmail.com.

Re: unexpected change

[Shaheed Haque]

To clarify, try creating an empty file (e.g. using the "touch" command)
with the right path.

On Thu, 3 Sep 2020, 17:39 Shaheed Haque,  wrote:

> Sorry to hear about your problems. One other avenue to possibly explore is
> Pycharm's "local history" facility. This keeps track of changes
> independently of any VCS such as git.
>
> It seems to be based on the original file paths. So, try creating one file
> with the EXACT path before the rename. Then go into the local history
> viewer. If you're in luck, you can try to recover the missing content. If
> that works, try the other files.
>
> On Thu, 3 Sep 2020, 17:11 rbarh...@gmail.com, 
> wrote:
>
>> Yeah, well great advice.  Too bad I didn't take it sooner.  I lost two
>> weeks of really hard work because I move a folder and somehow it erased all
>> the coding I had done in PyCharm.  I don't understand it, but there it is.
>> I am in the process of learning how to use Git and Github.  But.  Damn.
>>
>> On Monday, August 31, 2020 at 11:58:11 AM UTC-7 Kasper Laudrup wrote:
>>
>>> Hi rbarhydtsf,
>>>
>>> On 31/08/2020 20.21, rbarh...@gmail.com wrote:
>>> > Thank you, Kasper.  Good idea.  Sadly, I don't really know how to do
>>> that.
>>> >
>>>
>>> I would recommend finding some tutorial on how to use git (as that is
>>> probably what you'll use anyway). Cannot really recommend anything
>>> specific, but someone else might be able to.
>>>
>>> It takes some time to learn even the basics and it might seem useless as
>>> it is time not spend on writing code, but it can save you so much time
>>> later on, so it's definitely worth it.
>>>
>>> Has helped me many times in frustrating situations like the one you have
>>> right now, and you will have situations like that again, even after many
>>> years of writing code.
>>>
>>> We're only humans after all and most of us cannot remember the point of
>>> the code we wrote two days ago, so it's extremely helpful to have tools
>>> to help you remind you what you did (and why) two years ago.
>>>
>>> Kind regards,
>>>
>>> Kasper Laudrup
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/b5a51ad7-d4e1-4fa4-8ee2-cff352bcde13n%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-users/b5a51ad7-d4e1-4fa4-8ee2-cff352bcde13n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jeBJJ57o%2BLsjQERj0zicFCLcbEBmnQjKw-Nuypbfd6%2BPA%40mail.gmail.com.

Re: unexpected change

[Shaheed Haque]

Sorry to hear about your problems. One other avenue to possibly explore is
Pycharm's "local history" facility. This keeps track of changes
independently of any VCS such as git.

It seems to be based on the original file paths. So, try creating one file
with the EXACT path before the rename. Then go into the local history
viewer. If you're in luck, you can try to recover the missing content. If
that works, try the other files.

On Thu, 3 Sep 2020, 17:11 rbarh...@gmail.com,  wrote:

> Yeah, well great advice.  Too bad I didn't take it sooner.  I lost two
> weeks of really hard work because I move a folder and somehow it erased all
> the coding I had done in PyCharm.  I don't understand it, but there it is.
> I am in the process of learning how to use Git and Github.  But.  Damn.
>
> On Monday, August 31, 2020 at 11:58:11 AM UTC-7 Kasper Laudrup wrote:
>
>> Hi rbarhydtsf,
>>
>> On 31/08/2020 20.21, rbarh...@gmail.com wrote:
>> > Thank you, Kasper.  Good idea.  Sadly, I don't really know how to do
>> that.
>> >
>>
>> I would recommend finding some tutorial on how to use git (as that is
>> probably what you'll use anyway). Cannot really recommend anything
>> specific, but someone else might be able to.
>>
>> It takes some time to learn even the basics and it might seem useless as
>> it is time not spend on writing code, but it can save you so much time
>> later on, so it's definitely worth it.
>>
>> Has helped me many times in frustrating situations like the one you have
>> right now, and you will have situations like that again, even after many
>> years of writing code.
>>
>> We're only humans after all and most of us cannot remember the point of
>> the code we wrote two days ago, so it's extremely helpful to have tools
>> to help you remind you what you did (and why) two years ago.
>>
>> Kind regards,
>>
>> Kasper Laudrup
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/b5a51ad7-d4e1-4fa4-8ee2-cff352bcde13n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jeT8vMzp8xpRKJq6-i_5bXgLxoCMY%3Dr6xJBDCOwFS9xEw%40mail.gmail.com.

A question about running Channels and Django code under Celery

[Shaheed Haque]

I have a question about running Channels and Django code under Celery.
Originally, I had some code that used neither Channels nor Celery, but
simply run as part of the View code:

def import_all(...)

...database access...


This is an expensive operation and to avoid timing out the browser, I
needed to (a) run it under Celery but also (b) provide progress updates via
a Websocket to the client. First, I implemented (b) using a Channels
Consumer and the group_send, and that worked fine. I then tried to
implement (a) using a Celery Task:

@task

def import_all(...):
...database access...
async_to_sync(group_send)(...)


However, this caused Celery to be unhappy and the worker to die:

...
2020-08-22 13:33:08,303 [ERROR] MainProcess: Task handler raised error:
WorkerLostError('Worker exited prematurely: exitcode 0.')
Traceback (most recent call last):
 File
"/usr/local/lib/python3.8/dist-packages/billiard-3.6.3.0-py3.8.egg/billiard/pool.py",
line 1265, in mark_as_worker_lost
   raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0.

After quite some Googling, and various detours including where the DataBase
access caused issues (it seems it cannot be run from an async context), I
ended up with this double-wrapped monster:

@task

def import_all(...):
async_to_sync(_import_all)(...)

async def _import_all(...):
await database_sync_to_async(__import_all)(...)

def __import_all(...):   << original code
...database access...
async_to_sync(group_send)(...)


This seems to work (even though the Celery task eventually dies, it seems
to have done its work first, and is replaced). Is this double-wrapping
really the correct approach? Even if it is clumsy, is there any reason to
think it won't be reliable?

TIA, Shaheed

P.S. I should mention that I am using Celery's prefork library, as
opposed to say gevent or greenlets.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdxhaM%3Di6_aLRO-a3qdehkXt1tWYN%2B_oDpBXsB0Hp2jww%40mail.gmail.com.

Re: Django modelformset is_valid method retrieves entire model from database

[Shaheed Haque]

If you use the "empty_form
"
functionality, you may also wish to check that. In my case, I had added
queryset filtering for the normal form case, but recently noticed to my
horror that constructing the empty_form caused my filtering to be skipped.
The effect was similar to yours, returning every possible object for each
field in the form not only hammering the database, but the browser/renderer
too.

I *think* that if I had been able to use limit_choices_to

then the (undocumented?) get_limit_choices_to functionality on form fields
might have solved both problem cases. However, that was not an option for
me.

On Thu, 30 Jul 2020 at 00:47, Abdulrahman Alahaideb 
wrote:

> Fortunately; I found the missing part, in documentation it say (Changing
> the queryset
> 
> ):
>
>> By default, when you create a formset from a model, the formset will use
>> a queryset that includes all objects in the model (e.g.,
>> Author.objects.all()). You can override this behavior by using the queryset
>> argument:
>
> I was not changing changing the queryset before validating the formset
> which caused Django to retrieve the entire underlying model and include all
> objects in the model. I should have used the same queryset used to generate
> the formset in creating the form that will be validated. The validation
> should go like this:
>
> qs = prs.objects.filter(Q(cid = 1243) | Q(pid = 1243))
> form=prelations(data=datana, queryset = qs)
>
>
> On Wednesday, July 29, 2020 at 2:25:30 AM UTC+3, Abdulrahman Alahaideb
> wrote:
>>
>> I have a modelformset when is_valid method is called the entire model
>> objects of the underlying form is retrieved from the database which is not
>> efficient if the database table contains hundred of thousands records, I
>> believe Django was not designed with this deficiency and there must be
>> something wrong with my code, When is_valid is called these queries are
>> sent to the database, the first query has no WHERE clause which means it
>> will retrieve the entire table!:
>>
>> [{'sql': 'SELECT @@SQL_AUTO_IS_NULL', 'time': '0.000'},{'sql': 'SET SESSION 
>> TRANSACTION ISOLATION LEVEL READ COMMITTED', 'time': '0.000'},
>> **{'sql': 'SELECT `npr`.`rid`, `npr`.`PID`, `npr`.`rel`, `npr`.`CID` FROM 
>> `npr` ORDER BY `npr`.`rid` ASC', 'time': '0.037'}**{'sql': 'SELECT 
>> `person_details`.`ID`, `person_details`.`firstName` FROM `person_details` 
>> WHERE `person_details`.`ID` = 198 LIMIT 21', 'time': '0.001'},{'sql': 
>> 'SELECT `person_details`.`ID`, `person_details`.`firstName` FROM 
>> `person_details` WHERE `person_details`.`ID` = 1243 LIMIT 21', 'time': 
>> '0.000'},{'sql': 'SELECT `npr`.`rid`, `npr`.`PID`, `npr`.`rel`, `npr`.`CID` 
>> FROM `npr` WHERE `npr`.`rid` = 1377 LIMIT 21', 'time': '0.000'},{'sql': 
>> 'SELECT (1) AS `a` FROM `person_details` WHERE `person_details`.`ID` = 198 
>> LIMIT 1', 'time': '0.000'},{'sql': 'SELECT (1) AS `a` FROM `person_details` 
>> WHERE `person_details`.`ID` = 1243 LIMIT 1', 'time': '0.000'},{'sql': 
>> 'SELECT (1) AS `a` FROM `npr` WHERE (`npr`.`CID` = 1243 AND `npr`.`PID` = 
>> 198 AND NOT (`npr`.`rid` = 1377)) LIMIT 1', 'time': '0.000'},{'sql': 'SELECT 
>> `person_details`.`ID`, `person_details`.`firstName` FROM `person_details` 
>> WHERE `person_details`.`ID` = 200 LIMIT 21', 'time': '0.000'},{'sql': 
>> 'SELECT `person_details`.`ID`, `person_details`.`firstName` FROM 
>> `person_details` WHERE `person_details`.`ID` = 1243 LIMIT 21', 'time': 
>> '0.004'},{'sql': 'SELECT `npr`.`rid`, `npr`.`PID`, `npr`.`rel`, `npr`.`CID` 
>> FROM `npr` WHERE `npr`.`rid` = 1378 LIMIT 21', 'time': '0.000'},{'sql': 
>> 'SELECT (1) AS `a` FROM `person_details` WHERE `person_details`.`ID` = 200 
>> LIMIT 1', 'time': '0.000'},{'sql': 'SELECT (1) AS `a` FROM `person_details` 
>> WHERE `person_details`.`ID` = 1243 LIMIT 1', 'time': '0.000'},{'sql': 
>> 'SELECT (1) AS `a` FROM `npr` WHERE (`npr`.`CID` = 1243 AND `npr`.`PID` = 
>> 200 AND NOT (`npr`.`rid` = 1378)) LIMIT 1', 'time': '0.000'}]
>>
>> The models
>>
>>class PersonDetails(models.Model):
>> id = models.AutoField(db_column='ID', primary_key=True)
>> firstname = models.CharField(db_column='firstName', max_length=20)
>>
>> class Meta:
>> managed = True
>> db_table = 'person_details'
>>
>>
>> class Npr(models.Model):
>> rid = models.AutoField(db_column='rid', primary_key=True)
>> pid = models.ForeignKey(PersonDetails, on_delete=models.CASCADE, 
>> db_column='PID', related_name='pid')
>> rel = models.CharField(max_length=1)
>> cid = models.ForeignKey(PersonDetails, on_delete=models.CASCADE, 
>> db_column='CID', related_name='cid')
>>
>> class Meta:
>> managed = True
>

Re: Good usage of native JSON functions and operators in ORM queries

[Shaheed Haque]

On Sat, 30 May 2020 at 13:49, Shaheed Haque  wrote:

> Hi,
>
> I have a model MyModel which has a JSONField() called 'snapshot'. In
> Python terms, each snapshot looks like this:
>
> ==
> snapshot = {
> 'pay_definition' : {
> '1234': {..., 'name': 'foo', ...},
> '99': {..., 'name': 'bar', ...},
> }
> ==
>
> I'd like to find all unique values of 'name' in all instances of MyModel.
> I have this working using native JSON functions like this:
>
>class PayDefs(Func):
> function='to_jsonb'
> template="%(function)s(row_to_json(jsonb_each((%(expressions)s
> ->'pay_definition')))->'value'->'name')"
>
>MyModel.objects.annotate(xxx=PayDefs(F('snapshot')))
> .order_by().distinct('xxx').values_list('xxx', flat=True)
>
> My question is if this the best way to solve this problem? The way my
> current logic works, reading from insider out is, I think:
>
>1. Pass in the 'snapshot'.
>2. Since 'snapshot' is a JSON field, "->'pay_definition'" traverses
>this key.
>3. To skip the unknown numeric keys, "jsonb_each()" turns each key,
>value pair into an inner row like ['1234', {...}].
>4. To get to the value column of the inner row
>"row_to_json()->'value'".
>5. To get the name field's value "->'name'".
>6. A final call to "to_jsonb" in the PayDefs class.
>
> For example, since all I care about is the string value of 'name', is
> there a way to get rid of the PayDefs class, and its invocation of
> to_jsonb? Likewise, is there a better way to do the inner parts? To provide
> context on what "better" might be:
>
>- Snapshot JSONs might easily be 20MB in size.
>- Each 'pay_definition' is probablyonly about 1kB in size, and there
>might be 50 of them in a snapshot.
>- There might be 1000 MyModel instances in a given query.
>- I'm using PostgreSQL 12
>
> so my concern is not have the database server or Django perform extraneous
> work converting between strings and JSON for example.
>

It has been pointed out that a JOIN LATERAL like this can also perform the
iteration over the keys:

SELECT DISTINCT snapshot->’pay_definition’->k.value->’name’
FROM mymodel
JOIN LATERAL jsonb_object_keys(snapshot->’pay_definition’) AS k(value) ON
true

But Google suggests that a raw query might be the only way to get this to
work under the ORM. Pondering further, I was able to simplify that to this:

SELECT DISTINCT snapshot -> 'pay_definition' -> k.value -> 'name'
FROM mymodel, jsonb_object_keys(snapshot -> 'pay_definition') AS k(value);

Is there a way to wrap this form in the embrace of the ORM?

Thanks, Shaheed




=



>
> Thanks, Shaheed
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfv9z5AKQm8%3D%3DxWJD9e924%3DnANs7WSdaY_OEcPYyZOcMQ%40mail.gmail.com.

Good usage of native JSON functions and operators in ORM queries

[Shaheed Haque]

Hi,

I have a model MyModel which has a JSONField() called 'snapshot'. In Python
terms, each snapshot looks like this:

==
snapshot = {
'pay_definition' : {
'1234': {..., 'name': 'foo', ...},
'99': {..., 'name': 'bar', ...},
}
==

I'd like to find all unique values of 'name' in all instances of MyModel. I
have this working using native JSON functions like this:

   class PayDefs(Func):
function='to_jsonb'
template="%(function)s(row_to_json(jsonb_each((%(expressions)s
->'pay_definition')))->'value'->'name')"

   MyModel.objects.annotate(xxx=PayDefs(F('snapshot'))).order_by().distinct(
'xxx').values_list('xxx', flat=True)

My question is if this the best way to solve this problem? The way my
current logic works, reading from insider out is, I think:

   1. Pass in the 'snapshot'.
   2. Since 'snapshot' is a JSON field, "->'pay_definition'" traverses this
   key.
   3. To skip the unknown numeric keys, "jsonb_each()" turns each key,
   value pair into an inner row like ['1234', {...}].
   4. To get to the value column of the inner row "row_to_json()->'value'".
   5. To get the name field's value "->'name'".
   6. A final call to "to_jsonb" in the PayDefs class.

For example, since all I care about is the string value of 'name', is there
a way to get rid of the PayDefs class, and its invocation of to_jsonb?
Likewise, is there a better way to do the inner parts? To provide context
on what "better" might be:

   - Snapshot JSONs might easily be 20MB in size.
   - Each 'pay_definition' is probablyonly about 1kB in size, and there
   might be 50 of them in a snapshot.
   - There might be 1000 MyModel instances in a given query.
   - I'm using PostgreSQL 12

so my concern is not have the database server or Django perform extraneous
work converting between strings and JSON for example.

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jc24H-CKWH-3x2cEkRKMGnAz8s_HZt8%3Dd5LUrqKb%3D0r4A%40mail.gmail.com.

Re: Additional data added to the HTTP Get Request.

[Shaheed Haque]

I think you are missing a closing double quote on your data-url.

On Sat, 9 May 2020, 09:59 Ronald Kamulegeya, <
ronald.kamulegeya.2...@gmail.com> wrote:

> Hello Members.
>
> I am a django beginner and i am going trough online tutorials.
>
> I have the following view functions:
>
> # Create your views here.
> def save_tenant_form(request, form, template_name):
> data = dict()
> if request.method == 'POST':
> if form.is_valid():
> form.save()
> data['form_is_valid'] = True
> tenants = Tenants.objects.all()
> data['html_tenant_list'] = render_to_string(
> 'rentals/tenant_partial_list.html', {
> 'tenant_list': tenants
> })
> else:
> data['form_is_valid'] = False
> context = {'form': form}
> data['html_form'] = render_to_string(template_name, context, request=
> request)
> return JsonResponse(data)
>
>
>
>
> def createTenant(request):
> if request.method == 'POST':
> form = TenantsForm(request.POST)
> else:
> form = TenantsForm()
>
> return save_tenant_form(request, form,
> 'rentals/partial_tenant_create.html')
>
> Here is the url for the create function
> path("tenants/add/",views.createTenant, name='createTenant'),
>
> There is some java script to load the modal form shown below:
> $(function () {
>
>
>   /* Functions */
>
>
>   var loadForm = function () {
> var btn = $(this);
> $.ajax({
>   url: btn.attr("data-url"),
>   type: 'get',
>   dataType: 'json',
>   beforeSend: function () {
> $("#modal-tenant .modal-content").html("");
> $("#modal-tenant").modal("show");
>   },
>   success: function (data) {
> $("#modal-tenant .modal-content").html(data.html_form);
>   }
> });
>   };
>
>
>   var saveForm = function () {
> var form = $(this);
> $.ajax({
>   url: form.attr("action"),
>   data: form.serialize(),
>   type: form.attr("method"),
>   dataType: 'json',
>   success: function (data) {
> if (data.form_is_valid) {
>   $("#tenant-table tbody").html(data.html_book_list);
>   $("#modal-tenant").modal("hide");
> }
> else {
>   $("#modal-tenant .modal-content").html(data.html_form);
> }
>   }
> });
> return false;
>   };
>
>
>
>
>   /* Binding */
>
>
>   // Create tenant
>   $(".js-create-tenant").click(loadForm);
>   $("#modal-tenant").on("submit", ".js-tenant-create-form", saveForm);
>
> In the html form, i  try to load the modal form by clicking on this button
> 
>  data-url="{% url 'rentals:createTenant' %}>
>   
>   New Tenant
> 
>
>   
> I get a strange response shown below
> [09/May/2020 11:38:58] "GET
> /tenants/add/%3E%20%20%20%20%20%20%3Cspan%20class= HTTP/1.1" 404 2789
>
> I need the "GET /tenants/add/ part only. I do not know how the additional
> data is added to the Get request!
> Any bug in my code causing the strange data to be added?
>
> Ronald
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/2c8f5ac8-2109-49b4-ad8a-d75c2350a21c%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcsy6iacB-Yyo8mz1T_X%3DCq8WkZ02M4Tiv091JzGbONqQ%40mail.gmail.com.

Re: Channels/Daphne offloading SSL Certs to AWS ELB fails to establish socket

[Shaheed Haque]

On Tue, 21 Apr 2020 at 00:30, Tim Nelson  wrote:

> Well if you want the clients IP logged or need to do something with it on
> the request:
>
>set_real_ip_from 0.0.0.0/0;
>real_ip_header X-Forwarded-For;
>real_ip_recursive on;
>

Thanks for the tip. I'll have to look into those in due course.

Shaheed


>
> On Mon, Apr 20, 2020 at 5:55 PM Shaheed Haque 
> wrote:
>
>>
>>
>> On Mon, 20 Apr 2020 at 21:54, Filbert  wrote:
>>
>>> *Answering my own question.AWS classic ELBs never worked properly
>>> with websockets. Need to convert them to ALBs.*
>>>
>>
>> Thanks for closing the loop; this is quite likely to be in my future. Did
>> you need any nginx config changes?
>>
>> Shaheed
>>
>> On Tuesday, April 14, 2020 at 5:37:54 PM UTC-4, Filbert wrote:
>>>>
>>>> Using AWS ELB, I was able to serve HTTP (via uwsgi) and websockets
>>>> through Nginx with Nginx handling the SSL certs.
>>>>
>>>> Once I tried to offloading SSL certs to the ELB so I could capture
>>>> X-Forwarded-For, websockets fail to route/upgrade to Daphne and I get "Not
>>>> Found: /ws/" in uwsgi's logs.
>>>>
>>>> Here is the relevant (working) piece of Nginx .conf file using TCP/443
>>>> pass thru from the ELB:
>>>>
>>>> location /ws {
>>>> proxy_pass http://unix:/opt/toogo/run/r117.0.4.3720.ws;
>>>>  #Obviously daphne is listening here
>>>> proxy_set_header Upgrade $http_upgrade;
>>>> proxy_set_header Connection "upgrade";
>>>> proxy_http_version 1.1;
>>>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>> proxy_set_header Host $host;
>>>> }
>>>>
>>>> What change is required if I offload the SSL certs to the ELB and
>>>> introduce X-Forwarded-for?
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-users/305b31ae-ff69-425a-a173-645c55ba07a1%40googlegroups.com
>>> <https://groups.google.com/d/msgid/django-users/305b31ae-ff69-425a-a173-645c55ba07a1%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-users/iNoIPxDgv90/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAHAc2jf1KNYe-p2JM-XLD2q8HpC_29oNrJ24KkyjVFRj%2BpPo%2BQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-users/CAHAc2jf1KNYe-p2JM-XLD2q8HpC_29oNrJ24KkyjVFRj%2BpPo%2BQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAK09zoq2pD6MsMfXA6y_rj4CUvcxgrT-w11xYCSvaUjdm8TD_Q%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-users/CAK09zoq2pD6MsMfXA6y_rj4CUvcxgrT-w11xYCSvaUjdm8TD_Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jcGo1YFJhJNf%2Bt1BfhD60dG%3DwUaamU5M9Wg_ksAm4HWaQ%40mail.gmail.com.

Re: Channels/Daphne offloading SSL Certs to AWS ELB fails to establish socket

[Shaheed Haque]

On Mon, 20 Apr 2020 at 21:54, Filbert  wrote:

> *Answering my own question.AWS classic ELBs never worked properly with
> websockets. Need to convert them to ALBs.*
>

Thanks for closing the loop; this is quite likely to be in my future. Did
you need any nginx config changes?

Shaheed

On Tuesday, April 14, 2020 at 5:37:54 PM UTC-4, Filbert wrote:
>>
>> Using AWS ELB, I was able to serve HTTP (via uwsgi) and websockets
>> through Nginx with Nginx handling the SSL certs.
>>
>> Once I tried to offloading SSL certs to the ELB so I could capture
>> X-Forwarded-For, websockets fail to route/upgrade to Daphne and I get "Not
>> Found: /ws/" in uwsgi's logs.
>>
>> Here is the relevant (working) piece of Nginx .conf file using TCP/443
>> pass thru from the ELB:
>>
>> location /ws {
>> proxy_pass http://unix:/opt/toogo/run/r117.0.4.3720.ws;
>>  #Obviously daphne is listening here
>> proxy_set_header Upgrade $http_upgrade;
>> proxy_set_header Connection "upgrade";
>> proxy_http_version 1.1;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header Host $host;
>> }
>>
>> What change is required if I offload the SSL certs to the ELB and
>> introduce X-Forwarded-for?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/305b31ae-ff69-425a-a173-645c55ba07a1%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jf1KNYe-p2JM-XLD2q8HpC_29oNrJ24KkyjVFRj%2BpPo%2BQ%40mail.gmail.com.

Re: How to best secure environment variables (secret key, passwords etc.) stored in .yml files?

[Shaheed Haque]

I don't think you are overthinking this.

On Thu, 30 Jan 2020, 12:40 Tom Moore,  wrote:

> Hi there, I'm following the guidelines by making sure the environment
> variables are stored outside of the settings.py files.
>
> The project is "dockerised" and so the environment variables have been
> stored in files *docker-compose.yml* and *docker-compose-prod.yml*.
>

What we do isn't perfect but does limit our exposure:

- we don't store the production key(s) in git at all.
- instead, we fetch them at deployment time using a secure connection to
the store, and then inject them into our runtime (we are still in a vm)

The secure connection is the less than ideal part, since we require a human
to provide the private key. In theory, we can replace that with automation
at the cost of a short insecure connection (or some other complex key
management). But we can change that at will as our scale needs evolve.




> This includes things like the project's secret key, API keys, and database
> passwords.
>
> *My question is: *
> • Just because environment variables are stored in .yml files, won't they
> be equally insecure the moment I commit the project folder to a git repo
> (and especially if I push that repo to GitHub)?
> e.g. the Secret Key will forevermore be stored in the git repo (in earlier
> versions, even if I later move it to another file in subsequent commits).
>
> Is there an even more secure way of storing environment variables? Or am I
> overthinking it (as I'm the only developer and the GitHub repo is set to
> Private)?
>
> Many thanks in advance for your help.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/887bcd5b-4525-4a54-a4e5-5eae32b20041%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jdrdbroRUxFEtfiia7H0bGvuuWkEjg9XP2COumNqXk3kw%40mail.gmail.com.

Re: Make a Django query filter at runtime

[Shaheed Haque]

On Wed, 8 Jan 2020 at 20:09, Ezequias Rocha 
wrote:

> Hi everyone
>
> I am in a doubt about creating django filters dynamically.
>
> All you know a filter is made by using the parameters like:
>
> model.filter(name='John', age=42)
>
> But if I can't type all fields and values at design time but at runtime
> how to do that?
>

Populate a dictionary with what you need and pass that in:

query = {'name': 'John', 'age': 42}
model.filter(**query)



>
> Best regards
> Ezequias
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/9c843d02-235f-411f-8e83-7fea2156893b%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfQDKCv8gtqsnmJJrRm1XPvRp%3DCCQ3nP_3b8D16oyw65w%40mail.gmail.com.

Re: How to make Django Application more secure

[Shaheed Haque]

Hi,

On Thu, 2 Jan 2020, 19:30 Jody Fitzpatrick, 
wrote:

> Hi Balaji
>
> It's not necessarily template views.
>
> Let's come up with a scenario so you can see.
>
>
> Let's assume you have an order form, and your customers can view that
> order form by viewing:
>
> yoururl.com/orders/?order_id=101
>
> You think it's okay - after all the customer has to login and view their
> order.
>
> In your backend you use something like
>
>
> customer_order_id = request.GET.get('order_id')
> order = CustomerOrder.objects.get(id=customer_order_id)
>
>
> There is a couple of problems with this.
>
>  You should NEVER use numbers as your IDs that your user sees.
>
>  -- You can potentially let competitors know how many customers you have,
> or how many many orders you processed.
>
> But wait, if you look at the query - and I have seen this before... the
> query is not checking to see if the current user has permission to view the
> order...
> it just grabs the record with the ID
>
> Now assume that the end user changes 101 to 102, and to 103 -- if these
> records exist. They are going to get the data.
>
> use something like uuid as your primary key to prevent this...
>
> Also, add ownership to your queries, ex.) (id=customer_id,
> customer=request.user)
>

As a noob, I realised the "scope security" aspect but it took a LOT longer
to realise that Django's ORM has a nice pattern which can be of use here.
The idea is to use queries rooted on the user. Using Jody's example,
observe that there must be an FK between the request.user and
CustomerOrder.customer.

That means that the query could be written like this:

order = request.user.customerorder_set.get(...)

AFAIK there is no performance penalty to using this style and it seems to
me like a good habit to adopt (I'd be interested to hear if the experts
think differently?).

Thanks, Shaheed


> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/343105a0-4bd3-42f0-ba0d-c41d2482f9e0%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfhROAHioNrhOraiTa-S1XYBr_J-ouKDL3yZZVhm%2BOrRQ%40mail.gmail.com.

Re: Django 3.0 Released.

[Shaheed Haque]

On that subject...

On Thu, 5 Dec 2019, 15:12 Jonathan Morgan, 
wrote:

> Another thing to consider is whether you need to update the packages you
> use to build your application.  I've found that support for the LTS version
> of django is not consistent across packages, where support for the latest
> version is much more consistent, such that sometimes a needed update to a
> package required me to update django, either because of the package itself
> only working on a newer version, or one of its dependencies having this
> constraint.  Because of this, I tend to keep my applications running on the
> latest django
>

This is generally our preference too. But it does depend on the related
packages we use to stay compatible. I generally try to help here by trying
out early copies against our test suite and reporting issues early. This
time this approach worked for all but one related package (I have my
fingers crossed for that one).

, rather than an LTS, so I can also keep updating the related packages.  I
> also personally find the changes needed to move an application from version
> to version of django to be much more managable if you do it every version,
> in small chunks, rather than waiting between LTS versions, but your mileage
> and stomach for these updates may vary.
>
> On Wednesday, December 4, 2019 at 1:48:06 PM UTC-5, Andréas Kühne wrote:
>>
>> So your question is if you should use LTS versions or use the latest
>> blazing version.
>>
>> The answer is of course - it depends.
>>
>> Do you see anything in django 3 that you need to have? In that case -
>> upgrade to django 3. If you only want to make sure that your application is
>> safe and secure and you don't need the bleeding edge functionality - keep
>> with the LTS:es.
>>
>> It also depends on your application cycle. I currently have 2 projects
>> running - one is a commercial ecommerce platform. The other is a platform
>> for course administration. Both have completely different requirements on
>> technical functionality from django. On the ecommerce site we are running
>> on LTS versions, on the other site we are running with the latest version
>> of django.
>>
>> I think the most important thing is to make sure that you have a secure
>> site regardless - so if you run the lts versions, you need to update as
>> soon as the next LTS comes out. And the same goes for the other versions as
>> well - update all the time in that case.
>>
>> Regards,
>>
>> Andréas
>>
>>
>> Den mån 2 dec. 2019 kl 22:18 skrev o1bigtenor :
>>
>>> On Mon, Dec 2, 2019 at 6:18 AM Carlton Gibson 
>>> wrote:
>>> >
>>> > Django 3.0 is now available.
>>> >
>>> > https://www.djangoproject.com/weblog/2019/dec/02/django-3-released/
>>> >
>>> > With the release of Django 3.0, Django 2.2 has reached the end of
>>> > mainstream support. The final minor bug fix release (which is also a
>>> > security release), 2.2.8, was issued today. Django 2.2 is an LTS
>>> release and
>>> > will receive security and data loss fixes until April 2022. All users
>>> are
>>> > encouraged to upgrade before then to continue receiving fixes for
>>> > security issues.
>>> >
>>> > See the downloads page [1] for a table of supported versions and the
>>> > future release schedule.
>>> >
>>> > [1] https://www.djangoproject.com/download/#supported-versions
>>> >
>>> Greetings
>>>
>>> Thank you - - - -interesting news!
>>>
>>> I'm a little confused though - - - - I have, at least in many things
>>> so far that I've used,
>>> software I mean, tended to move from one long term service version to
>>> the next.
>>> Yet - - - Django is different - - - - it is suggested that I leave a
>>> long term version
>>> for a short term version, which will itself be followed by a short
>>> term version, and then
>>> after another period of time there will be a long term service
>>> version. Shortly after
>>> that long term service version has been reached, if the pattern used
>>> for the transition
>>> from version 2 to 3, version 4 will be offered.
>>>
>>> So - - - - what gives the best long term stability of function for use
>>> of Django?
>>>
>>> 1. forget long term service versions and expect to revise you code
>>> every 6 to 8 months
>>> 2. use LTS code but using it until the release of the next major
>>> version means that
>>>   interesting advancements again mean a relentless pattern of
>>> revision
>>> 3. don't worry about LTS and write the code and get the application to
>>> work well and
>>>  hope that I can hold things working without major security issues
>>> until its worth
>>>  redoing the application.
>>>
>>> I'm starting to think that option #3 is looking more attractive.
>>>
>>> Regards
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/ms

Re: Microservice with django

[Shaheed Haque]

At the end of the day, assuming you combine the Django "application server"
with a REST wrapper, you can build any level or type of service you could
want.

The crucial question is how you want to trade off the power of the ORM
against custom code? For example, let's say you wrapped every single table
(Django model) in a separate "nano"service... Then you could not use sql to
Jon or select anything and everything would be custom code.

OTOH, if you put everything in a single service, you can freely use the ORM
but loose whatever advantage you hope to fain from the modularity of soa.

Hth, Shaheed

On Fri, 8 Nov 2019, 20:22 salimon jamiu olashile, 
wrote:

> I’ve a group project to work on. It’s a bug tracking system & we decided
> to go with a microservice architecture which is the best solution for the
> assumed business scenario. I am to build the Reporting functionality as a
> seperate service.
>
> On Fri, 8 Nov 2019 at 7:26 PM, Murilo A. Gigliotti <
> mur...@gigliottitech.com.br> wrote:
>
>> All,
>>
>> Good question! I would like more details... Thank you!
>>
>>
>>
>> Rgds,
>>
>> _
>>
>> Murilo Gigliotti,
>>
>> *[image: Odoo icone2] Odoo Business Partner*
>>
>> Mais informações visite http://www.gigliottitech.com.br/
>>
>>
>>
>> *De:* django-users@googlegroups.com [mailto:django-users@googlegroups.com]
>> *Em nome de *Salim Kachemela
>> *Enviada em:* sexta-feira, 8 de novembro de 2019 15:04
>> *Para:* django-users@googlegroups.com
>> *Assunto:* Re: Microservice with django
>>
>>
>>
>> I was wondering the same thing
>>
>>
>>
>> On Fri, 8 Nov 2019, 20:15 salimon jamiu olashile, <
>> tunedae1shi...@gmail.com> wrote:
>>
>> Hello all,
>>
>>
>>
>> How can I build a Django project/app that follow the microservice
>> architecture?
>>
>>
>>
>> Looking to be pointed in the right direction.
>>
>>
>>
>> Best regards.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAFhdOCNZ%2BR084HQc1FD5a%2BT%3D_umuH87haQbe%3DyD2aAT6SKtUsg%40mail.gmail.com
>> 
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CACB6%2B1-9tNHnZ8X7U_EwqQx6%2B_msMhThvOif2RBuEuNbY3QVXQ%40mail.gmail.com
>> 
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/055701d59661%24ed290420%24c77b0c60%24%40com.br
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAFhdOCP8rGBzBoiYqQ_CtaQdNEN%2BKM5NqhBKBcAe0C9%2BP-ZKaw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jfkY1%2BzGbWBCHm3_pP4J3fzwg_wirrqDPQM%3DQ1f7OXWAw%40mail.gmail.com.

Testing the validity of a form which has inline formsets from a View subclass

[Shaheed Haque]

Hi all,

I have a set of views in a superclass/subclass hierarchy like this:

   - mobile_view is a subclass of
   - generic___view is a subclass of
   - django-extra-views formset support view

As expected, the django-extra-views formset support's "post()" logic does a
"if form.is_valid() then form.save(commit=False)", and then proceeds to
save the contents of the formsets after testing them with formset
"is_valid" logic, before returning an HttpResponse. AFAICS, no exceptions
are raised to signal the error.

Let's say I now want to do some additional work in generic_object_view,
based on whether the superclass had an error or not:

   - I could construct the form again, and the formsets again, and
   basically redo what the superclass did
   - And of course, it would need redoing again one layer up in
   "mobile_view".

Or have I missed something I can test in the view itself (i.e.
self.) or the return HTTPResponse?

Thanks, Shaheed

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jd842he83j77xP4phU4-CfyiqvNWhQ3T2wd2yWoGFuNJA%40mail.gmail.com.

Re: architecture for Blue/Green deployments

[Shaheed Haque]

I should clarify one thing...

On Thu, 2 May 2019 at 20:28, Shaheed Haque  wrote:

> On Thu, 2 May 2019 at 19:37,  wrote:
>
>> What gold!
>>
>> The essential piece of your logic is here:
>>
>>> Enforce a development process that ensures that (roughly speaking) all
>>> database changes result in a new column, and where the old cannot be
>>> removed until a later update cycle. All migrations populate the new column.
>>
>>
>> I assume that "enforce" is a software engineering management thing, not
>> clever CI/CD code.
>>
>
> Ack.
>

This response appears to contradict my original note, but what I actually
think is that *some* automation along the lines of Django's existing
migration capability is possible, and that careful software engineering
management would likely be needed to (a) maximise the chance the automation
will work, and (b) detect the cases when it won't.

As for the posited automation itself, one could imagine that the existing
Django logic to analyse the need for migrations would work as-is. What
would likely be needed is a different "backend" that targets blue-green
upgrades. I would guess the overall level of automation possible is less
than the near-magical current system, but I'm hopeful it might make the
problem manageable.

(Of course, I'm still crossing my fingers that some real Django migration
and SQL gurus will solve this before I have to).

Thanks, Shaheed


> To change a column, you essentially do it in two steps:
>>
>>- Create new column with migrations, and do a release to the
>>environment.
>>- Drop old column with migrations, and do a release to the
>>environment.
>>
>> Just so.
>
>
>> If you are only dropping an old column, it might go like this:
>>
>>- Drop the old column from  the model, but not from the database
>>(e.g. assure that makemigrations has not been run), test and deploy.
>>- Add the migration that does away with the old column - test and
>>deploy.
>>
>> Indeed.
>
> Personally, I have a similar career trajectory, but from systems
>> programming in C/C++ to software architect for webapps.   Understanding
>> C/C++  and Linux gave me a capability of working with both developers and
>> system guys. I think it was in 2013 that I did the presentation that
>> started to kill Adobe ColdFusion (which I didn't want to learn).   I
>> instead the next 6 years getting all of our applications moved to Django,
>> now we are in transition to Python 3.6 and Django 2.2, with our first cloud
>> system coming up.
>>
>
> LOL.I bet we could both tell some tales.
>
> On your slow cloud builds, I have an architecture that may help. The basic
>> idea is easy to explain:
>>
>>- Do system provisioning through ansible roles
>>- Install all the roles needed for all of your stacks when you build
>>the AMI, but only run some of them, e.g. the basics.
>>- If your build time in the cloud takes too long, the architecture
>>assures you can easily pivot to prebaking more into the AMI, but you are
>>not assuming you will have to.
>>
>> Of course, you still cannot go to milliseconds.   But it allows you to
>> trade building ahead of time and building during bring-up to nearly your
>> hearts content.  Even the role that knows how to install a Python webapp is
>> already on the AMI.
>>
>
> From the previous experience I alluded to, I appreciate that using Puppet
> (or Chef or Ansible) to pre-bake a VM image and such an image would, as you
> note, significantly reduce my re-spin "system down" window. Now, I hesitate
> to say the next bit out loud, because I don't yet know if I am right, or
> the received wisdom about freezing dependencies is right, but here
> goes...comments welcome!
>
> My current thinking is that the notion of trying to keep a public website
> secure by freezing dependencies (think virtualenv AND apt) is an Sisiphyian
> task given the implied need to track the transitive fanout of ALL
> dependencies down to the kernel. So, given that we have decent test
> coverage, and are frequently running those "live-to-test" upgrades, I can
> trade the security vulnerability tracking for compatibility tracking by
> having each upgrade cycle rebuild from the latest apt and pypi
> repositories. That's a win because we have to do the compatibility tracking
> anyway.
>
> Thus, I get early exposure to occasional issues (e.g. pgAdmin was broken
> recently by pyscopg2 2.8, and django-polymorphic is broken by Django 2.2,
> both of which I discovered within about a day of the 

Re: architecture for Blue/Green deployments

[Shaheed Haque]

On Thu, 2 May 2019 at 19:37,  wrote:

> What gold!
>
> The essential piece of your logic is here:
>
>> Enforce a development process that ensures that (roughly speaking) all
>> database changes result in a new column, and where the old cannot be
>> removed until a later update cycle. All migrations populate the new column.
>
>
> I assume that "enforce" is a software engineering management thing, not
> clever CI/CD code.
>

Ack.

To change a column, you essentially do it in two steps:
>
>- Create new column with migrations, and do a release to the
>environment.
>- Drop old column with migrations, and do a release to the environment.
>
> Just so.


> If you are only dropping an old column, it might go like this:
>
>- Drop the old column from  the model, but not from the database (e.g.
>assure that makemigrations has not been run), test and deploy.
>- Add the migration that does away with the old column - test and
>deploy.
>
> Indeed.

Personally, I have a similar career trajectory, but from systems
> programming in C/C++ to software architect for webapps.   Understanding
> C/C++  and Linux gave me a capability of working with both developers and
> system guys. I think it was in 2013 that I did the presentation that
> started to kill Adobe ColdFusion (which I didn't want to learn).   I
> instead the next 6 years getting all of our applications moved to Django,
> now we are in transition to Python 3.6 and Django 2.2, with our first cloud
> system coming up.
>

LOL.I bet we could both tell some tales.

On your slow cloud builds, I have an architecture that may help. The basic
> idea is easy to explain:
>
>- Do system provisioning through ansible roles
>- Install all the roles needed for all of your stacks when you build
>the AMI, but only run some of them, e.g. the basics.
>- If your build time in the cloud takes too long, the architecture
>assures you can easily pivot to prebaking more into the AMI, but you are
>not assuming you will have to.
>
> Of course, you still cannot go to milliseconds.   But it allows you to
> trade building ahead of time and building during bring-up to nearly your
> hearts content.  Even the role that knows how to install a Python webapp is
> already on the AMI.
>

>From the previous experience I alluded to, I appreciate that using Puppet
(or Chef or Ansible) to pre-bake a VM image and such an image would, as you
note, significantly reduce my re-spin "system down" window. Now, I hesitate
to say the next bit out loud, because I don't yet know if I am right, or
the received wisdom about freezing dependencies is right, but here
goes...comments welcome!

My current thinking is that the notion of trying to keep a public website
secure by freezing dependencies (think virtualenv AND apt) is an Sisiphyian
task given the implied need to track the transitive fanout of ALL
dependencies down to the kernel. So, given that we have decent test
coverage, and are frequently running those "live-to-test" upgrades, I can
trade the security vulnerability tracking for compatibility tracking by
having each upgrade cycle rebuild from the latest apt and pypi
repositories. That's a win because we have to do the compatibility tracking
anyway.

Thus, I get early exposure to occasional issues (e.g. pgAdmin was broken
recently by pyscopg2 2.8, and django-polymorphic is broken by Django 2.2,
both of which I discovered within about a day of the issue arising) while
ditching the need to continuously track the security perimeter of the whole
shooting match. Another way to look at it is that I leverage everybody
else's tracking of their security issues (fixes for functional issues are a
side benefit).

Assuming this analysis proves itself, the benefits trump the advantages of
a pre-baked VM image over rebuilds from live repos, at least for me (I
might think differently if we had more human bandwidth to burn).

And anyway, I'll fix the downtime with a cluster. One day. :-)


-- 
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/0799c1b8-04cc-424e-a08d-f124eab03ce7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@goog

Re: architecture for Blue/Green deployments

[Shaheed Haque]

Hi Dan,

I recently went through a similar exercise to the one you describe to move
our prototype code on AWS.

First, my background includes a stint building a control plane for
autoscaling VMs on OpenStack (and being generally long in tooth), but this
is my first attempt at a Web App, and therefore Django too. I also grew up
on VAXes, so the notion of an always-up cluster is deeply rooted.

Technical comments follow inline...

On Wed, 1 May 2019 at 21:35,  wrote:

> My organization is moving into the AWS cloud, and with some other projects
> using MongoDB, ElasticSearch and a web application framework that is not
> Django, we've had no problem.
>
> I'm our "Systems/Applications Architect", and some years ago I helped
> choose Django over some other solutions.   I stand by that as correct for
> us, but our Cloud guys want to know how to do Blue/Green deployments, and
> the more I look at it the less happy I am.
>
> Here's the problem:
>
>- Django's ORM has long shielded developers from simple SQL problems
>like "SELECT * FROM fubar ..." and "INSERT INTO fubar VALUES (...)" sorts
>of problems.
>- However, if an existing "Blue" deployment knows about a column, it
>will try to retrieve it:
>   - fubar = Fubar.objects.get(name='Samwise')
>- If a new "Green" deployment is coming up, and we want to wait until
>Selenium testing has passed, we have the problem of migrations
>
> I really don't see any simple way around a new database cluster/instance
> when we bring up a new cluster, with something like this:
>
>- Mark the live database as "in maintenance mode".The application
>now will not write to the database, but we can also make that user's access
>read/only to preserve this.
>- Take a snapshot
>- Restore the snapshot to build the new database instance/cluster.
>- Make the new database as "live", e.g. clear "maintenance mode".   If
>t he webapp user is read-only, they are restored to full read/write
>permissions.
>- Run migrations in production
>- Bring up new auto-scaling group
>
> We are not yet doing auto-scaling but otherwise your description applies
very well to us. Right now, we have a pair of VMs, a "logic" VM hosting
Django, and a "db" VM hosting Postgres (long term, we may move to Aurora
for the database, but we are not there right now). The logic VM is based on
an Ubuntu base image, but a load of extra stuff:

   - Django, our code and all Python dependencies
   - A whole host of non-Python dependencies starting with RabbitMQ (needed
   for Celery), nginx, etc
   - And a whole lot of configuration for the above (starting with
   production keys, passwords and the like)

The net result is that not only does it take 10-15 minutes for AWS to spin
up a new db VM from a snapshot, but it also takes several minutes to spin
gup, install, and configure the logic VM. So, we have a piece of code that
can do a "live-to-" upgrade:

   - Where scenario is "live-to-live" or "live-to-test".
   - The logic is the same in both except for a couple of small pieces only
   in the live-to-live case where we:
  - Pause the live system (db and celery queues) before snapshotting it
  for the new spin up
  - Create an archive of the database
  - Switch the Elastic IP on successful sanity test pass
   - We also have a small piece of run-time code in our project/settings.py
   that, on a live system, enables HTTPS and so on.

Before we do the "live-to-live" upgrade, we always to a "live-to-test"
upgrade. This ensure we have run all migrations and pre-release sanities on
virtually current data, and then perform a *separate* live-to-live.

While this works, it creates a window during when the service must be down.
There is also a finite window when all those 3rd party dependencies on apt
and pip/pypi expose the "live-to-live" to potential failure.

So in the "long term", I would prefer to attempt something like the
following:

   - Use a cluster of logic N VMs.
   - Use an LB at the front end.
   - Enforce a development process that ensures that (roughly speaking) all
   database changes result in a new column, and where the old cannot be
   removed until a later update cycle. All migrations populate the new column.
   - We spin up and N+1th VM with the new logic, and once sanity testing is
   passed, switch the N+1 machine on in the LB, and remove one of the original
   N.
  - Loop
   - Delete the old column

Of course, the $64k question is all around how to keep the old logic and
the new logic in sync with the two columns. For that, I can only wave my
arms at present and say that the old column cannot really be there in its
bare form, instead there will be some kind of a view that makes it look
like it is - possibly with some old school stored procedure/trigger logic
in support. Of course, I would love it if there were some magic tooling
developed by the Django and database gurus before I have to tackle this.
Then again, I don't bel

Re: Django reporting Library recommendations

[Shaheed Haque]

Hi Brian,

On Tue, 16 Apr 2019 at 12:21, Brian Mcnabola  wrote:
>
> Im looking for a Django reporting library something similar to Django report 
> builder ?

Can you clarify what you are looking for that Django report builder
does not provide? For example, we use JasperReports wrapped for use
from Python but that is a very different approach that Django report
builder.

Thanks, Shaheed

> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/b7f8225b-1c9a-4d67-b74c-5667c98aa4d2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHAc2jd%2BkMDKsc5wLPMPBjL534mTwQWS_osYBYu0mqZDCkt_7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Measuring code coverage when using Selenium and pytest

[Shaheed Haque]

Hi,

I'd like to get coverage.py running against my code base. The code consists 
of standard Django, some Celery code, some django-viewflow code and some 
Celery-and-django-viewflow code [1]. The main problem I see is that if I 
say "coverage -p manage.py runserver ...", I need a way for the server to 
exit in a way which allows coverage to end the run and write its output. 
For now, I achieve this by hacking in a specific view and arranging to call 
"_thread.interrupt_main()" at an appropriate time. This indeed causes a 
coverage output file to be written.

Unfortunately, the recorded results only show the static reading of the 
code files, so that class and function definitions are read, but no 
function body lines show up as covered. I suspected this might have 
something to do with spawned processes, so I have tried using "runserver 
--noreload" to avoid the second process, but to no avail.

Why is the normally executed code path not shown as executed? AFAIK, there 
should be no multi-processing or similar in play...am I mistaken in that 
regard?

Thanks, Shaheed

P.S. In separate attempts, I tried all the relevant techniques in this 
StackOverflow discussion 

 
such as using the coverage.py API in manage.py.

[1] I realise that:

   - Celery and coverage.py are not comfortable bedfellows (I believe this 
   is due to Celery using its own billard package, rather then generic 
   multiprocessing, which coverage.py apparently knows about).
   - There are standard recipies for getting coverage.py running when using 
   "manage.py test", but the tests in my current project are not run this way; 
   instead, pytest is used to invoke the tests, most of which use Selenium.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/37d3ac9b-b43c-4781-8809-1aac1274e6b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.