Re: Running migrations with multiple databases
Thanks: just what I needed to know!
On Tuesday, 11 July 2023 at 02:48:48 UTC+1 David Nugent wrote:
Migrations are executed according to how your db routers is setup.
See setting DATABASE_ROUTERS.
Note that normally you would be replicating secondaries, so migrations are
normally not applied on all bu the primary, so this is the default.
Regards,
David
On Mon, Jul 10, 2023 at 10:26 PM Shaheed Haque wrote:
Hi,
I'm on Django 4.2 atop Postgres. In my project settings.py, I have a
main/default database connection, and a second set up like this:
==
DATABASES = {
'default': {
...
'NAME': 'foo', # Postgres DATABASE NAME
...
},
'archive_restore': {
...
'NAME': 'archive_restore', # Postgres DATABASE NAME
...
},
# Other entries for specialised purposes such as custom Postgres
Foreign Data Wrappers.
===
As you can see, the Postgres database names are "foo" and "archive_restore"
respectively. For all normal Django purposes, we want to use "default", aka
"foo". The "archive_restore" connection/database is used in conjunction
with a bunch of psql commands to create a subset of the main Django ORM
data. Once populated via pg_restore, I need to run the DJango migrations on
them for eventual use under Django. I had assumed that a command like this:
./manage.py migrate --database archive_restore
would modify archive_restore/archive_restore. However, what seems to happen
is that the migrations are:
- Actually run in default/foo (the migrations include code generated by
makemigrations and custom RunPython stuff).
- But recorded in the django_migrations table in
archive_restore/archive_restore.
whereas I was expecting that they would be both run and recorded in the
latter. Have I overlooked some setting/restriction, or is this a bug?
Thanks, Shaheed
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-users...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAHAc2je%2B%3D67tZBeGBMUO8Dmqmq9SaCRuopLxqrECuXb1C8YriQ%40mail.gmail.com
.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/5e5abb96-10bc-43e9-bd33-6328e62f2c46n%40googlegroups.com.
Re: Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model
Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.
On Thursday, 15 October 2020 at 12:51:22 UTC+1 Jason wrote:
> this is pretty interesting, and may be something to report to the devs at
> https://groups.google.com/g/django-developers, or open up a ticket at
> https://code.djangoproject.com/ I don't see any tickets with the query
> *jsonfield
> foreign data* that are similar with your experience, so you may have
> found an edge case.
>
> On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 shahee...@gmail.com
> wrote:
>
>>
>> Hi,
>>
>> I have a Django model working fine under Django 3.0 (i.e. "Django<3.1")
>> which looks like this:
>>
>> ===
>> class Job(models.Model):
>> id = models.CharField(max_length=36, primary_key=True)
>> queue = models.CharField(max_length=40)
>> args = JSONField()
>> kwargs = JSONField()
>> type = models.CharField(max_length=80)
>> ...
>>
>> class Meta:
>> managed = False # <-- The table is implemented as a
>> Postgres FDW wrapper.
>> db_table = 'jobs'
>> ===
>>
>> I am testing the update to Django 3.1.2 and hit an error in executing
>> this line:
>>
>> jobs = list(models.Job.objects.filter(queue='celery',
>> state='scheduled'))
>>
>> The error is as follows from pytest (i.e. stack trace with local
>> variables too):
>>
>> ==
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>> _ _ _
>> /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in
>> __iter__
>>self._fetch_all()
>>self =
>> /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in
>> _fetch_all
>>self._result_cache = list(self._iterable_class(self))
>>self =
>> /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in
>> __iter__
>>for row in compiler.results_iter(results):
>>annotation_col_map = {}
>>compiler = > 0x7f8685e49160>
>>db = 'fdw'
>>init_list = ['id', 'queue', 'args', 'kwargs', 'type', 'state',
>> ...]
>>klass_info = {'model': ,
>> 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
>>known_related_objects = []
>>model_cls =
>>model_fields_end = 9
>>model_fields_start = 0
>>queryset =
>>related_populators = []
>>results= [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
>> ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
>> 'paiyroll.tasks.function_run', 'scheduled', ...)]]
>>select = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []),
>> None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None...,
>> paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs,
>> paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
>>select_fields = [0, 1, 2, 3, 4, 5, ...]
>>self = > 0x7f86836f3040>
>> /usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100:
>>
>> in apply_converters
>>value = converter(value, expression, connection)
>>connection = > object at 0x7f869a321670>
>>converter = > >
>>converters = [(2, ([> >], Col(jobs,
>> pa...NField.from_db_value of
>> >], Col(jobs,
>> paiyroll.Job.details)))]
>>convs = [> >]
>>expression = Col(jobs, paiyroll.Job.args)
>>pos= 2
>>row= ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
>> ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
>> 'paiyroll.tasks.function_run', 'scheduled', ...]
>>rows =
>>self = > 0x7f8685e49160>
>>value = ['paiyroll.tasks', 'call_to_string',
>> 'call_to_string', [], {}]
>>
>>
>>
>>
>>
>> */usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74:
>> in from_db_valuereturn json.loads(value, cls=self.decoder)
>>connection = > object at 0x7f869a321670>expression = Col(jobs, paiyroll.Job.args)
>>self =
>>value = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
Re: A question about running Channels and Django code under Celery
Perhaps I should have known, but didn't: Celery 4.x does not support
asyncio (celery#3883 <https://github.com/celery/celery/issues/3883>,
celery#3884 <https://github.com/celery/celery/issues/3884>). Luckily, this
was explained over at channels_rabbitmq#25
<https://github.com/CJWorkbench/channels_rabbitmq/issues/25>. From there, a
subprocess based fix was easy...once I became aware of channels_rabbitmq#13
<https://github.com/CJWorkbench/channels_rabbitmq/issues/13>. TL;DR:
- You cannot naively call into Channels (or any other) async code from
inside a Celery task (until Celery 5).
- You cannot naively call async_to_sync() et. al. Some awareness of the
scope of the event loop(s) that underpin the system is required, or a belt
and braces type workaround.
Thanks, Shaheed
P.S. I went down this rabbit hole partly because, at the time of my
original posting, I had not noticed that the exception noted below was
still happening AND when I moved off my development system onto an AWS
deployment setup, everything stopped working.
On Saturday, 22 August 2020 at 14:52:33 UTC+1 shahee...@gmail.com wrote:
> I have a question about running Channels and Django code under Celery.
> Originally, I had some code that used neither Channels nor Celery, but
> simply run as part of the View code:
>
> def import_all(...)
>
> ...database access...
>
>
> This is an expensive operation and to avoid timing out the browser, I
> needed to (a) run it under Celery but also (b) provide progress updates via
> a Websocket to the client. First, I implemented (b) using a Channels
> Consumer and the group_send, and that worked fine. I then tried to
> implement (a) using a Celery Task:
>
> @task
>
> def import_all(...):
> ...database access...
> async_to_sync(group_send)(...)
>
>
> However, this caused Celery to be unhappy and the worker to die:
>
> ...
> 2020-08-22 13:33:08,303 [ERROR] MainProcess: Task handler raised error:
> WorkerLostError('Worker exited prematurely: exitcode 0.')
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.8/dist-packages/billiard-3.6.3.0-py3.8.egg/billiard/pool.py",
>
> line 1265, in mark_as_worker_lost
>raise WorkerLostError(
> billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0.
>
> After quite some Googling, and various detours including where the
> DataBase access caused issues (it seems it cannot be run from an async
> context), I ended up with this double-wrapped monster:
>
> @task
>
> def import_all(...):
> async_to_sync(_import_all)(...)
>
> async def _import_all(...):
> await database_sync_to_async(__import_all)(...)
>
> def __import_all(...): <<<<<<<<<< original code
> ...database access...
> async_to_sync(group_send)(...)
>
>
> This seems to work (even though the Celery task eventually dies, it seems
> to have done its work first, and is replaced). Is this double-wrapping
> really the correct approach? Even if it is clumsy, is there any reason to
> think it won't be reliable?
>
> TIA, Shaheed
>
> P.S. I should mention that I am using Celery's prefork library, as
> opposed to say gevent or greenlets.
>
>
>
>
>
>
>
>
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/ef21c366-66b2-4760-a74f-415466695a55n%40googlegroups.com.
