Re: Running pending migrations in Middleware
On Sunday, July 28, 2019 at 2:35:03 PM UTC-7, David Grant wrote: > > > I'm using Google AppEngine and I have no access to the server. The > alternative is to have the exact same code checked out on my local machine. > Connect to the remote database with a special proxy tool that google > provides (so the db becomes available on a local port) and then run the > migrations. Problems with this are: 1) Hitting the remote database via the > proxy is extremely slow, like ridiculously slow and 2) there is a chance I > have the wrong local code. ie. this could mean I run some extra migrations > or not run enough migrations, thus causing production to be broken. > > I guess I can do this as part of the deploy, which makes the version is guarenteed to be the correct one. But I still need to use the google cloudsql proxy which sucks. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9ab39c2b-503c-44ff-89f3-bb5203ab799c%40googlegroups.com.
Re: Running pending migrations in Middleware
On Sunday, July 28, 2019 at 10:10:57 AM UTC-7, James Schneider wrote: > > > > On Sun, Jul 28, 2019, 12:47 AM David Grant > wrote: > >> Anyone see any problems with running migrations in Middleware? >> > > What's the actual problem you are trying to solve by doing this? This is a > bad idea for a number of reasons, and I can't think of any good ones. > I'm using Google AppEngine and I have no access to the server. The alternative is to have the exact same code checked out on my local machine. Connect to the remote database with a special proxy tool that google provides (so the db becomes available on a local port) and then run the migrations. Problems with this are: 1) Hitting the remote database via the proxy is extremely slow, like ridiculously slow and 2) there is a chance I have the wrong local code. ie. this could mean I run some extra migrations or not run enough migrations, thus causing production to be broken. Ideally I would just run `./manage.py migrate` on the server, but that's just not possibe. Also a colleague of mine has been doing this for a Java app for years. There's a quick check in middleware for current db version and then it runs migrations if necessary. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d05695c2-b537-4ad5-bbaf-6eb822a8f9e5%40googlegroups.com.
Re: Running pending migrations in Middleware
The reason I want to do this by the way, is because I'm using Google
AppEngine and I have no access to the server. The alternative is to have
the exact same code checked out on my local machine. Connect to the remote
database with a special proxy tool that google provides (so the db becomes
available on a local port) and then run the migrations. Problems with this
are: 1) Hitting the remote database via the proxy is extremely slow, like
ridiculously slow and 2) there is a chance I have the wrong local code. ie.
this could mean I run some extra migrations or not run enough migrations,
thus causing production to be broken.
Thanks for bringing up those points.
All middleware are run for every request
Yep, this is an issue if "migrate" is called on every request. I solved
this by recording whether or not migrations have been run or not on the
current version of the app, and also whether it was successful or not. So
the only hit I take on every request one query, which will be cached in the
db and could also be cached in memory if I really wanted to. Currently,
with just the one query it's like 10ms when I do it locally. Not the
bottleneck.
What you're doing can slow down your request response time to minutes,
> depending on how many migrations you have.
What's the alternative? Put the app in a maintenance mode? Either way the
app will be unusable for a bit. I guess there is an issue where the
requests could exceed the maximum request time (in my case in AppEngine it
is 60 seconds) and so if the migration takes longer that, that would be a
problem. I could kick off migration asynchonously?
Depending on your database, migrations might not be atomic thus causing
> conflicts
I'm using Postgresql so my understanding is my migrations can all run
atomically.
On Sunday, July 28, 2019 at 3:05:47 AM UTC-7, Markus Holtermann wrote:
>
> Yes, I do see several problems. Some of them:
>
> - All middleware are run for every request
> - What you're doing can slow down your request response time to minutes,
> depending on home many migrations you have
> - Depending on your database, migrations might not be atomic thus causing
> conflicts
> - You have no way to intervene when something goes south
> - ...
>
> I can elaborate in more detail when you want. But please, don't run
> migrations as part of the request-response cycle.
>
> /Markus
>
> On Sun, Jul 28, 2019, at 3:48 PM, David Grant wrote:
> > Anyone see any problems with running migrations in Middleware?
> >
> > import logging
> > import os
> > import time
> >
> > from django.core.management import call_command
> > from django.http import HttpResponse
> >
> > from products.models.models import AppVersion
> >
> > LOG = logging.getLogger(__name__)
> >
> >
> > # Used for testing locally when the GAE_VERSION environment is not set
> > FAKE_VERSION_TESTING = 8
> >
> >
> > class MigrateMiddleware:
> >
> > def __init__(self, get_response):
> > self.get_response = get_response
> >
> > def __call__(self, request):
> > start_time = time.time()
> > try:
> > LOG.info("Running migrate middleware")
> > # Figure out what version of our app is now deployed
> > app_version, created =
> > AppVersion.objects.get_or_create(version=os.getenv('GAE_VERSION',
> > FAKE_VERSION_TESTING))
> > if not app_version.migrations_succeeded or created:
> ># Previous attempt at this deployed version failed, or this was
> > a new deployment.
> > try:
> > call_command('migrate')
> > except:
> > return HttpResponse("Running a Django migration
> > failed! Investigate.", status=500)
> > app_version.migrations_succeeded = True
> > app_version.save()
> > finally:
> > LOG.info("Took: %ss", time.time() - start_time)
> > return self.get_response(request)
> >
> >
> > It seems to work just fine in a quick test.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to django...@googlegroups.com .
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/django-users/bdfd6393-6d68-4285-b5a6-eb1be98ef467%40googlegroups.com
>
> <
> https://groups.google.com/d/msgid/django-users/bdfd6393-6d68-4285-b5a6-eb1be98ef467%40googlegroups.com?utm_medium=email_source=footer>.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
Re: Running pending migrations in Middleware
On Sun, Jul 28, 2019, 12:47 AM David Grant wrote: > Anyone see any problems with running migrations in Middleware? > What's the actual problem you are trying to solve by doing this? This is a bad idea for a number of reasons, and I can't think of any good ones. -James > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2Be%2BciU%3DL5K-sBT6PkeUk4MpQYxxrSVLDLQ%3DZE2MmTF76BY7Uw%40mail.gmail.com.
Re: Running pending migrations in Middleware
Yes, I do see several problems. Some of them:
- All middleware are run for every request
- What you're doing can slow down your request response time to minutes,
depending on home many migrations you have
- Depending on your database, migrations might not be atomic thus causing
conflicts
- You have no way to intervene when something goes south
- ...
I can elaborate in more detail when you want. But please, don't run migrations
as part of the request-response cycle.
/Markus
On Sun, Jul 28, 2019, at 3:48 PM, David Grant wrote:
> Anyone see any problems with running migrations in Middleware?
>
> import logging
> import os
> import time
>
> from django.core.management import call_command
> from django.http import HttpResponse
>
> from products.models.models import AppVersion
>
> LOG = logging.getLogger(__name__)
>
>
> # Used for testing locally when the GAE_VERSION environment is not set
> FAKE_VERSION_TESTING = 8
>
>
> class MigrateMiddleware:
>
> def __init__(self, get_response):
> self.get_response = get_response
>
> def __call__(self, request):
> start_time = time.time()
> try:
> LOG.info("Running migrate middleware")
> # Figure out what version of our app is now deployed
> app_version, created =
> AppVersion.objects.get_or_create(version=os.getenv('GAE_VERSION',
> FAKE_VERSION_TESTING))
> if not app_version.migrations_succeeded or created:
># Previous attempt at this deployed version failed, or this was
> a new deployment.
> try:
> call_command('migrate')
> except:
> return HttpResponse("Running a Django migration
> failed! Investigate.", status=500)
> app_version.migrations_succeeded = True
> app_version.save()
> finally:
> LOG.info("Took: %ss", time.time() - start_time)
> return self.get_response(request)
>
>
> It seems to work just fine in a quick test.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/bdfd6393-6d68-4285-b5a6-eb1be98ef467%40googlegroups.com
>
> .
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/3d2c95c8-3fe6-4890-8c06-2206ad72c7b0%40www.fastmail.com.
Running pending migrations in Middleware
Anyone see any problems with running migrations in Middleware?
import logging
import os
import time
from django.core.management import call_command
from django.http import HttpResponse
from products.models.models import AppVersion
LOG = logging.getLogger(__name__)
# Used for testing locally when the GAE_VERSION environment is not set
FAKE_VERSION_TESTING = 8
class MigrateMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
start_time = time.time()
try:
LOG.info("Running migrate middleware")
# Figure out what version of our app is now deployed
app_version, created =
AppVersion.objects.get_or_create(version=os.getenv('GAE_VERSION',
FAKE_VERSION_TESTING))
if not app_version.migrations_succeeded or created:
# Previous attempt at this deployed version failed, or this was
a new deployment.
try:
call_command('migrate')
except:
return HttpResponse("Running a Django migration failed!
Investigate.", status=500)
app_version.migrations_succeeded = True
app_version.save()
finally:
LOG.info("Took: %ss", time.time() - start_time)
return self.get_response(request)
It seems to work just fine in a quick test.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/bdfd6393-6d68-4285-b5a6-eb1be98ef467%40googlegroups.com.
