Re: How to handle a migration that last too long for being deployed?

[Mike Dewhirst]

Adrien

May I start a new thread to discuss with you the costs and benefits for 
splitting your model. I have some big models (47 up to 76 columns) which 
I have long thought are just too big. The splits would all have to be 
1:1 with the core model.


Mike

On 13/04/2018 7:05 PM, Adrien Cossa wrote:


Hi everybody!

I would like to know what options exist when you have a huge migration 
that will obviously not run on your productive server.


I have spitted a model in two smaller ones and wrote then a migration 
to populate these new models. The number of original objects is around 
250,000 and I have also a few references to update. In the end, the 
migration lasted more than 30 mn on my machine (16 GB RAM and it was 
swapping a lot) and it failed on another machine because the RAM was 
out (the process was using then about 13 GB). On the productive server 
we have even less RAM so to run the migration as it is is really out 
of question.


I have tried to use all the Django mechanisms that I know to optimize 
the queries: select_related, prefetch_related, bulk_create, 
QuerySet.update... Now, the migration I am talking about use 
bulk_create(batch_size=None) and process the whole queryset at once. 
Before that, as the migration was not so long lasting because I had 2 
references less to update,  I tried other values for batch_size and 
also I processed the queryset as pages of a few hundreds or thousands 
objects. The results were not better then batch_size=None and "all at 
once", that's why I finally used "basic settings" (and the migration 
was lasting about 5 minutes). I will have to reintroduce some tweaks 
because the extra updates of the two relations I mentioned is making 
here a big difference.


I am wondering if someone already found him/herself in a similar 
situation, and with what solution you finally came to.


If the migration lasts very long, it's not a problem by itself but I 
don't want to lock the database for 15 mn. The fact is that I don't 
know what is happening during the migration process, what is locked by 
what? I will split the migration in "pages" to use less RAM anyway, 
but I was also thinking of migrating in two different steps *or* 
files, in order to process separately the objects that are not 
editable (basically most of them, that we keep for history but they 
are read-only) and the others (which should be much faster and thus 
people working will not be blocked for long). Does it make sense? Some 
other ideas?


Thanks a lot!

Adrien
--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-users+unsubscr...@googlegroups.com 
.
To post to this group, send email to django-users@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ca0c145a-8147-9bbd-e01e-e74355a16a2a%40init.at 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5236eda5-6a49-e3fe-5af0-f29eb327af97%40dewhirst.com.au.
For more options, visit https://groups.google.com/d/optout.

Django docs Polls App, Diferent queryset returns same object values but QuestionIndexViewTests works for one query and fails for other

[Himanshu Gamit]

In polls application I added annotation to check one to many relationship 
between Question and Choice models such that, Index only return questions 
with more than one choices to be displayed but my testcase fails all the 
time with annotation where both query returns same data.
views.py (Not Working: Added `annotate(num = 
Count('choice')).filter(num__gt=1))`
{{{
class IndexView(generic.ListView):
template_name = 'polls/index.html'
context_object_name = 'latest_question_list'
def get_queryset(self):
"""Return the last five published questions."""
return Question.objects.annotate(num = 
Count('choice')).filter(num__gt=1).filter(pub___lte=timezone.now()).order_by('-pub_date')[:5]
}}}
Error:
{{{
Traceback (most recent call last): File 
"C:\Users\hmnsh\repos\DjangoApp\mysite\polls\tests.py", line 80, in 
test_past_question ['']
File 
"C:\ProgramData\Anaconda3\envs\django\lib\site-packages\django\test\testcases.py",
 
line 940, in assertQuerysetEqual return self.assertEqual(list(items), 
values, msg=msg) AssertionError: Lists differ: [] != ['']
}}}

Please help.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/07017f35-aee4-486c-a84c-d9cad46ec36d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle a migration that last too long for being deployed?

[Mike Dewhirst]

Does it actually stop users reading? If the entire migration happens in a 
single transaction, the database (Postgres anyway) should remain accessible 
until the moment it is committed. 

Maybe you could announce a maintenance operation which will only interrupt 
certain actions for a few minutes?

Mike

Connected by Motorola

Adrien Cossa  wrote:

>Hi everybody!
>
>I would like to know what options exist when you have a huge migration that 
>will obviously not run on your productive server.
>
>I have spitted a model in two smaller ones and wrote then a migration to 
>populate these new models. The number of original objects is around 250,000 
>and I have also a few references to update. In the end, the migration lasted 
>more than 30 mn on my machine (16 GB RAM and it was swapping a lot) and it 
>failed on another machine because the RAM was out (the process was using then 
>about 13 GB). On the productive server we have even less RAM so to run the 
>migration as it is is really out of question.
>
>I have tried to use all the Django mechanisms that I know to optimize the 
>queries: select_related, prefetch_related, bulk_create, QuerySet.update... 
>Now, the migration I am talking about use bulk_create(batch_size=None) and 
>process the whole queryset at once. Before that, as the migration was not so 
>long lasting because I had 2 references less to update,  I tried other values 
>for batch_size and also I processed the queryset as pages of a few hundreds or 
>thousands objects. The results were not better then batch_size=None and "all 
>at once", that's why I finally used "basic settings" (and the migration was 
>lasting about 5 minutes). I will have to reintroduce some tweaks because the 
>extra updates of the two relations I mentioned is making here a big difference.
>
>I am wondering if someone already found him/herself in a similar situation, 
>and with what solution you finally came to.
>
>If the migration lasts very long, it's not a problem by itself but I don't 
>want to lock the database for 15 mn. The fact is that I don't know what is 
>happening during the migration process, what is locked by what? I will split 
>the migration in "pages" to use less RAM anyway, but I was also thinking of 
>migrating in two different steps *or* files, in order to process separately 
>the objects that are not editable (basically most of them, that we keep for 
>history but they are read-only) and the others (which should be much faster 
>and thus people working will not be blocked for long). Does it make sense? 
>Some other ideas?
>
>Thanks a lot!
>
>Adrien
>
>-- 
>You received this message because you are subscribed to the Google Groups 
>"Django users" group.
>To unsubscribe from this group and stop receiving emails from it, send an 
>email to django-users+unsubscr...@googlegroups.com.
>To post to this group, send email to django-users@googlegroups.com.
>Visit this group at https://groups.google.com/group/django-users.
>To view this discussion on the web visit 
>https://groups.google.com/d/msgid/django-users/ca0c145a-8147-9bbd-e01e-e74355a16a2a%40init.at.
>For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/f8e3qpjpuc9hm8k6oyp6fvi8.1523666325014%40email.android.com.
For more options, visit https://groups.google.com/d/optout.

Create custom widget in django-jet

[j . romeroc97]

Hey everybody, I'm here for some help, I'm using django-jet right now and 
trying to create a new widget with a view that I created before, somebody 
has knowledge about how to do this?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8c4799b8-a32d-40a9-b529-4fed246ea83e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to handle a migration that last too long for being deployed?

[Adrien Cossa]

Hi everybody!

I would like to know what options exist when you have a huge migration 
that will obviously not run on your productive server.


I have spitted a model in two smaller ones and wrote then a migration to 
populate these new models. The number of original objects is around 
250,000 and I have also a few references to update. In the end, the 
migration lasted more than 30 mn on my machine (16 GB RAM and it was 
swapping a lot) and it failed on another machine because the RAM was out 
(the process was using then about 13 GB). On the productive server we 
have even less RAM so to run the migration as it is is really out of 
question.


I have tried to use all the Django mechanisms that I know to optimize 
the queries: select_related, prefetch_related, bulk_create, 
QuerySet.update... Now, the migration I am talking about use 
bulk_create(batch_size=None) and process the whole queryset at once. 
Before that, as the migration was not so long lasting because I had 2 
references less to update,  I tried other values for batch_size and also 
I processed the queryset as pages of a few hundreds or thousands 
objects. The results were not better then batch_size=None and "all at 
once", that's why I finally used "basic settings" (and the migration was 
lasting about 5 minutes). I will have to reintroduce some tweaks because 
the extra updates of the two relations I mentioned is making here a big 
difference.


I am wondering if someone already found him/herself in a similar 
situation, and with what solution you finally came to.


If the migration lasts very long, it's not a problem by itself but I 
don't want to lock the database for 15 mn. The fact is that I don't know 
what is happening during the migration process, what is locked by what? 
I will split the migration in "pages" to use less RAM anyway, but I was 
also thinking of migrating in two different steps *or* files, in order 
to process separately the objects that are not editable (basically most 
of them, that we keep for history but they are read-only) and the others 
(which should be much faster and thus people working will not be blocked 
for long). Does it make sense? Some other ideas?


Thanks a lot!

Adrien

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ca0c145a-8147-9bbd-e01e-e74355a16a2a%40init.at.
For more options, visit https://groups.google.com/d/optout.

ImportError: No module named 'main'

[Артём Орлов]

Hello!


Django does not see application created with *django-admin startapp main*. 

./DjangoRestfulServer/
├── __init__.py
├── main
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── models.py
│   ├── __pycache__
│   │   ├── admin.cpython-35.pyc
│   │   ├── __init__.cpython-35.pyc
│   │   ├── models.cpython-35.pyc
│   │   └── views.cpython-35.pyc
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── __pycache__
│   ├── __init__.cpython-35.pyc
│   ├── models.cpython-35.pyc
│   ├── settings.cpython-35.pyc
│   ├── urls.cpython-35.pyc
│   └── wsgi.cpython-35.pyc
├── settings.py
├── urls.py
└── wsgi.py



*settings.py*

...
INSTALLED_APPS = [
...
'DjangoRestfulServer',
'DjangoRestfulServer.main',
]
...

*main/urls.py*

from django.conf.urls import url, includefrom django.urls import include, path, 
re_pathfrom . import views

urlpatterns = [
re_path(r'^facebook/$', views.FacebookLogin.as_view(), name='fb_login'),
]

*urls.py*

from django.contrib import adminfrom django.urls import include, path, re_path

urlpatterns = [
path('api/v1/admin/', admin.site.urls),
path('api/v1/accounts/', include('allauth.urls')),
path('api/v1/', include('main.urls')),  # <- path('api/v1/', 
include('DjangoRestfulServer.main.urls')) acts the same!!!
]

Traceback is the following:

(env3) artem@artem-inspirion:~/Projects/GoTogether/Server$ python3 manage.py 
checkTraceback (most recent call last):
  File "manage.py", line 15, in 
execute_from_command_line(sys.argv)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/__init__.py",
 line 371, in execute_from_command_line
utility.execute()
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/__init__.py",
 line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/base.py",
 line 288, in run_from_argv
self.execute(*args, **cmd_options)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/base.py",
 line 335, in execute
output = self.handle(*args, **options)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/commands/check.py",
 line 65, in handle
fail_level=getattr(checks, options['fail_level']),
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/base.py",
 line 364, in check
include_deployment_checks=include_deployment_checks,
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/management/base.py",
 line 351, in _run_checks
return checks.run_checks(**kwargs)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/checks/registry.py",
 line 73, in run_checks
new_errors = check(app_configs=app_configs)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/checks/urls.py",
 line 40, in check_url_namespaces_unique
all_namespaces = _load_all_namespaces(resolver)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/core/checks/urls.py",
 line 57, in _load_all_namespaces
url_patterns = getattr(resolver, 'url_patterns', [])
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/utils/functional.py",
 line 36, in __get__
res = instance.__dict__[self.name] = self.func(instance)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/urls/resolvers.py",
 line 536, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/utils/functional.py",
 line 36, in __get__
res = instance.__dict__[self.name] = self.func(instance)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/urls/resolvers.py",
 line 529, in urlconf_module
return import_module(self.urlconf_name)
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/importlib/__init__.py",
 line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 986, in _gcd_import
  File "", line 969, in _find_and_load
  File "", line 958, in _find_and_load_unlocked
  File "", line 673, in _load_unlocked
  File "", line 673, in exec_module
  File "", line 222, in _call_with_frames_removed
  File "/home/artem/Projects/GoTogether/Server/DjangoRestfulServer/urls.py", 
line 22, in 
path('api/v1/', include('main.urls')),
  File 
"/home/artem/Projects/GoTogether/Server/env3/lib/python3.5/site-packages/django/urls/conf.py",
 line 34, in include
urlconf_module = import_module(urlconf_module)
  File 

Re: I'm trying to edit address form for Saleor ecommerce platform

[ruslan.aldar]

Hello thank you for your reply.
the problem is when I'm removing fields like company_name, city, city_area, 
postal code from account/models.py , i18n.py. 

Mostly this error appears when I delete one line in I18N_MAPPING.

on the web:
KeyError at /checkout/shipping-address/

'company_name'

Request Method: GET
Request URL: http://localhost:8000/checkout/shipping-address/
Django Version: 2.0.3
Exception Type: KeyError
Exception Value: 

'company_name'

Exception Location: 
C:\Users\rusla\Documents\GitHub\zahial\saleor\saleor\account\i18n.py 
in _convert_to_bound_fields, line 199

Error during template rendering

In template 
C:\Users\rusla\Documents\GitHub\zahial\saleor\templates\account\snippets\address_form.html,
 
error at line *3*
company_name
1 {% load bootstrap_field from bootstrap4 %} 
2  
3 {% with address_form_lines=address_form.i18n_fields_order %}
I deleted commented out every company name in my source code. But I still 
having same issue.
I guess i18naddress requires those fields by default. 


On Sunday, April 8, 2018 at 8:38:14 PM UTC+8, Avraham Serour wrote:
>
> You seem to have pasted a line from your template, not the error you are 
> getting, no way of helping you there.
>
> Please post the actual error you are getting
>
> On Sat, Apr 7, 2018 at 2:00 PM, ruslan.aldar  > wrote:
>
>> hello. 
>> I'm trying to build an local merchant website. So I don't know how to 
>> edit address form for Saleor. It uses google i18n address form. 
>> and when i'm trying to delete some fields like company_name, city 
>> city_area, postal_code, country, country_area, i'm having problem.
>>
>> on the page : http://localhost:8000/checkout/shipping-address/
>>
>> i'm getting 
>> {% with address_form_lines=address_form.i18n_fields_order %} 
>> this kind of error. 
>>
>> Making migration, or deleting db, doesn't solve the problem and I don't 
>> know why it's still requesting the field which is doesn't exist!  I even 
>> tried deleting DB, migrating again!
>>
>  

>
>> If someone knows it please tell me. 
>> Thank you.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/5f44cebc-915e-4c08-b394-13efc4ca5c0c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c8a39ae6-d0fe-4714-b87b-733e89ade8f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ARGPARSE ERROR

[Michal Petrucha]

On Thu, Apr 12, 2018 at 07:58:08PM +, Ankush Sharma wrote:
> Hi Babatunde ,
> I installed other listed tools ! Here the main concern im talking about is
> Argsparse -
> the Progressbar2 3.6.2 requires argparse, which is not installed.
> Thanks in advance
> Ank

What version of Python are you running this on? Either it's a very old
one (2.6, 3.1), which would be unsupported by pretty much any
reasonably recent Python package, or your Python environment is messed
up somehow, and it lacks a part of the standard library. If the former
is the case, you need to migrate to a newer Python; if the latter,
then you need to fix your Python installation, and make sure that it
includes the entire standard library.

Good luck,

Michal

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20180413090646.GN1181%40konk.org.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Digital signature

Re: retain form data when a user session expires while filling a form

[Avraham Serour]

You may also always save to cookie before submitting the form and clear it
on the thank you page

On Thu, 12 Apr 2018, 14:20 Vijay Shanker,  wrote:

> I have to code this scenario:
>
> Some user comes to fill a form and while user is at it, session expires;
> User tries to submit form, as session has expired it will take him to login
> page after which he is rediredted to form page with a prefilled form with
> data he filled previously.
>
> my propsed solution: on form submit, check if user session is expired with
> an ajax call, if yes, drop a cookie with values of filled form, user comes
> back to same form, prefill the form from cookie value.
>
> how far I got: not much; I used ajax call to check session expiry with a
> backend call to this function:
>
> def check_valid_session(request):
> session_expiry_date = request.session.get_expiry_date()
> now = datetime.now()
> seconds_left = (session_expiry_date - now).total_seconds()
> if seconds_left <= 0:
> return JsonResponse({'session_expired': True, 'seconds_left': 
> seconds_left })
> else:
> return JsonResponse({'session_expired':False, 'seconds_left': 
> seconds_left})
>
> in my settings I have:
>
> SESSION_COOKIE_NAME = "sso-sessionid"
> SESSION_COOKIE_HTTPONLY = False
> SESSION_COOKIE_AGE = 10
> SESSION_COOKIE_DOMAIN = '.mydomain.com'
> SESSION_COOKIE_SECURE = True
> SESSION_EXPIRE_AT_BROWSER_CLOSE = True
>
> but in response, I always get seconds_left as something close to 9.999.
> and session_expiry_date keeps increasing itself by 10 seconds each time i
> hit submit and call the backend code above. How to get correct session
> expire time?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/e623b707-2014-4511-91f4-7a00420fe196%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFWa6tLyCywv3sZn%3D2ch9fdrxBLv-PCRdqPS3CK5ioiimypobg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.