Hi Dmitry,
Please check that documentation page
<https://docs.djangoproject.com/en/1.10/ref/models/querysets/#iterator>.
As you can see iterator() loads all in the memory and after that returns an
iterator.
Try to fetch records say by 50 - 100 items.
Hope that helps.
On Mon, Oct 10, 2016 at 2:14 PM, Горобец Дмитрий <dmitr...@gmail.com> wrote:
> Hello.
>
> Thanks for advices.
>
> Crash occures, when I run Django command through python manage.py
> my_command_name.
>
> It doesn't relate to nginx and even gunicorn, because, again, it's Django
> command, which I call from shell.
>
> I switched off redis caching (I use cacheops), but it' didn't help too.
>
> пятница, 7 октября 2016 г., 13:21:48 UTC+5 пользователь Горобец Дмитрий
> написал:
>
>> Hello.
>>
>> I have VPS with 2Gb RAM with django, gunicorn, mysql, nginx and redis.
>>
>> My app crashes with out of memory error, when I run django command on
>> model, which has approximately 7 million records. It takes each premise
>> object and updates one field by checking value with regular expression.
>> Please, help optimize that command.
>>
>> class Command(BaseCommand):
>> help = 'Updates premise.number_order'
>>
>> def handle(self, *args, **options):
>> for premise in Premise.objects.iterator():
>> premise.number_order = premise.set_number_order()
>> premise.save()
>>
>> self.stdout.write('Finished')
>>
>>
>> # Method of Premise model
>> def set_number_order(self):
>> tr = {
>> 'А': '.10',
>> 'A': '.10',
>> 'Б': '.20',
>> 'В': '.30',
>> 'Г': '.40',
>> 'Д': '.50',
>> 'Е': '.60',
>> 'Ж': '.70',
>> 'З': '.80',
>> 'И': '.90',
>> }
>>
>> only_digit = re.compile(r'^(?P<number>[0-9]{1,9})$')
>> digit_with_separator = re.compile(r'^(?P<number>[0-9]
>> {1,9})(?P<separator>[-|/])(?P<rest>\w+)$')
>> digit_with_letter = re.compile(r'^(?P<number>[0-9]
>> {1,9})(?P<letter>[А-Яа-я]+)')
>> result = 0
>> title = self.title.strip().upper()
>>
>> if only_digit.match(title):
>> number = only_digit.match(title).group('number')
>> result = number + '.00'
>>
>> elif digit_with_separator.match(title):
>> number = digit_with_separator.match(title).group('number')
>> rest = digit_with_separator.match(title).group('rest')
>> if rest[0].isalpha():
>> floating = tr.get(rest[0], '.90')
>> result = number + floating
>>
>> elif rest[0].isdigit():
>> try:
>> if rest[1].isdigit():
>> result = number + '.{}'.format(rest[:2])
>> else:
>> result = number + '.0{}'.format(rest[0])
>> except IndexError:
>> result = number + '.0{}'.format(rest[0])
>>
>> elif digit_with_letter.match(title):
>> number = digit_with_letter.match(title).group('number')
>> letter = digit_with_letter.match(title).group('letter')[0]
>>
>> floating = tr.get(letter, '.90')
>> result = number + floating
>>
>> return Decimal(result)
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/2e87323c-294c-4b82-89dd-783e36e23291%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/2e87323c-294c-4b82-89dd-783e36e23291%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAK52boUAy-8NfaoG8Hmvge2dNWps2baBc4tfHa0mOrV9MNEw8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
