Re: Request Timed Out When Generating Huge Data

[Babatunde Akinyanmi]

Hi,
How many records are we talking about? What code are you using to fetch the
records? In other words, add more information so that it's easier to
troubleshoot

On Mon, 2 Apr 2018, 02:27 tango ward,  wrote:

>
> Good day,
>
> I need suggestions on how to approach this problem.
>
> I am working on a task that needs to generate the list students per
> school. When the school has larger number of students, I am getting a
> request timed out error. I can filter the school per campus to lessen the
> number of students but the performance in displaying the list of students
> is still taking too long.
>
>
> Thanks,
> Jarvis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUzaCYm_un0zZFMXQC8ewzrg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2BWjgXN%2B2U5UtbjPGzTO5Aff-s%2BAeDKT%3DYg1HwFr72C4KGB4SQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Request Timed Out When Generating Huge Data

[tango ward]

Good day,

I need suggestions on how to approach this problem.

I am working on a task that needs to generate the list students per school.
When the school has larger number of students, I am getting a request timed
out error. I can filter the school per campus to lessen the number of
students but the performance in displaying the list of students is still
taking too long.


Thanks,
Jarvis

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUzaCYm_un0zZFMXQC8ewzrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: when i import large data to db using django orm, the django orm use too many memory

[jingniao]

thanks
在 2018年4月1日星期日 UTC+8下午9:35:43，Jani Tiainen写道：
>
> Hi,
>
> You need to set DEBUG = False to disable SQL logging.
>
> Better yet, don't use Django for large data import (it's not very suitable 
> for that) but use your database tools for that.
>
>
> On Sun, Apr 1, 2018 at 1:07 PM, jingniao  
> wrote:
>
>> django.db.backends.base.base.BaseDatabaseWrapper
>> self.queries_limit is 9000
>> which will effect self.queries_log, this is a query log cache
>> when I import a large data to db use Models.objects.bulk_create function
>> the long sql will be cache, so will be out out memory
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/b9f79199-181f-432f-af0e-6c36455ad282%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Jani Tiainen
>
> - Well planned is half done, and a half done has been sufficient before...
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/54e0f95e-1ce8-44cc-99de-a9d9b1902c27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: when i import large data to db using django orm, the django orm use too many memory

[Jani Tiainen]

Hi,

You need to set DEBUG = False to disable SQL logging.

Better yet, don't use Django for large data import (it's not very suitable
for that) but use your database tools for that.


On Sun, Apr 1, 2018 at 1:07 PM, jingniao  wrote:

> django.db.backends.base.base.BaseDatabaseWrapper
> self.queries_limit is 9000
> which will effect self.queries_log, this is a query log cache
> when I import a large data to db use Models.objects.bulk_create function
> the long sql will be cache, so will be out out memory
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/b9f79199-181f-432f-af0e-6c36455ad282%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Jani Tiainen

- Well planned is half done, and a half done has been sufficient before...

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHn91oeW4A_XSaXgcHz-uCDLXDU2_asaHxfPDfCm8rzQEyMhXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

when i import large data to db using django orm, the django orm use too many memory

[jingniao]

django.db.backends.base.base.BaseDatabaseWrapper
self.queries_limit is 9000
which will effect self.queries_log, this is a query log cache
when I import a large data to db use Models.objects.bulk_create function
the long sql will be cache, so will be out out memory

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b9f79199-181f-432f-af0e-6c36455ad282%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to run an external command that must run with the server

[Ken Whitesell]

We set up all our Django-related processes as a group under supervisor. 
This includes our celery-based processes. (We have some long-running 
tasks that are kicked off by the web site.) By setting it up as a group, 
we can manage all the different processes as a set.


Whether or not that's the "best" way is possibly debatable. But it works 
for us and doesn't give us any problems.


Ken

On 4/1/2018 7:06 AM, 'Alex' via Django users wrote:
I have a daphne server running a django channels application. I also 
have a python script that aggregates data from various sources, and 
sticks it into the channel layer (called source.py). At the moment, I 
run it as a management command (python manage.py source). It is nearly 
time for deployment(!), so I'm moving towards production solutions. 
Daphne itself currently runs under supervisor.


My question is, what is the best way to run source.py? As a management 
command, also under supervisor? Using celery? In some other way? Since 
source.py feeds into the channel layer, it needs access to settings.py 
in order to identify the details of the channel layer etc.

--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-users+unsubscr...@googlegroups.com 
.
To post to this group, send email to django-users@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b3acf9a3-41e9-08ee-1180-80fb5bc346d0%40comcast.net.
For more options, visit https://groups.google.com/d/optout.

How to run an external command that must run with the server

['Alex' via Django users]

I have a daphne server running a django channels application. I also have a 
python script that aggregates data from various sources, and sticks it into 
the channel layer (called source.py). At the moment, I run it as a 
management command (python manage.py source). It is nearly time for 
deployment(!), so I'm moving towards production solutions. Daphne itself 
currently runs under supervisor.

My question is, what is the best way to run source.py? As a management 
command, also under supervisor? Using celery? In some other way? Since 
source.py feeds into the channel layer, it needs access to settings.py in 
order to identify the details of the channel layer etc.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b3c22939-b520-4c9c-90f9-ef8cb7b8c661%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Overriding channel-layer's group_send and group_add to add persistence

['Alex' via Django users]

I have a partial solution to this, if anybody is interested. For some 
reason, creating a new version of the core.py file in which 
RedisChannelLayer is defined, and editing it directly, gave the desired 
results. It was something about overriding only part of it, but I have no 
idea what!

On Friday, 23 March 2018 09:56:13 UTC, Alex wrote:
>
> I've been trying to add persistence to channel layers, such that each new 
> consumer joining a group is sent the most recent message from that group, 
> on connect. Below are my attempts. For some reason, the message in the 
> highlighted line always seems to be of type 'None'. Am I going about this 
> completely incorrectly? I'd be really grateful for any help.
>
>
> from channels_redis.core import RedisChannelLayer
> from channels.exceptions import ChannelFull
> import time
>
>
> class RedisChannelLayerGroupPersistence(RedisChannelLayer):
>
>
>
>
> async def group_send(self, group, message):
> """
> Sends a message to the entire group.
> """
> assert self.valid_group_name(group), "Group name not valid"
> # Retrieve list of all channel names
> key = self._group_key(group)
> pers_key = str(key) + "_PERS"
> async with self.connection(self.consistent_hash(group)) as 
> connection:
> # Discard old channels based on group_expiry
> await connection.zremrangebyscore(key, min=0, max=int(time.
> time()) - self.group_expiry)
> # Return current lot
> channel_names = [
> x.decode("utf8") for x in
> await connection.zrange(key, 0, -1)
> ]
> # TODO: More efficient implementation (lua script per shard?) 
>  try:
> await connection.persist(pers_key)
> await connection.set(pers_key, str(message))
> print("TYPE = 
>  {}".format(type(str(
> message
>
>
> for channel in channel_names:
> try:
> await self.send(channel, message)
> except ChannelFull:
> pass
>
>
> async def group_add(self, group, channel):
>
>
> """
> Adds the channel name to a group.
> """
> # Check the inputs
> assert self.valid_group_name(group), "Group name not valid"
> assert self.valid_channel_name(channel), "Channel name not valid"
> # Get a connection to the right shard
> group_key = self._group_key(group)
> pers_key = str(group_key) + "_PERS"
> async with self.connection(self.consistent_hash(group)) as 
> connection:
> message = await connection.get(pers_key) #ISSUE HERE 
> -- MESSAGE IS NONE
> # Add to group sorted set with creation time as timestamp
> await connection.zadd(
> group_key,
> time.time(),
> channel,
> )
> # Set expiration to be group_expiry, since everything in
> # it at this point is guaranteed to expire before that
> try:
> await self.send(channel, str(message))
> except ChannelFull:
> pass
>
>
>
>
> await connection.expire(group_key, self.group_expiry)
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5fa1bdd4-07e1-42d8-8bc0-2662e754eb77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Decoupling Postgres database credentials in django for deployment.

[PASCUAL Eric]

How is github "security" going to help you keep your passwords safe


IMHO, it does not.


As you wrote, env vars configured on the target system (be it a bare server or 
a Dockerized environment in the cloud) seem to be safer on this point since not 
stored anywhere but on the target.


Eric


From: django-users@googlegroups.com  on behalf 
of Derek 
Sent: Saturday, March 31, 2018 3:46:53 PM
To: Django users
Subject: Re: Decoupling Postgres database credentials in django for deployment.

How is github "security" going to help you keep your passwords safe and why is 
this better than ENV variables (which most of use without any problems).?

On Friday, 30 March 2018 16:50:03 UTC+2, Bill Torcaso wrote:

I have a concern about using environment variables to hold secret information, 
and an opinion about it.

IF

DEBUG is enabled, and there is a 500 server internal error, and the default 500 
template is used to render the response,

THEN

all of your secret information is shown in the browser output

Of course, DEBUG should never be enabled in production.  But a single human 
error might make it happen.

I would prefer to trust Github security and long passwords than to think I am 
infallible about setting DEBUG.

Note that this is certainly what happens when I run on a Vagrant VM, and I 
think it would be the same in a Docker-like container.




On Thursday, March 29, 2018 at 4:24:40 PM UTC-4, prince gosavi wrote:
Hi,
I have made a django project and want to deploy it on cloud.
Before that i want to decouple all the private information.
I want to decouple the database info too, like the username password etc.
Any help is appreciated.

--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 
django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/cea076e8-4dcc-4c7e-a845-92f1d914b2c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/DB7P193MB0331EAF64F8BEC279D58BE618CA70%40DB7P193MB0331.EURP193.PROD.OUTLOOK.COM.
For more options, visit https://groups.google.com/d/optout.

Re: New Add-On: Django Model Lifecycle Hooks

[Jani Tiainen]

Hi,

Quite interesting feature have to say. You should use tox to run tests
against different versions of Django/Python to make it easier to make it
compatible easier.

I would like to see compatibilities with 1.11 and 2.0 since 1.11 is LTS and
2.0 is latest supported.

But it's a great start.


On Sun, Apr 1, 2018 at 12:54 AM, Robert Singer 
wrote:

>
> Thanks! Yup, should be compatible with Django 1.11.
>
> It is *only compatible* with Python 3.4 and up, though. There is no
> binding technical reason for this - it's just what my organization uses and
> I've been in habit of using type annotations.
>
> So if there is demand for Python 2.7 compatibility, I can certainly add
> it.
>
> On Saturday, March 31, 2018 at 3:33:28 PM UTC-5, tizonzon wrote:
>>
>> Very interesting! Good job. Is this module supported in 1.11?
>>
>>
>>
>> Sent from my iPhone
>>
>> On Mar 31, 2018, at 3:18 PM, George Silva  wrote:
>>
>> From Reading the readme this looks promising.
>>
>> Congrats!
>>
>> I might use this soon.
>>
>> Em sáb, 31 de mar de 2018 15:50, Robert Singer 
>> escreveu:
>>
>>> Hey there -
>>>
>>> I just wanted to let community know about a small add-on I wrote in
>>> order to add Rails-style hooks/callbacks to Django models.
>>> https://github.com/rsinger86/django-lifecycle
>>>
>>> Feedback welcome and appreciated :)
>>>
>>> Cheers.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users...@googlegroups.com.
>>> To post to this group, send email to django...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-users/f1fbb963-2d36-46b3-ba76-7781c074918a%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users...@googlegroups.com.
>> To post to this group, send email to django...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-users/CAGyPVTuRCtt3SHwTR7%3DehKZzwcV-kMOCbOkdiJyMH%3DFgu4uZnA%
>> 40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/d5bbe153-7f2f-4196-951e-0ffdf1200377%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Jani Tiainen

- Well planned is half done, and a half done has been sufficient before...

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHn91ocGSrks6Kfc5LC5RMje9mC14DKEWb46otddgvj3YMQs%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.