Re: How to fetch 10000 above record within second in django?

[Clive Bruton]

On 17 Jan 2023, at 05:01, Prashanth Patelc wrote:

I written query it's taking more time like 38 seconds, how to to  
fetch the lac of records within seconds any example in django using  
orm or MySQL query ?


Are you saying that you have a raw SQL query, or you are using a  
Django query?


My experience is that the biggest lag in database retrieval is the  
amount of data retrieved, so if you can reduce the number of fields  
returned, that would be a good start.


If you are using a Django query, and it has related elements, then  
the biggest boost you can give is using "select_related" and  
"prefetch_related".



-- Clive

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9EF46118-6870-46AD-BF58-12AEC8C6A518%40indx.co.uk.

Re: How to fetch 10000 above record within second in django?

[Prashanth Patelc]

I'm using orm query fetching all data and showing into html templates ,
it's taking more time for get the data ,

On Tue, Jan 17, 2023, 12:40 PM Clive Bruton  wrote:

>
> On 17 Jan 2023, at 05:01, Prashanth Patelc wrote:
>
> > I written query it's taking more time like 38 seconds, how to to
> > fetch the lac of records within seconds any example in django using
> > orm or MySQL query ?
>
> Are you saying that you have a raw SQL query, or you are using a
> Django query?
>
> My experience is that the biggest lag in database retrieval is the
> amount of data retrieved, so if you can reduce the number of fields
> returned, that would be a good start.
>
> If you are using a Django query, and it has related elements, then
> the biggest boost you can give is using "select_related" and
> "prefetch_related".
>
>
> -- Clive
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/9EF46118-6870-46AD-BF58-12AEC8C6A518%40indx.co.uk
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAMCU6CrAKQL6qA6hWHPDFbfpCT1R9v5%2BY_Qi-HOkPf%3DOXYt2yA%40mail.gmail.com.

Re: How to fetch 10000 above record within second in django?

[Clive Bruton]

The easiest way to debug this is to install django_debug_toolbar and  
look at the sql queries to see where the hold-up is.


As previously, restrict the number of fields returned using "values"  
and/or use "select_related" or "prefetch_related" to get related data  
in one or two queries, rather than having Django getting that data on  
(potentially) multiple queries per record.



-- Clive

On 17 Jan 2023, at 07:43, Prashanth Patelc wrote:

I'm using orm query fetching all data and showing into html  
templates , it's taking more time for get the data ,


On Tue, Jan 17, 2023, 12:40 PM Clive Bruton  wrote:

On 17 Jan 2023, at 05:01, Prashanth Patelc wrote:

> I written query it's taking more time like 38 seconds, how to to
> fetch the lac of records within seconds any example in django using
> orm or MySQL query ?

Are you saying that you have a raw SQL query, or you are using a
Django query?

My experience is that the biggest lag in database retrieval is the
amount of data retrieved, so if you can reduce the number of fields
returned, that would be a good start.

If you are using a Django query, and it has related elements, then
the biggest boost you can give is using "select_related" and
"prefetch_related".


-- Clive

--
You received this message because you are subscribed to the Google  
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it,  
send an email to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/ 
d/msgid/django-users/9EF46118-6870-46AD-BF58-12AEC8C6A518% 
40indx.co.uk.


--
You received this message because you are subscribed to the Google  
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it,  
send an email to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/ 
d/msgid/django-users/CAMCU6CrAKQL6qA6hWHPDFbfpCT1R9v5%2BY_Qi-HOkPf% 
3DOXYt2yA%40mail.gmail.com.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/52330EF7-BBD9-44E6-94E3-C0A733F4EF36%40indx.co.uk.

Re: How to fetch 10000 above record within second in django?

[Ayanfeoluwa Koyinsola]

You might now have to display all the data at once, see if you can use
paginisation to reduce the numbers of queries

On Tue, Jan 17, 2023, 12:39 Clive Bruton  wrote:

> The easiest way to debug this is to install django_debug_toolbar and
> look at the sql queries to see where the hold-up is.
>
> As previously, restrict the number of fields returned using "values"
> and/or use "select_related" or "prefetch_related" to get related data
> in one or two queries, rather than having Django getting that data on
> (potentially) multiple queries per record.
>
>
> -- Clive
>
> On 17 Jan 2023, at 07:43, Prashanth Patelc wrote:
>
> > I'm using orm query fetching all data and showing into html
> > templates , it's taking more time for get the data ,
> >
> > On Tue, Jan 17, 2023, 12:40 PM Clive Bruton  wrote:
> >
> > On 17 Jan 2023, at 05:01, Prashanth Patelc wrote:
> >
> > > I written query it's taking more time like 38 seconds, how to to
> > > fetch the lac of records within seconds any example in django using
> > > orm or MySQL query ?
> >
> > Are you saying that you have a raw SQL query, or you are using a
> > Django query?
> >
> > My experience is that the biggest lag in database retrieval is the
> > amount of data retrieved, so if you can reduce the number of fields
> > returned, that would be a good start.
> >
> > If you are using a Django query, and it has related elements, then
> > the biggest boost you can give is using "select_related" and
> > "prefetch_related".
> >
> >
> > -- Clive
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to django-users+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/django-users/9EF46118-6870-46AD-BF58-12AEC8C6A518%
> > 40indx.co.uk.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to django-users+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/django-users/CAMCU6CrAKQL6qA6hWHPDFbfpCT1R9v5%2BY_Qi-HOkPf%
> > 3DOXYt2yA%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/52330EF7-BBD9-44E6-94E3-C0A733F4EF36%40indx.co.uk
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAK-XCvfn84Xy%3D8kRpt%2BfJ_DmwUMxkeZ9iTnaGuUV5M3SNN-Twg%40mail.gmail.com.

Re: How to fetch 10000 above record within second in django?

[Prashanth Patelc]

I'm already using pagination but when I hiting particular url it is taking
more than 1minute for loading the data and displaying in html .

On Tue, Jan 17, 2023, 5:37 PM Ayanfeoluwa Koyinsola <
ayanfeoluwakoyins...@gmail.com> wrote:

> You might now have to display all the data at once, see if you can use
> paginisation to reduce the numbers of queries
>
> On Tue, Jan 17, 2023, 12:39 Clive Bruton  wrote:
>
>> The easiest way to debug this is to install django_debug_toolbar and
>> look at the sql queries to see where the hold-up is.
>>
>> As previously, restrict the number of fields returned using "values"
>> and/or use "select_related" or "prefetch_related" to get related data
>> in one or two queries, rather than having Django getting that data on
>> (potentially) multiple queries per record.
>>
>>
>> -- Clive
>>
>> On 17 Jan 2023, at 07:43, Prashanth Patelc wrote:
>>
>> > I'm using orm query fetching all data and showing into html
>> > templates , it's taking more time for get the data ,
>> >
>> > On Tue, Jan 17, 2023, 12:40 PM Clive Bruton  wrote:
>> >
>> > On 17 Jan 2023, at 05:01, Prashanth Patelc wrote:
>> >
>> > > I written query it's taking more time like 38 seconds, how to to
>> > > fetch the lac of records within seconds any example in django using
>> > > orm or MySQL query ?
>> >
>> > Are you saying that you have a raw SQL query, or you are using a
>> > Django query?
>> >
>> > My experience is that the biggest lag in database retrieval is the
>> > amount of data retrieved, so if you can reduce the number of fields
>> > returned, that would be a good start.
>> >
>> > If you are using a Django query, and it has related elements, then
>> > the biggest boost you can give is using "select_related" and
>> > "prefetch_related".
>> >
>> >
>> > -- Clive
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Django users" group.
>> > To unsubscribe from this group and stop receiving emails from it,
>> > send an email to django-users+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/
>> > d/msgid/django-users/9EF46118-6870-46AD-BF58-12AEC8C6A518%
>> > 40indx.co.uk.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Django users" group.
>> > To unsubscribe from this group and stop receiving emails from it,
>> > send an email to django-users+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/
>> > d/msgid/django-users/CAMCU6CrAKQL6qA6hWHPDFbfpCT1R9v5%2BY_Qi-HOkPf%
>> > 3DOXYt2yA%40mail.gmail.com.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/52330EF7-BBD9-44E6-94E3-C0A733F4EF36%40indx.co.uk
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAK-XCvfn84Xy%3D8kRpt%2BfJ_DmwUMxkeZ9iTnaGuUV5M3SNN-Twg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAMCU6CoWgqWc%2Bc%3DZe15nvZZc-cZ%3DcYG6jOUU2o3SXftPPRZTGg%40mail.gmail.com.

Re: How to fetch 10000 above record within second in django?

[Clive Bruton]

On 17 Jan 2023, at 12:06, Ayanfeoluwa Koyinsola wrote:

You might now have to display all the data at once, see if you can  
use paginisation to reduce the numbers of queries


I think this will only help if there are related lookups.


-- Clive

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3ADD9645-859E-4452-9460-0917C367669B%40indx.co.uk.

Re: How to fetch 10000 above record within second in django?

[Jason]

https://dev.mysql.com/doc/refman/8.0/en/using-explain.html is a good guide 
to showing how to use EXPLAIN to see what your query is running.

https://use-the-index-luke.com/ is a great introductory source for indexes

https://django-debug-toolbar.readthedocs.io/en/latest/ added to your 
project can show where your bottlenecks are.

On Tuesday, January 17, 2023 at 10:40:27 AM UTC-5 Clive Bruton wrote:

>
> On 17 Jan 2023, at 12:06, Ayanfeoluwa Koyinsola wrote:
>
> > You might now have to display all the data at once, see if you can 
> > use paginisation to reduce the numbers of queries
>
> I think this will only help if there are related lookups.
>
>
> -- Clive
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8924df06-94f1-411f-9c54-4758028f274fn%40googlegroups.com.