Re: Request Timed Out When Generating Huge Data

[tango ward]

I tried loading the site locally to see how long will it take to load a
record of 1800 students. It took about a minute and a half locally. The
request timed out problem occurs when I try to generate a CSV report and
PDF. Both result to timed out even locally.

On Tue, Apr 3, 2018 at 8:12 AM, tango ward  wrote:

> Hi,
>
> SLR.
>
> At the moment, the largest number of students we have is 2000 in one
> campus. I haven't played around with the code but when I check it we're
> using a get_queryset function to get the data and filter it using Q. Then
> display it using get_context_data
>
> Code:
>
> class FinancialReportView(ApplicantReportMixin, ListView):
> template_name = 'education/dashboard/reports/financial_report.html'
> paginate_by = 50
>
> def get_queryset(self):
>
> student_filter = self.request.GET.get('student_filter', '')
>
> filter_date = self.request.GET.get('date', '')
> if not filter_date and not student_filter:
> return Recipient.objects.none()
>
> if student_filter in ['None', None]:
> student_filter = ''
> recipients = self.get_recipients()
> recipients = recipients.filter(enrolled=1)
> recipients = recipients.filter(enrollmentinfo__isnull=False)
>
> campus = self.request.GET.get('campus', '')
> if campus:
> recipients = recipients.filter(department__school=campus)
> year_level = self.request.GET.get('year_level', '')
>
> if year_level:
> recipients = recipients.filter(year_level=year_level)
>
> if student_filter and not student_filter == 'None':
> recipients = recipients.filter(
> Q(first_name__icontains=student_filter) |
> Q(last_name__icontains=student_filter) |
> Q(student_id__icontains=student_filter))
> recipients = recipients.select_related('department', 'program')
> return recipients
>
> def get_context_data(self, **kwargs):
> context = super(FinancialReportView, self).get_context_data(**
> kwargs)
>
> filter_form = FinancialReportFilterForm(
> data=self.request.GET,
> school_manager=self.request.user.schoolmanager,
> )
> output = []
> recipients = self.get_queryset()
>
> filter_date = self.request.GET.get('date', '')
> if filter_date in ['None', None]:
> filter_date = ''
> if filter_date:
> filter_date = dateutil.parser.parse(filter_date).date()
> page = self.request.GET.get('page', 1)
>
> for ctr, recipient in enumerate(recipients):
> total_amount = 0
> enrollment_info = recipient.get_current_enrollment_info()
> if enrollment_info:
> if filter_date:
> invoices = enrollment_info.
> philsmileinvoice_set.filter(
> due_date__lte=filter_date)
> else:
> invoices = enrollment_info.philsmileinvoice_set.all()
>
> for invoice in invoices:
> total_amount += invoice.get_remaining_balance()
>
> output.append([ctr, recipient, total_amount])
>
> if self.paginate_by:
> paginator = Paginator(output, self.paginate_by)
> try:
> output = paginator.page(page)
> except PageNotAnInteger:
> output = paginator.page(1)
> except EmptyPage:
> output = paginator.page(paginator.num_pages)
> context.update({
> 'paginator': paginator
> })
>
> context.update({
> 'outputs': output,
> 'form': filter_form,
> 'school_system': self.request.user.
> schoolmanager.school_system,
> })
> return context
>
>
>
>
> On Mon, Apr 2, 2018 at 1:54 PM, Babatunde Akinyanmi 
> wrote:
>
>> Hi,
>> How many records are we talking about? What code are you using to fetch
>> the records? In other words, add more information so that it's easier to
>> troubleshoot
>>
>> On Mon, 2 Apr 2018, 02:27 tango ward,  wrote:
>>
>>>
>>> Good day,
>>>
>>> I need suggestions on how to approach this problem.
>>>
>>> I am working on a task that needs to generate the list students per
>>> school. When the school has larger number of students, I am getting a
>>> request timed out error. I can filter the school per campus to lessen the
>>> number of students but the performance in displaying the list of students
>>> is still taking too long.
>>>
>>>
>>> Thanks,
>>> Jarvis
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email 

Re: Request Timed Out When Generating Huge Data

[tango ward]

Hi,

SLR.

At the moment, the largest number of students we have is 2000 in one
campus. I haven't played around with the code but when I check it we're
using a get_queryset function to get the data and filter it using Q. Then
display it using get_context_data

Code:

class FinancialReportView(ApplicantReportMixin, ListView):
template_name = 'education/dashboard/reports/financial_report.html'
paginate_by = 50

def get_queryset(self):

student_filter = self.request.GET.get('student_filter', '')

filter_date = self.request.GET.get('date', '')
if not filter_date and not student_filter:
return Recipient.objects.none()

if student_filter in ['None', None]:
student_filter = ''
recipients = self.get_recipients()
recipients = recipients.filter(enrolled=1)
recipients = recipients.filter(enrollmentinfo__isnull=False)

campus = self.request.GET.get('campus', '')
if campus:
recipients = recipients.filter(department__school=campus)
year_level = self.request.GET.get('year_level', '')

if year_level:
recipients = recipients.filter(year_level=year_level)

if student_filter and not student_filter == 'None':
recipients = recipients.filter(
Q(first_name__icontains=student_filter) |
Q(last_name__icontains=student_filter) |
Q(student_id__icontains=student_filter))
recipients = recipients.select_related('department', 'program')
return recipients

def get_context_data(self, **kwargs):
context = super(FinancialReportView,
self).get_context_data(**kwargs)

filter_form = FinancialReportFilterForm(
data=self.request.GET,
school_manager=self.request.user.schoolmanager,
)
output = []
recipients = self.get_queryset()

filter_date = self.request.GET.get('date', '')
if filter_date in ['None', None]:
filter_date = ''
if filter_date:
filter_date = dateutil.parser.parse(filter_date).date()
page = self.request.GET.get('page', 1)

for ctr, recipient in enumerate(recipients):
total_amount = 0
enrollment_info = recipient.get_current_enrollment_info()
if enrollment_info:
if filter_date:
invoices = enrollment_info.philsmileinvoice_set.filter(
due_date__lte=filter_date)
else:
invoices = enrollment_info.philsmileinvoice_set.all()

for invoice in invoices:
total_amount += invoice.get_remaining_balance()

output.append([ctr, recipient, total_amount])

if self.paginate_by:
paginator = Paginator(output, self.paginate_by)
try:
output = paginator.page(page)
except PageNotAnInteger:
output = paginator.page(1)
except EmptyPage:
output = paginator.page(paginator.num_pages)
context.update({
'paginator': paginator
})

context.update({
'outputs': output,
'form': filter_form,
'school_system': self.request.user.schoolmanager.school_system,
})
return context




On Mon, Apr 2, 2018 at 1:54 PM, Babatunde Akinyanmi 
wrote:

> Hi,
> How many records are we talking about? What code are you using to fetch
> the records? In other words, add more information so that it's easier to
> troubleshoot
>
> On Mon, 2 Apr 2018, 02:27 tango ward,  wrote:
>
>>
>> Good day,
>>
>> I need suggestions on how to approach this problem.
>>
>> I am working on a task that needs to generate the list students per
>> school. When the school has larger number of students, I am getting a
>> request timed out error. I can filter the school per campus to lessen the
>> number of students but the performance in displaying the list of students
>> is still taking too long.
>>
>>
>> Thanks,
>> Jarvis
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUz
>> aCYm_un0zZFMXQC8ewzrg%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed 

Re: Request Timed Out When Generating Huge Data

[Babatunde Akinyanmi]

Hi,
How many records are we talking about? What code are you using to fetch the
records? In other words, add more information so that it's easier to
troubleshoot

On Mon, 2 Apr 2018, 02:27 tango ward,  wrote:

>
> Good day,
>
> I need suggestions on how to approach this problem.
>
> I am working on a task that needs to generate the list students per
> school. When the school has larger number of students, I am getting a
> request timed out error. I can filter the school per campus to lessen the
> number of students but the performance in displaying the list of students
> is still taking too long.
>
>
> Thanks,
> Jarvis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUzaCYm_un0zZFMXQC8ewzrg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2BWjgXN%2B2U5UtbjPGzTO5Aff-s%2BAeDKT%3DYg1HwFr72C4KGB4SQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Request Timed Out When Generating Huge Data

[tango ward]

Good day,

I need suggestions on how to approach this problem.

I am working on a task that needs to generate the list students per school.
When the school has larger number of students, I am getting a request timed
out error. I can filter the school per campus to lessen the number of
students but the performance in displaying the list of students is still
taking too long.


Thanks,
Jarvis

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUzaCYm_un0zZFMXQC8ewzrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.