Hi,
I’ve been using zlib compression on my custom view responses on my
site (https://www.futureclaw.com). I do this before I even cache, and
I store the compressed responses in my Redis cache. This effectively
increases my Redis cache size by about 10x. It also reduces response
times from my HTTP server by 10s of milliseconds, since my web server
doesn’t have to compress on-the-fly for cached responses - the view
fragments are served directly from the Redis cache.
I also use generators for my views, and I use Django’s streaming views
with my generators to stream responses, so that the web browser
receives early responses and start rendering immediately before the
view is completely processed. The web browser receives the first HTML
fragment before I even hit the database.
Ultimately my page load times are ridiculously fast, especially for a
fully graphically intensive site. For full cache miss, it might take
15ms to generate a page. For a cache hit, it might take .3ms. For a
full page load end-user, which loads all fonts and graphics, I'm down
to 381ms, faster than 98% of all sites (it was several seconds
before). See:
https://tools.pingdom.com/#!/cHCL3d/https://www.futureclaw.com
<https://tools.pingdom.com/#%21/cHCL3d/https://www.futureclaw.com>
(BTW I plan on reducing the full-page load times down to the 100-200ms
range using HTTP/2 server push)
Anyways, is this feature of interest to the Django community for
perhaps the next version?
Here is the relevant portion directly lifted from my code as an
example. I made this earlier this year, and I’m not sure how much
this needs to be changed for use by Django itself, especially
considering I don’t use the Django template system or ORM:
classZipView(View):
level = 9
wbits = zlib.MAX_WBITS | 16
decompressor = zlib.decompressobj(wbits=wbits)
@staticmethod
defstore(compressor,key,data,time=9,end=False):
zData = compressor.compress(data)
ifend == False:
zData += compressor.flush(zlib.Z_FULL_FLUSH)
else:
zData += compressor.flush(zlib.Z_FINISH)
cache.set(key, zData, time)
returnzData
definflate(self,decompressor,data):
returnself.decompressor.decompress(data)
pageData = []
defgetPageData(self,slug=None):
pass
defgenerator(self,request,slug=None,compress=True):
tuple = []
compressor = False
decompressor = False
zData = cache.get('pStart') #pStart contains zlib header. Header is
needed.
ifzData isNone:
html = bytes(self.get_page_start_stream(),'utf-8')
compressor = zlib.compressobj(level=self.level,
wbits=self.wbits)
zData = self.store(compressor,'pStart',html,time=None)
else:
ifcompress == False:
decompressor = zlib.decompressobj(wbits=self.wbits)
html = decompressor.decompress(zData)
ifcompress == False:
zData = html
yieldzData
pageKey = self.getPageKey(slug)
zPageData = cache.get(pageKey)
ifzPageData isNone:
expire = 3600
zPage = b''
self.getPageData(slug)
#Fragment tuples contain HTML generator & associated parameters, as
well as cache settings for each fragment.
fortuple inself.fragments(request):
iftuple[2]:
iftuple[2] < expire:
expire = tuple[2]
k = tuple[0]+str(tuple[1])
zData = cache.get(k)
ifzData isNone:
func = tuple[3]
html = bytes(func(*tuple[4:]),'utf-8')
ifcompressor == False:
compressor =
zlib.compressobj(level=self.level, wbits=self.wbits)
self.store(compressor,'zHeader',b"")
zData = self.store(compressor,k,html,tuple[2])
else:
ifcompress == False:
ifdecompressor == False:
decompressor =
zlib.decompressobj(wbits=self.wbits)
html = decompressor.decompress(zData)
zPage += zData
ifcompress == False:
zData = html
yieldzData
zData = cache.get('pEnd')
ifzData isNone:
html = bytes(self.get_closing_stream(),'utf-8')
ifcompressor == False:
compressor = zlib.compressobj(level=self.level,
wbits=self.wbits)
self.store(compressor,'zHeader',b"")
zData =
self.store(compressor,'pEnd',html,time=None,end=True)
else:
ifcompress == False:
ifdecompressor == False:
decompressor =
zlib.decompressobj(wbits=self.wbits)
html = decompressor.decompress(zData)
zPage += zData
cache.set(pageKey, zPage, expire)
self.newtime = time.perf_counter()
ifcompress == False:
zData = html
yieldzData
else:
ifcompress == False:
ifdecompressor == False:
decompressor = zlib.decompressobj(wbits=self.wbits)
zPageData = decompressor.decompress(zPageData)
yieldzPageData
classFastView(ZipView):
c = connection.cursor()
defget(self, request):
returnself.html(request)
defhtml(self, request, slug=None):
try:
encoding = request.META['HTTP_ACCEPT_ENCODING']
except:
encoding = ''
if'gzip'inencoding:
compress = True
else:
compress = False
ifsettings.BENCHMARK:
count = 1
logger.info <http://logger.info>('//Starting Benchmark')
logger.info <http://logger.info>(' Class name: %s'%
self.__class__.__name__)
start_time = timing = time.perf_counter()
ifcompress:
page = b''
forhtml inself.generator(request,slug,compress):
page += html
newtime = time.perf_counter()
logger.info <http://logger.info>(' Step %i: %fms'% (count,
(newtime-timing)*1000))
count = count + 1
timing = newtime
logger.info <http://logger.info>(' Total Time: %fms'%
((newtime-start_time)*1000))
logger.info <http://logger.info>('//End Benchmark')
response = HttpResponse(page)
ifcompress:
response['content-encoding'] = 'gzip'
returnresponse
else:
response =
StreamingHttpResponse(self.generator(request,slug,compress))
ifcompress:
response['content-encoding'] = 'gzip'
returnresponse
classIndexView(FastView,factory):
title = 'FutureClaw'
@staticmethod
defgetPageKey(slug=None):
return'pIndex'
deffragments(self,request=None,slug=None):
return[
['pMetaIndex','',3600,self.get_page_meta_stream,self.title, None, None],
['pCover','',3600*24,self.get_cover_page_stream,self.c],
['pStartCategory','',None,self.get_start_category_page_stream],
['pHeadlines','',3600,self.get_headlines_stream,self.c],
['pLatest','',3600,self.get_latest_collection_stream,self.c],
['pFullSeasonStart','',None,self.get_full_season_start_stream],
['pFullSeason',None,3600,self.get_full_season_stream,None,None],
['pAllSeasons',0,3600*72,self.get_all_seasons_stream,self.c,0,''],
['pGallery',0,None,self.get_gallery_stream,None,None,None,None],
['pArticle',0,None,self.get_article_stream,None],
]
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to django-developers+unsubscr...@googlegroups.com
<mailto:django-developers+unsubscr...@googlegroups.com>.
To post to this group, send email to
django-developers@googlegroups.com
<mailto:django-developers@googlegroups.com>.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/5B4D7D1E-F21B-4CA0-AABD-95610A8DAF40%40gmail.com
<https://groups.google.com/d/msgid/django-developers/5B4D7D1E-F21B-4CA0-AABD-95610A8DAF40%40gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
