Re: Dynamic settings variables (dynamic global variable)

[bfrederi]

Makes sense. I actually ended up using your example and it worked well
as far as I can tell. And you actually explained the importing too,
which is where I messed up previously.

Also, I made the mistake of putting it in a file that was already
importing a bunch of other things, so I got wrapped up in import hell,
but I think I found a better place to keep that class.

Thanks!

On Feb 9, 2:13 pm, Doug Ballance  wrote:
> > Doug, I don't see how the LazyFetch you wrote is much different than
> > what akaariai shared? Can you explain to me what the difference is?
> > And I'm not saying that in a condescending way, I'm saying I'm just
> > not sharp enough on my Python to recognize the difference.
>
> Slightly different implementation of the same idea.  From your reply
> to his post I thought there was still some confusion on the concept so
> I did a modification of a class we use in a project so that you could
> have a tested/working sample.  They are fundamentally the same
> approach to the problem, it just seemed to me that treating an
> instance of the class as the 'page' variable rather than 'cache' used
> to fetch a url might be clearer as an example.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[Doug Ballance]

> Doug, I don't see how the LazyFetch you wrote is much different than
> what akaariai shared? Can you explain to me what the difference is?
> And I'm not saying that in a condescending way, I'm saying I'm just
> not sharp enough on my Python to recognize the difference.

Slightly different implementation of the same idea.  From your reply
to his post I thought there was still some confusion on the concept so
I did a modification of a class we use in a project so that you could
have a tested/working sample.  They are fundamentally the same
approach to the problem, it just seemed to me that treating an
instance of the class as the 'page' variable rather than 'cache' used
to fetch a url might be clearer as an example.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[bfrederi]

I tried akaariai's method, but I may have done the importing wrong. It
seemed to make apache struggle mightily to the point where it was
unusable for some reason.

Doug, I don't see how the LazyFetch you wrote is much different than
what akaariai shared? Can you explain to me what the difference is?
And I'm not saying that in a condescending way, I'm saying I'm just
not sharp enough on my Python to recognize the difference.

On Feb 9, 12:42 am, Doug Ballance  wrote:
> I made an error when I changed a variable name just before posting.
> Replace "self._age" with "self._last_updated".

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[bfrederi]

Thanks Andy, I didn't know you could use a cache by name. That will be
helpful.

On Feb 8, 9:22 pm, Andy McKay  wrote:
> You can define multiple caches:
>
> https://docs.djangoproject.com/en/1.3/topics/cache/#django-s-cache-fr...
>
> You could then use a file system cache or your own local memcache.
>
> https://docs.djangoproject.com/en/1.3/topics/cache/#filesystem-caching
>
> You can then get that cache and access it in your code:
>
> from django.core.cache import get_cache
> cache = get_cache('my-filesystem-cache-defined-in-settings')
>
>
>
>
>
>
>
> On Wed, Feb 8, 2012 at 3:29 PM, bfrederi  wrote:
> > I'm already using memcached over http so I can have multiple memcached
> > instances. But there isn't any way to set priority or which cache the
> > item goes to or is pulled from. So if I had a local memcached
> > instance, there's no way I could be sure which memcached instance the
> > data was coming from. I already store the data on disk as a back-up in
> > case the connection to my memcached servers fails. But I'm really
> > looking to store it in local memory. And hopefully without adding any
> > other tools to my stack.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[Doug Ballance]

I made an error when I changed a variable name just before posting.
Replace "self._age" with "self._last_updated".

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[Doug Ballance]

A method similar to what Anssi describes is what we use for local
caching of data that is too large to comfortably fetch from cache/
parse from file every time we need it. A  Doing a lazy fetch with a
maximum age like that also helps with load times since it is only
fetched when accessed. Of course the fetch needs to be reasonably
possible with the time frame of a single request in order to not
frustrate users.

# Put this class somewhere and import into your settings.py
import datetime
import urllib2
class LazyFetch(object):
def __init__(self,url,max_age=60*60):
self.url=url
self.reload_delta=datetime.timedelta(seconds=max_age)

def _get_content(self):
last_updated=getattr(self,'_last_updated',None)
if not last_updated or datetime.datetime.now()-last_updated >
self.reload_delta:
self._content=urllib2.urlopen(self.url).read()
self._age=datetime.datetime.now()
return self._content
content=property(_get_content)

somepage=LazyFetch("http://www.example.com",30)

then elsewhere in your code you can import settings and access the
page content

current_content=settings.somepage.content

Accessing the property "content" would call get_content() which checks
to see if the content hasn't been loaded, or expired.. and reloads on
demand.  This is just a barebones example though, you'd want to throw
in some exception handing to the urlopen call at the very least.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[Andy McKay]

You can define multiple caches:

https://docs.djangoproject.com/en/1.3/topics/cache/#django-s-cache-framework

You could then use a file system cache or your own local memcache.

https://docs.djangoproject.com/en/1.3/topics/cache/#filesystem-caching

You can then get that cache and access it in your code:

from django.core.cache import get_cache
cache = get_cache('my-filesystem-cache-defined-in-settings')

On Wed, Feb 8, 2012 at 3:29 PM, bfrederi  wrote:
> I'm already using memcached over http so I can have multiple memcached
> instances. But there isn't any way to set priority or which cache the
> item goes to or is pulled from. So if I had a local memcached
> instance, there's no way I could be sure which memcached instance the
> data was coming from. I already store the data on disk as a back-up in
> case the connection to my memcached servers fails. But I'm really
> looking to store it in local memory. And hopefully without adding any
> other tools to my stack.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[bfrederi]

I'm already using memcached over http so I can have multiple memcached
instances. But there isn't any way to set priority or which cache the
item goes to or is pulled from. So if I had a local memcached
instance, there's no way I could be sure which memcached instance the
data was coming from. I already store the data on disk as a back-up in
case the connection to my memcached servers fails. But I'm really
looking to store it in local memory. And hopefully without adding any
other tools to my stack.

On Feb 8, 5:04 pm, Brett Epps  wrote:
> I'd recommend caching the data using Django's caching framework and either
> the local memory or memcached backend.  To update the cache, write a
> management command that runs periodically as a cron job.  (If you find
> yourself needing more sophisticated background task management, check out
> Celery.)
>
> Brett
>
> On 2/8/12 4:14 PM, "bfrederi"  wrote:
>
>
>
>
>
>
>
> >I have some data that I retrieve using urllib and then cache on a
> >different server that I frequently use through my site. Since it gets
> >used so frequently, I was wondering why it might be a good/bad idea to
> >make the urllib request in the settings.py. It's data that gets
> >changed infrequently, but should update every once in a while, but I'd
> >prefer that it stayed in local memory as much as possible, since it
> >gets used so much. Any advice on how to make a dynamic global variable
> >like that is much appreciated.
>
> >Thanks,
> >Brandon
>
> >--
> >You received this message because you are subscribed to the Google Groups
> >"Django users" group.
> >To post to this group, send email to django-users@googlegroups.com.
> >To unsubscribe from this group, send email to
> >django-users+unsubscr...@googlegroups.com.
> >For more options, visit this group at
> >http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[Brett Epps]

I'd recommend caching the data using Django's caching framework and either
the local memory or memcached backend.  To update the cache, write a
management command that runs periodically as a cron job.  (If you find
yourself needing more sophisticated background task management, check out
Celery.)

Brett


On 2/8/12 4:14 PM, "bfrederi"  wrote:

>I have some data that I retrieve using urllib and then cache on a
>different server that I frequently use through my site. Since it gets
>used so frequently, I was wondering why it might be a good/bad idea to
>make the urllib request in the settings.py. It's data that gets
>changed infrequently, but should update every once in a while, but I'd
>prefer that it stayed in local memory as much as possible, since it
>gets used so much. Any advice on how to make a dynamic global variable
>like that is much appreciated.
>
>Thanks,
>Brandon
>
>-- 
>You received this message because you are subscribed to the Google Groups
>"Django users" group.
>To post to this group, send email to django-users@googlegroups.com.
>To unsubscribe from this group, send email to
>django-users+unsubscr...@googlegroups.com.
>For more options, visit this group at
>http://groups.google.com/group/django-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[bfrederi]

So would I put that class in my settings.py? Where would I put that
class to make sure that the data is frequently retrieved from local
memory?

On Feb 8, 4:47 pm, akaariai  wrote:
> On Feb 9, 12:14 am, bfrederi  wrote:
>
> > I have some data that I retrieve using urllib and then cache on a
> > different server that I frequently use through my site. Since it gets
> > used so frequently, I was wondering why it might be a good/bad idea to
> > make the urllib request in the settings.py. It's data that gets
> > changed infrequently, but should update every once in a while, but I'd
> > prefer that it stayed in local memory as much as possible, since it
> > gets used so much. Any advice on how to make a dynamic global variable
> > like that is much appreciated.
>
> Don't do that in settings.py. Use Django's caching framework. Or
> something like:
> class MyUrlLoader(object):
>     cache = {} # map of url -> (timeout cutpoint, data)
>     def get_data(url):
>            val = self.cache.get(url)
>            if val is None or datetime.now() > val[0]:
>                 val = (datetime.now() + timedelta(seconds=60),
> loaded_data)
>                 self.cache[url]  = val
>            return val[1]
>
> No idea if the above actually works, but the idea should work. But you
> really are duplicating what the cache framework does, so first try
> that.
>
>  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dynamic settings variables (dynamic global variable)

[akaariai]

On Feb 9, 12:14 am, bfrederi  wrote:
> I have some data that I retrieve using urllib and then cache on a
> different server that I frequently use through my site. Since it gets
> used so frequently, I was wondering why it might be a good/bad idea to
> make the urllib request in the settings.py. It's data that gets
> changed infrequently, but should update every once in a while, but I'd
> prefer that it stayed in local memory as much as possible, since it
> gets used so much. Any advice on how to make a dynamic global variable
> like that is much appreciated.

Don't do that in settings.py. Use Django's caching framework. Or
something like:
class MyUrlLoader(object):
cache = {} # map of url -> (timeout cutpoint, data)
def get_data(url):
   val = self.cache.get(url)
   if val is None or datetime.now() > val[0]:
val = (datetime.now() + timedelta(seconds=60),
loaded_data)
self.cache[url]  = val
   return val[1]

No idea if the above actually works, but the idea should work. But you
really are duplicating what the cache framework does, so first try
that.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.