Re: Django scalability with files

[James Bennett]

On Thu, Aug 27, 2009 at 2:22 PM, Rafael Ferreira wrote:
> The queue idea is a good one, and you can use Gearman to do that really
> easily. Another, even simpler, way to handle this is to use some kind of
> shared NFS mount for the storage. All things being equal, scaling a filer or
> SAN will be much easier than scaling your DB, specially if - like most
> similar cases I've seen in the past - most of you db becomes blob data.

There are also distributed file stores like MogileFS.


-- 
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Django scalability with files

[Rafael Ferreira]

The queue idea is a good one, and you can use Gearman to do that really
easily. Another, even simpler, way to handle this is to use some kind of
shared NFS mount for the storage. All things being equal, scaling a filer or
SAN will be much easier than scaling your DB, specially if - like most
similar cases I've seen in the past - most of you db becomes blob data.

On Thu, Aug 27, 2009 at 8:32 AM, David De La Harpe Golden <
david.delaharpe.gol...@ichec.ie> wrote:

>
> Lewis Taylor wrote:
>
> > Is there a better solution django offers, and why are
> > blobs not supported. ideally i would have the image stored in DB and
> > that way there would never be sync issues.
> >
>
> FWIW (which given you're concerned about scaling mightn't be all that
> much), it's fairly straightforward to stash small to middling sized*
> binary stuff naively to a database rather than fs without even writing a
> full custom file storage backend: just base64 encode it to text (using a
> postgres bytea would be preferable, but BinaryField or whatever isn't in
> Django yet, so my get_internal_type returns "TextField").
>
> e.g. (...not efficient (watch it slurp the whole thing into memory),
> some might say pretty dumb):
> http://python.pastebin.com/d1c577e71
>
> * postgres text and bytea go up to 1GB IIRC.
>
>
>
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Django scalability with files

[David De La Harpe Golden]

Lewis Taylor wrote:

> Is there a better solution django offers, and why are
> blobs not supported. ideally i would have the image stored in DB and
> that way there would never be sync issues.
> 

FWIW (which given you're concerned about scaling mightn't be all that
much), it's fairly straightforward to stash small to middling sized*
binary stuff naively to a database rather than fs without even writing a
full custom file storage backend: just base64 encode it to text (using a
postgres bytea would be preferable, but BinaryField or whatever isn't in
Django yet, so my get_internal_type returns "TextField").

e.g. (...not efficient (watch it slurp the whole thing into memory),
some might say pretty dumb):
http://python.pastebin.com/d1c577e71

* postgres text and bytea go up to 1GB IIRC.




--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Django scalability with files

[David Zhou]

On Thu, Aug 27, 2009 at 6:13 AM, Lewis
Taylor wrote:
>
> I have an imagefield that i want to create a thumbnail for on approval
> of the image. The problem is (setup is 2 django instances, load
> balancer and DB) if the image is stored on one machine it's not the
> cleanest solution to have to get the image from the other server and

We ran into a similar issue in a past project, and if you have
scalability concerns, setting up a proper queue system is pretty much
the only practical method we thought of.  For every image uploaded,
add a new task to the queue detailing where the image is located. Then
have a separate image processing service running somewhere, ping the
queue for new tasks, and sync the processed image to your static
server(s).

If you really want to store uploaded images directly to the database
in blobs, look into writing a custom storage backend:

http://docs.djangoproject.com/en/dev/howto/custom-file-storage/

That will lessen syncing issues without needing a task queue, but will
make your db server work harder.

-- dz

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Django scalability with files

[Lewis Taylor]

Question:

I have an imagefield that i want to create a thumbnail for on approval
of the image. The problem is (setup is 2 django instances, load
balancer and DB) if the image is stored on one machine it's not the
cleanest solution to have to get the image from the other server and
process it. Is there a better solution django offers, and why are
blobs not supported. ideally i would have the image stored in DB and
that way there would never be sync issues.

Thanks

Lewis
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---