On 05/23/2017 03:16 PM, Edward Leafe wrote:
On May 23, 2017, at 1:43 PM, Jay Pipes <jaypi...@gmail.com> wrote:
[1] Witness the join constructs in Golang in Kubernetes as they work around
etcd not being a relational data store:
Maybe it’s just me, but I found that Go code more understandable than some of
the SQL we are using in the placement engine. :)
I assume that the SQL in a relational engine is faster than the same thing in
code, but is that difference significant? For extremely large data sets I think
that the database processing may be rate limiting, but is that the case here?
Sometimes it seems that we are overly obsessed with optimizing data handling
when the amount of data is relatively small. A few million records should be
fast enough using just about anything.
When you write your app fresh, put some data into it, a few hundred
rows, not at all. Pull it all into memory and sort/filter all you want,
SQL is too hard. Push it to production! works great. send the
customer your bill.
6 months later. Customer has 10K rows. The tools their contractor
wrote seem a little sticky. Not sure when that happened?
A year later. Customer is at 300K rows, nowhere near "a few million"
records. Application regularly crashes when asked to search and filter
results. Because Python interpreter uses a fair amount of memory for a
result set, multiplied by the overhead of Python object() / dict() per
row == 100's / 1000's of megs of memory to have 300000 objects in memory
all at once. Multiply by dozens of threads / processes handling
concurrent requests, Python interpreter rarely returns memory. Then add
latency of fetching 300K rows over the wire, converting to objects.
Concurrent requests pile up because they're slower; == more processes,
== more memory.
New contractor is called in to rewrite the whole thing in MongoDB. Now
it's fast again! Proceed to chapter 2, "So you decided to use
MongoDB...." :)
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
