hehe. That's a very good MySQL observation. :) I was trying to avoid hadoop & map reduce because my data doesn't grow more than ~5 million / device (appliance) .. At 5 million, I am still able to run my queries at a very reasonable time and most if not all the queries need to be realtime and not batched.
If I want to create a table per device. Would it be possible to bind the ORM object to a table dynamically? Any sqlalchemy examples I can look it that does something similar? Thanks! On Saturday, June 30, 2012 2:10:55 PM UTC-7, Michael Bayer wrote: > > OK well it's MySQL, so sure if you want to make a table per customer, its > not a terrible drain on MySQL....the "create tables on the fly" thing makes > DBAs very upset but then again, MySQL DBs are usually not DBA controlled... > > still, it seems like these tables aren't referred to by any other tables, > otherwise table-per-customer would be quite unwieldy. I have a vague > recollection that Solr can be used for this sort of thing too....and google > says yes ! in fact it's explicitly a group (well, it's rackspace!) that > chose a solr/hadoop solution over "partitioned MySQL": > > > http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data > > Solr is a very good product and worth looking into here. > > > On Jun 30, 2012, at 2:20 PM, espresso maker wrote: > > 1: The logs are selected quite often based on indexed columns, and this is > done via a web portal graphing tool. Maybe I shouldn't refer to them as > logs for clarity, but the data is specific snmp fields where a schema fits > well and the queries are fairly basic with some group by aggregation. > > 2. I see. I was thinking of using myisam to store the "log" tables which > makes dropping a table equivalent to deleting a file. > > 3. Anytime I wanted to delete a device, the operation took some time, and > that table is read/write intensive so I am afraid that deleting 5M records > would interrupt things. (using innodb) > > 4. I am not expecting more than 5k customers. But I can imagine how > managing that can be a hassle. > > > Should I keep things the way it is, single database, single table for all > device logs of all customers? Or there is a better approach I can take? > > Thanks > > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/sqlalchemy/-/yviCZpEc-fwJ. > To post to this group, send email to sqlalchemy@googlegroups.com. > To unsubscribe from this group, send email to > sqlalchemy+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/R4ZUB7CcgQMJ. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
