Re: [RDBO] client out of memory with large table

2007-01-05 Thread Neal Clark
On Jan 5, 2007, at 8:36 PM, John Siracusa wrote: > On 1/5/07 11:29 PM, John Siracusa wrote: >> On 1/5/07 10:53 PM, Neal Clark wrote: >>> my $sql = "SELECT * FROM malware_logs"; >>> $Rose::DB::Object::Manager::Debug = 1; >>> my $iterator = MalwareLog::Manager->get_objects_iterator_from_sql >>> ($s

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Perrin Harkins
John Siracusa wrote: > Finally, if you only want to turn this on for particular calls, remember > that the db is an optional argument to all Manager methods. You can always > make/get a $db, yank out the $dbh, set the attribute manually, and pass the > $db as a parameter to your Manager calls. >

Re: [RDBO] client out of memory with large table

2007-01-05 Thread John Siracusa
On 1/5/07 11:29 PM, John Siracusa wrote: > On 1/5/07 10:53 PM, Neal Clark wrote: >> my $sql = "SELECT * FROM malware_logs"; >> $Rose::DB::Object::Manager::Debug = 1; >> my $iterator = MalwareLog::Manager->get_objects_iterator_from_sql($sql); > > Just an aside: in the case of this simple query, the

Re: [RDBO] client out of memory with large table

2007-01-05 Thread John Siracusa
On 1/5/07 10:53 PM, Neal Clark wrote: > my $sql = "SELECT * FROM malware_logs"; > $Rose::DB::Object::Manager::Debug = 1; > my $iterator = MalwareLog::Manager->get_objects_iterator_from_sql($sql); Just an aside: in the case of this simple query, there's no reason to use a *_from_sql() Manager metho

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Neal Clark
Hi. I just want to thank everyone for their interest and advice on this thread. As far as Postgre/MySQL goes, the reason I'm using mysql is most certainly ignorance. I work for a pretty small company, and the scope and requirements were very loosely defined when I began this project. Th

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Perrin Harkins
Jonathan Vanasco wrote: > i simply don't know of anyone handling large > amounts of data with mysql and NOT having several dedicated staff to > deal with mysql and data integrity alone. Now you do. We have multiple databases with 80 million+ row tables, no dedicated DBAs, and no data integr

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Ask Bjørn Hansen
On Jan 5, 2007, at 6:08 PM, Jonathan Vanasco wrote: > for small stuff, not a problem. for large stuff, i think it is -- > which is why i'm a little shocked at the original poster. i'd be > scared shitless to be dealing with an 18M row mysql db myself. 5+ years ago I worked at a place with many

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Jonathan Vanasco
On Jan 5, 2007, at 8:30 PM, Perrin Harkins wrote: > Randal L. Schwartz wrote: >> Anyone using MySQL these days is for one of two reasons: >> >> 1) Legacy >> 2) Ignorance >> >> Both of those are curable. > > There's no cure for speed, flexibility, and ease of use though. :) completely off topic b

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Perrin Harkins
Randal L. Schwartz wrote: > Anyone using MySQL these days is for one of two reasons: > > 1) Legacy > 2) Ignorance > > Both of those are curable. There's no cure for speed, flexibility, and ease of use though. :) - Perrin -

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Perrin Harkins
Neal Clark wrote: > Ideally, I will be making iterators from SQL calls. The SQL will > limit the number of rows as best as I can, but there will be times I > need to iterate over every row. I read in one of the CPAN docs that > Rose::DB::Object::Iterator returns 'true iterators', i.e. the row

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Randal L. Schwartz
> "Jonathan" == Jonathan Vanasco <[EMAIL PROTECTED]> writes: Jonathan> note that i'm not familiar with mysql anymore. i dropped it long Jonathan> ago for postgres. Amen. Anyone using MySQL these days is for one of two reasons: 1) Legacy 2) Ignorance Both of those are curable. -- Randal

Re: [RDBO] client out of memory with large table

2007-01-05 Thread Jonathan Vanasco
On Jan 5, 2007, at 6:22 PM, Neal Clark wrote: > when I make a new table (called 'abc') from the same schema > definition and load it with 1,000,000 rows from the malware_logs > table, everything works fine. is it just a limitation of Rose::DB > that it won't work with tables of this size? or is i

Re: [RDBO] client out of memory with large table

2007-01-05 Thread John Siracusa
On 1/5/07 6:22 PM, Neal Clark wrote: > The last row in the table has an id of 23503496. Am I wrong to think > I can say: > > my $objects = MalwareLog::Manager->get_malwarelogs(id => { gt => > 23503495 }); > > and get an array of two objects back? You're not wrong. That should work fine. The er

[RDBO] client out of memory with large table

2007-01-05 Thread Neal Clark
Hello. I have a somewhat large database table in MySQL. It's a MyISAM table with about 21,000,000 rows. It was working great at around 18,000,000 rows, and I've written a ton of code using Rose::DB... but now that the table has that mayn rows, I can't do anything without running out of mem