Re: LiveCode and SQLite performace

2015-07-28 Thread Bob Sneidar
You do not remember correctly. What I said (if I am remembering correctly) is that you need at least the size of your memory free and then some. This is because your OS will page out your memory into virtual memory, which is just a disk cache. But other apps also need room to grow for other

Re: LiveCode and SQLite performace

2015-07-28 Thread JB
Okay I did not remember your statement correctly. But I have noticed after 50% things start slowing down a little. And if I get to 20% they have slowed down a lot. I do agree at 10% you are risking damage but I would say from the poor performance I have seen at 20% it suggest the drive is rising

Re: LiveCode and SQLite performace

2015-07-28 Thread JB
Another thing to consider is how many files you are working with and creating in a day. I am easily in the hundreds and that could make things worse than for someone who is just working with the same few files. John Balgenorth On Jul 28, 2015, at 12:59 PM, JB sund...@pacifier.com wrote: Okay

Re: LiveCode and SQLite performace

2015-07-26 Thread Kay C Lan
On Sat, Jul 25, 2015 at 1:03 AM, Richard Gaskin ambassa...@fourthworld.com wrote: Yes, I also use Time Machine, but relying on any single backup isn't enough; TimeMachine can easily be set up to use multiple HDs so that you are not relying on a single backup. Even so I fully agree with

Re: LiveCode and SQLite performace

2015-07-24 Thread Robert Brenstein
A rule of thumb for Mac is 10% of drive being free... I find iStat Menus a useful tool for continuous monitoring of vital parameters (just a happy user). RObert On 23.07.2015 at 15:37 Uhr -0700 JB apparently wrote: If I remember correctly Bob Sneidar said that a you need at least 1/2 of

Re: LiveCode and SQLite performace

2015-07-24 Thread EED-wp Email
I've found DiskWarrior to be an indispensable tool on my Mac. It fixes disc corruption that diskutil can't and optimizes the directory. It might be worth a try. Bill William Prothero http://ed.earthednet.org On Jul 24, 2015, at 4:50 AM, JB sund...@pacifier.com wrote: 10% might work for

Re: LiveCode and SQLite performace

2015-07-24 Thread Mike Doub
I'm z On Jul 24, 2015, 10:23 AM, at 10:23 AM, EED-wp Email proth...@earthednet.org wrote: I've found DiskWarrior to be an indispensable tool on my Mac. It fixes disc corruption that diskutil can't and optimizes the directory. It might be worth a try. Bill William Prothero

Re: LiveCode and SQLite performace

2015-07-24 Thread Mike Doub
@ On Jul 24, 2015, 10:23 AM, at 10:23 AM, EED-wp Email proth...@earthednet.org wrote: I've found DiskWarrior to be an indispensable tool on my Mac. It fixes disc corruption that diskutil can't and optimizes the directory. It might be worth a try. Bill William Prothero

Re: LiveCode and SQLite performace

2015-07-24 Thread Richard Gaskin
The guidelines I'd read were about 15% IIRC; I believe there used to be a tech note on this, but I can no longer find it at apple.com. Most third-party sites discussing this cite 15%. If OS X required us to never use half of our disk space that would be quite a public controversy, since

Re: LiveCode and SQLite performace

2015-07-24 Thread JB
Thanks to everyone for the info. It looks like I need to run some test on it. John Balgenorth On Jul 24, 2015, at 8:19 AM, Richard Gaskin ambassa...@fourthworld.com wrote: The guidelines I'd read were about 15% IIRC; I believe there used to be a tech note on this, but I can no longer

Re: LiveCode and SQLite performace

2015-07-24 Thread Mike Kerner
As long as we're on this topic, I can't get over how much faster my POC MacBook Air is than my not-so POC Mac Mini. The former has an ssd, the latter, a regular HD. The former has a 1.4 ghz i5 with 4GB RAM, and the latter a 2.3 ghz i5 with 16 GB RAM. That SSD makes the air scream. On Fri, Jul

Re: LiveCode and SQLite performace

2015-07-24 Thread Richard Gaskin
Mark Waddingham wrote: I'd get your hdd checked out asap. This is a good opportunity for all of us to remember that portable large-capacity hard drives are dirt cheap compared to the cost of lost data. A USB 3.0 1TB drive can be picked up at the corner market for about US$60, and a 2TB

Re: LiveCode and SQLite performace

2015-07-24 Thread Mark Waddingham
Hi Mike, The only reference to slow opening of SQLite dbs on the Internet was to do with file permission issues on windows so if you are seeing this on Mac it very much sounds like a hard disk issue. One of my colleagues had an issue recently with the hard drive in a 2011 iMac which didn't

Re: LiveCode and SQLite performace

2015-07-24 Thread JB
10% might work for you but it definitely does not work for me. I have a 1tb drive and 348 gb free space. Most operations run slower than normal. I deleted about 150gb of music to bring it up to 350gb because it ran too slow to use. Now it runs fast enough to use but I still have a lot of wasted

Re: LiveCode and SQLite performace

2015-07-23 Thread Peter Haworth
Thanks for letting us know Mike. You might try running Diskutil to check out your hard drive. Or maybe the disk is getting full enough that there's not enough paging space? On Thu, Jul 23, 2015 at 2:12 PM Michael Doub miked...@gmail.com wrote: Hey guys, I think that I have a system issue of

Re: LiveCode and SQLite performace

2015-07-23 Thread JB
If I remember correctly Bob Sneidar said that a you need at least 1/2 of your hard drive as free space to run efficiently. So if you have a drive with 500 GB you need 250 GB or more free space on the drive. Anything below that and it normal operations like opening files will be slower. I have

Re: LiveCode and SQLite performace

2015-07-23 Thread Michael Doub
Hey guys, I think that I have a system issue of some type. We had a nasty thunderstorm last night and we lost power. Obviously my Mac was re-booted. When I ran my timing tests this morning. Open was 3 seconds and loading all the data in to a variable was 1. Much better. I just got

Re: LiveCode and SQLite performace

2015-07-23 Thread Bob Sneidar
I use .db without any problems. By the way, why would you want to load the entire data set? I agree that to open any sql database should not take that long. It almost sounds like something is not working as it should and it times out. But on the query side, loading 100’s of 1000’s of records

Re: LiveCode and SQLite performace

2015-07-23 Thread Peter Haworth
Hi Michael, Those are ridiculously long times! I can't think of anything obvious that would cause them. Mark's question about whether the automatic indexes are being recreated is a possibility but I'm pretty sure that doesn't happen. I couldn't tell from your original post if your table has a

Re: LiveCode and SQLite performace

2015-07-23 Thread Peter Haworth
Hi Mark, I couldn't say for sure but I'm 99% sure the indexes are not recreated each time the database is opened. On Wed, Jul 22, 2015 at 4:58 PM Mark Wieder mwie...@ahsoftware.net wrote: On 07/22/2015 02:11 PM, Peter Haworth wrote: You'll see auto indexes for any columns that are defined as

Re: LiveCode and SQLite performace

2015-07-23 Thread Peter Haworth
One more thing Michael - do the names of the auto indexes you're seeing begin with sqlite? If not, your sqlitebrowser program is creating them and may be doing something to contribute to the problem. On Wed, Jul 22, 2015 at 4:33 PM Michael Doub miked...@gmail.com wrote: I just updated the

Re: LiveCode and SQLite performace

2015-07-22 Thread Peter Haworth
Hi Michael, Haven't come across that before. 600k records is a large number but I've seen dbs with millions of records in them so it's not unusual. In any case, almost every performance problem I've seen revolves around either selecting or changing data not simply opening the database. Try

Re: LiveCode and SQLite performace

2015-07-22 Thread Michael Doub
Kay and Mark, the database file size is 250Mb. The performance issue that surprised me was the open time. Adds and Queries are as expected. ID's currently range from 1 to 60400 URL is a typical URL that ranges from 50 to 130 characters, average is 80 char. tag is currently not being used

Re: LiveCode and SQLite performace

2015-07-22 Thread Mike Bonner
If its on an external usb drive, part of the wait is most likely the drive wake up time. To test, move the file to a local drive and try it from there, OR, open the drive in finder first and poke around a little to make sure the drive is live, then try to open the database and see if it solves the

Re: LiveCode and SQLite performace

2015-07-22 Thread Peter Haworth
Hi Michael, Out of interest, when you say it takes a long time to open the database, how long do you mean? Also, where is the database located? On your Mac's hard drive, external drive, on a network? I'm a bit confused as to the number of records. Your original email said 600,000+ records, but

Re: LiveCode and SQLite performace

2015-07-22 Thread Michael Doub
I just updated the database and it now has 608436 records. Sorry for the typo. it was 604000. How long to open - 216 seconds. I timed put revDataFromQuery(,,db_id,select * from mydatabase) into tResult it was 26 seconds. 216 to open and 26 to copy all of the data into a variable. This

Re: LiveCode and SQLite performace

2015-07-22 Thread Mark Wieder
On 07/22/2015 02:11 PM, Peter Haworth wrote: You'll see auto indexes for any columns that are defined as UNIQUE, that's how sqlite enforces that constraint. Doh! You're right. I could swear the documentation said you had to compile sqlite with a non-default compiler option to get this to

Re: LiveCode and SQLite performace

2015-07-21 Thread Kay C Lan
An example of half a dozen records please. Maybe a couple of very short, a couple of the longest and a couple of average. How big is the db file size - MB not record count? On Wed, Jul 22, 2015 at 7:55 AM, Michael Doub miked...@gmail.com wrote: I need some database consulting help. I have

Re: LiveCode and SQLite performace

2015-07-21 Thread Mark Wieder
On 07/21/2015 04:55 PM, Michael Doub wrote: I need some database consulting help. I have created an SQLite database that has 4 fields: ID, URL, tag, local. IS is integer, unique, URL is text not null unique and tag and local are just text. I have 600,000+ records. I use the ID to access

LiveCode and SQLite performace

2015-07-21 Thread Michael Doub
I need some database consulting help. I have created an SQLite database that has 4 fields: ID, URL, tag, local. IS is integer, unique, URL is text not null unique and tag and local are just text. I have 600,000+ records. I use the ID to access each record in most cases or I am filtering