On Jul 14, 2011, at 11:24, dschnur wrote: > > Hi Debbie, > > You don't need to start over with a new database. > > When gathering data from IM, the bug caused the importer to create duplicate > entries for certain datapoints. But there's a safety mechanism where, when > it tries to actually insert those entries into the database, it discovers the > problem and discards the duplicates. Even so, this resulted in a significant > decrease in performance, due to having to re-run the import, as well as a > bunch of warnings in the log. > > 5.4.4 just prevents those duplicates from being created.
okay, thanks. Glad I don't have to start again. > > Regarding the time it's taking to import old data, how many datasets are you > exporting, and what retention policy do you have assigned to them? > > For comparison, we have a machine here - a couple of years old - that we use > for performance testing. It has 1500 devices at a 30-second poll, 60000 > interfaces and 8500 exported datasets. Starting with a fresh database, it > processes roughly 3-4 weeks of historical data per day. we're connecting with only one server, which has 2547 devices today, some poll at 30 seconds, more poll at 1 or 2 minutes. we've got some chart data that goes back to Dec. 2006. most charts start in 2008 or later. I'm not sure how to get Intermapper server to tell me about interfaces and datasets. When I try and get IMDC reports to "check all" everything to tell me, Safari starts to warn me about slow scripts and it takes a long time to see anything. We noticed this kind of behavior when we try and query it too. If there are tuning things we can do to make it more responsive, that would be great. I just tried it in FireFox 3.6.19 instead, and it doesn't draw the screens right in the popup windows, so the "check all" and "uncheck all" overlap the device list and are tricky to select (Mac OS 10.6.8). When I finally get them all checked, it just says "All Devices" "All Interfaces" etc. so I can't tell. IMDC version 5.4.2 is running on a separate server from the actual monitoring station (it brought monitoring down when we tried putting them together a year or so ago) and is on a dedicated VM with 1 core and 4G of RAM, and 2T of SAN-attached disk. If we try to query the database it tends to run the load average up to 6 or 7, so we're thinking about asking the system people for a second core on the VM, but if it shouldn't do this, that would be interesting to know. my typical log file entries look like this: 2011-07-14 09:58:58 CONN IMPORT (netmon9): Import complete for current reports (114578 rows, 3737551 bytes, 107.779224s elapsed) 2011-07-14 10:04:17 CONN IMPORT (netmon9): Import complete for old reports (293512 rows, 3915247 bytes, 319.090344s elapsed) 2011-07-14 10:06:23 CONN IMPORT (netmon9): Import complete for current reports (123146 rows, 3863440 bytes, 125.712027s elapsed) 2011-07-14 10:11:44 CONN IMPORT (netmon9): Import complete for old reports (293624 rows, 3917669 bytes, 320.807978s elapsed) the policy on the IMDC side is: original data 2 weeks 5-minute samples 1 month Hourly samples 2 years daily samples forever we will adjust it once we get all the data imported, but we were hoping to get some granular reports for the last 2 years to make comparisons with before setting the hourly same data down smaller. -debbie > > That machine does have the advantage of not doing much besides IM & IMDC, so > real-world performance will be less. But unless something else on the > machine is hitting the disk, that's roughly what we expect for that number of > datasets. If your experience is way off that, please let me know. Thanks, > > David > > > Debbie Fligor wrote: >> In reading the release notes to finally get upgraded from 5.3 to 5.4, I >> noticed that 5.4.4 fixes a problem where import thinks that there are >> duplicate data entries when DaylightSavings time shifts. Do I need to >> delete my data-base and start the import again? > > > ----------- > David Schnur > Dartware, LLC > > > -------------------- m2f -------------------- > > Read this topic online here: > http://forums.dartware.com/viewtopic.php?p=3551#3551 > > > > > > ____________________________________________________________________ > List archives: > http://www.mail-archive.com/intermapper-talk%40list.dartware.com/ > To unsubscribe: send email to: [email protected] > ----- -debbie Debbie Fligor, n9dn Network Engineer, CITES, Univ. of Il email: [email protected] <http://www.uiuc.edu/ph/www/fligor> "My turn." -River Tam ____________________________________________________________________ List archives: http://www.mail-archive.com/intermapper-talk%40list.dartware.com/ To unsubscribe: send email to: [email protected]
