[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 caulier.gil...@gmail.com changed: What|Removed |Added Status|REPORTED|RESOLVED Version Fixed In||7.6.0 Resolution|--- |FIXED --- Comment #57 from caulier.gil...@gmail.com --- Thanks for your instructive feedback Mark I close this file now. Gilles Caulier -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #56 from Mark Stoughton --- Everything is fine on this end now. As far as I'm concerned this can be closed -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #55 from Mark Stoughton --- I think I fixed it. Two different solutions, one for PC one for Mac. Both versions seem to be running fine now. For the PC -Uninstall and reinstall Digikam. For the Mac - Uninstall and reinstall Digikam - Create a new database form scratch and reimport the photos into discrete albums, one album per folder They were both really slow, but the reinstall worked by itself for the PC, but I'm lost as to why. So it got me thinking about how I built the database each one. n the PC, I imported folders one at a time into discrete albums. On the Mac I did it all at once into one huge master album. The existing database worked on the PC, but not on the Mac. So when I created a new database on the Mac, I imported one folder at a time into discrete albums. Digikam is now working perfectly on both platforms. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 Mark Stoughton changed: What|Removed |Added OS|All |macOS --- Comment #54 from Mark Stoughton --- I tried uninstalling and reinstalling Digikam on both my Mac nd PC. After that, the PC runs normally, and NAS performance is excellent. No change on the Mac, still slow as molasses. I've changed the platform to Mac only. Hope this helps. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #53 from Mark Stoughton --- I've found one more thing. I swapped databases between the one that uses local files and the one that uses files on the NAS. The only difference was the location of the photo files. The databases were both on local folders on the SSD attached to my Mac. Using photo files stored locally: Digicam scans the folders in about 2 seconds and starts normally Using photo files stored on the NAS Digicam scanned for over an hour at 0% before I cancelled it. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 Mark Stoughton changed: What|Removed |Added OS|Linux |All -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #52 from Mark Stoughton --- I ran a few more tests. I think that I've isolated the issue to occur only when photos are stored on the NAS. Test 1: Move the database off the System drive to a Thunderbolt Attached SSD - Photos still on the NAS -Startup time reduced to 21 minutes -Changing folders took 3 minutes each time Test 2: Copied files to the SSD Drive and Created a new database on the SSD Drive -Startup time took about 10 seconds -Changing folders took 3-5 seconds if the folder had no thumbnails, 1 second if it had thumbnails Test 3: Read all database tables using SQL Lite - Nearly instant response in all cases for 118k photo records Test 4: Opening photo files using Finder/Windows Explorer from the NAS - File on SSD - 3 seconds - File on NAS - 3 seconds I can reproduce this behavior on both my Mac and my PC. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #51 from Mark Stoughton --- I ran a few more tests. I think that I've isolated the issue to occur only when photos are stored on the NAS. Test 1: Move the database off the System drive to a USB Attached SSD -Startup time reduced to 21 minutes -Changing folders took 3 minutes each time Test 2: Copied files to the SSD Drive and Created a new database on the SSD Drive -Startup time took about 10 seconds -Changing folders took 3-5 seconds if the folder had no thumbnails, 1 second if it had thumbnails Test 3: Read all database tables using SQL Lite - Nearly instant response in all cases for 118k photo records Test 4: Opening photo files using Finder/Windows Explorer from the NAS - File on SSD - 3 seconds - File on NAS - 3 seconds I can reproduce this behavior on both my Mac and my PC. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #50 from Maik Qualmann --- If they change an album, there is no relevant access to the NAS/collection. All information comes only from the database. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #49 from Mark Stoughton --- (In reply to Maik Qualmann from comment #48) > If you change an album it's a pure database operation. It doesn't matter > whether the album is on a NAS. As a test, I would move the digiKam databases > to a completely different directory. > > Maik Hi Maik, I moved the database to a thunderbolt attached external SSD. Digikam took 21 minutes to load, down from ~40, and the splash screen stayed visiblele the entire time. It still consistently took 3 minutes to change folders, regardless of the number of photos in the folder. I'm backing up the folders on the NAS to a USB hard drive. I'm going to try using a directly attached drive to see if that works. I'm thinking that if that works normally, the issue is somewhere between Digikam and files on the NAS. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #48 from Maik Qualmann --- If you change an album it's a pure database operation. It doesn't matter whether the album is on a NAS. As a test, I would move the digiKam databases to a completely different directory. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #47 from Mark Stoughton --- Hi, That is correct, it takes about 3 minutes to change folders while using the program. While in a single folder the program works fine, but takes about a minute to read the metadata, particularly the EXIF data. I've checked the permissions, and Digikam has RW on all folders on the NAS. My database is on the system drive, which is an SSD. Querying the database using SQLite Studio returns results instantly. Just to investigate further, I duplicated Giles' setup and installed Digikam on a different Mac, put about 1000 photos on the system drive and created a new database on the system drive. Digikam performed flawlessly. This makes me think that there is some issue when files are stored on a NAS. I don't think it's a latency issue because I have been investigating DAM software and have installed several of them (ACDSee, Photo Mechanic, Apple Photos, Adobe Bridge, Luminar) on this machine, and Digikam is the only one that behaves this way. It's unfortunate because Digikam would be my favorite otherwise. The log that I have annotated shows where Digikam pauses during execution, and for how long. It pauses in the same places and for roughly the same amount of time whether monitoring for external changes is on or off. (In reply to Maik Qualmann from comment #44) > I understand you correctly that the change from an album takes 3 minutes > after the start? > This is a database operation, in principle there is no loss of time. You > must have very, very slow access to your database. Since there are no error > messages in the log, the behavior cannot be explained, check the rights as > Gilles already recommends. > > Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #46 from Mark Stoughton --- I checked the rights and digicam does have RW on the entire drive. The only thing that I can think of is that there may be some issue because the files are on a NAS. So this morning I installed it on another Mac, with photos on the system drive, and Digikam works perfectly. Hope this helps -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #45 from Mark Stoughton --- (In reply to Maik Qualmann from comment #44) > I understand you correctly that the change from an album takes 3 minutes > after the start? > This is a database operation, in principle there is no loss of time. You > must have very, very slow access to your database. Since there are no error > messages in the log, the behavior cannot be explained, check the rights as > Gilles already recommends. > > Maik In both cases (Mac and PC), the database is on the system drive, which are SSDs in both cases. The photos themselves are stored on regular hard drives on a NAS. Each folder has anywhere between 100 to 1000 photos in it. When I query the database using SQLite studio, the results come back instantaneously. I can reproduce this exact behavior on both my Mac and PC installs. While the watching Digikam write the log, there are several long pauses as I documented in the first log that I sent. I didn't annotate the second one because the pauses were all in the same places and for similar times. It appears to me that Digikam is taking an abnormally long time to complete whatever it does at those points. I've tried several DAM packages and Digikam is the only one that behaves this way. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #44 from Maik Qualmann --- I understand you correctly that the change from an album takes 3 minutes after the start? This is a database operation, in principle there is no loss of time. You must have very, very slow access to your database. Since there are no error messages in the log, the behavior cannot be explained, check the rights as Gilles already recommends. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #43 from Mark Stoughton --- This is a run with the boxes unchecked. Load time was down to 30 minutes exactly Last login: Thu Feb 10 12:53:11 on ttys000 markstoughton@Marks-Mac-mini ~ % export QT_LOGGING_RULES="digikam*=true" /Applications/digiKam.org/digikam.app/Contents/MacOS/digikam digikam.widgets: Breeze icons resource file found digikam.widgets: Breeze-dark icons resource file found digikam.general: Qt standard translations removed: 0 digikam.general: Qt standard translations path: "/Applications/digiKam.org/digikam.app/Contents/Resources/translations" digikam.general: Loaded Qt standard translations "en_US" from catalog "qt" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtbase" digikam.general: Loaded Qt standard translations "en_US" from catalog "qt_help" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtdeclarative" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtquickcontrols" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtquickcontrols2" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtmultimedia" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtwebengine" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtxmlpatterns" digikam.general: Loaded Qt ECM translations "en" from catalog "kcoreaddons5_qt" digikam.general: Loaded Qt ECM translations "en" from catalog "kwidgetsaddons5_qt" digikam.general: Switch to widget style: "Fusion" digikam.general: AlbumWatch is disabled digikam.general: Database Parameters: Type: "QSQLITE" DB Core Name: "/Users/markstoughton/Pictures/Digikam/digikam4.db" DB Thumbs Name: "/Users/markstoughton/Pictures/Digikam/thumbnails-digikam.db" DB Face Name: "/Users/markstoughton/Pictures/Digikam/recognition.db" DB Similarity Name: "/Users/markstoughton/Pictures/Digikam/similarity.db" Connect Options: "" Host Name: "" Host port: -1 Internal Server: false Internal Server Path: "" Internal Server Admin Cmd: "" Internal Server Serv Cmd: "" Internal Server Init Cmd: "" Username: "" Password: "" digikam.dbengine: Loading SQL code from config file "/Applications/digiKam.org/digikam.app/Contents/Resources/digikam/database/dbconfig.xml" digikam.dbengine: Checking XML version ID => expected: 3 found: 3 digikam.coredb: Core database: running schema update digikam.coredb: Core database: have a structure version 15 digikam.coredb: Core database: makeUpdates 15 to 15 digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2021" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2020" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2019" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2018" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2017" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2016" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2015" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2014" digikam.database: location for "/Volumes/OurPhotos/Photo Library/2021" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2020" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2019" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2018" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2017" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2016" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2015" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2014" is available true digikam.general: KMemoryInfo: Platform identified : "Unknown" digikam.general: KMemoryInfo: TotalRam: -1 digikam.general: Allowing a cache size of 60 MB digikam.thumbsdb: ThumbDB SelectThumbnailSetting val ret = 0 digikam.thumbsdb: ThumbDB SelectThumbnailSetting val ret = 0 digikam.thumbsdb: Thumbs database: have a structure version "3" digikam.general: Thumbnails database ready for use digikam.similaritydb: SimilarityDb SelectSimilaritySetting val ret = 0 digikam.similaritydb: SimilarityDb SelectSimilaritySetting val ret = 0 digikam.s
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #42 from Mark Stoughton --- I unchecked the Settings-Collections I unchecked the box that says "Monitor the ablbums for external changes(requires restart). Is that the right one? If not, please direct me to the right one. Thanks. P.S. The log that I sent was before I unchecked that box. I'm rerunning it now and will post when it's done -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #41 from Maik Qualmann --- I don't think you have the right option unchecked. It was enabled anyway and adding every album to QFilesSystemWatcher is known to cause extreme startup times on MacOS. If the option is deactivated, digiKam must start "quickly" under MacOS. digikam.general: AlbumWatch use QFileSystemWatcher Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #40 from caulier.gil...@gmail.com --- oups, sorry wrong bugzilla entry. Ignore my last comment. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #39 from caulier.gil...@gmail.com --- I use a macbook pro here, with last big sur, and i never seen this dyfunction here. I tried sqlite or mysql internal database, and all work as expected. Of course my disk is a NVME SSD. Image are also located on the system disk. Did you give right to digiKam to read / write on disk where images are stored ? Gilles Caulier -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #38 from Mark Stoughton --- After some use, the album switching got down to. fairly repeatable three minutes, regardless of the number of photos in the album -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #37 from Mark Stoughton --- I deactivated the monitor external changes and restarted. I saw no difference in either startup time (40 minutes) or album switching (15 minutes). -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #36 from Mark Stoughton --- It was activated. I'll try again after deactivating it (In reply to Maik Qualmann from comment #34) > I assume you have activated the option to monitor for external changes in > the digiKam settings under Collection. This option must be deactivated under > MacOS, as it causes the albums to load extremely slowly. > > Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #35 from Mark Stoughton --- I ran the debug session. I've annotated it because digicam stopped for several minutes in different places along the way, and the log does not show the pauses. It took 41 minutes from start to finish markstoughton@Marks-Mac-mini ~ % export QT_LOGGING_RULES="digikam*=true" /Applications/digiKam.org/digikam.app/Contents/MacOS/digikam digikam.widgets: Breeze icons resource file found digikam.widgets: Breeze-dark icons resource file found digikam.general: Qt standard translations removed: 0 digikam.general: Qt standard translations path: "/Applications/digiKam.org/digikam.app/Contents/Resources/translations" digikam.general: Loaded Qt standard translations "en_US" from catalog "qt" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtbase" digikam.general: Loaded Qt standard translations "en_US" from catalog "qt_help" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtdeclarative" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtquickcontrols" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtquickcontrols2" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtmultimedia" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtwebengine" digikam.general: Loaded Qt standard translations "en_US" from catalog "qtxmlpatterns" digikam.general: Loaded Qt ECM translations "en" from catalog "kcoreaddons5_qt" digikam.general: Loaded Qt ECM translations "en" from catalog "kwidgetsaddons5_qt" digikam.general: Switch to widget style: "Fusion" digikam.general: AlbumWatch use QFileSystemWatcher digikam.general: Database Parameters: Type: "QSQLITE" DB Core Name: "/Users/markstoughton/Pictures/Digikam/digikam4.db" DB Thumbs Name: "/Users/markstoughton/Pictures/Digikam/thumbnails-digikam.db" DB Face Name: "/Users/markstoughton/Pictures/Digikam/recognition.db" DB Similarity Name: "/Users/markstoughton/Pictures/Digikam/similarity.db" Connect Options: "" Host Name: "" Host port: -1 Internal Server: false Internal Server Path: "" Internal Server Admin Cmd: "" Internal Server Serv Cmd: "" Internal Server Init Cmd: "" Username: "" Password: "" digikam.dbengine: Loading SQL code from config file "/Applications/digiKam.org/digikam.app/Contents/Resources/digikam/database/dbconfig.xml" digikam.dbengine: Checking XML version ID => expected: 3 found: 3 digikam.coredb: Core database: running schema update digikam.coredb: Core database: have a structure version 15 digikam.coredb: Core database: makeUpdates 15 to 15 digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2021" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2020" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2019" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2018" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2017" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2016" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2015" digikam.database: Creating new Location "/" uuid "networkshareid:?mountpath=/Volumes/OurPhotos/Photo Library/2014" digikam.database: location for "/Volumes/OurPhotos/Photo Library/2021" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2020" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2019" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2018" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2017" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2016" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2015" is available true digikam.database: location for "/Volumes/OurPhotos/Photo Library/2014" is available true digikam.general: KMemoryInfo: Platform identified : "Unknown" digikam.general: KMemoryInfo: TotalRam: -1 digikam.general: Allowing a cache size of 60 MB digikam.thumbsdb: ThumbDB SelectThumbnailSetting val ret = 0 digikam.thumbsdb: ThumbDB SelectThumbnailSetting val ret = 0 digikam.thumbsdb: Thumbs database: have a structure version "3" digikam.general: Thumbnails database ready for use digikam.similaritydb: SimilarityDb SelectSimilaritySetting val re
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #34 from Maik Qualmann --- I assume you have activated the option to monitor for external changes in the digiKam settings under Collection. This option must be deactivated under MacOS, as it causes the albums to load extremely slowly. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #33 from Mark Stoughton --- sorry for the slow reply. It's 10 minutes after the smash screen disappears. The app shows up as non-responsive in Task manager in Windows or in Force Quit on the Mac. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #32 from Mark Stoughton --- Hi, It's from the time the splash screen disappears until the interface shows up. During that time there is no indicator that Digikam is even running at all. The photos were already imported. It shows up as "not responding" in my Mac Force Quit Applications box or Windows Task Manager. It also takes several minutes of appearing to be frozen when I switch between folders. As long as I'm working in one folder, the performance is slow but useable. The minute I switch folders, I have to wait another 10 minutes or so for Digikam to wake up again. Latency on my network to the SAN is 0.5ms. Throughput is about 100MB/s. I don't have tools to test latency installed right now. I also have ACDSee installed on both the Mac and PC, and neither of those exhibits this issue. From: MarcP Sent: Saturday, January 22, 2022 1:23 PM To: geoph...@hotmail.com Subject: [digikam] [Bug 405235] Slow startup and new file scan on low latency networks https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #31 from MarcP --- Hi Mark, you mean 10-minutes since the digikam splash disappears and the interface shows up, or until the progress bar reaches 100%? Are the pictures in your library already imported or it's the first scan? I'm now in a high latency network (~120ms to my NAS), and it's not nearly as bad as it was with v6.x versions. The interface shows up rather quickly and it's true that the initial scan can take a while, but if there aren't many new pictures it's just a matter of minutes. -- You are receiving this mail because: You are on the CC list for the bug. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #31 from MarcP --- Hi Mark, you mean 10-minutes since the digikam splash disappears and the interface shows up, or until the progress bar reaches 100%? Are the pictures in your library already imported or it's the first scan? I'm now in a high latency network (~120ms to my NAS), and it's not nearly as bad as it was with v6.x versions. The interface shows up rather quickly and it's true that the initial scan can take a while, but if there aren't many new pictures it's just a matter of minutes. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #30 from Maik Qualmann --- Create a start log, under Windows with the program DebugView from Microsoft, under Mac in the terminal, each with an active Qt-Debug environment variable. The procedure is described here: https://www.digikam.org/contribute/ Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 Mark Stoughton changed: What|Removed |Added Version|7.4.0 |7.5.0 CC||geoph...@hotmail.com --- Comment #29 from Mark Stoughton --- I am seeing this on the 7.5.0 install. I have my photos stored on a Synology NAS in a shared set of folders. They contain approximately 50,000 photos. It takes 10-20 minutes for DigiKam to start up. The splash screen shows for about 20 seconds, then disappears. In fact, I started Digikam, went and got a cup of coffee, logged into the bugs site, searched for the problem, found this bug and am making this entry while waiting. I'm seeing this behavior on both my Mac and Windows installs. I'm willing to do whatever logging that you require, and to run experiments if you want. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #28 from Maik Qualmann --- Git commit 6545c2e25eb5f5104d2069f34ccc8b2d5d094273 by Maik Qualmann. Committed on 06/09/2021 at 17:50. Pushed by mqualmann into branch 'master'. added fast scan as an option in the settings M +2-1core/libs/database/collection/collectionscanner_scan.cpp M +3-0core/libs/metadataengine/engine/metaenginesettingscontainer.cpp M +1-0core/libs/metadataengine/engine/metaenginesettingscontainer.h M +47 -14 core/utilities/setup/setupmisc.cpp https://invent.kde.org/graphics/digikam/commit/6545c2e25eb5f5104d2069f34ccc8b2d5d094273 -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #27 from Maik Qualmann --- Hmm, yes I can reproduce, e.g. change an image with Gimp. For programs that do not create temporary files to change the image, the album modification date remains the same. Let's see, either the quick scan is optional, which detects new, deleted and renamed items, or I can think of something else. ((:-)) Even in a slower scan, at least the halving of the scan time would be retained. However, you must also note that with the old scan method, items whose modification date was reset (for example another digiKam instance with the active option in the metadata settings), possibly no change was detected either. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #26 from MarcP --- Hi Maik. Just one note. Yesterday I added a bunch of tags to the pictures of an album using Digikam 7.3 in another computer. Today when I started the flatpak preview of flatpak 7.4, it did not detect these changes, so I had to manually re-read the metadata from that album. Maybe the new system of storing the last modified date did not work properly this time? Does it check the last modified date of each individual picture or for each folder? Checking a random picture on the affected album, I see that the last modified date for the picture was yesterday (correct), but for the album still says 20/08/2021. If I am not mistaken, the folder modified date will only change if items are added or removed from it, but not if an existing file is modified. Am I right? -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #25 from Maik Qualmann --- But it must also be said that having to read all of the metadata will not speed up the first scan of a collection. For this I would recommend a NAS that can also be connected via USB. It is now very easy to switch the collection to network later. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #24 from caulier.gil...@gmail.com --- I agree, Maik has done a wonderful optimization in source code. Congratulations... Gilles -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #23 from MarcP --- After trying thursday's build (Rev.: 61cee1cc9536b88b073acce47d37dc85bb1bbfdd), the initial scan was further reduced to 2 minutes and 29 seconds. That's only 17% of the time it took two weeks ago (14m20s), so more than 5 times quicker. I'm impressed! -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #22 from Maik Qualmann --- Git commit 9a443ffcae0f417d355a16e5040225f3a8e7075a by Maik Qualmann. Committed on 18/08/2021 at 17:33. Pushed by mqualmann into branch 'master'. use album date cache to reduce file reading Related: bug 437771, bug 431037, bug 265241 M +2-0core/libs/database/collection/collectionscanner_p.h M +37 -3core/libs/database/collection/collectionscanner_scan.cpp M +1-1core/libs/database/utils/scan/scancontroller_progress.cpp M +10 -2core/libs/database/utils/scan/scancontroller_start.cpp https://invent.kde.org/graphics/digikam/commit/9a443ffcae0f417d355a16e5040225f3a8e7075a -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #21 from Maik Qualmann --- In the first section we look for possible deleted albums or whether digiKam albums have just been moved. That means we read the database and look directly in the collection to see whether the folder still exists, if not it is deleted from the database. This process is currently not counted in the progress display. This is so fast on local drives that it is unnecessary. It would also only effect a few percent in the ad, because, for example, with 5,000 albums against 100,000 images, it would be around 5 percent. In the second section, the change date of the albums is now determined and compared with the DB and a scan is initiated if necessary; this is recorded by the progress display. It is not so easy to combine both sections. A cache might still be possible so that we don't have to look twice at the network collection folders ... Comparing how fast console tools are when listing a network collection, I think we are now almost at the end of the optimization. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #20 from MarcP --- By the way, when the initial scan starts, it stays for a while at 0%, before the progress bar starts to move towards the 100%. I must measured it, from 0% to 1%: 2m26s >From 1% to 100%: 2m16s That's about half of the time. What is digikam doing during that period of time? I noticed that wait is also longer the higher the latency in the network. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #19 from MarcP --- Hi again, I could not find to run digikam in debug mode using the flatpak version, but I measured the date when I launched it and when it finished the initial scan. After upgrading to today's version (with no new pictures): First run: 10 min and 42 seconds Second run: 4 minutes and 43 seconds. I don't know if the first time was a bit slower because it had to populate the database with the new field (10 minutes instead of the 7 and a half that took last week), but the second run was even faster than that, so, so far so good! -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #18 from Maik Qualmann --- By the way, digiKam print the scan time in the terminal with active debug variable. digikam.database: Complete scan took: 22110 msecs. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #17 from MarcP --- Ok! I'll run some tests this week and get back to you. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #16 from Maik Qualmann --- Hi Marc, Feedback welcome again. The first scan with this patch will not change the scan time, as the database must first be filled with the albums modification date. The next digiKam starts will be interesting. Please test whether changes are still recognized in the collection after the start. My initial scan time in this setup is now 22 seconds. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #15 from Maik Qualmann --- Git commit d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6 by Maik Qualmann. Committed on 16/08/2021 at 20:09. Pushed by mqualmann into branch 'master'. use album modification date to speed up the initial scan Database update from V13 to V14 is carried out, the column "modificationDate" is added to the Albums table. Related: bug 437771, bug 431037, bug 265241 M +14 -4core/data/database/dbconfig.xml.cmake.in M +1-1core/libs/database/collection/collectionscanner.h M +5-0core/libs/database/collection/collectionscanner_p.cpp M +53 -16 core/libs/database/collection/collectionscanner_scan.cpp M +45 -0core/libs/database/coredb/coredb.cpp M +19 -1core/libs/database/coredb/coredb.h M +9-1core/libs/database/coredb/coredbschemaupdater.cpp https://invent.kde.org/graphics/digikam/commit/d14487fc0da4e26bc7ba2a82730b7a36d1c4cff6 -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #14 from Maik Qualmann --- This means that the percentage display in the status bar, depending on whether images have been removed or added, e.g. jumps from 90% to 100% or lingers on 100% for a while before it is finished. If you have a large collection with 100,000 images, you will probably not notice anything, even if 1,000 images have been removed or added. The next step I plan to do is to monitor the folder modified date instead of always checking all images for the modified date. This will, when implemented, mean a database update. However, the first tests also show that it does not bring as much as hoped, the current scan code is already quite highly optimized, we will see. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #13 from MarcP --- Today I updated the flatpak version (v.: 03f233383d94c20099afe07e75edb727694e5c3c) and tried again: Same library, same network, no new items: 7 minutes and 37 seconds. That's about half the time it took yesterday! Good work! When you said the new method is less precise, what does it mean exactly? What kind of inconsistencies can I expect? -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #12 from MarcP --- Ok, so, as a reference, version 7.4.0 rev 90854f4a4e9e06ed2f1196afbee0b92fe5bf9fe8 took 14 minutes and 20 seconds since it was started until it finished checking for changes (there were none since I already started it a few minutes earlier). My library right now comprises around 13 pictures, connects to a NAS via internet through a VPN (two-way 100mbps) and a latency of ~41ms. I will try these days if I see any changes. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #11 from MarcP --- I'll test with my collections these days and provide some feedback as well. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #10 from Maik Qualmann --- As a test environment, I use my Fritz-Box router with a connected 2.5 inch WD hard drive via USB with 100Mb/s LAN network. Mounted via Cifs-SMB. So not really a fast NAS. There are about 30,000 images on the hard drive. Scan before: 268 seconds Scan now: 170 seconds Feedback is welcome. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #9 from Maik Qualmann --- Git commit 03f233383d94c20099afe07e75edb727694e5c3c by Maik Qualmann. Committed on 09/08/2021 at 19:29. Pushed by mqualmann into branch 'master'. first step for a faster scan of network drives We only run through a network collection once in the first scan to determine the number of files. The possible number of images and albums is now taken from the database. This can be a little less precise under certain circumstances, we will test it. Related: bug 437771, bug 431037, bug 265241 M +1-1core/libs/database/collection/collectionscanner.h M +11 -13 core/libs/database/collection/collectionscanner_scan.cpp M +25 -4core/libs/database/collection/collectionscanner_utils.cpp M +33 -0core/libs/database/coredb/coredb.cpp M +5-0core/libs/database/coredb/coredb.h https://invent.kde.org/graphics/digikam/commit/03f233383d94c20099afe07e75edb727694e5c3c -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 caulier.gil...@gmail.com changed: What|Removed |Added Version|6.0.0 |7.4.0 -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #8 from MarcP --- I believe nothing has changed on this regards. The total startup time still heavily depends on the latency of the medium where pictures are stored. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #7 from caulier.gil...@gmail.com --- With next digiKam 7.4.0 release, AppImage bundle is compiled using a more recent Linux Mageia 7.1 host. Last stable Qt 5.15.2 and KF5 5.84 are used. ImageMagick codec 7 and libav 58 (ffmpeg) are used to supports extra image and video formats. https://i.imgur.com/XV1tZkL.png Please check if problem still reproducible with this version available as pre-release here: https://files.kde.org/digikam/ Gilles Caulier -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #6 from MarcP --- Ok, so I experienced this problem when I lived abroad, and had a ~100ms ping, but it's no longer the case, so I tried to find ways to assess startup and scanning speed by simulating high network latencies. So I used this command to add a 100ms delay to my computer's wifi and replicate the results: # tc qdisc add dev eth0 root netem delay 100ms The good news: digikam takes as long to show the interface in a slow network as in a fast one (so, not super fast, but acceptable). However, once the interface has loaded, the initial scan for changes is still quite slow. I am just testing it, and it took me around 45 minutes to finish scanning (~10 photos). But that's much faster than the 12 hours it took before. I also think it's remarkable that, while digikam is scanning for new pictures, the interface is responsive and I can browse albums and tags just fine (solved in bug #389652, but I have not tried if new thumbnails are generated during the initial scan, though). Overall, I would say that 3/4 of the bug has been resolved, as digikam is usable much quicker, and from the start. However, the initial scan is still too dependent on the network latency, as I guess it needs to access every single one of the pictures to check if something has changed. Couldn't that be done in bulk? Or maybe requesting the changes to the filesystem instead of checking every file individually (I'm not sure if that's possible, I'm just wondering). So, if you want to close this bug, it's fine for me, there have been big improvements since I opened it. The only issue is how long it takes to scan for a large library, specially being dependent on the network latency, but maybe that could be addressed separately. In any case, thanks for your time and effort! -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #5 from caulier.gil...@gmail.com --- https://bugs.kde.org/show_bug.cgi?id=417121 --- Comment #3 from caulier.gil...@gmail.com --- digiKam 7.0.0 stable release is now published: https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/ We need a fresh feedback on this file using this version. Best Regards Gilles Caulier -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #4 from IWBR --- I see. Don't worry about it. It's mostly a network problem. However, as I mentioned, due to the high latency, Digikam thinks the path is not available, although I can browse to that location using Ubuntu's file manager. Btw, checking for the availability of the trash is a smart solution. However, during those periods of "non availability", Digikam freezes the interface and Ubuntu constantly asks me to force close the app (related to bug #389652?). Not even a menu or a picture can be selected, even when Digikam is seemingly doing nothing. I understand if pictures couldn't be opened, but browsing catched thumbnails or opening menus should be possible even if the collection is momentarily non-available (considering the database is stored locally, of course). Maybe I should open a bug just for this issue? Anyway, I understand it's not Digikam's fault that it takes so long to scan new files on a remote location. It's just a bit annoying because the main interface is frozen during that process. Regarding the slow startup, even with new file scan disabled, yep, it can take several minutes even to show the splashscreen and the tray icon, and about 10-12 minutes to show the main interface in my case. Do you think something could be done in this regard? -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #3 from Maik Qualmann --- Another thing is the latency of over 100 ms (10 times an internet connection). Assuming that the query on whether a file exists would take so long, and probably longer, 100,000 files would take over 2.7 hours! DigiKam must also check the file date, if there are side cars and also get the date, etc. I think the 12 hours are then completely normal. I think we can not speed up the scan. The launch of digiKam is another thing under these conditions and I'll take a look. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 Maik Qualmann changed: What|Removed |Added CC||metzping...@gmail.com --- Comment #2 from Maik Qualmann --- If digiKam is instructed externally or internally to scan a folder, we have changed something because of Bug 405764. In order to get a grip on problems with lost network connections under Windows, we checked the location path to see if it still exists. This works fine under Windows but apparently not under Linux. We now check if the trash folder is available (.dtrash). If not, the location list will be updated. When using network location, it only checks if at least one file exists in the location path, if not, we set the location to read only. So your network is not only slow, but also not always available, because a simple QFileInfo::exists() function returns false. I do not see how we can fix it. Maik -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 --- Comment #1 from IWBR --- Just a heads up. Even when the main interface has shown up and new file scan is disabled, the interface freezes intermittently. In the console, I see messages like these every time it happens: Digikam::CollectionManager::locationForAlbumRootPath: Album root path not exist "/media/sshfs/box_ext/path/to/picture/library" Digikam::CollectionManager::locationForAlbumRootPath: Drive or network connection broken? And after a few seconds: Digikam::CollectionManager::updateLocations: location for "/media/sshfs/box_ext/path/to/picture/library" is available true (I can confirm that indeed, that path has been available all that time, albeit the access time latency can range from 100 to 250ms). I know technically it's not Digikam's fault, but I wanted to share this information in case some optimization could be made regarding responsiveness in high latency networks. -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 caulier.gil...@gmail.com changed: What|Removed |Added Component|Database|Database-Scan CC||caulier.gil...@gmail.com -- You are receiving this mail because: You are watching all bug changes.
[digikam] [Bug 405235] Slow startup and new file scan on low latency networks
https://bugs.kde.org/show_bug.cgi?id=405235 IWBR changed: What|Removed |Added CC||iwannaber...@gmail.com -- You are receiving this mail because: You are watching all bug changes.
