Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Dear Nicholas, Le jeudi 4 février 2021, 04:23:36 CET Nicholas D Steeves a écrit : > Norbert Preining writes: > > > Control: tag -1 +unreproducible > > > > Hi Nicholas, > > > > to be clear, tagging this bug as "unreproducible" means that **we** the > > developer cannot reproduce it. So please don't change it. Thanks. > > > > Also, moreinfo is still needed, and you didn't provide it. > > > > I did. The minimum dataset needed to trigger was with an extensive set > of exclusion patterns that result in 436767 indexed files, with the > following results returned for balooctl indexSize > > Actual Size: 3.27 GiB > Expected Size: 2.12 GiB > >PostingDB: 424.52 MiB19.516 % > PositionDB: 845.84 MiB38.884 % > DocTerms: 344.65 MiB15.844 % > DocFilenameTerms: 47.45 MiB 2.181 % >DocXattrTerms:0 B 0.000 % > IdTree: 7.64 MiB 0.351 % > IdFileName: 35.77 MiB 1.645 % > DocTime: 22.29 MiB 1.024 % > DocData: 23.21 MiB 1.067 % >ContentIndexingDB:0 B 0.000 % > FailedIdsDB:0 B 0.000 % > MTimeDB: 2.59 MiB 0.119 % I beat you although by a small margin ;) Baloo File Indexer is not running Total files indexed: 438,883 Files waiting for content indexing: 0 Files failed to index: 0 Current size of index is 4.26 GiB File Size: 4,26 Gio Used: 2,17 Gio PostingDB: 584,37 Mio26.301 % PositionDB: 946,16 Mio42.585 % DocTerms: 499,82 Mio22.496 % DocFilenameTerms: 57,64 Mio 2.594 % DocXattrTerms:0 o 0.000 % IdTree: 9,84 Mio 0.443 % IdFileName: 45,09 Mio 2.029 % DocTime: 28,65 Mio 1.290 % DocData: 43,68 Mio 1.966 % ContentIndexingDB:0 o 0.000 % FailedIdsDB:0 o 0.000 % MTimeDB: 6,57 Mio 0.296 % … and I can’t reproduce the bug either… :( I’m using it for searches on a nearly daily basis and it’s been working reliably for as long as I can remember. Both with file names and file contents. […] > >> Has anyone on the team tried testing baloo with a large $HOME? Mine is > >> 131GB, in 142460 mixed files and directories. At the moment my > >> baloo_file process is consuming over a terabyte (!) of memory. After > >> resetting baloo and forcing a full reindex it takes a couple of weeks > >> before it gets out of control again and crashes. > > > > Well, maybe you should disable the service since it obviously is not > > able to cope with the amount of data/files/size you are asking it to > > index. > > > > Right, that's why I filed this bug. With a non-toy dataset, baloo is > more often than not unusable. Typical lore says to disable baloo and > use the more robust recoll. I'm willing to conceded that it's possible > that there's something special about my data/files, and that they're > causing baloo to fail in a corner-case. If that's the case, what data > do you need? Other than all my files ;-) As said above, this has not been my experience on a definitely non-toy dataset. Maybe there’s something more specific to the crash you experienced that would need investigating before it can be fixed. I had an issue on a a secondary machine where it would crash / stop due to starvation of inotify watches. This was clearly visible in .xsession-errors with the advices to increase fs.inotify.max_user_watches plainy written, and baloo was not the only piece of software having issues in this situation. Probably not your case but I’m mentioning it here just in case. > >> Please don't sweep baloo bugs under the rug by tagging bugs containing > >> crashes with obvious misbehaviour as severity "normal", eg: > > > > Again, since it is not reproducible and nobody else has either reported > > this problem and none of us see this problem, it is normal, would you > > please kindly trust the decision of the maintainers? > > And thanks, you don't have to educate me about severities. > > > > Important: "a bug which has a major effect on the usability of a > package, without rendering it completely unusable to everyone." It is > clear that this bug meets Debian criteria for "important". If no one > else on the team (I'm a team member too) is testing baloo with a medium > size set of diverse file types than no one else one the team will be > able to reproduce it. > > A politic of "normal + unreproducible" is a contributing factor to > why baloo is never fixed. In other words, it's a politic of ignoring > problems in baloo. Given that we failed to reproduce the issue so far, with little indication as to what could actually help reproducing it, this looks really overkill to me. Happy hacking ! -- Aurélien
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Hey, > The crash is not specific to jack_capture_90.mp3; except for a period of > time a couple of weeks long leading up to the original bug. But do you see any pattern? I cannot reproduce it, too. > Has anyone on the team tried testing baloo with a large $HOME? Yes I think myself. Baloo indexed 1.387.086 files (mixed content) ; 650GB source file =>10GB baloo database. According your description of the bug, that is bigger than your dataset and be enough to reproduce. Why do you think we do not use the software they maintain? Baloo is working great for me, I haven't seen issues for a long long time. Okay what can you do to tackle down the bug: * debug symbols for libKF5BalooEngine.so.5 and liblmdb.so.0, so that we know what line triggers the segfault. because the interesting part is without debug symbols. * can you reproducible trigger the crash with jack_capture_90.mp3, or any other file? * did you checked upstream bug tracker - to find any matching bug? As this is an upstream issue, it needs to be fixed by upstream anyways, so we need to clarify if the issue is still at a current version. Otherwise create a bugreport at bugs.kde.org, they should also know better what infos they need to fix the issue. * start a Debian unstable VM/ KDE neon VM and try to reproduce the issue > It could take up to a month to reproduce this bug, and by then the reply will be "ignore for bullseye". This is a contributing factor to why baloo is in perpetual beta (I'd argue alpha, but subjectivity ;-)) Yes I know the timeframe small between the freeze and release to tackle down complicated issues. But please don't forget the other side. What should maintainers do if, they cannot reproduce the issue and upstream does not have stable LTS branches and doing monthly releases? The first thing is checking the current version: If it is fixed, than one can search for the bugfix to backport it or file the bug upstream otherwise. But without checking against the current version upstream reply most of the time: please update first. So I cannot reproduce it nor Norbert. Mark the bug as "unreproducible" is a valid approach, as we cannot tell if it is fixed or not. With more information it may be possible to decide. That's why the question is so important: Can you reproduce it somehow, so we are able to see the same issue? hefee signature.asc Description: This is a digitally signed message part.
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Hi Norbert, Norbert Preining writes: > Control: tag -1 +unreproducible > > Hi Nicholas, > > to be clear, tagging this bug as "unreproducible" means that **we** the > developer cannot reproduce it. So please don't change it. Thanks. > > Also, moreinfo is still needed, and you didn't provide it. > I did. The minimum dataset needed to trigger was with an extensive set of exclusion patterns that result in 436767 indexed files, with the following results returned for balooctl indexSize Actual Size: 3.27 GiB Expected Size: 2.12 GiB PostingDB: 424.52 MiB19.516 % PositionDB: 845.84 MiB38.884 % DocTerms: 344.65 MiB15.844 % DocFilenameTerms: 47.45 MiB 2.181 % DocXattrTerms:0 B 0.000 % IdTree: 7.64 MiB 0.351 % IdFileName: 35.77 MiB 1.645 % DocTime: 22.29 MiB 1.024 % DocData: 23.21 MiB 1.067 % ContentIndexingDB:0 B 0.000 % FailedIdsDB:0 B 0.000 % MTimeDB: 2.59 MiB 0.119 % >> > Can you reproduce this with current frameworks 5.77 in unstable? > > Since you haven't answered this questions, that BTW is alread at 5.78, > we cannot really do anything. > > You don't expect us to try to triage bugs that we cannot reproduce on a > system that is not running the current version? > I will upgrade this system in mid to late February, and hope to be pleasantly surprised that this bug was fixed upstream :-) At that time, how would you like me run baloo to generate a higher quality debugging info? It could take up to a month to reproduce this bug, and by then the reply will be "ignore for bullseye". This is a contributing factor to why baloo is in perpetual beta (I'd argue alpha, but subjectivity ;-)) >> Has anyone on the team tried testing baloo with a large $HOME? Mine is >> 131GB, in 142460 mixed files and directories. At the moment my >> baloo_file process is consuming over a terabyte (!) of memory. After >> resetting baloo and forcing a full reindex it takes a couple of weeks >> before it gets out of control again and crashes. > > Well, maybe you should disable the service since it obviously is not > able to cope with the amount of data/files/size you are asking it to > index. > Right, that's why I filed this bug. With a non-toy dataset, baloo is more often than not unusable. Typical lore says to disable baloo and use the more robust recoll. I'm willing to conceded that it's possible that there's something special about my data/files, and that they're causing baloo to fail in a corner-case. If that's the case, what data do you need? Other than all my files ;-) >> Please don't sweep baloo bugs under the rug by tagging bugs containing >> crashes with obvious misbehaviour as severity "normal", eg: > > Again, since it is not reproducible and nobody else has either reported > this problem and none of us see this problem, it is normal, would you > please kindly trust the decision of the maintainers? > And thanks, you don't have to educate me about severities. > Important: "a bug which has a major effect on the usability of a package, without rendering it completely unusable to everyone." It is clear that this bug meets Debian criteria for "important". If no one else on the team (I'm a team member too) is testing baloo with a medium size set of diverse file types than no one else one the team will be able to reproduce it. A politic of "normal + unreproducible" is a contributing factor to why baloo is never fixed. In other words, it's a politic of ignoring problems in baloo. Cheers, Nicholas signature.asc Description: PGP signature
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Control: tag -1 +unreproducible Hi Nicholas, to be clear, tagging this bug as "unreproducible" means that **we** the developer cannot reproduce it. So please don't change it. Thanks. Also, moreinfo is still needed, and you didn't provide it. > > Can you reproduce this with current frameworks 5.77 in unstable? Since you haven't answered this questions, that BTW is alread at 5.78, we cannot really do anything. You don't expect us to try to triage bugs that we cannot reproduce on a system that is not running the current version? > Has anyone on the team tried testing baloo with a large $HOME? Mine is > 131GB, in 142460 mixed files and directories. At the moment my > baloo_file process is consuming over a terabyte (!) of memory. After > resetting baloo and forcing a full reindex it takes a couple of weeks > before it gets out of control again and crashes. Well, maybe you should disable the service since it obviously is not able to cope with the amount of data/files/size you are asking it to index. > Please don't sweep baloo bugs under the rug by tagging bugs containing > crashes with obvious misbehaviour as severity "normal", eg: Again, since it is not reproducible and nobody else has either reported this problem and none of us see this problem, it is normal, would you please kindly trust the decision of the maintainers? And thanks, you don't have to educate me about severities. Best Norbert -- PREINING Norbert https://www.preining.info Fujitsu Research Labs + IFMGA Guide + TU Wien + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Control: tag -1 -unreproducible -moreinfo Control: severity -1 important Hi Norbert, Norbert Preining writes: > Hi Nicholas, > > On Wed, 09 Oct 2019, Nicholas D Steeves wrote: >> Version: 5.54.0-1 >> Severity: serious >> Justification: poor experience will cause user to give up on baloo; worse >> than GNOME > > Can you reproduce this with current frameworks 5.77 in unstable? > > Does it always crash at the same file (jack_capture_90.mp3) or always in > different files? > The crash is not specific to jack_capture_90.mp3; except for a period of time a couple of weeks long leading up to the original bug. Thank you for following up in this bug (finally someone did!) It's still too early in the freeze for me to upgrade this system to bullseye. Has anyone on the team tried testing baloo with a large $HOME? Mine is 131GB, in 142460 mixed files and directories. At the moment my baloo_file process is consuming over a terabyte (!) of memory. After resetting baloo and forcing a full reindex it takes a couple of weeks before it gets out of control again and crashes. Please don't sweep baloo bugs under the rug by tagging bugs containing crashes with obvious misbehaviour as severity "normal", eg: > id seems to have changed. Perhaps baloo was > not running, and this file was deleted + re-created > mdb.c:3124: Assertion 'pglast <= env->me_pglast' failed in > mdb_freelist_save() which is somewhere between important and serious https://www.debian.org/Bugs/Developer#severities Regards, Nicholas signature.asc Description: PGP signature
Processed: Re: Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Processing commands for cont...@bugs.debian.org: > severity 942078 normal Bug #942078 [baloo-kf5] [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f Severity set to 'normal' from 'serious' > tags 942078 + moreinfo unreproducible Bug #942078 [baloo-kf5] [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f Added tag(s) moreinfo and unreproducible. > thanks Stopping processing here. Please contact me if you need assistance. -- 942078: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942078 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
severity 942078 normal tags 942078 + moreinfo unreproducible thanks Hi Nicholas, I cannot reproduce this, nor anyone else in the team, and you haven't given an update to whether this is reproducible or not on your side. I thus downgrade this bug and tag it appropriately. Thanks for your understanding Norbert On Wed, 23 Dec 2020, Norbert Preining wrote: > Can you reproduce this with current frameworks 5.77 in unstable? > > Does it always crash at the same file (jack_capture_90.mp3) or always in > different files? -- PREINING Norbert https://www.preining.info Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Hi Nicholas, On Wed, 09 Oct 2019, Nicholas D Steeves wrote: > Version: 5.54.0-1 > Severity: serious > Justification: poor experience will cause user to give up on baloo; worse > than GNOME Can you reproduce this with current frameworks 5.77 in unstable? Does it always crash at the same file (jack_capture_90.mp3) or always in different files? Thanks Norbert -- PREINING Norbert https://www.preining.info Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Hi all, On Wed, 09 Oct 2019 18:53:25 -0400 Nicholas D Steeves wrote: > Recently baloo has been crashing whenever I log in. Today I decided it > was a persistent and serious enough problem that a serious bug was > warranted. I've attached the backtrace produced by drkonqi. Here is > some additional info that I hope will help quickly resolve this bug. Any progress on this? The bullseye freeze is approaching. baloo-kf5 is a key package so isn't autoremoved. Hence it's on the Release Team's radar. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Package: baloo-kf5 Version: 5.54.0-1 Severity: serious Justification: poor experience will cause user to give up on baloo; worse than GNOME Hi, Recently baloo has been crashing whenever I log in. Today I decided it was a persistent and serious enough problem that a serious bug was warranted. I've attached the backtrace produced by drkonqi. Here is some additional info that I hope will help quickly resolve this bug. $ balooctl start QCoreApplication::arguments: Please instantiate the QApplication object first QCoreApplication::applicationDirPath: Please instantiate the QApplication object first This process needs a QCoreApplication instance in order to use KCrash "/home/sten/jack_capture_90.mp3" id seems to have changed. Perhaps baloo was not running, and this file was deleted + re-created mdb.c:3124: Assertion 'pglast <= env->me_pglast' failed in mdb_freelist_save() KCrash: crashing... crashRecursionCounter = 2 KCrash: Application Name = baloo_file_extractor path = /usr/bin pid = 15575 KCrash: Arguments: /usr/bin/baloo_file_extractor KCrash: Attempting to start /usr/lib/x86_64-linux-gnu/libexec/drkonqi from kdeinit QSocketNotifier: Invalid socket 9 and type 'Read', disabling... QSocketNotifier: Invalid socket 18 and type 'Read', disabling... QSocketNotifier: Invalid socket 10 and type 'Read', disabling... $ balooctl clear /home/sten/jack_capture_90.mp3 Skipping: /home/sten/jack_capture_90.mp3 Reason: Not yet indexed File(s) cleared $ balooctl index /home/sten/jack_capture_90.mp3 Indexing /home/sten/jack_capture_90.mp3 mdb.c:3124: Assertion 'pglast <= env->me_pglast' failed in mdb_freelist_save() Aborted $ balooctl status Baloo File Indexer is running Indexer state: Indexing file content Indexed 57061 / 57062 files Current size of index is 703.08 MiB $ balooctl indexSize Actual Size: 703.08 MiB Expected Size: 456.43 MiB PostingDB: 111.95 MiB24.527 % PositionDB: 189.31 MiB41.476 % DocTerms: 58.71 MiB12.862 % DocFilenameTerms: 6.20 MiB 1.357 % DocXattrTerms:0 B 0.000 % IdTree: 996.00 KiB 0.213 % IdFileName: 4.65 MiB 1.019 % DocTime: 2.57 MiB 0.563 % DocData: 3.51 MiB 0.769 % ContentIndexingDB: 4.00 KiB 0.001 % FailedIdsDB:0 B 0.000 % MTimeDB: 1.20 MiB 0.263 % ...wait a while, then $ balooctl stop ...wait a while. Baloo fails to stop. $ ps xa | grep /usr/bin/baloo_file $ gdb -p $THAT_PID $ kill $THAT_PID Result: no backtrace $ gdb /usr/bin/baloo_file Starting program: /usr/bin/baloo_file [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x72c7e700 (LWP 17185)] [New Thread 0x7215b700 (LWP 17186)] [Detaching after fork from child process 17187] QCoreApplication::arguments: Please instantiate the QApplication object first QCoreApplication::applicationDirPath: Please instantiate the QApplication object first This process needs a QCoreApplication instance in order to use KCrash "/home/sten/jack_capture_90.mp3" id seems to have changed. Perhaps baloo was not running, and this file was deleted + re-created mdb.c:3124: Assertion 'pglast <= env->me_pglast' failed in mdb_freelist_save() KCrash: crashing... crashRecursionCounter = 2 KCrash: Application Name = baloo_file_extractor path = /usr/bin pid = 17187 KCrash: Arguments: /usr/bin/baloo_file_extractor KCrash: Attempting to start /usr/lib/x86_64-linux-gnu/libexec/drkonqi from kdeinit QSocketNotifier: Invalid socket 9 and type 'Read', disabling... QSocketNotifier: Invalid socket 18 and type 'Read', disabling... QSocketNotifier: Invalid socket 10 and type 'Read', disabling... Result: CRASH. Drkonqi provides a bt for baloo_file_extractor. I've attached that one too. Regards, Nicholas -- System Information: Debian Release: 10.1 APT prefers stable-debug APT policy: (500, 'stable-debug'), (500, 'proposed-updates'), (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.19.72-rt25 (SMP w/4 CPU cores; PREEMPT) Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8), LANGUAGE=en_CA:en (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages baloo-kf5 depends on: ii kio 5.54.1-1 ii libc62.28-10 ii libkf5baloo5 5.54.0-1 ii libkf5balooengine5 5.54.0-1 ii libkf5configcore55.54.0-1+deb10u1 ii libkf5coreaddons55.54.0-1 ii libkf5crash5 5.54.0-1 ii libkf5dbusaddons55.54.0-1 ii libkf5filemetadata3 5.54.0-1 ii libkf5i18n5 5.54.0-1 ii libkf5idletime5 5.54.0-1 ii libkf5kiocore5 5.54.1-1 ii libkf5solid5 5.54.0-1 ii libqt5core5a 5.11.3+dfsg1-1 ii libqt5dbus5 5.11.3+dfsg1-1 ii libqt5gui5
