[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=359119

Nate Graham  changed:

   What|Removed |Added

 CC||n...@kde.org
 Status|CONFIRMED   |RESOLVED
 Resolution|--- |DUPLICATE

--- Comment #8 from Nate Graham  ---


*** This bug has been marked as a duplicate of bug 400704 ***

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Arthur Țițeică]

https://bugs.kde.org/show_bug.cgi?id=359119

Arthur Țițeică  changed:

   What|Removed |Added

 CC||arthur+...@cloud.titeica.ro

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Alexander Zhigalin]

https://bugs.kde.org/show_bug.cgi?id=359119

--- Comment #7 from Alexander Zhigalin  ---
I'm no longer experiencing this issue.
However, when I do copy a _large_ amount of files my system becomes somewhat
slow for a while and I think this is due to baloo indexing them all.
Is there some kind of a limit to make it work less but longer in background?

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=359119

dow...@hotmail.com changed:

   What|Removed |Added

 CC||dow...@hotmail.com

--- Comment #6 from dow...@hotmail.com ---
I just installed KDE neon 5.9 and can confirm that baloo_file_extractor still
consumes 1.3 Gb ram and excessive IO, to the point of taking 30 seconds or more
to get a login prompt at a console.  This is on a system with 4 Gb ram, 2
cores, 2.16 Ghz.

After leaving it running all night it completed the initial indexing of my 2
Terrabyte hard drive.  But even though the system was now responsive,
baloo_file_extractor soon took off again when I tried to do anything on the
system, and I ended up having to kill and disable it (as I have had to do ever
since it was introduced, on every new install, on every system).

Something appears fundamentally wrong in its design...

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Rainer Sabelka]

https://bugs.kde.org/show_bug.cgi?id=359119

Rainer Sabelka  changed:

   What|Removed |Added

 CC||sabe...@iue.tuwien.ac.at

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Christoph Cullmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=359119

Christoph Cullmann  changed:

   What|Removed |Added

 CC||noahdessa...@gmail.com

--- Comment #5 from Christoph Cullmann  ---
*** Bug 356176 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Christoph Cullmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=359119

Christoph Cullmann  changed:

   What|Removed |Added

 CC||bpgu...@its.caltech.edu

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Christoph Cullmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=359119

--- Comment #4 from Christoph Cullmann  ---
e.g. see conversation on framworks-devel:

Hi,

> Hi Christoph,
> 
> On 10 September 2016 at 23:46, Christoph Cullmann  wrote:
 Would it be a good idea to restrict the file extractor process to some
>>> fixed amount of memory
 to use via setrlimit? (or more fancy stuff?)
>>>
>>> That would probably just make Baloo crash, so fixing the bug is probably
>>> the better option.
>> Actually, that we don't limit the resources + sandbox baloo_file_indexer is 
>> the
>> bug,
>> not that some meta info extractor is buggy (which should be fixed, too).
>>
>> ATM, the state is:
>>
>> 1) baloo is on per default
>> 2) it will index at least your home
>> 3) if it encounters any "bad" file, it will OOM you, in my case in a way
>> that a normal user is doomed, as 1-2 seconds after login the machine is 
>> already
>> halted.
>>
>> Given that e.g. your "Downloads" might even contain "evil" files from the 
>> net,
>> at least some resource limit would be good and even better some sandbox, to
>> avoid that
>> the indexer which is easily pwn'd pwn's your session.
> 
> You've got a point there. In that case, what I'd do is:
> 
> 1) Limit resources on baloo_file_extractor.
> 2) Try to detect if it crashes because it exceeded limits, not sure if
> this is easily possible.
> 3) Mark files causing such crashes as files that should be skipped,
> and the user notified somehow (?).
I think just mark any files as "skipped for the future" for which the indexer
crashs.
(if by hinting resource limit or just other index fail doesn't really matter,
IMHO,
beside that it perhaps should be logged somewhere)

Other problem: after indexer crash, the DB is corrupted or locked.

It seems one not really does any proper lmdb locking, baloo_file even just
kills the lockfile
on startup.

> 
>> Beside that, a real other problem is, that baloo has close to zero error
>> handling for its
>> database, once one error happens, all further things will go down and never
>> recover.
>>
>> e.g. one time balooctl wrongly use => goodbye
>> https://bugs.kde.org/show_bug.cgi?id=368557
>>
>> Interesting too: We use lmdb, which means, we memory map always, aka 32-bit
>> machines will
>> be out of memory if you have large indices like > 2GB :/
> 
> Nah, 32bit machines should have PAE. If they don't... I'm not willing
> to make fundamental changes to how indexes are kept to support edge
> cases like this. Disabling Baloo automatically if you detect machines
> with a 32-bit address space is the way to go.
PAE doesn't help there at all. (it only helps that your system can use more
than 3/4GB,
not one application)

If you use lmdb, and lets say the file is 2GB, your applications 4GB of virtual
spaces is halfed (and even more, as some parts are anyways used otherwise).

Beside, other issue: ATM the index is fixed to max 5GB, after that, all things
will fail,
see bug https://bugs.kde.org/show_bug.cgi?id=364475, as I have seen index sizes
> 2 GB,
that will hit people, too.

We should increase that limit IMHO and out-of-space should be handled at all I
guess.

> 
> I'd still wait for Pinak to comment on all of the above though.
Sure.

Greetings
Christoph

-- 
You are receiving this mail because:
You are watching all bug changes.

[frameworks-baloo] [Bug 359119] baloo_file_extractor out-of-memory handling (resource limits and more)

[Christoph Cullmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=359119

Christoph Cullmann  changed:

   What|Removed |Added

Summary|baloo_file_extractor got|baloo_file_extractor
   |mad and ate 3 gigabytes of  |out-of-memory handling
   |my RAM  |(resource limits and more)
 CC||cullm...@kde.org
 Status|UNCONFIRMED |CONFIRMED
 Ever confirmed|0   |1

-- 
You are receiving this mail because:
You are watching all bug changes.