Bug#929048: tracker-extract: Allocates between 5 and 10 GiB of memory when examining certain DDS files

[Andreas Henriksson]

Control: tags -1 + unreproducible

Hello Bálint Kovács,

Thanks for your bug report. I'm not able to reproduce this issue.
Possibly I don't have things installed that's actually used to
parse this file format. (The bug is likely in some underlying
library for the file format rather than in tracker(-extract) itself.)
Would be great if you could gather some more detailed debug information
as your current info is apparently not enough to reproduce the issue.
I'd suggest you install dbgsym packages and run gdb on the
tracker-extract process and try to break it and see where it's at, then
continue running, then breaking again repeat a number of times and
make sure you have all the dbgsym packages installed.

The output I get below FWIW.

On Thu, May 16, 2019 at 01:42:44AM +0100, Bálint Kovács wrote:
> Package: tracker-extract
> Version: 2.1.6-1
> Severity: critical
> Justification: breaks the whole system
[...]
> Full output from an affected file:
> 
> $ /usr/lib/tracker/tracker-extract -v 2 -f bad.dds
> 00:57
> ** Message: 00:57:52.901: Starting tracker-extract 2.1.6
> ** Message: 00:57:52.901: General options:
> ** Message: 00:57:52.901:   Verbosity    2
> ** Message: 00:57:52.901:   Sched Idle  ...  1
> ** Message: 00:57:52.901:   Max bytes (per file)  .  1048576
> (tracker-extract:9171): dconf-DEBUG: 00:57:52.901: watch_established:
> "/org/freedesktop/tracker/extract/" (establishing: 1)
> Setting scheduler policy to SCHED_IDLE
> Setting priority nice level to 19
> Loading extractor rules... (/usr/share/tracker-miners/extract-rules)
> Extractor rules loaded
> MIME type guessed as 'image/x-dds' (from GIO)
> ../../../glib/gmem.c:105: failed to allocate 65687 bytes
> 
[...]

For me the same commands finishes in about no time and without putting
any memory pressure on the system at all.

$ /usr/lib/tracker/tracker-extract -v 2 -f /tmp/dds_examples/bad.dds 
** Message: 16:40:49.858: Starting tracker-extract 2.1.6
** Message: 16:40:49.858: General options:
** Message: 16:40:49.858:   Verbosity    2
** Message: 16:40:49.858:   Sched Idle  ...  1
** Message: 16:40:49.858:   Max bytes (per file)  .
1048576
Setting scheduler policy to SCHED_IDLE
Setting priority nice level to 19
(tracker-extract:3683): dconf-DEBUG: 16:40:49.858: watch_established:
"/org/freedesktop/tracker/extract/" (establishing: 1)
Loading extractor rules... (/usr/share/tracker-miners/extract-rules)
Extractor rules loaded
MIME type guessed as 'image/x-dds' (from GIO)
@prefix rdf:  .
@prefix nmm:  .
@prefix nfo: 
.

 a nfo:Image , nmm:Photo .

Regards,
Andreas Henriksson

Bug#929048: tracker-extract: Allocates between 5 and 10 GiB of memory when examining certain DDS files

[Bálint Kovács]

Two more things:

1. I have a few more samples in both categories, if needed

2. I do realize that this bears resemblance to #679581. However, there the 
offending process was different.

On 16 May 2019 01:42:44 BST, "Bálint Kovács"  wrote:
>Package: tracker-extract
>Version: 2.1.6-1
>Severity: critical
>Justification: breaks the whole system
>
>Dear Maintainer,
>
>   * What led up to the situation?
>I have been extracting resources from a game. When I extracted a
>handful of DDS
>files, my memory usage shot up and in about 5 seconds the entire system
>locked
>up, no moving to virtual consoles, no nothing. This kept happening,
>every time
>I logged into my profile. To investigate I killed all tracker-related
>processes.
>
>   * What exactly did you do (or not do) that was effective (or
> ineffective)?
>I have been able to identify that the misbehaving process is
>tracker-extract. I
>started running tracker-extract with an rlimit on the DDS files.
>
>   * What was the outcome of this action?
>Some files worked fine, some failed under an RLIMIT_AS of 5 GiB but not
>under
>an RLIMIT_AS of 10 GiB (but it still allocated a not insignificant
>amount of
>memory)
>
>Full output from an affected file:
>
>$ /usr/lib/tracker/tracker-extract -v 2 -f bad.dds
>00:57
>** Message: 00:57:52.901: Starting tracker-extract 2.1.6
>** Message: 00:57:52.901: General options:
>** Message: 00:57:52.901:   Verbosity    2
>** Message: 00:57:52.901:   Sched Idle  ...  1
>** Message: 00:57:52.901:   Max bytes (per file)  . 
>1048576
>(tracker-extract:9171): dconf-DEBUG: 00:57:52.901: watch_established:
>"/org/freedesktop/tracker/extract/" (establishing: 1)
>Setting scheduler policy to SCHED_IDLE
>Setting priority nice level to 19
>Loading extractor rules... (/usr/share/tracker-miners/extract-rules)
>Extractor rules loaded
>MIME type guessed as 'image/x-dds' (from GIO)
>../../../glib/gmem.c:105: failed to allocate 65687 bytes
>
>   * What outcome did you expect instead?
>Full output from an unaffected file:
>
>$ /usr/lib/tracker/tracker-extract -v 2 -f good.dds
>00:57
>** Message: 00:57:46.382: Starting tracker-extract 2.1.6
>** Message: 00:57:46.382: General options:
>** Message: 00:57:46.382:   Verbosity    2
>** Message: 00:57:46.382:   Sched Idle  ...  1
>** Message: 00:57:46.382:   Max bytes (per file)  . 
>1048576
>(tracker-extract:9113): dconf-DEBUG: 00:57:46.382: watch_established:
>"/org/freedesktop/tracker/extract/" (establishing: 1)
>Setting scheduler policy to SCHED_IDLE
>Setting priority nice level to 19
>Loading extractor rules... (/usr/share/tracker-miners/extract-rules)
>Extractor rules loaded
>MIME type guessed as 'image/x-dds' (from GIO)
>@prefix rdf:  .
>@prefix nmm:  .
>@prefix nfo:
> .
>
> a nfo:Image ,
>nmm:Photo .
>
>   * Other information
>All of the files work with imagemagick display perfectly well, they are
>even
>the same resolution.
>
>To remedy the situation, I have deleted all affected DDS files, but I
>have kept
>a tarball of an affected and an unaffected sample (along with one that
>is not
>recognized for some reason.) I have attached it to my report. Handle
>with care,
>especially if you have a desktop with tracker on it.
>
>-- System Information:
>Debian Release: buster/sid
>  APT prefers testing
>APT policy: (990, 'testing'), (500, 'testing-debug'), (500, 'unstable')
>Architecture: amd64 (x86_64)
>Foreign Architectures: i386
>
>Kernel: Linux 4.19.0-4-amd64 (SMP w/4 CPU cores)
>Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
>Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
>LANGUAGE=en_US:en (charmap=UTF-8)
>Shell: /bin/sh linked to /bin/dash
>Init: systemd (via /run/systemd/system)
>LSM: AppArmor: enabled
>
>Versions of packages tracker-extract depends on:
>ii  dconf-gsettings-backend [gsettings-backend]  0.30.1-2
>ii  libc62.28-10
>ii  libcue2  2.2.1-2
>ii  libexempi8   2.5.0-2
>ii  libexif120.6.21-5.1
>ii  libflac8 1.3.2-3
>ii  libgexiv2-2  0.10.9-1
>ii  libgif7  5.1.4-3
>ii  libglib2.0-0 2.58.3-1
>ii  libgsf-1-114 1.14.45-1
>ii  libgstreamer-plugins-base1.0-0   1.14.4-1
>ii  libgstreamer1.0-01.14.4-1
>ii  libgxps2 0.3.1-1
>ii  libicu63 63.1-6
>ii  libiptcdata0 1.0.5-2.1
>ii  libjpeg62-turbo  1:1.5.2-2+b1
>ii