ok, I've added a commit with a #define TIFF_MAX_DIR_COUNT 1048576
Le 12/12/2022 à 20:50, Christoph Gohlke a écrit :
This is great! Thank you. I was able to read a TIFF file with 100,000
pages. It's not uncommon to find such files in bio-imaging.
Could the 1048576 limit be added to the public API, e.g. add `#define
TDIR_MAX 1048576` in tiffio.h?
Christoph
On 12/12/2022 10:31 AM, Even Rouault wrote:
Well, so fixing that particular issue was not too hard, but I then
discovered that TIFFSetDirectory() that used to be fast before was
slow now due to IFD looping detection having been added. So I've
decided to bite the bullet and add a hashset/hashmap based detection
of IFD (luckily I had a C one at hand), which considerably speeds up
things, and also bump tdir_t type to uint32_t. So now the limit is
raised to 1048576 IFDs (could be larger, but as this is untested
area, I preferred to enforce a not too big limit).
Now on your test file, one can loop through all directories beyond
the 65535th one with TIFFReadDirectory() and seek to an arbitrary one
with TIFFSetDirectory().
Work in https://gitlab.com/libtiff/libtiff/-/merge_requests/436 .
Please test
Le 11/12/2022 à 23:34, Christoph Gohlke a écrit :
One way to reproduce this regression is:
```
#include "tiffio.h"
int main()
{
TIFF* tiff = TIFFOpen("gray.movie.u2.tif", "r");
TIFFSetDirectory(tiff, 65534);
TIFFSetDirectory(tiff, 0);
}
```
This passes for me on Windows with libtiff 4.4 and raises an error
and prints the warning with 4.5.
I'm OK if this is not considered a blocker and can just reduce the
maximum supported IFDs in the imagecodecs library by one.
Christoph
On 12/11/2022 1:04 PM, Even Rouault wrote:
I've just skimmed through Debian backported patches on top of 4.1.0
referenced at https://packages.ubuntu.com/focal/libtiff-dev
(http://archive.ubuntu.com/ubuntu/pool/main/t/tiff/tiff_4.1.0+git191117-2ubuntu0.20.04.7.debian.tar.xz)
and none of them are IFD looping detection related.
Anyway, given the following test program
"""
int main()
{
TIFF* tiff =
TIFFOpen("/home/even/Téléchargements/gray.movie.u2.tif", "r");
int incr = 1;
while(TIFFReadDirectory(tiff))
{
incr ++;
}
printf("%d\n", incr);
}
""",
with 4.4.0 I get
TIFFCheckDirOffset: Cannot handle more than 65535 TIFF directories.
65535
and with master I get:
_TIFFCheckDirNumberAndOffset: Cannot handle more than 65535 TIFF
directories.
TIFFReadDirectory: Warning, Didn't read next directory due to IFD
looping at offset 0xa7ff1e (11009822) to offset 0xa7ffc4 (11009988).
65535
So except the extra error message that is a bit misleading, we
didn't really regress. I'd be tempted to release with the current
state of master.
Le 11/12/2022 à 21:43, Bob Friesenhahn a écrit :
Before people get too excited about this issue, I should mention
that it appears that the libtiff provided with Ubuntu 20.04
already produces the same error condition.
This means that either the issue was in libtiff before, or else
Ubuntu applied patches to introduce it.
Bob
On Sun, 11 Dec 2022, Sulau wrote:
Hi Christoph
Is the actual issue having more than 65534 IFDs in the file or
just the error message when changing from IFD number 65534 to IFD
number 0? I understood that more than 65534 IFDs (even more than
65535) are to be addressed.
Best regards
Su
-----Ursprüngliche Nachricht-----
Von: Tiff [mailto:[email protected]] Im Auftrag von Sulau
Gesendet: Sonntag, 11. Dezember 2022 20:58
An: 'Even Rouault' <[email protected]>; 'Bob
Friesenhahn' <[email protected]>; 'Christoph Gohlke'
<[email protected]>
Cc: [email protected]
Betreff: Re: [Tiff] libtiff v4.5.0 release candidate available
Hi,
I had a quick look at it.
Within libitff, the number of directories / directory number is
stored in an uint16_t and the number 65535 was treated as a magic
number for a "non-existing directory"! Thus, libtiff can only
handle 65534 directories.
However, many libtiff functions do not depend on the directory
number. Therefore, it is in principle possible to handle TIFF
files with more than 65535 IFDs if none of these functions that
use the directory number are used.
Like the test file gray.movie.u2.tif containing 65567 pages.
Before version 4.5.0, the existing IFD loop check was
ineffective. Therefore, no warnings occur for test files with
more than 65534 directories.
Since the directory number is contained as a uint16_t in public
libtiff interfaces, I see no quick solution at the moment other
than to disable the IFD loop check again.
I will setup a gitlab issue.
Su
-----Ursprüngliche Nachricht-----
Von: Tiff [mailto:[email protected]] Im Auftrag von
Even Rouault
Gesendet: Sonntag, 11. Dezember 2022 17:57
An: Bob Friesenhahn <[email protected]>; Christoph
Gohlke <[email protected]>
Cc: [email protected]
Betreff: Re: [Tiff] libtiff v4.5.0 release candidate available
The unfortunate quadratic performance of IFD looping detection is
not something new. I can reproduce it with older libtiff too.
We'd probably need the equivalent of std::set<uint64_t>
setOfIFDOffsets to have good performance ...
But it is not time to fix that at release candidate stage. The
priority would be to avoid the regression of the error message
when reading the 65535th IFD. I've CC'ed Su Laus who did work in
that area during this dev cycle.
Even
Le 11/12/2022 à 17:37, Bob Friesenhahn a écrit :
"TIFFReadDirectory: Warning, Didn't read next directory due to IFD
looping at offset 0xa7ff1e (11009822) to offset 0x8 (8)."
Error: "Cannot handle more than 65535 TIFF directories"
The test file at
<https://github.com/cgohlke/imagecodecs/blob/master/tests/tiff/gray.m
ovie.u2.tif>
contains 65567 pages.
This test file reveals something else which is concerning. I ran a
profiler on GraphicsMagick while it was reading this file and I see
that 53% of the time is being spent in
_TIFFCheckDirNumberAndOffset().
Most of the time is being spent in the code associated with
"Check if
offset is already in the list".
I assumed that most of the time would be spent in GraphicsMagick
code.
Bob
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
Tiff mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/tiff
_______________________________________________
Tiff mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/tiff
_______________________________________________
Tiff mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/tiff
_______________________________________________
Tiff mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/tiff
_______________________________________________
Tiff mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/tiff
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
Tiff mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/tiff
