Hi Bob, Even and Paavo, Thanks a lot for your super quick and helpful replies. đ
@Bob â understood! I was going to create a bug request but then I saw Evenâs reply and started trying out the patches/fixes he mentioned. @Even â You are amazing!! Thank you so much for sharing both the fixes. I applied those in my current version of tiff (4.2.0) and they have fixed the following issues: 1. With the âstrip choppingâ fix, I can see now Windows is mimicking the same behavior as Linux where it is automatically changing the âRowsPerStripâ from 25000 to 1 (for uncompressed files) in my example. It also explains why I saw the same behavior on Windows with tiff version 4.0.0 as Linux because for 4.0.0, we were using NMAKE but we switched to CMAKE (where the bug was) since 4.0.10. 2. With the âlongâ fix, I can now read back the LZW compressed file successfully on Windows so indeed the root cause was âlongâ datatype used in tif_lzw.c which worked on Linux (64 bit) but failed on Windows (32 bit) as Paavo had suspected. Not sure which response you referred to when you said âSee my previous responseâ? Do you mean explanation here: https://www.asmail.be/msg0054930920.html? @Paavo â Thanks again for your explanation. I have now understood the recommended way to write large TIFF files. One last thing remaining for me to investigate and understand that I can observe the changing value of âRowsPerStripâ tag for uncompressed files when I write a simple plain C++ file with tiff dynamic libraries. I am using simple call to TIFFGetField for RowsPerStrip. However, when I place the same call in my application which links against the tiff dynamic libraries on different platforms, I get the value that I wrote explicitly and I observe no change in the âRowsPerStripâ tag. I have carefully inspected that I am calling the exact same function at both places but the results differ. The dynamic libraries are the exact same ones used in both cases. Thanks, Nalini From: [email protected] <[email protected]> Sent: Wednesday, December 8, 2021 5:51 PM To: Nalini Vishnoi <[email protected]>; [email protected] Subject: RE: [Tiff] Windows - difference in behavior when large RowsPerStrip is specified while writing Uncompressed vs LZW compressed TIFF file Hi Nalini, Nowadays the main culprit for inconsistencies between Windows and Linux is the âlongâ data type, which is 32-bit in 64-bit Windows (at least with MSVC) and 64-bit in Linux. Not sure if this is the case here or not. Silent production of an invalid TIFF file would of course mean a bug which should be fixed. As others have explained, the fix might be an error return with âstrip too largeâ. Regardless of above, writing multi-gigabyte strips is not a good idea and should be avoided even if it worked. Cheers Paavo From: Nalini Vishnoi <[email protected]<mailto:[email protected]>> Sent: kolmapäev, 8. detsember 2021 23:39 To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [Tiff] Windows - difference in behavior when large RowsPerStrip is specified while writing Uncompressed vs LZW compressed TIFF file Hi Paavo, Thanks a lot for your response. I understand that writing a single strip of large size is not recommended. I will keep that in mind. However, 1. One correction to my previous email. The results that I shared in my last email are from libTIFF version 4.0.0 (an older version). When I tried to use 4.2.0, I didnât get the weird results of RowsPerStrip changing after writing Uncompressed TIFF file on Windows, the LZW compression issue remained the same. However, with 4.2.0 and using Linux, I see the same issue of changed RowsPerStrip when writing uncompressed data. Not sure why this is happening. 2. In case of LZW compression, as you said there is some 32-bit counter which is not capable of handling this large strip size â could it be different on different platforms? I am not seeing the issue of corrupted file on Linux platform, only on Windows. May be some overflow happening depending on the size of the counter and how that size is interpreted on different platforms? 3. Keeping in mind your recommendation, do you think this behavior should be reported as a libTIFF bug (functionality wise)? Thanks again, Nalini From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 8, 2021 10:00 AM To: Nalini Vishnoi <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: RE: [Tiff] Windows - difference in behavior when large RowsPerStrip is specified while writing Uncompressed vs LZW compressed TIFF file Hi Nalini, You are attempting to write a single strip of ca 2.5 G pixels. This is not a wise thing to do. Most probably there is some 32-bit counter somewhere which cannot cope with such sizes, probably in the LZW compression routines. For such large files one should use the tiled format and write e.g. 512x512 pixel tiles. This enables the reader to âzoom inâ into the image by reading only a few tiles. This is widely used in âpyramidalâ TIFF files of various kinds. Another option would be to just write smaller strips, but for such large images the tiles would be the right approach. HTH Paavo From: Tiff <[email protected]<mailto:[email protected]>> On Behalf Of Nalini Vishnoi Sent: kolmapäev, 8. detsember 2021 05:38 To: [email protected]<mailto:[email protected]> Subject: [Tiff] Windows - difference in behavior when large RowsPerStrip is specified while writing Uncompressed vs LZW compressed TIFF file Hello everyone, I am writing to understand a behavior I observed on Windows with libTIFF. I am using libTIFF 4.2.0. I write a TIFF file with RowsPerStrip=25000 using two different ways: 1. Uncompressed 2. Using LZW compression. When the file is written without any compression, RowsPerStrip is automatically changed to 1 from 25000 while writing the file. See the output of the code added below (in case of writing uncompressed file): Status after setting rowsPerStrip = 1 Status after reading rowsPerStrip = 1 Rows per strip = 25000 Size of strip = 2500000000 NumStrips = 1 Status after writing the file = 2500000000 NumStrips = 25000 Size of strip = 100000 Rows per strip = 1 Status after reading the image = 100000 However, when I use LZW compression, the written file become corrupted: Status after setting rowsPerStrip = 1 Status after reading rowsPerStrip = 1 Rows per strip = 25000 Size of strip = 2500000000 NumStrips = 1 Status after writing the file = 2500000000 NumStrips = 1 Size of strip = 2500000000 Rows per strip = 25000 Status after reading the image = -1 Are these results expected (RowsPerStrip automatically changing its value AND the file getting corrupted with LZW compression)? Please let me know if I any additional information is needed. I look forward to your reply. Thanks, Nalini Here is my code: #include<stdlib.h> #include<tiffio.h> #include<tiff.h> #include<iostream> #include<vector> #define FILENAME "mytest.tif" int main() { TIFF* tif = TIFFOpen(FILENAME, "w"); std::vector<uint16_t> v(1,0); tsize_t rowsPerStrip=0, stripSize; int64 status = -10; TIFFSetField(tif,TIFFTAG_PHOTOMETRIC, 2); TIFFSetField(tif,TIFFTAG_BITSPERSAMPLE, 8); TIFFSetField(tif,TIFFTAG_IMAGELENGTH, 25000); TIFFSetField(tif,TIFFTAG_IMAGEWIDTH, 25000); TIFFSetField(tif,TIFFTAG_SAMPLESPERPIXEL, 4); TIFFSetField(tif,TIFFTAG_PLANARCONFIG, 1); status = TIFFSetField(tif,TIFFTAG_ROWSPERSTRIP, uint32_t(25000)); std::cout << "\nStatus after setting rowsPerStrip = "<< status ; status = TIFFGetField(tif,TIFFTAG_ROWSPERSTRIP, &rowsPerStrip); std::cout << "\nStatus after reading rowsPerStrip = "<< status ; std::cout<<"\nRows per strip = "<< rowsPerStrip; // LZW compression // TIFFSetField(tif,TIFFTAG_COMPRESSION, 5); TIFFSetField(tif,TIFFTAG_EXTRASAMPLES, v.size(), &v[0]); tsize_t numBytes = tsize_t(25000)*25000*4; stripSize = TIFFStripSize(tif); std::cout<<"\nSize of strip = "<<stripSize; std::vector<uint8_t> writeBuffer(stripSize, 10); tstrip_t numStrips = TIFFNumberOfStrips(tif); std::cout << "\nNumStrips = " << numStrips; tdata_t buffer = static_cast<tdata_t>(&writeBuffer[0]); status = TIFFWriteEncodedStrip(tif, 0, buffer, numBytes ); std::cout << "\nStatus after writing the file = "<< status <<'\n'; TIFFClose(tif); tif = TIFFOpen(FILENAME, "r"); numStrips = TIFFNumberOfStrips(tif); std::cout<< "\nNumStrips = "<<numStrips; stripSize = TIFFStripSize(tif); std::cout<<"\nSize of strip = "<<stripSize; TIFFGetField(tif,TIFFTAG_ROWSPERSTRIP, &rowsPerStrip); std::cout<<"\nRows per strip = "<<rowsPerStrip; status = TIFFReadEncodedStrip(tif, 0, &writeBuffer[0], -1); std::cout<<"\nStatus after reading the image = "<<status<<'\n'; TIFFClose(tif); }
_______________________________________________ Tiff mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/tiff
