[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Christian Knoke]

URL:
  

 Summary: Loading XZ compressed savefiles goes fail
 Project: Freeciv
Submitted by: chrisk
Submitted on: Sun 20 Sep 2015 03:13:39 PM CEST
Category: general
Severity: 3 - Normal
Priority: 5 - Normal
  Status: None
 Assigned to: None
Originator Email: 
 Open/Closed: Open
 Release: 2.5.1
 Discussion Lock: Any
Operating System: GNU/Linux
 Planned Release: 

___

Details:


Vers. 2.5.1 compiled from source on debian oldstable, gtk3 client

In server options, I set compression=xz and complevel=9

Starting game, auto save file writing all fine.

Loading savefile, client console:

chris: 'load /home/chris/.freeciv/saves/mouldy-T0032-Y-2400-manual.sav.xz'
/load: Could not load savefile:
/home/chris/.freeciv/saves/mouldy-T0032-Y-2400-manual.sav.xz

I can read savefile with xzless, looks sane on first view

savefile attached

debug level 3 log attached

all saves are affected so far

Christian





___

File Attachments:


---
Date: Sun 20 Sep 2015 03:13:39 PM CEST  Name:
mouldy-T0032-Y-2400-manual.sav.xz  Size: 12kB   By: chrisk


---
Date: Sun 20 Sep 2015 03:13:39 PM CEST  Name: xz-noload.debug3.log  Size: 4kB 
 By: chrisk



___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev

[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Marko Lindqvist]

Update of bug #23891 (project freeciv):

  Status:None => In Progress
 Planned Release: => 2.5.2, 2.6.0, 3.0.0

___

Follow-up Comment #1:

It seems that this file loads fine once one makes xz decompression buffers big
enough (70Mb is not enough, 100Mb is).

So the file is fine (saved ok), but cannot be loaded. As a workaround (if you
need this file) you can decompress it outside freeciv and load as plain file.

___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev

[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Marko Lindqvist]

Update of bug #23891 (project freeciv):

  Status: In Progress => Ready For Test 

___

Follow-up Comment #2:

Well, it IS enough to increase memory limit to 65Mb (for loading this
particular file).

Attached fix has an implementation to increase it further if needed, but in my
tests falling back to that implementation never really helped as it always
resulted in corrupted stream error then. Regardless having that part does not
hurt (failing already failing case in a different way) but might help in some
case.

(file #25167, file #25168)
___

Additional Item Attachment:

File name: Xz65Mb.patch   Size:3 KB
File name: Xz65Mb-S2_5.patch  Size:3 KB


___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev

[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Christian Knoke]

Follow-up Comment #3, bug #23891 (project freeciv):

> Well, it IS enough to increase memory limit to 65Mb (for loading this
particular file). 

The savegame file 135 Kb uncompressed.

Christian


___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev

[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Marko Lindqvist]

Follow-up Comment #4, bug #23891 (project freeciv):

>> Well, it IS enough to increase memory limit to 65Mb (for loading this
particular file). 

> The savegame file 135 Kb uncompressed.

The main reason I'm not so thrilled about the idea of dropping .bz2 support
completely.

___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev

[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Marko Lindqvist]

Update of bug #23891 (project freeciv):

  Status:  Ready For Test => Fixed  
 Assigned to:None => cazfi  
 Open/Closed:Open => Closed 


___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev

[Freeciv-Dev] [bug #23891] Loading XZ compressed savefiles goes fail

[Jacob Nevins]

Follow-up Comment #8, bug #23891 (project freeciv):

From the xz man page:


For example, decompressing  a  file  created with xz -9 currently 
requires 65 MiB of memory.

[...]

-7 ... -9
   These are like -6 but with higher compressor  and  decom‐
   pressor  memory requirements.  These are useful only when
   compressing files bigger than 8 MiB, 16 MiB, and  32 MiB,
   respectively.

[...]

The following table summarises the features of the presets:

   Preset   DictSize   CompCPU   CompMem   DecMem

 -6   8 MiB   6   94 MiB9 MiB

 -9  64 MiB   6  674 MiB   65 MiB


Examining the compressed file:


$ unxz -lvv mouldy-T0032-Y-2400-manual.sav.xz
  Blocks: 1
  Compressed size:11.9 KiB (12,136 B)
  Uncompressed size:  133.8 KiB (136,986 B)
  Ratio:  0.089
  Check:  CRC32
  Stream padding: 0 B
  Streams:
StreamBlocks  CompOffsetUncompOffsetCompSize 
UncompSize  Ratio  Check  Padding
 1 1   0   0  12,136
136,986  0.089  CRC320
  Blocks:
Stream Block  CompOffsetUncompOffset   TotalSize 
UncompSize  Ratio  Check  CheckVal  Header  FlagsCompSize   
MemUsage  Filters
 1 1  12   0  12,100
136,986  0.088  CRC32  c2ccb49c  12  -- 12,084  65 MiB
 --lzma2=dict=64MiB
  Memory needed:  65 MiB
  Sizes in headers:   No
  Minimum XZ Utils version: 5.0.0


From which I think:
* The server that saved the file must have had "compress" set to 9 (I infer
from MemUsage and "--lzma2=dict=64MiB"). Default is 6, which is much more
sensible.
* 65 MiB should always be enough, I think?
* If we could predict the uncompressed size of the savefile, we could dial
down pointless memory usage in the decompressor. (But we can't, and in any
case the decompressor still has to handle the worst case anyway.)

___

Reply to this item at:

  

___
  Message sent via/by Gna!
  http://gna.org/


___
Freeciv-dev mailing list
Freeciv-dev@gna.org
https://mail.gna.org/listinfo/freeciv-dev