Re: [Pharo-users] Working with a compressed Fuel file?

[stephan]

A long time ago, Mariano wrote:
> And what happened with the vm gzip crash? Still around?

Yep, still crashing the vm.

I revisited my MonticelloProjects experiment yesterday, and am still 
crashing
the vm when serializing to a compressed fuel file. I wrote about it at 
the time,

and then gave up as the vm was less stable during that period.

To reproduce the crash in a pharo6 image:

|project|
Gofer it
smalltalkhubUser: 'StephanEggermont' project: 
'MonticelloProjects';

package: 'MonticelloProjects';
load.

project := (Smalltalk at: #MCProject) new location:
'http://smalltalkhub.com/mc/StephanEggermont/Documentation/main'.
project read.
project saveToFile.

That reliably crashes in saveToFile.

It seems to be independent of 32/64 bit and mac/linux, as I tried some 
variations.

Crash dumps in a mail to the vm list

Stephan

Re: [Pharo-users] Working with a compressed Fuel file?

[Mariano Martinez Peck]

And what happened with the vm gzip crash? Still around?

On Tue, Jan 5, 2016 at 3:43 PM, Stephan Eggermont  wrote:

> On 05-01-16 16:29, Stephan Eggermont wrote:
>
>> Writing out the 71712 unique definitions in this project separately is
>> 54MB. The project consists of 17 packages with 2723 package versions,
>> the total number of pointers to the definitions is 2.2 million.
>> The total should be below 100MB
>>
>
> Thanks to the help of Max and Thierry I found the problem.
>
> Set withAll: (MCVersionInfo allInstances collect: [  :each | each id ])
> 7155
>
> MCVersionInfo allInstances size 2726588
>
> 2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally...
>
> Stephan
>
>
>
>
>


-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-users] Working with a compressed Fuel file?

[Stephan Eggermont]

On 03-01-16 20:31, Mariano Martinez Peck wrote:

I may have a hint. Which is the error upon materialization? A dnu on the
stream? Which message?


Here is a further problem. I've done:

|project|
Gofer it
smalltalkhubUser: 'StephanEggermont' project: 'MonticelloProjects';
package: 'MonticelloProjects';
load.

project := (Smalltalk at: #MCProject) new location: 
'http://source.squeak.org/VMMaker'.

project read.
project repository: nil.
FLSerializer serialize: project toFileNamed: 'VMMaker.fl'.

That takes a while (downloads 5 GB or so).
It results in a 1.2 GB Fuel file.
That is way too much, so somewhere I'm pulling in too many objects,
or they are not recognized as equal.

Writing out the 71712 unique definitions in this project separately is 
54MB. The project consists of 17 packages with 2723 package versions, 
the total number of pointers to the definitions is 2.2 million.

The total should be below 100MB

Stephan

Re: [Pharo-users] Working with a compressed Fuel file?

[Stephan Eggermont]

On 05-01-16 16:29, Stephan Eggermont wrote:

Writing out the 71712 unique definitions in this project separately is
54MB. The project consists of 17 packages with 2723 package versions,
the total number of pointers to the definitions is 2.2 million.
The total should be below 100MB


Thanks to the help of Max and Thierry I found the problem.

Set withAll: (MCVersionInfo allInstances collect: [  :each | each id ]) 7155

MCVersionInfo allInstances size 2726588

2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally...

Stephan

Re: [Pharo-users] Working with a compressed Fuel file?

[Stephan Eggermont]

On 04-01-16 14:29, Mariano Martinez Peck wrote:

Ahh I reported that years ago:
http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html


I can't seem to find an issue for this. Did we lose it?

Stephan

Re: [Pharo-users] Working with a compressed Fuel file?

[Mariano Martinez Peck]

On Mon, Jan 4, 2016 at 10:16 AM, Stephan Eggermont  wrote:

> On 03-01-16 20:31, Mariano Martinez Peck wrote:
>
>> I may have a hint. Which is the error upon materialization? A dnu on the
>> stream? Which message?
>>
>
> A segmentation fault when saving
>

Ahh I reported that years ago:
http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html

Cheers,



>
> Smalltalk stack dump:
> 0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n)
> GZipWriteStream
> 0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt:
> 0xe66c808: a(n) GZipWriteStream
> 0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n)
> FLBufferedWriteStream
> 0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n)
> FLBufferedWriteStream
> 0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder
> 0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString
> 0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8:
> a(n) FLHookPrimitiveCluster
> 0xffcb746c M [] in
> FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith:
> 0xe713af8: a(n) FLHookPrimitiveCluster
> 0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet
> 0xffcb74b0 M
> FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith:
> 0xe713af8: a(n) FLHookPrimitiveCluster
> 0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n)
> FLSerialization
> 0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n)
> FLSerialization
> 0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection
> 0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization
> 0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization
> 0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n)
> FLSerializer
> 0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer
> 0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878:
> a(n) FLEncoder class
> 0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure
> 0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n)
> FLEncoder class
> 0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer
> 0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n) FLSerializer
> class
> 0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject
> 0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File
> 0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure
> 0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File
> 0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject
> 0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject
>  0xb73ad00 s OpalCompiler>evaluate
>  0xb92dec8 s RubSmalltalkEditor>evaluate:andDo:
>  0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo:
>  0xb743418 s [] in
> GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate:
>  0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit:
>  0xb92dff8 s [] in
> GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate:
>  0xb743478 s WorldState>runStepMethodsIn:
>  0xb73adc0 s WorldMorph>runStepMethods
>  0xb92e070 s WorldState>doOneCycleNowFor:
>  0xb7434d8 s WorldState>doOneCycleFor:
>  0xb73ae20 s WorldMorph>doOneCycle
>  0x944db08 s [] in MorphicUIManager>(nil)
>  0x944da80 s [] in BlockClosure>(nil)
>
>
>
>
>


-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-users] Working with a compressed Fuel file?

[Stephan Eggermont]

On 04-01-16 14:29, Mariano Martinez Peck wrote:

Ahh I reported that years ago:
http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html


Ah. That is also when writing a string. Is the automatic conversion of 
ByteString to WideString safe here?


Stephan

Re: [Pharo-users] Working with a compressed Fuel file?

[Stephan Eggermont]

On 03-01-16 20:31, Mariano Martinez Peck wrote:

I may have a hint. Which is the error upon materialization? A dnu on the
stream? Which message?


A segmentation fault when saving

Smalltalk stack dump:
0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n) 
GZipWriteStream
0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt: 
0xe66c808: a(n) GZipWriteStream
0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n) 
FLBufferedWriteStream
0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n) 
FLBufferedWriteStream

0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder
0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString
0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8: 
a(n) FLHookPrimitiveCluster
0xffcb746c M [] in 
FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 
0xe713af8: a(n) FLHookPrimitiveCluster

0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet
0xffcb74b0 M 
FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 
0xe713af8: a(n) FLHookPrimitiveCluster
0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n) 
FLSerialization
0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n) 
FLSerialization

0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection
0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization
0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization
0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n) 
FLSerializer

0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer
0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878: 
a(n) FLEncoder class

0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure
0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) 
FLEncoder class

0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer
0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n) 
FLSerializer class

0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject
0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File
0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure
0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File
0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject
0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject
 0xb73ad00 s OpalCompiler>evaluate
 0xb92dec8 s RubSmalltalkEditor>evaluate:andDo:
 0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo:
 0xb743418 s [] in 
GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate:

 0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit:
 0xb92dff8 s [] in 
GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate:

 0xb743478 s WorldState>runStepMethodsIn:
 0xb73adc0 s WorldMorph>runStepMethods
 0xb92e070 s WorldState>doOneCycleNowFor:
 0xb7434d8 s WorldState>doOneCycleFor:
 0xb73ae20 s WorldMorph>doOneCycle
 0x944db08 s [] in MorphicUIManager>(nil)
 0x944da80 s [] in BlockClosure>(nil)

Re: [Pharo-users] Working with a compressed Fuel file?

[Mariano Martinez Peck]

I may have a hint. Which is the error upon materialization? A dnu on the
stream? Which message?
On Jan 3, 2016 4:19 PM, "Stephan Eggermont"  wrote:

> I'm trying to work with a compressed Fuel file, without having to expand
> everything at the same time. It looks like not all streams are created
> equally. I seem to need a File instead of a FileReference:
>
> saveToFile
> |zipStream fileRef|
> fileRef := File named: (FileSystem disk workingDirectory / (self
> fileName, '.zip')) pathString.
> fileRef writeStreamDo: [ :s |
> zipStream := GZipWriteStream on: s.
> repository := nil.
> FLSerializer serialize: self on: zipStream.
> zipStream close].
>
>
> And materializing doesn't like this stream
>
> from: aFileName
> |zipStream fileRef theNew|
> fileRef := File named: aFileName.
> fileRef readStreamDo: [ :s |
> zipStream := GZipReadStream on: s.
> theNew := (FLMaterializer new materializeFrom: zipStream)
> root.
> zipStream close].
> ^theNew
>
> How am I supposed to work with compressed Fuel files?
>
> Stephan
>
>
>