Re: [Pharo-users] Working with a compressed Fuel file?
A long time ago, Mariano wrote: > And what happened with the vm gzip crash? Still around? Yep, still crashing the vm. I revisited my MonticelloProjects experiment yesterday, and am still crashing the vm when serializing to a compressed fuel file. I wrote about it at the time, and then gave up as the vm was less stable during that period. To reproduce the crash in a pharo6 image: |project| Gofer it smalltalkhubUser: 'StephanEggermont' project: 'MonticelloProjects'; package: 'MonticelloProjects'; load. project := (Smalltalk at: #MCProject) new location: 'http://smalltalkhub.com/mc/StephanEggermont/Documentation/main'. project read. project saveToFile. That reliably crashes in saveToFile. It seems to be independent of 32/64 bit and mac/linux, as I tried some variations. Crash dumps in a mail to the vm list Stephan
Re: [Pharo-users] Working with a compressed Fuel file?
And what happened with the vm gzip crash? Still around? On Tue, Jan 5, 2016 at 3:43 PM, Stephan Eggermont wrote: > On 05-01-16 16:29, Stephan Eggermont wrote: > >> Writing out the 71712 unique definitions in this project separately is >> 54MB. The project consists of 17 packages with 2723 package versions, >> the total number of pointers to the definitions is 2.2 million. >> The total should be below 100MB >> > > Thanks to the help of Max and Thierry I found the problem. > > Set withAll: (MCVersionInfo allInstances collect: [ :each | each id ]) > 7155 > > MCVersionInfo allInstances size 2726588 > > 2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally... > > Stephan > > > > > -- Mariano http://marianopeck.wordpress.com
Re: [Pharo-users] Working with a compressed Fuel file?
On 05-01-16 16:29, Stephan Eggermont wrote: Writing out the 71712 unique definitions in this project separately is 54MB. The project consists of 17 packages with 2723 package versions, the total number of pointers to the definitions is 2.2 million. The total should be below 100MB Thanks to the help of Max and Thierry I found the problem. Set withAll: (MCVersionInfo allInstances collect: [ :each | each id ]) 7155 MCVersionInfo allInstances size 2726588 2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally... Stephan
Re: [Pharo-users] Working with a compressed Fuel file?
On 03-01-16 20:31, Mariano Martinez Peck wrote: I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message? Here is a further problem. I've done: |project| Gofer it smalltalkhubUser: 'StephanEggermont' project: 'MonticelloProjects'; package: 'MonticelloProjects'; load. project := (Smalltalk at: #MCProject) new location: 'http://source.squeak.org/VMMaker'. project read. project repository: nil. FLSerializer serialize: project toFileNamed: 'VMMaker.fl'. That takes a while (downloads 5 GB or so). It results in a 1.2 GB Fuel file. That is way too much, so somewhere I'm pulling in too many objects, or they are not recognized as equal. Writing out the 71712 unique definitions in this project separately is 54MB. The project consists of 17 packages with 2723 package versions, the total number of pointers to the definitions is 2.2 million. The total should be below 100MB Stephan
Re: [Pharo-users] Working with a compressed Fuel file?
On 04-01-16 14:29, Mariano Martinez Peck wrote: Ahh I reported that years ago: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html I can't seem to find an issue for this. Did we lose it? Stephan
Re: [Pharo-users] Working with a compressed Fuel file?
On 04-01-16 14:29, Mariano Martinez Peck wrote: Ahh I reported that years ago: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html Ah. That is also when writing a string. Is the automatic conversion of ByteString to WideString safe here? Stephan
Re: [Pharo-users] Working with a compressed Fuel file?
On Mon, Jan 4, 2016 at 10:16 AM, Stephan Eggermont wrote: > On 03-01-16 20:31, Mariano Martinez Peck wrote: > >> I may have a hint. Which is the error upon materialization? A dnu on the >> stream? Which message? >> > > A segmentation fault when saving > Ahh I reported that years ago: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html Cheers, > > Smalltalk stack dump: > 0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n) > GZipWriteStream > 0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt: > 0xe66c808: a(n) GZipWriteStream > 0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n) > FLBufferedWriteStream > 0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n) > FLBufferedWriteStream > 0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder > 0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString > 0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8: > a(n) FLHookPrimitiveCluster > 0xffcb746c M [] in > FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: > 0xe713af8: a(n) FLHookPrimitiveCluster > 0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet > 0xffcb74b0 M > FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: > 0xe713af8: a(n) FLHookPrimitiveCluster > 0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n) > FLSerialization > 0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n) > FLSerialization > 0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection > 0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization > 0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization > 0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n) > FLSerializer > 0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer > 0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878: > a(n) FLEncoder class > 0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure > 0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) > FLEncoder class > 0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer > 0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n) FLSerializer > class > 0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject > 0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File > 0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure > 0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File > 0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject > 0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject > 0xb73ad00 s OpalCompiler>evaluate > 0xb92dec8 s RubSmalltalkEditor>evaluate:andDo: > 0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo: > 0xb743418 s [] in > GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: > 0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit: > 0xb92dff8 s [] in > GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: > 0xb743478 s WorldState>runStepMethodsIn: > 0xb73adc0 s WorldMorph>runStepMethods > 0xb92e070 s WorldState>doOneCycleNowFor: > 0xb7434d8 s WorldState>doOneCycleFor: > 0xb73ae20 s WorldMorph>doOneCycle > 0x944db08 s [] in MorphicUIManager>(nil) > 0x944da80 s [] in BlockClosure>(nil) > > > > > -- Mariano http://marianopeck.wordpress.com
Re: [Pharo-users] Working with a compressed Fuel file?
On 03-01-16 20:31, Mariano Martinez Peck wrote: I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message? A segmentation fault when saving Smalltalk stack dump: 0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n) GZipWriteStream 0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt: 0xe66c808: a(n) GZipWriteStream 0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n) FLBufferedWriteStream 0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n) FLBufferedWriteStream 0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder 0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString 0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb746c M [] in FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet 0xffcb74b0 M FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n) FLSerialization 0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization 0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection 0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization 0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization 0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n) FLSerializer 0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer 0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) FLEncoder class 0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure 0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) FLEncoder class 0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer 0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n) FLSerializer class 0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject 0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File 0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure 0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File 0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject 0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject 0xb73ad00 s OpalCompiler>evaluate 0xb92dec8 s RubSmalltalkEditor>evaluate:andDo: 0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo: 0xb743418 s [] in GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit: 0xb92dff8 s [] in GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb743478 s WorldState>runStepMethodsIn: 0xb73adc0 s WorldMorph>runStepMethods 0xb92e070 s WorldState>doOneCycleNowFor: 0xb7434d8 s WorldState>doOneCycleFor: 0xb73ae20 s WorldMorph>doOneCycle 0x944db08 s [] in MorphicUIManager>(nil) 0x944da80 s [] in BlockClosure>(nil)
Re: [Pharo-users] Working with a compressed Fuel file?
I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message? On Jan 3, 2016 4:19 PM, "Stephan Eggermont" wrote: > I'm trying to work with a compressed Fuel file, without having to expand > everything at the same time. It looks like not all streams are created > equally. I seem to need a File instead of a FileReference: > > saveToFile > |zipStream fileRef| > fileRef := File named: (FileSystem disk workingDirectory / (self > fileName, '.zip')) pathString. > fileRef writeStreamDo: [ :s | > zipStream := GZipWriteStream on: s. > repository := nil. > FLSerializer serialize: self on: zipStream. > zipStream close]. > > > And materializing doesn't like this stream > > from: aFileName > |zipStream fileRef theNew| > fileRef := File named: aFileName. > fileRef readStreamDo: [ :s | > zipStream := GZipReadStream on: s. > theNew := (FLMaterializer new materializeFrom: zipStream) > root. > zipStream close]. > ^theNew > > How am I supposed to work with compressed Fuel files? > > Stephan > > >
[Pharo-users] Working with a compressed Fuel file?
I'm trying to work with a compressed Fuel file, without having to expand everything at the same time. It looks like not all streams are created equally. I seem to need a File instead of a FileReference: saveToFile |zipStream fileRef| fileRef := File named: (FileSystem disk workingDirectory / (self fileName, '.zip')) pathString. fileRef writeStreamDo: [ :s | zipStream := GZipWriteStream on: s. repository := nil. FLSerializer serialize: self on: zipStream. zipStream close]. And materializing doesn't like this stream from: aFileName |zipStream fileRef theNew| fileRef := File named: aFileName. fileRef readStreamDo: [ :s | zipStream := GZipReadStream on: s. theNew := (FLMaterializer new materializeFrom: zipStream) root. zipStream close]. ^theNew How am I supposed to work with compressed Fuel files? Stephan