This means it would be healthy to do a cleanup (at least the non aggressive one, ChangeSets and MC stuff) on each of the images we produce and not just the latest one.
On Wed, Aug 16, 2017 at 8:35 AM, Tim Mackinnon <tim@testit.works> wrote: > Yes you were on to something there (and at the same time, by poking around > with #pointersTo I noticed some chains of objects too). So I ran the > following script (partially borrowed from ImageCleaner) and this has got me > down to a 14mb image (instance sizes listed below, which is looking much > healthier - and those MethodChangeRecords are gone too) !!! > > I suspect there are more monti/metacello things that are still lurking > around. > > I also wonder if I need some the character sorting strings too. > > Tim > > "CmdLine script to debug the initial minimal image" > > | logger repo version | > > logger := FileStream stderr. > logger cr; nextPutAll: 'Starting Minimal Cleanup Script...'. > > logger cr; nextPutAll: '>Resetting Class Comments'. > Smalltalk allClasses do: [ :c | c classComment: '' stamp: '' ]. > > logger cr; nextPutAll: '>Removing MC holders'. > MCMethodDefinition allInstances do: [:each | each become: String new ]. > MCClassDefinition allInstances do: [:each | each become: String new ]. > MCVersionInfo allInstances do: [:each | each become: String new ]. > > logger cr; nextPutAll: '>ImageCleaner release routines'. > Smalltalk organization removeEmptyCategories. > Smalltalk > allClassesAndTraitsDo: [ :class | > [ :each | > each > removeEmptyCategories; > sortCategories ] > value: class organization; > value: class class organization ]. > > (RPackageOrganizer default packages select: #isEmpty) > do: #unregister. > > Smalltalk organization sortCategories. > Smalltalk garbageCollect. > Smalltalk cleanOutUndeclared. > Smalltalk fixObsoleteReferences. > Smalltalk cleanUp: true except: #() confirming: false. > > logger cr; nextPutAll: '>GC'. > 3 timesRepeat: [ > Smalltalk garbageCollect. > Smalltalk cleanOutUndeclared. > Smalltalk fixObsoleteReferences]. > > logger cr; nextPutAll: 'Finished Script.'; cr; cr. > > > My top instances are now: > > Class code space # instances > inst space percent inst average size > CompiledMethod 19159 30481 > 2912968 21.60 95.57 > Array 3742 36495 > 2852448 21.10 78.16 > ByteString 2640 24018 > 2517168 18.60 104.80 > ByteSymbol 1698 20722 > 759208 5.60 36.64 > Association 1148 19786 > 633152 4.70 32.00 > IdentitySet 408 15452 > 494464 3.70 32.00 > MethodDictionary 3310 3520 > 350192 2.60 99.49 > Protocol 1679 8382 > 268224 2.00 32.00 > WeakArray 1758 265 > 232304 1.70 876.62 > OrderedCollection 6555 5043 > 201720 1.50 40.00 > ClassOrganization 5281 3520 > 168960 1.30 48.00 > Metaclass 7184 1748 > 153824 1.10 88.00 > > > > > On 15 Aug 2017, at 23:00, Guillermo Polito <guillermopol...@gmail.com> > wrote: > > Just a hunch: could you inspect ur MethodChangeRecord instances ? > > Le mar. 15 août 2017 à 23:55, Tim Mackinnon <tim@testit.works> a écrit : > >> A weird observation - is it possible that source code is being stored in >> the image as strings somehow? When I do >> >> ./pharo PharoLambda.image eval "ByteString allInstances inject: >> (OrderedCollection new) into: [:r :i | i size > 500 ifTrue: [r add: i]. r]" >> >> I see to see reams of what looks like method source - but I thought >> source code was stored in the .sources file and the .changes file (and I >> haven’t been bundling those in my deployed image). >> >> I’m trying to figure out how you find references to a string object, to >> chase down what is pointing to these strings as maybe there is a quick 4mb >> win by simply nil’ing out some obvious things. (This doesn’t of course help >> with a default minimal image - but maybe a few tricks for packaging and >> deploying something). >> >> Tim >> >> On 15 Aug 2017, at 22:26, Tim Mackinnon <tim@testit.works> wrote: >> >> Hi Guille/Ben - I got a quick moment to try the SpaceTally (aside: it >> seems very convoluted to load a single package into the image, I was trying >> to avoid having to create a baselineOf for something so simple - I ended up >> with: >> >> repo := MCFileTreeRepository new directory: './bootstrap' asFileReference. >> >> version := repo loadVersionFromFileNamed: 'Tool-Profilers.package'. >> version load. >> >> >> Anyway - in my minimal image, like in the fat image there seems to be a >> surprising amount of bytestrings (4mb worth?). I think that might need some >> digging into? It seems like a lot somehow. Although Ben’s neat experiment >> of zipping strings shows that’s not a real route. >> >> In a deployed minimal image - maybe I can get rid of some other things >> like MethodChangeRecords or MCMethodDefiniion’s (but they are smaller wins >> - but noticeable) >> >> Class code space # instances >> inst space percent inst average size >> ByteString 2640 37365 >> 4823848 21.50 129.10 >> Array 3742 53002 >> 3961944 17.60 74.75 >> CompiledMethod 19159 30481 >> 2912968 13.00 95.57 >> Association 1148 58348 >> 1867136 8.30 32.00 >> MethodChangeRecord 431 34312 >> 1097984 4.90 32.00 >> ByteArray 4605 290 >> 908728 4.00 3133.54 >> ByteSymbol 1698 22689 >> 840168 3.70 37.03 >> IdentitySet 408 19076 >> 610432 2.70 32.00 >> MethodDictionary 3310 3520 >> 608688 2.70 172.92 >> WeakArray 1758 3024 >> 597824 2.70 197.69 >> MCMethodDefinition 4318 6659 >> 426176 1.90 64.00 >> Protocol 1679 8382 >> 268224 1.20 32.00 >> OrderedCollection 6555 5509 >> 220360 1.00 40.00 >> >> As an aside - my Gitlab project is public, the scripts that load things >> up are in ./scripts (build.sh, and minimal.st and loadlocal.st) >> >> Tim >> >> On 15 Aug 2017, at 08:02, Guillermo Polito <guillermopol...@gmail.com> >> wrote: >> >> >> >> On Mon, Aug 14, 2017 at 4:42 PM, Tim Mackinnon <tim@testit.works> wrote: >> >>> Hi Guille - just running SpaceTally on my dev image to get a feel for >>> it. It turns out that in the minimal images you’ve been creating, its not >>> loaded (makes sense). >>> >> >> Yup, it's loaded afterwards. >> >> All packages are loaded through metacello baselines. We should start >> refactoring and making standalone projects, each one with a baseline for >> himself, and his own dependencies described. >> >> I was checking on your gitlab and I have probably no access: how are you >> finally loading packages in the bootstrap image? Can you share that with us >> in text? I'd like to improve that situation. >> >> >>> I’m wondering if there is an easy way to import it in (I guess that >>> package should be in the Pharo git tree I cloned to get Fuel loaded right? >>> Or is there a separate standalone source?). >>> >> >> Yes it is, you can get the package programatically doing >> >> SpaceTally package name >> >> And furthermore, get the baseline that currently is loading by doing >> >> package := SpaceTally package name. >> BaselineOf subclasses select: [ :e | >> e project version packages anySatisfy: [ :p | p name = package ]]. >> >> >>> >>> Thanks for all the support, and your email about why the contexts stack >>> up is very well received (I will comment over there). >>> >>> By the way - it looks like Martin Fowler picked up on this announcement >>> - so maybe we might get some interest from his mass of followers. >>> >>> Tim >>> >>> On 14 Aug 2017, at 10:49, Guillermo Polito <guillermopol...@gmail.com> >>> wrote: >>> >>> Hi Tim, >>> >>> On Mon, Aug 14, 2017 at 11:41 AM, Tim Mackinnon <tim@testit.works> w >>> rote: >>> >>>> Hey guys, thanks for your enthusiasm around this - and I cannot stress >>>> enough how this was only possible because of the work that has gone into >>>> making Pharo (in particular the 64bit image, as well as having a minimal >>>> image, and some great blog posts on serialising contexts) as well as the >>>> patience from everyone in answering questions and helping me get it all >>>> working. >>>> >>>> I’m still quite keen to get my execution time back down under 800ms and >>>> I’d like to actually get back to writing a few skills to automate a few >>>> things around my house. >>>> >>>> To Answer Denis’ question - >>>> >>>> My final footprint is 30.4mb - thats composed of a 22mb image (with a >>>> simple example that pulls in Fuel, ZTimestamp and the S3 Library which >>>> depends on XMLParser) and then the VM (from which I removed obvious dll’s). >>>> >>>> In my original experiments with a 6.0 minimal image - I did manage to >>>> get to a 13.4mb image (which started out as 12mb original size, and then >>>> loaded in STON and had only a simple clock example). I think the sweet spot >>>> is around 20mb total footprint as that seems to get me into the 450ms-900ms >>>> range. >>>> >>>> The 7.0 min image now starts out at 15mb and then I’m not sure why >>>> loading Fuel, S3 and XMLParser takes 7mb (it seems big to me - but I’ve not >>>> dug into that). >>>> >>> >>> You can do further space analysis using the following expression >>> >>> SpaceTally new printSpaceAnalysis >>> >>> You can do that in an eval and check what's taking space. With measures >>> we can iterate and improve :). >>> >>> >>>> I’ve also found (and this on the back of unserialising the context in >>>> my example) that the way we build images has 15+ saved stack sessions that >>>> have saved on top of each other from the way we build up the images. I >>>> don’t yet know the implications of size/speed of these - but we need a >>>> better way of folding executions when we snapshot headless images. I’m also >>>> not clear if there are any other startup tasks that take precious time >>>> (this also has implications for our fat development images as they take >>>> much longer to appear than they really should). >>>> >>> >>> I'm working on this as I'm writing this mail ;) >>> >>> https://pharo.fogbugz.com/f/cases/20309 >>> https://github.com/pharo-project/pharo/pull/196 >>> >>> I'll write down the implications further in a different thread. >>> >>> >>>> I’ll be exploring some of these size/speed tradeoff’s in follow on >>>> messages. >>>> >>>> But once again, a big thanks - I’ve not enjoyed programming like this >>>> for ages. >>>> >>>> Tim >>>> >>>> On 12 Aug 2017, at 16:26, Ben Coman <b...@openinworld.com> wrote: >>>> >>>> hi Tim, >>>> >>>> That is..... AWESOME! >>>> >>>> Very nice delivery - it flowed well with great narration. >>>> >>>> I loved @2:17 "this is the interesting piece, because PharoLambda has >>>> serialized the execution context of its application and saved it into [my >>>> S3 bucket] ... [then on the local machine] rematerializes a debugger [on >>>> that context]." >>>> >>>> There is a clarity in your video presentation that really may intrigue >>>> outsiders. As a community we should push this on the usual hacker forums - >>>> ycombinator could be a good starting point (but I'm locked out of my >>>> account there). >>>> An enticing title could be... >>>> "Debugging Lambdas by re-materializing saved execution contexts on your >>>> local machine." >>>> >>>> cheers -ben >>>> >>>> On Fri, Aug 11, 2017 at 3:37 PM, Denis Kudriashov <dionisi...@gmail.com >>>> > wrote: >>>> >>>>> This is cool Tim. >>>>> >>>>> So what image size you deployed at the end? >>>>> >>>>> 2017-08-10 15:47 GMT+02:00 Tim Mackinnon <tim@testit.works>: >>>>> >>>>>> I just wanted to thank everyone for their help in getting my pet >>>>>> project further along, so that now I can announce that PharoLambda is now >>>>>> working with the V7 minimal image and also supports post mortem debugging >>>>>> by saving a zipped fuel context onto S3. >>>>>> >>>>>> This latter item is particularly satisfying as at a recent serverless >>>>>> conference (JeffConf) there was a panel where poor development tools on >>>>>> serverless platforms was highlighted as a real problem. >>>>>> >>>>>> In our community we’ve had these kinds of tools at our fingertips for >>>>>> ages - but I don’t think the wider development community has really >>>>>> noticed. Debugging something short lived like a Lambda execution is quite >>>>>> startling, as the current answer is “add more logging”, and we all know >>>>>> that sucks. To this end, I’ve created a little screencast showing this in >>>>>> action - and it was pretty cool because it was a real example I >>>>>> encountered >>>>>> when I got everything working and was trying my test application out. >>>>>> >>>>>> I’ve also put a bit of work into tuning the excellent GitLab CI >>>>>> tools, so that I can cache many of the artefacts used between different >>>>>> build runs (this might also be of interest to others using CI systems). >>>>>> >>>>>> The Gitlab project is on: https://gitlab.com/macta/PharoLambda >>>>>> And the screencast: https://www.youtube.com/watch?v=bNNCT1hLA3E >>>>>> >>>>>> Tim >>>>>> >>>>>> >>>>>> On 15 Jul 2017, at 00:39, Tim Mackinnon <tim@testit.works> wrote: >>>>>> >>>>>> Hi - I’ve been playing around with getting Pharo to run well on AWS >>>>>> Lambda. It’s early days, but I though it might be interesting to share >>>>>> what >>>>>> I’ve learned so far. >>>>>> >>>>>> Usage examples and code at https://gitlab.com/macta/PharoLambda >>>>>> >>>>>> With help from many of the folks here, I’ve been able to get a simple >>>>>> example to run in 500ms-1200ms with a minimal Pharo 6 image. You can >>>>>> easily >>>>>> try it out yourself. This seems slightly better than what the GoLang >>>>>> folks >>>>>> have been able to do. >>>>>> >>>>>> Tim >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> Guille Polito >>> >>> Research Engineer >>> French National Center for Scientific Research - *http://www.cnrs.fr* >>> <http://www.cnrs.fr/> >>> >>> >>> *Web:* *http://guillep.github.io* <http://guillep.github.io/> >>> *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> >>> >>> >>> >> >> >> -- >> >> Guille Polito >> >> Research Engineer >> French National Center for Scientific Research - *http://www.cnrs.fr* >> <http://www.cnrs.fr/> >> >> >> *Web:* *http://guillep.github.io* <http://guillep.github.io/> >> *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> >> >> >> >> -- > > Guille Polito > > Research Engineer > French National Center for Scientific Research - *http://www.cnrs.fr* > <http://www.cnrs.fr/> > > > *Web:* *http://guillep.github.io* <http://guillep.github.io/> > *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013> > > > -- Guille Polito Research Engineer French National Center for Scientific Research - *http://www.cnrs.fr* <http://www.cnrs.fr> *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
