Re: [Pharo-users] Problem cleaning up image

[Petr Fischer]

Hello, when I remember correctly when I tested cleaning last time, the problem 
was in the method ImageCleaner>>cleanUpForRelease (called form 
cleanUpForProduction).

Try to comment "self cleanUpMethods." in ImageCleaner>>cleanUpForRelease - 
"cleanUpMethods" method never finish.

Have you tried "minimal image" or "kernel image" (instead of cleaning full 
image)?

pf


> Hello,
> 
> Isn't cleanup the headless image working right now? I know it has some
> problems, but I tried to:
> 
> curl get.pharo.org | bash
> ./pharo Pharo.image clean --production
> 
> It runs for hours and don't complete. Last time I leave it for about 5
> hours before I cancelled.
> 
> Regards,
> Vitor

Re: [Pharo-users] Problem cleaning up image

[Marcus Denker]

Hello,

Yes, this is a bug that we really need to fix.
(sadly no information to add other thant that).

> On 09 Sep 2016, at 15:16, Vitor Medina Cruz  wrote:
> 
> Hello,
> 
> Isn't cleanup the headless image working right now? I know it has some 
> problems, but I tried to:
> 
> curl get.pharo.org  | bash 
> ./pharo Pharo.image clean --production
> 
> It runs for hours and don't complete. Last time I leave it for about 5 hours 
> before I cancelled.
> 
> Regards,
> Vitor

Re: [Pharo-users] Problem cleaning up image

[stepharo]

Ok may be this is just an endless loop :)



Le 12/9/16 à 10:22, Marcus Denker a écrit :

Hello,

Yes, this is a bug that we really need to fix.
(sadly no information to add other thant that).

On 09 Sep 2016, at 15:16, Vitor Medina Cruz > wrote:


Hello,

Isn't cleanup the headless image working right now? I know it has 
some problems, but I tried to:


curl get.pharo.org  | bash
./pharo Pharo.image clean --production

It runs for hours and don't complete. Last time I leave it for about 
5 hours before I cancelled.


Regards,
Vitor

Re: [Pharo-users] Problem cleaning up image

[Vitor Medina Cruz]

Ok, I did some extra tests and I think I figured what is happening: the
code I load on the image subject to clean has a startUp method that starts
a Teapot server on port 8080. I tried various scenarios:

1- Load a fresh image from get.pharo.org and execute clean --> Ok
2- Load a fresh image from get.pharo.org only with the dependencies from my
project (ConfigurationOfTwitterChallenge), one at a time, and execute clean
--> Ok
3- Load a fresh image from get.pharo.org, apply
ConfigurationOfTwitterChallenge and execute clean --> Fail
4- Load a fresh image from get.pharo.org, apply
ConfigurationOfTwitterChallenge and execute eval "TwitterChallengeMain
stopDefaultExecution. ImageCleaner cleanUpForProduction. Smalltalk
snapshot: true andQuit: true." --> *Ok *(Assuming  ImageCleaner
cleanUpForProduction. Smalltalk snapshot: true andQuit: true. does the same
as clean --production)

It appears I cannot clean an Image with the server started. Question: is
that an error or I really should not try to clean an image with a server
on? If the former I open an issue on FogBuz and I will try on a
not-headless vm to see if some error happen.

Regards,
Vitor



On Mon, Sep 12, 2016 at 6:59 AM, stepharo  wrote:

> Ok may be this is just an endless loop :)
>
>
>
> Le 12/9/16 à 10:22, Marcus Denker a écrit :
>
> Hello,
>
> Yes, this is a bug that we really need to fix.
> (sadly no information to add other thant that).
>
> On 09 Sep 2016, at 15:16, Vitor Medina Cruz  wrote:
>
> Hello,
>
> Isn't cleanup the headless image working right now? I know it has some
> problems, but I tried to:
>
> curl get.pharo.org | bash
> ./pharo Pharo.image clean --production
>
> It runs for hours and don't complete. Last time I leave it for about 5
> hours before I cancelled.
>
> Regards,
> Vitor
>
>
>
>

Re: [Pharo-users] Problem cleaning up image

[Marcus Denker]

Hi,

I think it is not really realistic to expect this to work *after* loading code 
of others.

Why not 

-> do image clean
-> *then* load the packages you need?

Marcus

> On 13 Sep 2016, at 03:19, Vitor Medina Cruz  wrote:
> 
> Ok, I did some extra tests and I think I figured what is happening: the code 
> I load on the image subject to clean has a startUp method that starts a 
> Teapot server on port 8080. I tried various scenarios:
> 
> 1- Load a fresh image from get.pharo.org  and execute 
> clean --> Ok
> 2- Load a fresh image from get.pharo.org  only with 
> the dependencies from my project (ConfigurationOfTwitterChallenge), one at a 
> time, and execute clean --> Ok
> 3- Load a fresh image from get.pharo.org , apply 
> ConfigurationOfTwitterChallenge and execute clean --> Fail
> 4- Load a fresh image from get.pharo.org , apply 
> ConfigurationOfTwitterChallenge and execute eval "TwitterChallengeMain 
> stopDefaultExecution. ImageCleaner cleanUpForProduction. Smalltalk snapshot: 
> true andQuit: true." --> Ok (Assuming  ImageCleaner cleanUpForProduction. 
> Smalltalk snapshot: true andQuit: true. does the same as clean --production)
> 
> It appears I cannot clean an Image with the server started. Question: is that 
> an error or I really should not try to clean an image with a server on? If 
> the former I open an issue on FogBuz and I will try on a not-headless vm to 
> see if some error happen.
> 
> Regards,
> Vitor
> 
> 
> 
> On Mon, Sep 12, 2016 at 6:59 AM, stepharo  > wrote:
> Ok may be this is just an endless loop :)
> 
> 
> 
> Le 12/9/16 à 10:22, Marcus Denker a écrit :
>> Hello,
>> 
>> Yes, this is a bug that we really need to fix.
>> (sadly no information to add other thant that).
>> 
>>> On 09 Sep 2016, at 15:16, Vitor Medina Cruz >> > wrote:
>>> 
>>> Hello,
>>> 
>>> Isn't cleanup the headless image working right now? I know it has some 
>>> problems, but I tried to:
>>> 
>>> curl get.pharo.org  | bash 
>>> ./pharo Pharo.image clean --production
>>> 
>>> It runs for hours and don't complete. Last time I leave it for about 5 
>>> hours before I cancelled.
>>> 
>>> Regards,
>>> Vitor
>> 
> 
> 

Re: [Pharo-users] Problem cleaning up image

[Vitor Medina Cruz]

Humm, why not? When I load the configuration it loads several other
packages, wouldn't that need a cleanup too? When I first see that I could
cleanup an image my first thought was to put it in a production state with
my code, then I would cleanup to leave it as lean as possible. Isn't that
correct?

If I should not clean after load my packages why provide the functionality?
Why not provide clean images to be used in production so that people could
download and load it's ready to production code?

Regards,
Vitor

On Tue, Sep 13, 2016 at 4:00 AM, Marcus Denker 
wrote:

> Hi,
>
> I think it is not really realistic to expect this to work *after* loading
> code of others.
>
> Why not
>
> -> do image clean
> -> *then* load the packages you need?
>
> Marcus
>
> On 13 Sep 2016, at 03:19, Vitor Medina Cruz  wrote:
>
> Ok, I did some extra tests and I think I figured what is happening: the
> code I load on the image subject to clean has a startUp method that starts
> a Teapot server on port 8080. I tried various scenarios:
>
> 1- Load a fresh image from get.pharo.org and execute clean --> Ok
> 2- Load a fresh image from get.pharo.org only with the dependencies from
> my project (ConfigurationOfTwitterChallenge), one at a time, and execute
> clean --> Ok
> 3- Load a fresh image from get.pharo.org, apply
> ConfigurationOfTwitterChallenge and execute clean --> Fail
> 4- Load a fresh image from get.pharo.org, apply
> ConfigurationOfTwitterChallenge and execute eval "TwitterChallengeMain
> stopDefaultExecution. ImageCleaner cleanUpForProduction. Smalltalk
> snapshot: true andQuit: true." --> *Ok *(Assuming  ImageCleaner
> cleanUpForProduction. Smalltalk snapshot: true andQuit: true. does the same
> as clean --production)
>
> It appears I cannot clean an Image with the server started. Question: is
> that an error or I really should not try to clean an image with a server
> on? If the former I open an issue on FogBuz and I will try on a
> not-headless vm to see if some error happen.
>
> Regards,
> Vitor
>
>
>
> On Mon, Sep 12, 2016 at 6:59 AM, stepharo  wrote:
>
>> Ok may be this is just an endless loop :)
>>
>>
>>
>> Le 12/9/16 à 10:22, Marcus Denker a écrit :
>>
>> Hello,
>>
>> Yes, this is a bug that we really need to fix.
>> (sadly no information to add other thant that).
>>
>> On 09 Sep 2016, at 15:16, Vitor Medina Cruz  wrote:
>>
>> Hello,
>>
>> Isn't cleanup the headless image working right now? I know it has some
>> problems, but I tried to:
>>
>> curl get.pharo.org | bash
>> ./pharo Pharo.image clean --production
>>
>> It runs for hours and don't complete. Last time I leave it for about 5
>> hours before I cancelled.
>>
>> Regards,
>> Vitor
>>
>>
>>
>>
>
>

Re: [Pharo-users] Problem cleaning up image

[Marcus Denker]

> On 13 Sep 2016, at 13:15, Vitor Medina Cruz  wrote:
> 
> Humm, why not? When I load the configuration it loads several other packages, 
> wouldn't that need a cleanup too? When I first see that I could cleanup an 
> image my first thought was to put it in a production state with my code, then 
> I would cleanup to leave it as lean as possible. Isn't that correct? 
> 
> If I should not clean after load my packages why provide the functionality? 
> Why not provide clean images to be used in production so that people could 
> download and load it's ready to production code?
> 

No, the #cleanUpForProduction just unloads tests, examples and things like 
that… it only touches loaded code in the sense that it
unloads all packages named -Tests and -Examples.

But if the external package happens to be structured in a way that this is bad, 
it would brake.

I never expected this to be run *after* more packages are loaded, just 
*before*. 

What it does right now it this (you can see that it could be simpler if our 
packages are better structured):

cleanUpForProduction
"self new cleanUpForProduction"
"delete logo"   
PolymorphSystemSettings showDesktopLogo: false.

self packagesForCleanUpInProduction do: [ :each | (MCPackage named: 
each) unload ]. 
self helpPackages do: [ :each | (MCPackage named: each) unload ].   
self testPackages do: [ :each | (MCPackage named: each) unload ].
self examplePackages do: [ :each | (MCPackage named: each) unload ].
(MCPackage named: 'ScriptLoader40') unload. 


(MCPackage named: 'MonticelloMocks') unload.
(MCPackage named: 'ToolsTest') unload.  
(MCPackage named: 'Announcements-Tests-Core') unload.   
(MCPackage named: 'AST-Tests-Core') unload. 
(MCPackage named: 'AST-Interpreter-Test') unload.   
(MCPackage named: 'Ring-Tests-Containers') unload.  
(MCPackage named: 'Ring-Tests-Kernel') unload.  
(MCPackage named: 'Ring-Tests-Monticello') unload.  
(MCPackage named: 'Regex-Tests-Core') unload.   
(MCPackage named: 'Refactoring-Tests-Changes') unload.  
(MCPackage named: 'Refactoring-Tests-Core') unload. 
(MCPackage named: 'Refactoring-Tests-Critics') unload.  
(MCPackage named: 'Refactoring-Tests-Environment') unload.  
(MCPackage named: 'FileSystem-Tests-Core') unload.  
(MCPackage named: 'FileSystem-Tests-Disk') unload.  
(MCPackage named: 'FileSystem-Tests-Memory') unload.

(MCPackage named: 'GT-Tests-Inspector') unload. 
(MCPackage named: 'Glamour-Tests-Core') unload. 
(MCPackage named: 'Glamour-Tests-Morphic') unload.  
(MCPackage named: 'Glamour-Tests-Resources') unload.
(MCPackage named: 'Glamour-Tests-Rubric') unload.   

(MCPackage named: 'Versionner-Core-Model') unload.
(MCPackage named: 'Versionner-Core-DependenciesModel') unload.  
(MCPackage named: 'Versionner-Core-Announcements') unload.  
(MCPackage named: 'Versionner-Core-Commands') unload.   
(MCPackage named: 'Versionner-Spec-Browser') unload.

(MCPackage named: 'Versionner-Tests-Core-Commands') unload. 
(MCPackage named: 'Versionner-Tests-Core-DependenciesModel') unload.
(MCPackage named: 'Versionner-Tests-Core-Model') unload.

(MCPackage named: 'Versionner-Tests-Resources') unload. 


self cleanUpForRelease.


Marcus

Re: [Pharo-users] Problem cleaning up image

[Vitor Medina Cruz]

I understand. I thought it did more stuff, like remove necessary
development tools or executing processes, I don't know, but considering its
implementation it should affect only the image size, right?

Thanks,
Vitor

On Tue, Sep 13, 2016 at 8:30 AM, Marcus Denker 
wrote:

>
> > On 13 Sep 2016, at 13:15, Vitor Medina Cruz 
> wrote:
> >
> > Humm, why not? When I load the configuration it loads several other
> packages, wouldn't that need a cleanup too? When I first see that I could
> cleanup an image my first thought was to put it in a production state with
> my code, then I would cleanup to leave it as lean as possible. Isn't that
> correct?
> >
> > If I should not clean after load my packages why provide the
> functionality? Why not provide clean images to be used in production so
> that people could download and load it's ready to production code?
> >
>
> No, the #cleanUpForProduction just unloads tests, examples and things like
> that… it only touches loaded code in the sense that it
> unloads all packages named -Tests and -Examples.
>
> But if the external package happens to be structured in a way that this is
> bad, it would brake.
>
> I never expected this to be run *after* more packages are loaded, just
> *before*.
>
> What it does right now it this (you can see that it could be simpler if
> our packages are better structured):
>
> cleanUpForProduction
> "self new cleanUpForProduction"
> "delete logo"
> PolymorphSystemSettings showDesktopLogo: false.
>
> self packagesForCleanUpInProduction do: [ :each | (MCPackage
> named: each) unload ].
> self helpPackages do: [ :each | (MCPackage named: each) unload ].
> self testPackages do: [ :each | (MCPackage named: each) unload ].
> self examplePackages do: [ :each | (MCPackage named: each) unload
> ].
> (MCPackage named: 'ScriptLoader40') unload.
>
>
> (MCPackage named: 'MonticelloMocks') unload.
> (MCPackage named: 'ToolsTest') unload.
> (MCPackage named: 'Announcements-Tests-Core') unload.
> (MCPackage named: 'AST-Tests-Core') unload.
> (MCPackage named: 'AST-Interpreter-Test') unload.
> (MCPackage named: 'Ring-Tests-Containers') unload.
> (MCPackage named: 'Ring-Tests-Kernel') unload.
> (MCPackage named: 'Ring-Tests-Monticello') unload.
> (MCPackage named: 'Regex-Tests-Core') unload.
> (MCPackage named: 'Refactoring-Tests-Changes') unload.
> (MCPackage named: 'Refactoring-Tests-Core') unload.
> (MCPackage named: 'Refactoring-Tests-Critics') unload.
> (MCPackage named: 'Refactoring-Tests-Environment') unload.
> (MCPackage named: 'FileSystem-Tests-Core') unload.
> (MCPackage named: 'FileSystem-Tests-Disk') unload.
> (MCPackage named: 'FileSystem-Tests-Memory') unload.
>
> (MCPackage named: 'GT-Tests-Inspector') unload.
> (MCPackage named: 'Glamour-Tests-Core') unload.
> (MCPackage named: 'Glamour-Tests-Morphic') unload.
> (MCPackage named: 'Glamour-Tests-Resources') unload.
> (MCPackage named: 'Glamour-Tests-Rubric') unload.
>
> (MCPackage named: 'Versionner-Core-Model') unload.
> (MCPackage named: 'Versionner-Core-DependenciesModel') unload.
> (MCPackage named: 'Versionner-Core-Announcements') unload.
> (MCPackage named: 'Versionner-Core-Commands') unload.
> (MCPackage named: 'Versionner-Spec-Browser') unload.
> (MCPackage named: 'Versionner-Tests-Core-Commands') unload.
> (MCPackage named: 'Versionner-Tests-Core-DependenciesModel')
> unload.
> (MCPackage named: 'Versionner-Tests-Core-Model') unload.
> (MCPackage named: 'Versionner-Tests-Resources') unload.
>
> self cleanUpForRelease.
>
>
> Marcus
>
>
>

Re: [Pharo-users] Problem cleaning up image

[Marcus Denker]

> On 13 Sep 2016, at 18:54, Vitor Medina Cruz  wrote:
> 
> I understand. I thought it did more stuff, like remove necessary development 
> tools or executing processes,

It does call #cleanUpForRelease, but this we call at *every* update on the CI, 
too. This calls  the #cleanUp method of all classes
and does some more (rehashing all dictionaries, restarting all? methods holding 
on to blocks…).

> I don't know, but considering its implementation it should affect only the 
> image size, right?
> 
Yes. Of course right now Cog is not compacting the somewhat empty allocated  
memory regions, so e.g. the
47 MB image should be around 15 MB empty space and the cleanup will not shrink 
it.

But when this is fixed, you should see a change. A saved image is too a large 
extend methods and literals they
point to (especially for things like Tutorials). In the past the normal image 
was ca. 25MB, while the shrieked one
after cleanup for deployment maybe 15? I do not remember.

But that was already a bit of “moored law” time ago ;-)

Marcus

Re: [Pharo-users] Problem cleaning up image

[Vitor Medina Cruz]

Thanks for the answers!

On Tue, Sep 13, 2016 at 2:46 PM, Marcus Denker 
wrote:

>
> > On 13 Sep 2016, at 18:54, Vitor Medina Cruz 
> wrote:
> >
> > I understand. I thought it did more stuff, like remove necessary
> development tools or executing processes,
>
> It does call #cleanUpForRelease, but this we call at *every* update on the
> CI, too. This calls  the #cleanUp method of all classes
> and does some more (rehashing all dictionaries, restarting all? methods
> holding on to blocks…).
>
> > I don't know, but considering its implementation it should affect only
> the image size, right?
> >
> Yes. Of course right now Cog is not compacting the somewhat empty
> allocated  memory regions, so e.g. the
> 47 MB image should be around 15 MB empty space and the cleanup will not
> shrink it.
>
> But when this is fixed, you should see a change. A saved image is too a
> large extend methods and literals they
> point to (especially for things like Tutorials). In the past the normal
> image was ca. 25MB, while the shrieked one
> after cleanup for deployment maybe 15? I do not remember.
>
> But that was already a bit of “moored law” time ago ;-)
>
> Marcus
>
>
>