Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
Doru,
That is a good idea. It might also make sense from a Pharo perspective to have
a global fallback defined. I am expecting that the mechanisms that we come up
with will be useful for individual projects as well as Pharo overall, so
building in flexibility at load time (instead of embedded in the configuration)
is a good idea.
To summarize my takeaway on this issue:
- A pharo backup or secondary repository is a good solution to ensure
that configurations and packages are guaranteed to survive over long
periods
of time regardless of the state of the primary project repositories
- The secondary repository will be maintained by running a scraper script
that
scans the Metacello configurations that are critical to Pharo.
- The script will copy the packages referenced by the configurations into the
secondary repository. The configurations will also be copied into the
repository.
- Whether or not the configuration copies are modified to reference the
secondary
repository directly or not is still undefined.
- This issue will be managed as a separate collection of packages to keep the
core
of Metacello clean.
- The initial short term goal will be to provide a mechanism to do a
one-shot
checkpoint of configurations and packages (no configuration modification) so
that a Pharo 1.0 repository of all of the important build artifacts can be
archived in the event that the longer term goals aren't met in time. This
initial short term goal will be based upon code already in use at GemStone
to
perform a similar function.
- The second phase will be to work out the details of using the repository in
real life conditions ... and at this point there are a number of
possibilities
under consideration.
- More details when we get this far:)
If I've missed something important, I'm sure you'll let me know:)
Dale
- Tudor Girba tudor.gi...@gmail.com wrote:
| Hi Dale,
|
| I believe the original problem is that you do not know upfront about
|
| where the actual files will be. In this case, perhaps the easiest
| solution is to provide some scripting facility to provide a repository
|
| mapping that should could be modified at loading time, and not be
| stored in the actual configuration.
|
| Suppose you have something like this:
|
| ConfigurationOfXXXbaselineX: spec
| version: 'X-baseline'
| spec package: 'SomePackage' with: [ spec repository: 'SOME_URL' ].
| ...
|
| ConfigurationOfXXXversionX: spec
| version: 'X' imports: #('X-baseline')
| spec package: 'SomePackage' with: 'SomePackage.tg.1'.
| ...
|
|
| Maybe a solution would be that when loading you could say:
|
| (ConfigurationOfXXX project version: 'X')
| withRepositoryMapping: 'SOME_URL' - 'SOME_NEW_URL';
| withRepositoryMapping: '...' - '...';
| load
|
| What do you think?
|
| In this way, the original configuration specifies everything, and the
|
| URL works either as the actual URL or as an alias that can be mapped
|
| to something else later on when you know the actual URL. Actually,
| this can already be accomplished at engine level. I believe a simple
|
| scripting like above would just make it practical. What do you think?
|
| Cheers,
| Doru
|
|
| On 23 Feb 2010, at 21:18, Dale Henrichs wrote:
|
|
| - Lukas Renggli reng...@gmail.com wrote:
|
| | Perhaps Gofer can be provided with a secondary repository
| location
| | in the case when the primary repository fails.
| |
| | Yes, it can. You just specify multiple repositories and it will
| use
| | them all. If you tell Gofer to #disableRepositoryErrors then it
| will
| | silently ignore repositories that are not available.
|
| Lukas,
|
| The only thing that worries about including such repository all of
|
| the time, is the cost of building the cache of repository when
| resolving repositories ... the secondary repository would get pretty
|
| large over time and most of the time (only in the case of errors)
| would it need to be scanned.
|
| I suppose Metacello could handle load errors in the code that calls
|
| Gofer and do a retry using the secondary repository...the secondary
|
| repository _could_ be specified as part of the configuration, so
| this probably isn't a bad idea...
|
| Thanks,
|
| Dale
|
| ___
| Pharo-project mailing list
| Pharo-project@lists.gforge.inria.fr
| http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
|
| --
| www.tudorgirba.com
|
| Every thing has its own flow.
___
Pharo-project mailing list
Pharo-project@lists.gforge.inria.fr
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
[Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
Hi all I would like that we have a discussion about how do we put in place an infrastructure to make sure that we can get metacello configuration working in 3/5 years from now. I think that there are two scenarios: MyProject == in my project I used metacello and I publish a config and if it breaks in the future this is my responsibility PublishedMyProjectIn = When I publish my project (for now I copy the configuration file from my repository to the one of metacello) in the metacelloPharo1.0 project, we could automatically pulled up all the necessary files and copy them in the 1.0 repository. This way we can shield and make sure that a project can be loaded. I would like to have MetacelloMetaRepository1.0 MetacelloMetaRepository1.1 MetacelloMetaRepository1.2 ... what do you think? Stef ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió: I was thinking about this problem yesterday in the light of the packages moved by Lukas. We have a problem with the packages referenced by a ConfigurationOfXXX. The can disappear anytime. The repository can disappear anytime. So I thought that we need a dedicated hosting server where: - We have all the ConfigurationOfXXX (in the MetacelloRepository repo, just as now) - We copy there *all* the packages that conform a giver ConfigurationOfXXXversionXX: from the original repo to this dedicated repo. - There is *no way* to delete anything. This is a *write only, read always, delete never* repository. So maybe we need a modified squeaksource image there that disables delete/rename/move packages. Or maybe the metasource from Torsten (is that the name of the new app designed for Metacello) can add this requirement. Besides, the app can scan all the ConfigurationOfXXX and check for what ones are affected by a deletion of a package before permiting delete of the package. This way we can delete packages but also see what that will affect and correctly promptly. The point is, we need an independent, mostly never breaking metacello repository. Other idea will be to use google or github versioned repositories and from them build the published repo, so that when a package is deleted by accident or by mistake, it can be restored quickly. But I like more the previous version. What do you think? -- Miguel Cobá http://miguel.leugim.com.mx ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
Miguel Enrique Cobá Martinez wrote: El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió: I was thinking about this problem yesterday in the light of the packages moved by Lukas. We have a problem with the packages referenced by a ConfigurationOfXXX. The can disappear anytime. The repository can disappear anytime. So I thought that we need a dedicated hosting server where: - We have all the ConfigurationOfXXX (in the MetacelloRepository repo, just as now) - We copy there *all* the packages that conform a giver ConfigurationOfXXXversionXX: from the original repo to this dedicated repo. - There is *no way* to delete anything. This is a *write only, read always, delete never* repository. Sidenote: SqueakMap actually caches all URLs used for versions and if the original URL fails an SHA check it falls back on the cached file on the SqueakMap server. In that way it was meant to handle original URLs going stale or disappearing. regards, Göran ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
[Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
make sure that we can get metacello configuration working in 3/5 years from now. We should also think about - mirroring. Currently in Metacello you give the repo hard coded in the baseline. Maybe it would be better to have an alias there and define the repo elsewhere (for instance in #versionXXX method). - offline work where on is working offline, behind a closed proxy. There are also companies that want to have an internal server. They do not want to rely on a public server even when it is 24*7. Did I already ask if we have the infrastructure/resources to run an own server (PharoSource, ...)? Bye T. c -- NEU: Mit GMX DSL über 1000,- ¿ sparen! http://portal.gmx.net/de/go/dsl02 ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
El mar, 23-02-2010 a las 14:12 +0100, Göran Krampe escribió: Sidenote: SqueakMap actually caches all URLs used for versions and if the original URL fails an SHA check it falls back on the cached file on the SqueakMap server. In that way it was meant to handle original URLs going stale or disappearing. Does that mean that everything pushed to squeakmap can never be deleted? If that isn't true, then the problem is just pushed one step further. regards, Göran Thanks -- Miguel Cobá http://miguel.leugim.com.mx ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
Hi, I am very interested in the whole configuration management, metacello, monticello, public/private repository, project, library/package, object, project, environment, release, version stuff ... some clarity around is badly needed. My first contact with the whole Squeak/Pharo packages and versioning was very confusing with SqueakSource, SqueakMap, Sake/Packages, Package Universe, Monticello. Why not look at other distributed version control systems like Darcs, Bazaar, GIT ... and see what could work for smalltalk? -- View this message in context: http://n4.nabble.com/Metacello-How-to-make-sure-that-we-will-be-able-to-load-in-3-years-from-now-tp1565830p1566000.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com. ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
and with that I did not mean to use those other DVCS but maybe take ideas from them :) -- View this message in context: http://n4.nabble.com/Metacello-How-to-make-sure-that-we-will-be-able-to-load-in-3-years-from-now-tp1565830p1566007.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com. ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
and with that I did not mean to use those other DVCS but maybe take ideas from them :) This is not a problem any DVCS tackles. Rather take ideas of package management systems like aptitude, rpm, portage, fink, ... As you see in all such system, they ship the code (binary or source) together with the package definition. None of these mechanisms refer to the VCS where the code is developed, but they include all the dependencies they need. I was thinking more along of something of a SAR file that would include the Metacello configuration and all the referenced mcz files in one file. Dragging and dropping that file onto the image (or using some GUI) would bootstrap Metacello, execute the Metacello definition, and if necessary download and pull in other SAR files from a well known place. That's how Linux works quite successful for over a decade. Lukas -- Lukas Renggli http://www.lukas-renggli.ch ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
neat! El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió: I was thinking about this problem yesterday in the light of the packages moved by Lukas. We have a problem with the packages referenced by a ConfigurationOfXXX. The can disappear anytime. The repository can disappear anytime. So I thought that we need a dedicated hosting server where: - We have all the ConfigurationOfXXX (in the MetacelloRepository repo, just as now) - We copy there *all* the packages that conform a giver ConfigurationOfXXXversionXX: from the original repo to this dedicated repo. - There is *no way* to delete anything. This is a *write only, read always, delete never* repository. Sidenote: SqueakMap actually caches all URLs used for versions and if the original URL fails an SHA check it falls back on the cached file on the SqueakMap server. In that way it was meant to handle original URLs going stale or disappearing. regards, Göran ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
On Feb 23, 2010, at 2:25 PM, Torsten Bergmann wrote: make sure that we can get metacello configuration working in 3/5 years from now. We should also think about - mirroring. Currently in Metacello you give the repo hard coded in the baseline. Maybe it would be better to have an alias there and define the repo elsewhere (for instance in #versionXXX method). - offline work where on is working offline, behind a closed proxy. There are also companies that want to have an internal server. They do not want to rely on a public server even when it is 24*7. Did I already ask if we have the infrastructure/resources to run an own server (PharoSource, ...)? I requested inria to get a server with the correct webdav configuration but so far they were busy... Bye T. c -- NEU: Mit GMX DSL über 1000,- ¿ sparen! http://portal.gmx.net/de/go/dsl02 ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
I like the notion of a secondary repository for packages. It is absolutely necessary to protect us from a catastrophic failure with any one of the SqueakSource repositories. Perhaps Gofer can be provided with a secondary repository location in the case when the primary repository fails. With this kind of approach, the metacello configurations can continue to reference the primary development repository, in this way we might just be able to have our cake and eat it too:) The Pharo secondary repository would protect users from repository volatility and package volatility ... Once a version is published in Pharo you can count of being able to load that version over time. This secondary repository could be maintained by periodically scanning the Pharo configurations and copying the referenced packages from the primary repository to a known secondary repository (this repository could just be done with Apache file access since a fancy UI for the secondary repository isn't essential). Having code and processes in place means that individuals can reuse the code and process to maintain their own secondary repository for their own production environment. I have code that does most of what is needed, as I use this process at GemStone for archiving the configurations and mcz packages in our company source code control system. Dale - Göran Krampe go...@krampe.se wrote: | Miguel Enrique Cobá Martinez wrote: | El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió: | I was thinking about this problem yesterday in the light of the | packages | moved by Lukas. | We have a problem with the packages referenced by a | ConfigurationOfXXX. | The can disappear anytime. The repository can disappear anytime. | | So I thought that we need a dedicated hosting server where: | | - We have all the ConfigurationOfXXX (in the MetacelloRepository | repo, | just as now) | - We copy there *all* the packages that conform a giver | ConfigurationOfXXXversionXX: from the original repo to this | dedicated | repo. | - There is *no way* to delete anything. This is a *write only, read | always, delete never* repository. | | Sidenote: SqueakMap actually caches all URLs used for versions and if | the original URL fails an SHA check it falls back on the cached file | on | the SqueakMap server. | | In that way it was meant to handle original URLs going stale or | disappearing. | | regards, Göran | | | ___ | Pharo-project mailing list | Pharo-project@lists.gforge.inria.fr | http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
Perhaps Gofer can be provided with a secondary repository location in the case when the primary repository fails. Yes, it can. You just specify multiple repositories and it will use them all. If you tell Gofer to #disableRepositoryErrors then it will silently ignore repositories that are not available. Lukas -- Lukas Renggli http://www.lukas-renggli.ch ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
- Lukas Renggli reng...@gmail.com wrote: | Perhaps Gofer can be provided with a secondary repository location | in the case when the primary repository fails. | | Yes, it can. You just specify multiple repositories and it will use | them all. If you tell Gofer to #disableRepositoryErrors then it will | silently ignore repositories that are not available. Lukas, The only thing that worries about including such repository all of the time, is the cost of building the cache of repository when resolving repositories ... the secondary repository would get pretty large over time and most of the time (only in the case of errors) would it need to be scanned. I suppose Metacello could handle load errors in the code that calls Gofer and do a retry using the secondary repository...the secondary repository _could_ be specified as part of the configuration, so this probably isn't a bad idea... Thanks, Dale ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
What if use redudancy features of Internet itself? As you know the Internet was designed to withstand a nuclear attack, with a lot of redudant features, which we can explore to harden the Smalltalk infracture as well. In the similar ways as guyss like Google do. You know, they have always a single entry, http://www.google.com, but actually they direct your search to 1000s and milions of their servers behind. A mini version of Google can be use for Squeak or PharoSource as well. As I can see the Monticello repositories are basically the stateless WebDAV enabled Apache web servers. So having a mirror is a first thing to do. To still have a single entry point like http://www.squeaksource.com), we can exploit the redundancy feature of DNS: round robin entries. That is, you enter two entries in DNS zone, with the same name but different IP addresses, say: squeaksource.com www IN A87.54.32.108 www IN A212.18.45.87 This way in the case of failure of one server, the another one will responds. If both are working, DNS will return one of them in roun robin fashion. This will work for WebDAV part of the story, but we still need to tackle of the statefull web application of SqueakSource... hmm, a bit more thinking is needed therefore... Best regards Janko On 23. 02. 2010 19:48, Dale Henrichs wrote: I like the notion of a secondary repository for packages. It is absolutely necessary to protect us from a catastrophic failure with any one of the SqueakSource repositories. Perhaps Gofer can be provided with a secondary repository location in the case when the primary repository fails. With this kind of approach, the metacello configurations can continue to reference the primary development repository, in this way we might just be able to have our cake and eat it too:) The Pharo secondary repository would protect users from repository volatility and package volatility ... Once a version is published in Pharo you can count of being able to load that version over time. This secondary repository could be maintained by periodically scanning the Pharo configurations and copying the referenced packages from the primary repository to a known secondary repository (this repository could just be done with Apache file access since a fancy UI for the secondary repository isn't essential). Having code and processes in place means that individuals can reuse the code and process to maintain their own secondary repository for their own production environment. I have code that does most of what is needed, as I use this process at GemStone for archiving the configurations and mcz packages in our company source code control system. Dale - Göran Krampe go...@krampe.se wrote: | Miguel Enrique Cobá Martinez wrote: | El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió: | I was thinking about this problem yesterday in the light of the | packages | moved by Lukas. | We have a problem with the packages referenced by a | ConfigurationOfXXX. | The can disappear anytime. The repository can disappear anytime. | | So I thought that we need a dedicated hosting server where: | | - We have all the ConfigurationOfXXX (in the MetacelloRepository | repo, | just as now) | - We copy there *all* the packages that conform a giver | ConfigurationOfXXXversionXX: from the original repo to this | dedicated | repo. | - There is *no way* to delete anything. This is a *write only, read | always, delete never* repository. | | Sidenote: SqueakMap actually caches all URLs used for versions and if | the original URL fails an SHA check it falls back on the cached file | on | the SqueakMap server. | | In that way it was meant to handle original URLs going stale or | disappearing. | | regards, Göran | | | ___ | Pharo-project mailing list | Pharo-project@lists.gforge.inria.fr | http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project -- Janko Mivšek Svetovalec za informatiko Eranova d.o.o. Ljubljana, Slovenija www.eranova.si tel: 01 514 22 55 faks: 01 514 22 56 gsm: 031 674 565 ___ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Re: [Pharo-project] Metacello --- How to make sure that we will be able to load... in 3 years from now
Hi Dale,
I believe the original problem is that you do not know upfront about
where the actual files will be. In this case, perhaps the easiest
solution is to provide some scripting facility to provide a repository
mapping that should could be modified at loading time, and not be
stored in the actual configuration.
Suppose you have something like this:
ConfigurationOfXXXbaselineX: spec
version: 'X-baseline'
spec package: 'SomePackage' with: [ spec repository: 'SOME_URL' ].
...
ConfigurationOfXXXversionX: spec
version: 'X' imports: #('X-baseline')
spec package: 'SomePackage' with: 'SomePackage.tg.1'.
...
Maybe a solution would be that when loading you could say:
(ConfigurationOfXXX project version: 'X')
withRepositoryMapping: 'SOME_URL' - 'SOME_NEW_URL';
withRepositoryMapping: '...' - '...';
load
What do you think?
In this way, the original configuration specifies everything, and the
URL works either as the actual URL or as an alias that can be mapped
to something else later on when you know the actual URL. Actually,
this can already be accomplished at engine level. I believe a simple
scripting like above would just make it practical. What do you think?
Cheers,
Doru
On 23 Feb 2010, at 21:18, Dale Henrichs wrote:
- Lukas Renggli reng...@gmail.com wrote:
| Perhaps Gofer can be provided with a secondary repository location
| in the case when the primary repository fails.
|
| Yes, it can. You just specify multiple repositories and it will use
| them all. If you tell Gofer to #disableRepositoryErrors then it will
| silently ignore repositories that are not available.
Lukas,
The only thing that worries about including such repository all of
the time, is the cost of building the cache of repository when
resolving repositories ... the secondary repository would get pretty
large over time and most of the time (only in the case of errors)
would it need to be scanned.
I suppose Metacello could handle load errors in the code that calls
Gofer and do a retry using the secondary repository...the secondary
repository _could_ be specified as part of the configuration, so
this probably isn't a bad idea...
Thanks,
Dale
___
Pharo-project mailing list
Pharo-project@lists.gforge.inria.fr
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
--
www.tudorgirba.com
Every thing has its own flow.
___
Pharo-project mailing list
Pharo-project@lists.gforge.inria.fr
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
