Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
On Wed, Jan 12, 2011 at 04:29, Michael Scherer wrote: > Le mardi 11 janvier 2011 à 21:45 -0500, andre999 a écrit : >> Some suggestions for faster downloads without bittorrent. >> 1) use aria2c (or a similar application), which uses multiple >> connections, defaulting to 5, and allows multiple mirrors. >> By default it starts by allocating space for the file to be downloaded, >> which allows non-sequential downloading of the file, facilitating faster >> downloading from multiple sites. >> >> 2) use mirrors which allow multiple connexions. >> (Of course, with download software that takes advantage of this.) >> >> 3) use multiple mirrors. >> (Again, according to download software.) > > Theses 3 suggestions basically put X time the load of the mirror for > each client. ( or on more mirror, for that matters ). However, if 1) was to open 5 connections on 5 distinct servers, that would make more sense, no? But then I'm not sure there is so much more value than using a P2P protocol. Romain
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Michael Scherer a écrit : Le mardi 11 janvier 2011 à 21:45 -0500, andre999 a écrit : Michael Scherer a écrit : Le mardi 11 janvier 2011 à 20:03 +0100, Marcello Anni a écrit : hi all, i have one question (maybe it can be a proposal): is it possible to implement the torrent protocol to faster download the updates of the distro? it could be an interesting features for the coming Mageia releases I think the issue of faster download could be simply taken care by having more mirror, or faster one. I had under the impression that some ISP throttle down bittorrent, and that it may not be very nat and firewall friendly.. Some suggestions for faster downloads without bittorrent. 1) use aria2c (or a similar application), which uses multiple connections, defaulting to 5, and allows multiple mirrors. By default it starts by allocating space for the file to be downloaded, which allows non-sequential downloading of the file, facilitating faster downloading from multiple sites. 2) use mirrors which allow multiple connexions. (Of course, with download software that takes advantage of this.) 3) use multiple mirrors. (Again, according to download software.) Theses 3 suggestions basically put X time the load of the mirror for each client. ( or on more mirror, for that matters ). And that's quite bad from the point of view of a mirror manager. Why would it make _any_ difference to the mirrors ? Besides spreading the download over multiple mirrors. We download the same amount of data from the mirrors in any case. So we want to accelerate the download. Some mirrors might want to throttle the download, or limit the connexions, to spread the download over a longer period of time, and they still can. This approach just works around such limitations for the end user. BTW, aria2c monitors the download speed, to choose the faster mirrors for multiple connexions. (From the urls put on the command line.) For example, distrib-coffee could blacklist you if you do this, if you are not alone on your network connexion. And when we deployed this measure to protect the server, the limit was 2 connexion per address, since this was taking too much ressources on the old server ( each http request taking 1 process and so memory ). Hopefully, the hardware was upgraded but not everybody can afford 32g of ram and 8*2 ghz CPU. Such restrictions are perfectly legitimate for a mirror. [r...@distrib-coffee ~]# grep -B 6 -A 3 MaxConnPerIP /etc/httpd/conf.d/distrib-coffee.conf order allow,deny Allow from all Options +Indexes +MultiViews +SymLinksIfOwnerMatch MaxConnPerIP 5 ErrorDocument 503 "5 connections at the same time only allowed." So I think pissing off mirror maintainers is likely the wrong way of solving the problem ( who was not properly explained nor looked at besides "it should be faster" ). Again, the same amount of data is downloaded collectively from the mirrors used. And I agree totally with mirrors using whatever download speed controls they wish. (It is strongly advisable for mirrors with limited resources to put such controls in place.) Just as downloaders are free to work within the limits of such controls to their advantage. 4) use ftp instead of http Based on ? My experience. Most downloads are noticeably faster downloading larger files via ftp. (Not just Mandriva downloads by any means.) And not just my current provider, either. I use a very fast internet access for most of my ISO downloads. (The local library.) If this is based on using d-c, again, that's our custom QOS rules. If this is because of throttling on your provider, not everybody have the same provider, and so the same throttling. The only difference between http and ftp is that ftp server will likely scale better server side. An explanation ... And maybe at least some http servers don't allow out-of-order packet requests ? 5) use closer mirrors. (less delay in handshaking, etc.) I think tcp handshake is not much a problem, given the fact it happen once per rpm, compared to the number of tcp packet for the rest of the download. Use wireshark to see. Again based on my experience. BTW, this is mostly ISOs, since the download time of smaller files -- like typical .rpms -- is relatively fast. Isn't there some sort of handshaking and verification by tcp packet ? Also propagation time in the event of errors should explain at least part of the difference. I could have added download to the fastest disk available, but that seemed obvious. (e.g. one could download to a usb disk, but that would be painfully slow, unless one has a usb3 disk on a usb3 port.) In my case, using aria2c with 2 mirrors and the default 5 connexions is at least 3 times as fast as a single connexion (to my closest mirror). And a much greater improvement over other download options I've tried. I also have configured rpmdrake to use aria2c -- it seems to give me faster and more reliable upda
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
> aria2 amongst others has bittorrent support though, so metalinks with > ftp/http/++ mirrors together with torrents could help combine them all > together. :) Well choosing the best one following a given (configurable) criteria is good idea, i believe :) signature.asc Description: This is a digitally signed message part.
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
mercoledì 12 gennaio 2011 alle 00:12, Michael Scherer ha scritto: > I had under the impression that some ISP throttle down bittorrent Yes. In Italy it's a common use. Since the most usage of it is for... well we all know that... They think all the traffic is for that, and they cut the band in any case despite of what we're really download... Angelo signature.asc Description: This is a digitally signed message part.
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Romain d'Alverny a écrit : On Wed, Jan 12, 2011 at 04:29, Michael Scherer wrote: Le mardi 11 janvier 2011 à 21:45 -0500, andre999 a écrit : Some suggestions for faster downloads without bittorrent. 1) use aria2c (or a similar application), which uses multiple connections, defaulting to 5, and allows multiple mirrors. By default it starts by allocating space for the file to be downloaded, which allows non-sequential downloading of the file, facilitating faster downloading from multiple sites. 2) use mirrors which allow multiple connexions. (Of course, with download software that takes advantage of this.) 3) use multiple mirrors. (Again, according to download software.) Theses 3 suggestions basically put X time the load of the mirror for each client. ( or on more mirror, for that matters ). However, if 1) was to open 5 connections on 5 distinct servers, that would make more sense, no? Right. Another way to look at the question : If 1000 people are downloading from mirrors allowing a total of 2000 connexions, if no-one uses multiple connexions, then 1000 connexions are wasted. These unused connexions would likely be from faster mirrors. The advantage of an application like aria2c, is that it detects automatically the speed of whatever the connexions are available from the urls (mirrors) specified, and chooses the fastest connexions. Thus downloading at an optimal speed for the user. At the same time, it makes the best use of the resources available, since the slowest connexions won't be used. Thus relieving slower mirrors. (Considering direct downloads, not options like P2P.) Of course each user will have their own set of fastest connexions, depending on their location. A user in Australia would have different connexions from someone in France, or here in Canada. So in my view, the approach of aria2c is a win/win for both users and mirrors. With aria2c, 3 mirrors which support a total of 5 connexions gives me optimal speed. (The limit being the speed of my computer.) But then I'm not sure there is so much more value than using a P2P protocol. P2P is great if one has (essentially) unlimited bandwidth, and many others are downloading at more or less the same time, and accessible to the Internet when you are downloading. And it does relieve bandwidth from the mirrors. But it's not as good for bandwidth limited users (which included many of us), or those downloading at a time when not many corresponding P2P downloaders are available. (I knew someone who had a surprise 100$ plus surcharge due to P2P uploading, before understanding the bandwidth usage factor.) An aria2c type solution doesn't require any cooperation from mirrors. Although resource-limited mirrors should protect themselves from this approach, to remain readily accessible. I'm not sure what is required (from Mageia) for P2P, but it could be worth considering. another 2 cents :) Romain -- André
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Angelo Naselli a écrit : mercoledì 12 gennaio 2011 alle 00:12, Michael Scherer ha scritto: I had under the impression that some ISP throttle down bittorrent Yes. In Italy it's a common use. Since the most usage of it is for... well we all know that... They think all the traffic is for that, and they cut the band in any case despite of what we're really download... Angelo Bell Canada does that too -- but only to give priority to other users. (So in non-peak times there is no limitation.) They used the reputed usage, among other arguments, to get precedent-setting approval from regulatory authorities. (They serve maybe half of Canadian users.) A number of smaller internet providers have followed suit. -- André
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
* andre999 (and...@laposte.net) wrote: > Michael Scherer a écrit : >> >> Le mardi 11 janvier 2011 à 20:03 +0100, Marcello Anni a écrit : > > In my case, using aria2c with 2 mirrors and the default 5 connexions is > at least 3 times as fast as a single connexion (to my closest mirror). > And a much greater improvement over other download options I've tried. This is a selfish point of view, nothing is important except the result on your side. But as I can see the results of such thing on distrib-coffee I can tell: - often the bandwidth limititation is client side (eg you, with your DSL connection, and not the mirror, with 100mbits/s or more) - all mirrors have limit, and when the max connection will be reached the mirror will stop serving you (download will be very faster !) Of course, if in the case I don't blacklist mageia and deny download via http/ftp to save my mirror (d-c is already under heavy load). So, no, don't donwload several times from a mirror. BTW: on distrib-coffee bandwidth is regulated by IP, not connection, downloading 5 times will not be faster than 1. -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpuL80ShI4lW.pgp Description: PGP signature
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
* andre999 (and...@laposte.net) wrote: > Romain d'Alverny a écrit : >> >> On Wed, Jan 12, 2011 at 04:29, Michael Scherer wrote: >> >> However, if 1) was to open 5 connections on 5 distinct servers, that >> would make more sense, no? > > Right. > Another way to look at the question : > If 1000 people are downloading from mirrors allowing a total of 2000 > connexions, if no-one uses multiple connexions, then 1000 connexions are > wasted. These unused connexions would likely be from faster mirrors. The upload given by a serveur is split in term of: - connection available - bandwidth available. If the mirrors is connected to Gb (which is more likelly the size of bandwidth for the whole university...), and you split this Gb/s onto 2000 connection you obtain 500kb/s. However, I know only few server in the world really able to read Gb/s from their disk. The top rate I obtain on dc is 400mbis/s (the memory cache help a lot to obtain more). The number of connection is set high because some people download at only 50kb/s and other 2Mbit/s, so the spare bandwidth from the 50kb/s can be given to others. Nothing more. According the graph on distrib-coffee: http://distrib-coffee.ipsl.jussieu.fr/munin/distrib-coffee/distrib-coffee/index.html there is an average of 100 http download at a time, if each connection become 5 (500), you'll reach the limit (200 on this server). What will be the gain on your side ? I talked here only about http, but apply the same to ftp in the same internet connection, so it mean I have 100mbits/s to share between 400 connections. As Mickael explain, I voluntary limit the count of connection per IP, before I did this, the server was overload 12 hour a day, which mean stop serving ! > With aria2c, 3 mirrors which support a total of 5 connexions gives me > optimal speed. (The limit being the speed of my computer.) Can we know the speed of your connection ? Maybe your connection is good enough to provide a mirror for Mageia ? And then you'll be able to test what you're suggesting. What is the result with 5 mirrors and 1 connection per mirror ? Obviously, any mirror in France (I am in France) is able to fulfill my personnal connection... (10mbits/s). -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpMqejuntQRN.pgp Description: PGP signature
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Le mercredi 12 janvier 2011 à 08:05 +0100, Thierry Vignaud a écrit : > On 12 January 2011 04:29, Michael Scherer wrote: > > For example, distrib-coffee could blacklist you if you do this, if you > > are not alone on your network connexion. And when we deployed this > > measure to protect the server, the limit was 2 connexion per address, > > since this was taking too much ressources on the old server ( each http > > request taking 1 process and so memory ). > > switching to nginx would help that According to the current mirror maintainer, the problem is not only on http server memory usage ( I do use nginx myself on my own servers and indeed, that would alleviate this part of the issue ). It is also that doing random access to file result in random seek on the raid array, and on a regular scsi disk ( not on ssd but ssd is not cheap for the moment ), this is more disrupting since the head of the disk is constantly moving to seek the block to fetch. Fetching continuous block from the disk is faster, since there is no delay. Fetching from random part of the disk is more problematic. And the same goes when done on multiple servers. It is lighter on the overall ressources to download on one mirror and stick to it than open 5 connexions and do 5 random seeks on 5 different disk. -- Michael Scherer
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Olivier Thauvin a écrit : * andre999 (and...@laposte.net) wrote: Michael Scherer a écrit : Le mardi 11 janvier 2011 à 20:03 +0100, Marcello Anni a écrit : In my case, using aria2c with 2 mirrors and the default 5 connexions is at least 3 times as fast as a single connexion (to my closest mirror). And a much greater improvement over other download options I've tried. This is a selfish point of view, nothing is important except the result on your side. Sorry, but this can't be serious. If a user is to download say a 4,7G DVD iso, it is 4,7G if the user uses one or 5 connexions, from one or 5 mirrors. And if the 5 connexions go to 5 different mirrors, that is only 20% per mirror. Globally, if the same files are downloaded, the same amount of data is downloaded. What we are talking about here, from the user side, is downloading faster, if the capacity is _available_ on the network of mirrors. Nothing more, nothing less. There are many mirrors which provide multiple connexions, with a very large bandwidth. If users don't use the available connexions on these servers, other mirrors will be more heavily impacted. So, as I said in another post on this thread, it is win-win for both users and mirrors. But as I can see the results of such thing on distrib-coffee I can tell: - often the bandwidth limititation is client side (eg you, with your DSL connection, and not the mirror, with 100mbits/s or more) - all mirrors have limit, and when the max connection will be reached the mirror will stop serving you (download will be very faster !) That is the advantage of applications like aria2c. They search available connexions from the urls (mirrors) given, and choose which give the fastest downloads. The number of connexions vary according to the conditions. With the default of 5 connexions, aria2c tries to maintain 5 connexions, but in practice fluctuates (dynamically) between 3 and 5, according to conditions. (From my experience.) Think of it another way. Scenerio 1) Suppose at this moment the mirror network has 20% free capacity. We download without multiple connections or multiple mirrors, without using this excess capacity. In one hour we are still downloading, at a time when there is no free capacity. Scenerio 2) We download using multiple connexions and multiple mirrors, accelerating the downloads with the otherwise unused 20% free capacity. In one hour, most of our downloads are finished, thus leaving free capacity. Just who is served by scenerio 1 ? The users that don't have access, or the mirrors that are overloaded ? Of course, if in the case I don't blacklist mageia and deny download via http/ftp to save my mirror (d-c is already under heavy load). So, no, don't donwload several times from a mirror. The solution ? If a mirror can't handle multiple connexions, restrict it in some way. Many mirrors do. Either all the time, or from time to time. Or throttle the transfer speed. Just do whatever works best for the mirror in question. However let users use the capacity available. Because it may not be available in one hour, or even 5 minutes. BTW: on distrib-coffee bandwidth is regulated by IP, not connection, downloading 5 times will not be faster than 1. Essentially the same thing. Great, if it works for the mirror. aria2c senses the the bandwidth dynamically, and reacts accordingly. I'm sure that many other applications do the same thing. Don't forget that much of the increase in download speed comes from using multiple mirrors. (Clearly indicated in the aria2c documentation, and verified in my usage.) Why not think globally ? another 2 cents :) -- André
[Mageia-dev] 12/01/2011 meeting
Our next weekly meeting will happen tonight, 20h UTC on #mageia-dev Topics == - mentoring: review on current process - build system status: summary of current status - relation with mdv: asked by several people - speak about what to do on that topic - mageia policies: review - organize maintainership for packages - short notice about FOSDEM Cheers -- Anne http://www.mageia.org
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
* andre999 (and...@laposte.net) wrote: > Olivier Thauvin a écrit : >> * andre999 (and...@laposte.net) wrote: >>> Michael Scherer a écrit : >> This is a selfish point of view, nothing is important except the result >> on your side. > > Sorry, but this can't be serious. > If a user is to download say a 4,7G DVD iso, it is 4,7G if the user uses > one or 5 connexions, from one or 5 mirrors. > And if the 5 connexions go to 5 different mirrors, that is only 20% per > mirror. > Globally, if the same files are downloaded, the same amount of data is > downloaded. > What we are talking about here, from the user side, is downloading > faster, if the capacity is _available_ on the network of mirrors. Exactly, it is seriously a selfish point of view as: - you're talking about users side, me both - IF the capacity, obviously no one really know the capacity of mirrors except admin of them (and I am one of them). Moreover you're mixing "downloading on several server" and "downloading several time on one or more servers". > There are many mirrors which provide multiple connexions, with a very > large bandwidth. If users don't use the available connexions on these > servers, other mirrors will be more heavily impacted. > So, as I said in another post on this thread, it is win-win for both > users and mirrors. All mirrors provides multiple connection, otherwise they would serve only one client a time... And no, it is not win-win, I am trying to explain, munin statistics in hand, only the users think to win something. And you're ignoring my comment as mirror admin. > Think of it another way. > Scenerio 1) Suppose at this moment the mirror network has 20% free > capacity. We download without multiple connections or multiple mirrors, > without using this excess capacity. In one hour we are still > downloading, at a time when there is no free capacity. > > Scenerio 2) We download using multiple connexions and multiple mirrors, > accelerating the downloads with the otherwise unused 20% free capacity. > In one hour, most of our downloads are finished, thus leaving free > capacity. > > Just who is served by scenerio 1 ? Mirrors, they are not serving only you and your 5 connections, they already serving 100 users other than just you ! I cannot see how the mirror will give more bandwidth with 1 or 5 connections. Except in the case each connection take same amount of bandwidth, then you're 5 part while other take one. And this is not a fair use of ressource. > The users that don't have access, or the mirrors that are overloaded ? > >> Of course, if in the case I don't blacklist mageia and deny download via >> http/ftp to save my mirror (d-c is already under heavy load). >> >> So, no, don't donwload several times from a mirror. > > The solution ? If a mirror can't handle multiple connexions, restrict > it in some way. Many mirrors do. Either all the time, or from time to > time. Or throttle the transfer speed. > Just do whatever works best for the mirror in question. > However let users use the capacity available. Because it may not be > available in one hour, or even 5 minutes. The problem is (maybe I am not clear) your 5 connections will get queue letting other people having to wait more to download their file, or getting "too many connections" error. Indeed, this maybe does not matter for you, except when you'll complain because the distribution cannot be download. > > Why not think globally ? I do, especially, I prefectly when http stop to reply, it stop to reply globally, for Mageia but also for Fedora / Ubuntu / Scientific Linux / Arklinux / Gentoo / ... and all other project hosted on mirrors. This is really a more global issue. I also think my mirror have first to serve my work, and time to time my colleague complain about slowdown cause by external downloader (eg YOU). Here my POV: 1) I am mirror admin and get tired of abuse 2) I am mirror manager for the distro a) According the fact doing this create an huge load on some mirrors, b) according we'll have to search mirror around the world and the distrib size is already a problem c) according doing this can discourage people offering space and bandwidth for us I am aginst such things. I cannot deny you to do this, but don't complain if no mirror agree to serve you. -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpRMusvavXnj.pgp Description: PGP signature
Re: [Mageia-dev] Java-Policy first draft published
Farfouille wrote:
> Hi
>
> I have published a first draft of Java application package policy.
> http://mageia.org/wiki/doku.php?id=java_applications_policy
>
> Corrections and comments are welcome.
>
The bit about pre-packaged JARs may cause trouble. In theory, it's
great, but many applications depend upon certain versions of their
utility JARs, and can't all run with the latest versions. Any such app
would have a Requires for its specific version, which would prevent the
utility JAR from being updated with a newer version for other apps.
This is why EJB allows EJB apps to include their own specific versions
of utility JARs, which are visible to them but not to other apps or the
container itself, and also why Maven uses versioned artifacts.
An extreme example of such an app is NetBeans, which includes its own
versions of Ant and Maven.
Also, for such apps, upstream developers may refuse to investigate
issues unless the shipped versions of supplemental JARs are being used.
> As I have never used maven, a deep reread is required
>
>
>
Maven POMs allow the packager to specify required other objects
("artifacts") not only for building the package but for execution as
well. There are central Maven repositories which contain versioned
artifacts for commonly-used projects, e.g. JUnit, and many companies
have site-wide repositories of their own. Finally, every user of Maven
has a personal repository located in $HOME/.m2, which is why the policy
has code for creating this directory.
Repositories are seached for needed artifacts from the most local (the
user's personal repository) to the most remote (the central Maven
repositories) as directed by a settings.xml file in the user's .m2
directory or tags in the individual pom.xml files. The
general intent is to obtain artifacts from the "closest" repository.
Company repositories are not just the central location for
company-specific artifacts, but also a local cache for central Maven
repository artifacts.
>From the policy, it looks like the personal repository for the ID under
which RPM builds the package is wiped clean for every package, and thus
every needed artifact will be downloaded from the remote repositories
for every build. If that's the case, it's an awful waste of bandwidth,
since many of these artifacts are used for every Maven project.
We might want to consider having a central repository for each system,
which would be one level above the individual users' personal
repositories, and would be used before more remote repositories. There
is no harm to allowing artifacts to accumulate there, since all
artifacts are versioned. Cleanup might be possible through filetriggers
if a tool identified all dependent artifacts for each installed Maven
package, and deleted ones no longer used by anything installed. If
there were such a repository, say under /var, then it should be used for
builds rather than an individual repository tied to the RPM userid.
There's also the issue of allowing company or site repositories to be
used if they exist. For packages installed during system installation,
there would be no way for a sysadmin to specify local repositories
unless the install were modified to allow it, but few if any Java
packages (much less Maven ones) are installed at that time. For later
package installs, perhaps a dynamic settings.xml could be created for
the build using XML fragments in something like an
/etc/maven/repositories.d with the same sort of numeric prefix
preference that chkconfig uses to establish precedence.
Finally, there's the issue of using repository artifacts on the system
in the execution-time CLASSPATHs for the Maven applications. Maven has
a plugin which will build a single huge JAR containing an application's
classes as well as classes from every JAR artifact on which it depends,
and many docs recommend this as the way to distribute an application,
but it consumes quite a bit of space as every app is carrying its own
copy of every supplemental utility class it needs when it could probably
find the identical classes in the versioned artifacts in a system
repository. This ties in to the build-classpath and
build-jar-repository capabilities for non-Maven apps.
By the way, the policy should probably contain full usage instructions
for build-classpath and build-jar-repository as well as a description of
the uses to which they are put.
Good Work !
Re: [Mageia-dev] 12/01/2011 meeting
Le mercredi 12 janvier 2011 à 16:51 +0100, Anne nicolas a écrit : > Our next weekly meeting will happen tonight, 20h UTC on #mageia-dev I have another exceptional appointment tonight, sorry for not being there. ( as said you should not have voted for me ). -- Michael Scherer
Re: [Mageia-dev] Java-Policy first draft published
Le mercredi 12 janvier 2011 à 11:24 -0500, Frank Griffin a écrit : > Farfouille wrote: > > Hi > > > > I have published a first draft of Java application package policy. > > http://mageia.org/wiki/doku.php?id=java_applications_policy > > > > Corrections and comments are welcome. > > > > The bit about pre-packaged JARs may cause trouble. In theory, it's > great, but many applications depend upon certain versions of their > utility JARs, and can't all run with the latest versions. Any such app > would have a Requires for its specific version, which would prevent the > utility JAR from being updated with a newer version for other apps. > This is why EJB allows EJB apps to include their own specific versions > of utility JARs, which are visible to them but not to other apps or the > container itself, and also why Maven uses versioned artifacts. That's the same issue for everything. Shipping binary jar given by upstream tarball cause trouble because you 1) cannot patch them in case of bug 2) cannot see how and what was compiled That's not very free software friendly, and I think we should refuse that. -- Michael Scherer
Re: [Mageia-dev] 12/01/2011 meeting
Le 12/01/2011 17:29, Michael Scherer a écrit : > Le mercredi 12 janvier 2011 à 16:51 +0100, Anne nicolas a écrit : >> Our next weekly meeting will happen tonight, 20h UTC on #mageia-dev > > I have another exceptional appointment tonight, sorry for not being > there. > ( as said you should not have voted for me ). I'll be here :) -- Sandro Cazzaniga IRC: Kharec (irc.freenode.net) Twitter: @Kharec
Re: [Mageia-dev] Java-Policy first draft published
Michael Scherer wrote: > Le mercredi 12 janvier 2011 à 11:24 -0500, Frank Griffin a écrit : > >> The bit about pre-packaged JARs may cause trouble. > That's the same issue for everything. > > Shipping binary jar given by upstream tarball cause trouble because you > 1) cannot patch them in case of bug > 2) cannot see how and what was compiled > > That's not very free software friendly, and I think we should refuse > that. > > Granted, but unless every free software project migrates to Maven, you'd be refusing a lot of popular apps. In theory, the packager of such an application could create supplementary packages for the specific versions of included JARs and build them first from source. But for something like NetBeans or Eclipse, that's going to be a lot of work
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Thanks Olivier, for saying this much better than I could wobo, yet another mirror maintainer
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
> Thanks Olivier, for saying this much better than I could > > wobo, yet another mirror maintainer > yes wobo, but except Per Øyvind no one has really answered to my question... i think if that if we want Mageia to become the most popular distro over the world -yep!- we should find a way to augment the overall bandwith available for updates download, and what's better than bittorrent protocol to use user available bandwith to do this? i would like to know: - if it is possible from a techincal pov - if it is convenient and useful (overall in a long-term vision) - if we could plan it for the coming releases (and who could take in charge this) cheers, Marcello
Re: [Mageia-dev] Java-Policy first draft published
Le 12/01/2011 17:45, Frank Griffin a écrit : > > Michael Scherer wrote: >> Le mercredi 12 janvier 2011 à 11:24 -0500, Frank Griffin a écrit : >> >>> The bit about pre-packaged JARs may cause trouble. >> That's the same issue for everything. >> >> Shipping binary jar given by upstream tarball cause trouble because you >> 1) cannot patch them in case of bug >> 2) cannot see how and what was compiled >> >> That's not very free software friendly, and I think we should refuse >> that. >> >> > Granted, but unless every free software project migrates to Maven, you'd > be refusing a lot of popular apps. > > In theory, the packager of such an application could create > supplementary packages for the specific versions of included JARs and > build them first from source. But for something like NetBeans or > Eclipse, that's going to be a lot of work > > > I propose to keep the restriction, but to allow some exception, mainly blockbuster like Eclipse and Netbeans in order to build an appealing distro. Do you know other softwares that are worth to be exception ? -- Farfouille
Re: [Mageia-dev] Java-Policy first draft published
On mercredi 12 janvier 2011 at 17:45, Frank Griffin wrote : > In theory, the packager of such an application could create > supplementary packages for the specific versions of included JARs and > build them first from source. But for something like NetBeans or > Eclipse, that's going to be a lot of work I don't know for netbeans (maybe it is the same as eclipse), but eclipse needs anyway all of its dependencies inside his own plugins directory, so it would anyway be hard to use other system installed jars. But the source of the required version could probably be included in the src.rpm of the requiring plugins and built/installed before them (but I actually have no idea how eclipse plugins are built from source). The problem is, many eclipse plugins (or plugins collections actually) have dependencies on the same libs (for example xalan and xerces) which they all provide in their zip archive. We cannot have them in each package that need them as it would create file conflicts between those packages. So we would need to create separate packages for those "external dependencies plugins". They could be built from their own source (from the official project) and then the real plugins would depend on them. That would probably require a special naming convention for dependencies between plugins (as the package doesn't provide "xalan" but "xalan as an eclipse plugin") Or maybe, if the jar in the external dependency plugin is actually the same (or compatible) version as the on from the normal package maybe we could make a plugin package only containing the eclipse related files (plugin.properties) and creating a symlink to the system wide jar. But that mean that eclipse plugins, which are now frequently provided as single jar files, would need to be packaged as files and directories like in older eclipse releases. What do you think? -- Renaud Michel
Re: [Mageia-dev] Java-Policy first draft published
On mercredi 12 janvier 2011 at 22:41, Renaud MICHEL wrote : > Or maybe, if the jar in the external dependency plugin is actually the > same (or compatible) version as the on from the normal package maybe we > could make a plugin package only containing the eclipse related files > (plugin.properties) and creating a symlink to the system wide jar. But > that mean that eclipse plugins, which are now frequently provided as > single jar files, would need to be packaged as files and directories > like in older eclipse releases. Looking at fedora's eclipse plugins packages, it seems they are symlinking directly the jars from their system bundle in the eclipse plugins directory, with full package and version as symlink name. I thought some eclipse specific files (like plugin.properties) were required inside the jar, but it seems not. -- Renaud Michel
Re: [Mageia-dev] Java-Policy first draft published
Farfouille wrote: > > I propose to keep the restriction, but to allow some exception, mainly > blockbuster like > Eclipse and Netbeans in order to build an appealing distro. > > Do you know other softwares that are worth to be exception ? > The only one that comes to mind is JBoss, but I'm sure there will be others. Any servlet or EJB container is going to be prone to this.
Re: [Mageia-dev] Java-Policy first draft published
Renaud MICHEL wrote: > > Or maybe, if the jar in the external dependency plugin is actually the same > (or compatible) version as the on from the normal package maybe we could > make a plugin package only containing the eclipse related files > (plugin.properties) and creating a symlink to the system wide jar. But that > mean that eclipse plugins, which are now frequently provided as single jar > files, would need to be packaged as files and directories like in older > eclipse releases. > > What do you think? > > I think we want to consider carefully before we try to replace complex existing plugin installation procedures in various products. This issue is also being discussed relative to Firefox in another thread. Users of things like NetBeans and Eclipse are used to the mechanisms provided by those tools for managing plugin installation, and are not going to take kindly to having to use rpmdrake instead. And in some cases, e.g. Maven, you don't have a choice - if Maven needs it and it isn't there, it will get it from repositories whether or not you have packaged a plugin as an RPM. At some point, you just give up and accept the fact that you can't reform the entire world. As more and more projects migrate to Maven, the problem (sort of) goes away. All of our suggestions here are basically trying to recreate what Maven does without requiring projects to use it.
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
* Marcello Anni (marcello.a...@alice.it) wrote: > > Thanks Olivier, for saying this much better than I could > > > > wobo, yet another mirror maintainer > > > yes wobo, but except Per Øyvind no one has really answered to my question... > i think if that if we want Mageia to become the most popular distro over the > world -yep!- we should find a way to augment the overall bandwith available > for > updates download, and what's better than bittorrent protocol to use user > available bandwith to do this? i would like to know: > > - if it is possible from a techincal pov > - if it is convenient and useful (overall in a long-term vision) > - if we could plan it for the coming releases (and who could take in charge > this) I can quickly answer to this: - bittorent is denied where distrib-coffee is hosted. This close every discuss on this side. But I can probably setup a bittorent server elsewhere (it is nice to work with two different university). But first I have to ask to network administrator. From my own experience, I never saw bittorent so efficient for downloading, except it reduce the bandwidth for the team making the iso available (and this is still not sure). BTW: It's time to find mirrors for the distribution ! Any volunteer ? http://mirrors.mageia.org/ Regards. > > > cheers, > Marcello -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpl5rxcrKs6o.pgp Description: PGP signature
Re: [Mageia-dev] Java-Policy first draft published
On mercredi 12 janvier 2011 at 23:54, Frank Griffin wrote : > Users of things like NetBeans and Eclipse are used to the mechanisms > provided by those tools for managing plugin installation, and are not > going to take kindly to having to use rpmdrake instead. Well, I use eclipse everyday (arrive at work, start eclipse and close it at the end of day before leaving) but I don't like its trying to duplicate what package managers do better. I want to know what' installed on may computer, how and where. For years I used to grab the pre-build archives and manage their dependencies by hand. A few month ago, I decided to repackage those as rpms with proper dependencies between packages to have urpmi do the work for me. It was a little complicated to find in the features and packages where their dependencies were stored, to extract them automatically at package build. So I was able to split it in many sub-packages to be able to install only what I need. The only downside is that it is built from binary archives and not from source, but for my own use it does work well. Back to your remark, people who don't like to have eclipse installed via rpm simply won't. They will continue to download the tarball from eclipse.org and install the plugins they want, either from archives or update sites. They can also install only the platform from the rpm and, if the - configuration and -data options are properly added to eclipse startup, they will be able to install in their home dir the plugins they want using the update sites. I won't comment on maven as I don't know how it works (the last time I tried to build from source a program using maven I did not find what options I was supposed to give to the mvn command to have it built). -- Renaud Michel
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
2011/1/13 Olivier Thauvin : > > BTW: It's time to find mirrors for the distribution ! Any volunteer ? > http://mirrors.mageia.org/ I will set up our mirror at mandrivauser.de within next week, will use dc as source as I already do for Mandriva. -- wobo
Re: [Mageia-dev] Java-Policy first draft published
Renaud MICHEL wrote: > On mercredi 12 janvier 2011 at 23:54, Frank Griffin wrote : > >> Users of things like NetBeans and Eclipse are used to the mechanisms >> provided by those tools for managing plugin installation, and are not >> going to take kindly to having to use rpmdrake instead. >> > Well, I use eclipse everyday (arrive at work, start eclipse and close it at > the end of day before leaving) but I don't like its trying to duplicate what > package managers do better. I want to know what' installed on may computer, > how and where. > I think you're the exception. > For years I used to grab the pre-build archives and manage their > dependencies by hand. A few month ago, I decided to repackage those as rpms > with proper dependencies between packages to have urpmi do the work for me. > It was a little complicated to find in the features and packages where their > dependencies were stored, to extract them automatically at package build. > So I was able to split it in many sub-packages to be able to install only > what I need. > The only downside is that it is built from binary archives and not from > source, but for my own use it does work well. > I think you're exceptionally industrious. > Back to your remark, people who don't like to have eclipse installed via rpm > simply won't. They will continue to download the tarball from eclipse.org > and install the plugins they want, either from archives or update sites. > They can also install only the platform from the rpm and, if the - > configuration and -data options are properly added to eclipse startup, they > will be able to install in their home dir the plugins they want using the > update sites. > No they won't. They won't know anything about the distinction between installing via RPM or via tarball. They'll look in rpmdrake for Eclipse initially, install it from there, and then use the normal Eclipse mechanism to install plugins. They'll do this because we tell them to use rpmdrake to look for software, and they'll do the plugins the way they're used to or the way the Eclipse docs tell them to. > I won't comment on maven as I don't know how it works (the last time I tried > to build from source a program using maven I did not find what options I was > supposed to give to the mvn command to have it built). > > Maven, like Eclipse and NetBeans, has its own infrastructure for obtaining artifacts (sort of like plugins). The difference is that it isn't an active thing like Eclipse plugins. Eclipse won't decide you need something and automatically install it; Maven will. If Maven needs an artifact and it isn't already in your repository (because you installed some RPM that put it there), it will fetch it via HTTP automatically regardless of whether you have an RPM for it or not. The result of all this is that you're wasting your time trying to package plugins and such as RPMs for these products. People aren't going to use rpmdrake for this because they are used to doing it a different way, and you'll end up with an unmaintainable mix of stuff installed via RPM and stuff installed via whatever.
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Olivier Thauvin a écrit : * andre999 (and...@laposte.net) wrote: Romain d'Alverny a écrit : On Wed, Jan 12, 2011 at 04:29, Michael Scherer wrote: However, if 1) was to open 5 connections on 5 distinct servers, that would make more sense, no? Right. Another way to look at the question : If 1000 people are downloading from mirrors allowing a total of 2000 connexions, if no-one uses multiple connexions, then 1000 connexions are wasted. These unused connexions would likely be from faster mirrors. The upload given by a serveur is split in term of: - connection available - bandwidth available. If the mirrors is connected to Gb (which is more likelly the size of bandwidth for the whole university...), and you split this Gb/s onto 2000 connection you obtain 500kb/s. Please note that my example was hypothetical, for multiple sites :) Sorry if that wasn't clear. I don't know of _any_ single sites able to do that. (Although I don't doubt that they exist.) (When I started the example, I wrote 20 mirrors, but then erased it because it didn't seem relevant to my point.) However, I know only few server in the world really able to read Gb/s from their disk. The top rate I obtain on dc is 400mbis/s (the memory cache help a lot to obtain more). The number of connection is set high because some people download at only 50kb/s and other 2Mbit/s, so the spare bandwidth from the 50kb/s can be given to others. Nothing more. exactly According the graph on distrib-coffee: http://distrib-coffee.ipsl.jussieu.fr/munin/distrib-coffee/distrib-coffee/index.html there is an average of 100 http download at a time, if each connection become 5 (500), you'll reach the limit (200 on this server). What will be the gain on your side ? In that context, obviously. The idea wasn't that most sites would have 5 connexions - rather to allow up to 5 (depending on the traffic at the moment. I would expect most sites to have one or 2 connexions. And preferably use multiple mirrors, as recommended by aria2c documentation. I talked here only about http, but apply the same to ftp in the same internet connection, so it mean I have 100mbits/s to share between 400 connections. As Michael explain, I voluntary limit the count of connection per IP, before I did this, the server was overload 12 hour a day, which mean stop serving ! A very good idea. I would do the same thing in your situation. Actually, aria2c would work somewhat better if all sites with more limited capacity did the same thing. Because the benefit comes from making use of the unused capacity (of the moment) of mostly larger mirror sites. And trying to access sites already at capacity slows somewhat the downloading. Unfortunately not all download software monitors the connexion speed to dynamically choose the fastest connexions, as does aria2c. With aria2c, 3 mirrors which support a total of 5 connexions gives me optimal speed. (The limit being the speed of my computer.) Can we know the speed of your connection ? Fast, but I don't know exactly. It is at the local library, via wi-fi. (I use it for downloading ISOs because my home connexion has limited bandwidth.) It is on the municipal network with telephony and all, and I access it in the evening, when essentially only the library is open. I have seen another user downloading at over 900 KB/sec while I was downloading, at a somewhat slower speed. (A while back.) At the library with up to 5 connexions on 2 or 3 mirrors, it is always over 600 KB/sec, often over 700 KB/sec. The last time I did a comparison (1 connexion/1 mirror vs max 5 connexions/2 mirrors) for a DVD, the ratio of speed was approx 3,5. (From summary notes on my computer.) Maybe your connection is good enough to provide a mirror for Mageia ? And then you'll be able to test what you're suggesting. Unfortunately, my home connexion tops out at 400KB/sec (officially 512KB/sec), but usually is less than half that. (Variable-speed slow DSL.) And the upload speed is severely limited by my FAI, as well as a limited bandwidth. Wouldn't mind trying a mirror when better connexions become available. (Don't know if I would go for something as big as Mageia, though.) What is the result with 5 mirrors and 1 connection per mirror ? At the library, it tops out with 3 mirrors and 5 connexions. The best was over 700KB/sec. (4 mirrors wasn't any faster.) (I borrowed a faster computer from someone once, and it averaged well over 800 KB/sec on 1 mirror/5 connexions, for download of a DVD.) Obviously, any mirror in France (I am in France) is able to fulfill my personnal connection... (10mbits/s). J'aimerais bien ça :) Where I live (banlieue de Montréal), I couldn't get more than 2Mbits/s (officially), if that. And fairly expensive, as well. With very limited upload speeds. -- André
Re: [Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download
Olivier Thauvin a écrit : * Marcello Anni (marcello.a...@alice.it) wrote: Thanks Olivier, for saying this much better than I could wobo, yet another mirror maintainer yes wobo, but except Per Øyvind no one has really answered to my question... i think if that if we want Mageia to become the most popular distro over the world -yep!- we should find a way to augment the overall bandwith available for updates download, and what's better than bittorrent protocol to use user available bandwith to do this? i would like to know: - if it is possible from a techincal pov - if it is convenient and useful (overall in a long-term vision) - if we could plan it for the coming releases (and who could take in charge this) I can quickly answer to this: - bittorent is denied where distrib-coffee is hosted. This close every discuss on this side. But I can probably setup a bittorent server elsewhere (it is nice to work with two different university). But first I have to ask to network administrator. From my own experience, I never saw bittorent so efficient for downloading, except it reduce the bandwidth for the team making the iso available (and this is still not sure). I agree (for what it's worth). Technically, it is possible, and a number of applications such as aria2c can use bittorrent. However the only time something like bittorrent is significantly beneficial is when a large number of users with high uploading bandwidth download the same large files (such as ISOs) at about the same time. Which is only likely to occur shortly after a new release. When in addition to reducing bandwidth load on participating mirrors, makes downloads for such users faster. (I would say largely due to reduced traffic on the mirrors.) At other times, it tends to just add a bit of bandwidth usage to such mirrors. Still, it could be useful to have a few such mirrors, especially later on when Mageia is more established, At least around release time. BTW: It's time to find mirrors for the distribution ! Any volunteer ? http://mirrors.mageia.org/ I'll contact some local mirrors to see if they will. (Mandriva had one, up to 2009.) Regards. cheers, Marcello regards -- André
