[ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Hi all, Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory modules, our Buildserver has enough RAM for the Win7 buildslave VM now. I've fired it up and while builds are much slower than before, they are at least being done again! Note that Doxygen, ISO hosting, and all three buildslaves are on a single HDD-backed server right now, so don't expect any performance miracles. I'm counting on Aleksey here to get us our remaining servers back this month :) Unfortunately, our HDDs were also lacking space, so I had to move the public "bootcd_old" folder from iso.reactos.org to another (private) place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. Any idea how to deal with them in the long run? I can basically think of 4 possibilities: * We remove ISOs older than 4 years, because nobody would really do regression-testing with them. * We buy additional HDDs every year and continue to host them ourselves just next to the newer ISOs. * We find a free file hosting service for OSS projects that can cope with these amounts of data. Not sure SF.net is the right choice here.. * We choose a paid data storage service like Amazon AWS. As always, comments are very welcome! - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Hi Colin, On our side at triplecheck we are increasing the infrastructure, exactly with the intention of filling it up with software. Would happily make available 1Tb of long term storage or more as needed to save the old ISOs at no cost. The only nuisance is that we can't give direct access to the storage for security reasons (it is cut off from the Internet). Still, it is possible to download through a request mechanism that puts the needed files on a public server for download. We are negotiating the infrastructure growth, around the 1st of November should be possible to confirm that we can host these files. Nuno On 04/10/16 21:10, Colin Finck wrote: Hi all, Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory modules, our Buildserver has enough RAM for the Win7 buildslave VM now. I've fired it up and while builds are much slower than before, they are at least being done again! Note that Doxygen, ISO hosting, and all three buildslaves are on a single HDD-backed server right now, so don't expect any performance miracles. I'm counting on Aleksey here to get us our remaining servers back this month :) Unfortunately, our HDDs were also lacking space, so I had to move the public "bootcd_old" folder from iso.reactos.org to another (private) place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. Any idea how to deal with them in the long run? I can basically think of 4 possibilities: * We remove ISOs older than 4 years, because nobody would really do regression-testing with them. * We buy additional HDDs every year and continue to host them ourselves just next to the newer ISOs. * We find a free file hosting service for OSS projects that can cope with these amounts of data. Not sure SF.net is the right choice here.. * We choose a paid data storage service like Amazon AWS. As always, comments are very welcome! - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev -- Managing Director, TripleCheck GmbH twitter @nn81 | UK +44 7453 648423 | DE +49 162 327 2929 http://triplecheck.net ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
What about deduplication? Am 04.10.2016 um 22:10 schrieb Colin Finck: Hi all, Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory modules, our Buildserver has enough RAM for the Win7 buildslave VM now. I've fired it up and while builds are much slower than before, they are at least being done again! Note that Doxygen, ISO hosting, and all three buildslaves are on a single HDD-backed server right now, so don't expect any performance miracles. I'm counting on Aleksey here to get us our remaining servers back this month :) Unfortunately, our HDDs were also lacking space, so I had to move the public "bootcd_old" folder from iso.reactos.org to another (private) place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. Any idea how to deal with them in the long run? I can basically think of 4 possibilities: * We remove ISOs older than 4 years, because nobody would really do regression-testing with them. * We buy additional HDDs every year and continue to host them ourselves just next to the newer ISOs. * We find a free file hosting service for OSS projects that can cope with these amounts of data. Not sure SF.net is the right choice here.. * We choose a paid data storage service like Amazon AWS. As always, comments are very welcome! - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Am 05.10.2016 um 19:48 schrieb Christoph von Wittich: > What about deduplication? We would need to test if this yields any good results on our already compressed data. Even if we extract the currently 7zip-compressed ISOs, the "reactos.cab" inside the ISOs would still be a compressed file. What deduplication solution do you suggest anyway? ZFS? - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
On 2016-10-05 03.10, Colin Finck wrote: > Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory > modules, our Buildserver has enough RAM for the Win7 buildslave VM now. (y) > Unfortunately, our HDDs were also lacking space, so I had to move the > public "bootcd_old" folder from iso.reactos.org to another (private) > place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. > Any idea how to deal with them in the long run? I can basically think of > 4 possibilities: > > * We remove ISOs older than 4 years, because nobody would really do > regression-testing with them. A reasonable option, imo. > * We buy additional HDDs every year and continue to host them ourselves > just next to the newer ISOs. Sounds right to me.. HDDs are not too expensive. I'm willing to pitch in, despite my decrepit pension. Speaking of which, maybe we could all pitch in to build a new buildserver? I built my latest workstation for just over 500 euro. (Xeon E3 3.1GHz, 16 GB RAM, 2x1 TB HDD) > * We find a free file hosting service for OSS projects that can cope > with these amounts of data. Not sure SF.net is the right choice here.. > > * We choose a paid data storage service like Amazon AWS. Oh god, no.. Amazon is a nightmare security-wise. It's practically impossible to define reasonable firewall rules for Amazon clients, since AWS shifts their IP range around all the time. > As always, comments are very welcome! And if fun is permissible.. Someone who likes to tinker could have fun building a CD changer archive :) Some Al rails, stepper motors, drive belts, servos, a normal BR burner drive, and an Arduino or two to control the mechanics and take commands from the PC. E.g. pater noster stores don't require too much space. While it would be slow to fetch old stuff, it would practically never run out of storage space. Best Regards L. > > > - Colin > > ___ > Ros-dev mailing list > Ros-dev@reactos.org > http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Am 14.10.2016 um 05:55 schrieb Neo Love: >> * We buy additional HDDs every year and continue to host them ourselves >> just next to the newer ISOs. > > Sounds right to me.. HDDs are not too expensive. > I'm willing to pitch in, despite my decrepit pension. > > Speaking of which, maybe we could all pitch in to build a new buildserver? > I built my latest workstation for just over 500 euro. > (Xeon E3 3.1GHz, 16 GB RAM, 2x1 TB HDD) Having additional servers is not a problem of hardware costs, but of hosting and logistics. Our critical services are all hosted on dedicated servers at a decent hosting provider. These servers handle standard tasks very well. But as soon as we have special requirements (e.g. lots of RAM, HDD space or multiple physical servers for VM testing), it is impossible or too expensive. Therefore, we also bought some specialized servers. They have always been hosted in countries where electricity and fast internet connections are cheap and where we have a project member to take care of them. But as you can imagine, this can never match the availability of a real hosting provider. In the next few days, I'm expecting some more servers to be set up at our project member's location. If that turns out well, we can think about getting additional storage for them. I'm still open for alternative/failover hosting solutions though as this shouldn't depend on a single location and person. - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
On 2016-10-18 01.51, Colin Finck wrote: > Am 14.10.2016 um 05:55 schrieb Neo Love: >>> * We buy additional HDDs every year and continue to host them ourselves >>> just next to the newer ISOs. >> Sounds right to me.. HDDs are not too expensive. >> I'm willing to pitch in, despite my decrepit pension. >> Speaking of which, maybe we could all pitch in to build a new buildserver? > Having additional servers is not a problem of hardware costs, but of > hosting and logistics. Oh.. It sounded like there was a hardware issue, since you didn't have enough RAM and the build server was too slow. And yes, of course running costs will always dominate over hardware investment (which can be cheap using Asian sources). > Our critical services are all hosted on dedicated servers at a decent > hosting provider. These servers handle standard tasks very well. > ... > Therefore, we also bought some specialized servers. They have always > been hosted in countries where electricity and fast internet connections > are cheap and where we have a project member to take care of them. Yes, web/FTP/mail services &c. can be had quite cheaply these days. It makes sense to move the bandwidth load for e.g. web off our own machines. I just thought that all build server(s) and test server(s) were our own machines due to the special programming requirements they have. South Korea, Ukraine, and Russia have well developed internet infrastructure at low/competitive cost (per my research at least). Hmm.. Guess that's why you pointed sideways at Aleksey :) > But as you can imagine, this can never match the availability of a real > hosting provider. Availability, yes.. But requirement-wise I doubt a provider like AWS or Akamai is a feasible way to go (economically). They would most likely charge an arm and a leg to get all our special programming on their (virtual) machines, even if they would be able to manage it (in an agnostic manner). > In the next few days, I'm expecting some more servers to be set up at > our project member's location. If that turns out well, we can think > about getting additional storage for them. Rented machines? Or is someone lending us the use of hardware (and/or bandwidth) out of the goodness of their hearts? :) > I'm still open for alternative/failover hosting solutions though as this > shouldn't depend on a single location and person. Thailand, where I live, have a consumer peak speed of 58 Mb/s (avg ~10), and electricity is fairly cheap; 6-13 US cent/kWh. (A home 10M/640k ADSL connect like mine is ~16 US $/month.) If you'd like, I can look into cost and availability for a fixed line or fiber. The idea with a self-built CD changer archive was something I just came up with since I like to tinker myself, and thought it would be fun to build.. Never mind ;) Best Regards L. > > > - Colin > > ___ > Ros-dev mailing list > Ros-dev@reactos.org > http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
