On Sun, Aug 26, 2012 at 3:50 PM, Dave Fisher <dave2w...@comcast.net> wrote: > > On Aug 26, 2012, at 12:38 PM, Rob Weir wrote: > >> On Sun, Aug 26, 2012 at 3:20 PM, Dave Fisher <dave2w...@comcast.net> wrote: >>> Hi, >>> >>> We need to do more work to have proper compliance with Apache >>> Infrastructure policy in managing external dependencies. >>> >>> I may not be precisely correct and am looking for confirmation, but In >>> general i think we need to >>> >>> (1) Completely avoid using svn.apache.org. I don't think we are allowed to >>> do this even as a backup URL. >>> >>> (2) Use mirrors or maven for ASF dependencies where we use the current >>> release. If we use mirrors then archive.apache.org should be the backup for >>> the mirror so that we aren't in trouble if the project has a release. If a >>> maven repository were used then there would be no issue. >>> >>> (3) If we use mirrors then we should allow the user to choose which mirror. >>> >>> If we decide to take the time to go the maven route. I can use the example >>> of ant and maven repos from the Apache POI build.xml. >>> >>> Notes about maven repos. Infra [1], maven central [2] and example of an >>> externally hosted repo [3] >>> >>> This area needs careful attention. >>> > > Please take the time to study the Maven project before you attempt to > reinvent it. [1] >
I think one could fairly look at the relative creation dates and see that Maven reinvented OpenOffice's dependency tracking system ;-) >> Note that this move is exactly the wrong thing to do if we want have >> buildbots build binaries that are assumed to be safe and therefore >> signable. Instead of the security and verifiability of ASF-run host, >> we're putting the dependencies off to a dozen different remote sites, >> with no visibility into their site's mechanisms for vetting changes, >> access controls, auditability of changes, even basics like ensuring >> domain names are renewed and not poached by others. >> >> Do we really think other websites are as secure as the ones that Infra >> operates? If so we should move the source code to the other sites as >> well, right? > > This is not the question. My comments were about using the ASF sites > properly. Joe has told us that we are doing this out of policy. > My point is that if we seek to have build-bot automated signed binaries it is entirely foreseeable that what we're proposing to do with 3rd party dependencies will also be out of policy, due to the security concerns. To do this we'll need to do more than just link to random sites on the web for dependencies. >> >> No easy resolution of this, but we might mitigate the risk by putting >> all of the dependencies to Apache-Extras and load from there >> primarily. And if at all possible make sure all change notifications >> from there get echoed to the ooo-committs lis. We have a better >> chance of exercising now screwing up if we control rather than having >> multiple 3rd parties control. > > We need to differentiate between ASF projects and third party projects. Look > at the current process before you go nuts assuming that everything is > changing. > Look at Andre's note from August 3rd. First locations searched for dependencies is the original website where it is hosted. Then it checks Apache Extras. Then it checks SVN. Taking SVN out of the loop solves one issue. But we still have the other issue -- I think Dennis groks this as well -- that our primary search location is less secure than the alternatives. >> >> Another option would be to use cryptographic means to ensure the >> integrity of the remote dependencies, e.g., detached signatures. That >> doesn't protect us from another website going down, temporarily or >> permanently, but it does allow us to verify that what we are >> downloading has not been tampered with. > > This is already part of the current process. The signatures are in > download_external_dependencies.pl. The Central Maven Repository uses these as > well. > Those are MD5 hashes, not signatures. MD5 has been broken since 1996: http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities Again, two relatively simple improvements: 1) Search Apache Extras first or 2) Have the current script verify a detached signature rather than relying on MD5 hashes Note: the MD5 hashes were OK previously, since we also maintained control over the actually files. This was true under Sun/Oracle, as well as earlier in the Podling. But the risk profile changes significantly once we start retrieving these files from a website over which we have no control. -Rob > [1] http://maven.apache.org/ > >> > > >> >> -Rob >> >> >>> The current script is here: >>> main/solenv/bin/download_external_dependencies.pl >>> >>> Regards, >>> Dave >>> >>> [1] http://apache.org/dev/repository-faq.html and >>> [2] http://maven.apache.org/guides/mini/guide-central-repository-upload.html >>> [3] >>> http://repo.maven.apache.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.pom >>> >>> >>> On Aug 26, 2012, at 11:58 AM, w...@apache.org wrote: >>> >>>> Author: wave >>>> Date: Sun Aug 26 18:58:08 2012 >>>> New Revision: 1377482 >>>> >>>> URL: http://svn.apache.org/viewvc?rev=1377482&view=rev >>>> Log: >>>> one more small step to infra compliance. still to do removing use of svn >>>> as a backup and for current releases of ASF software the archive is not >>>> proper - either a mirror or the maven repository is required. >>>> >>>> Modified: >>>> incubator/ooo/trunk/main/external_deps.lst >>>> >>>> Modified: incubator/ooo/trunk/main/external_deps.lst >>>> URL: >>>> http://svn.apache.org/viewvc/incubator/ooo/trunk/main/external_deps.lst?rev=1377482&r1=1377481&r2=1377482&view=diff >>>> ============================================================================== >>>> --- incubator/ooo/trunk/main/external_deps.lst (original) >>>> +++ incubator/ooo/trunk/main/external_deps.lst Sun Aug 26 18:58:08 2012 >>>> @@ -72,7 +72,7 @@ if ( true ) >>>> if (SOLAR_JAVA == TRUE) >>>> MD5 = 17960f35b2239654ba608cf1f3e256b3 >>>> name = lucene-2.9.4-src.tar.gz >>>> - URL1 = >>>> http://www.us.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz >>>> + URL1 = >>>> http://archive.apache.org/dist/lucene/java/2.9.4/lucene-2.9.4-src.tar.gz >>>> URL2 = $(OOO_EXTRAS)$(MD5)-$(name) >>>> # Fall back to a version in SVN from a previous revsion. >>>> URL3 = >>>> http://svn.apache.org/repos/asf/!svn/bc/1337615/incubator/ooo/trunk/ext_sources/$(MD5)-$(name) >>>> >>>> >>> >
