Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Freitag, 16. Oktober 2015, 16:13:32 schrieb Matthew Toseland:
> Then we should fix the build script to auto-detect their locations?
> First we need a build script so we can have such one-time logic...

No, we should just fix the README.

Best wishes,
Arne
-- 
Ein Mann wird auf der Straße mit einem Messer bedroht. 
Zwei Polizisten sind sofort da und halten ein Transparent davor. 

"Illegale Szene. Niemand darf das sehen."

Der Mann wird ausgeraubt, erstochen und verblutet, 
denn die Polizisten haben beide Hände voll zu tun. 

Willkommen in Deutschland. Zensur ist schön. 
  ( http://draketo.de/stichwort/zensur )



signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Matthew Toseland]

On 15/10/15 21:29, Arne Babenhauserheide wrote:
> Am Donnerstag, 15. Oktober 2015, 18:26:49 schrieb Florent Daigniere:
>> On Thu, 2015-10-15 at 17:38 +0200, Arne Babenhauserheide wrote:
>>> Though maybe we could also get the same done with easier to follow
>>> documentation (just hosting the jars where people can download them)
>> Like https://javadoc.freenetproject.org/ ?
> No (though that’s good to have), rather like a 5 step guide for
> setting up a Freenet development environment, written in the
> README. There is README.building.md, but the junit4 and hamcrest
> packages never worked for me, and the non-eclipse text is somehow like
> a maze to navigate.
Then we should fix the build script to auto-detect their locations?
First we need a build script so we can have such one-time logic...
> There’s “you need these packages” and then two lines are missing:
>
> build Freenet by calling `ant`.
> Stop freenet, copy dist/freenet.jar into your freenet folder, then start 
> Freenet to run your own bulid.
>
> And a line in README.md which points to README.building.md.
>
> Best wishes,
> Arne



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Dienstag, 13. Oktober 2015, 18:07:02 schrieb Ian:
> If your bootstrap gcc isn't compiled from source then it's irrelevant what
> else is.  http://c2.com/cgi/wiki?TheKenThompsonHack

There are ways to counter that to some degree:
http://www.dwheeler.com/trusting-trust/

Fully Countering Trusting Trust through Diverse Double-Compiling (DDC)
  - Countering Trojan Horse attacks on Compilers

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein, 
ohne es zu merken. 
- Arne (http://draketo.de)




signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Dienstag, 13. Oktober 2015, 21:12:28 schrieb Steve Dougherty:
> > Why not? It could simplify getting into development a lot if people
> > would only have to install a generic java development environment.
> It takes extra storage and bandwidth for something the vast majority of
> people will never need.

Can we estimate how much that is? (whether is’t worth it)

Though maybe we could also get the same done with easier to follow
documentation (just hosting the jars where people can download them)

Best wishes,
Arne

signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Donnerstag, 15. Oktober 2015, 18:26:49 schrieb Florent Daigniere:
> On Thu, 2015-10-15 at 17:38 +0200, Arne Babenhauserheide wrote:
> > Though maybe we could also get the same done with easier to follow
> > documentation (just hosting the jars where people can download them)
> 
> Like https://javadoc.freenetproject.org/ ?

No (though that’s good to have), rather like a 5 step guide for
setting up a Freenet development environment, written in the
README. There is README.building.md, but the junit4 and hamcrest
packages never worked for me, and the non-eclipse text is somehow like
a maze to navigate.

There’s “you need these packages” and then two lines are missing:

build Freenet by calling `ant`.
Stop freenet, copy dist/freenet.jar into your freenet folder, then start 
Freenet to run your own bulid.

And a line in README.md which points to README.building.md.

Best wishes,
Arne
--
Ein Würfel System - einfach saubere Regeln: 

- http://1w6.org



signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Florent Daigniere]

On Thu, 2015-10-15 at 17:38 +0200, Arne Babenhauserheide wrote:
> Am Dienstag, 13. Oktober 2015, 21:12:28 schrieb Steve Dougherty:
> > > Why not? It could simplify getting into development a lot if
> > > people
> > > would only have to install a generic java development
> > > environment.
> > It takes extra storage and bandwidth for something the vast
> > majority of
> > people will never need.
> 
> Can we estimate how much that is?

Of course, feel free to.

>  (whether is’t worth it)
> 

It's not. What you're suggesting is just insane. We used to ship the
source code (and unit tests) within the released jar; It's never lead
to a surge in contributors.

> Though maybe we could also get the same done with easier to follow
> documentation (just hosting the jars where people can download them)
> 
> 

Like https://javadoc.freenetproject.org/ ?

Florent

signature.asc
Description: This is a digitally signed message part
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[nextgens]

On Mon, Oct 12, 2015 at 01:09:48PM -0400, Ian wrote:
> On Mon, Oct 12, 2015 at 12:27 PM, Arne Babenhauserheide 
> wrote:
> 
> > Am Sonntag, 11. Oktober 2015, 20:37:32 schrieb Zlatin Balevsky:
> > > Developers who care about their anonymity can force gradle or maven to
> > use
> > > a tor proxy
> >
> > Can we make Tor or repo-over-freenet the default for people who build
> > freenet?
> >
> 
> Since Freenet can (in theory) do it, I think it would be much better to use
> Freenet from an "eat your own dogfood" perspective.  I think the only
> danger here is that we further complicate things for a developer trying to
> get into the project.
> 
> Ian.

That's far from the only danger...

If we make "building freenet" depend on "running freenet" to "download freenet 
over freenet", I'd like someone to explain to me how it can work without 
trusting someone else's binaries.

...

As for using Tor, sure we could. We could also recompile the JVM and reinvent a 
source-only distro...

I'm sure we need more scope creep; The project's goals are simple enough that 
we can afford it.

Florent
PS: My take on it is that we should document what the build process does... and 
if people are not happy with it they can run it in a throwaway Tails instance.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[nextgens]

On Mon, Oct 12, 2015 at 07:47:22PM +0100, Matthew Toseland wrote:
> The second stage is this: Do we want to write a Gradle plugin to manage
> dependencies.properties? Essentially this would make adding external
> run-time dependencies significantly easier. The cost is we'd need to
> maintain a repository, including obtaining and checking signatures etc.
> 

That's the type of things gradle shines at... The question is do we have 
someone who:
  1) wants to do it?
  2) knows how to speak gradle?
  3) knows/want to learn the current process?

IMHO it's the type of tasks for which we should have a bounty. There's a 
long-term reason on why the project should have it (Ian has highlighted it 
enough), it improves on the status quo (we all agree on that) and it's 
self-contained... but it's not a small task, the skillset required is very 
specialized (spoilers: it's not just ant that you need to know, we also have 
native code... all in all we're talking about almost every single build system 
you can think of) and the people who could do it have arguably "other/better" 
things to do.

Florent
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[nextgens]

On Tue, Oct 13, 2015 at 10:13:07AM -0400, Ian wrote:
> On Tue, Oct 13, 2015 at 3:26 AM,  wrote:
> >
> > That's far from the only danger...
> >
> > If we make "building freenet" depend on "running freenet" to "download
> > freenet over freenet", I'd like someone to explain to me how it can work
> > without trusting someone else's binaries.
> >
> 
> Can you explain to me how anyone can use a modern computer for anything
> without trusting someone else's binaries?  Let's live in reality here.
> 

The people who insist on building freenet (and everything else) from source are 
likely the same who insit on "not downloading the dependencies insecurely from 
the internet"

What I am saying is that we should explicitely make it a non-goal and let 
$peopleWhoCare deal with it (by installing Tails).

> Regardless, the top priority (as far as this part of the discussion is
> concerned) is to migrate the project over to a modern build system, most
> likely Gradle.  Support for downloading dependencies from Freenet (or
> through Tor) would be nice but should be way down the list of priorities.
> 

Gradle uses ant internally to "do the build" (if you manage to use it without 
ant, I'd like to know how). If you don't need dependency management, a 3 line 
gradle script is enough to "call ant" and provide the newbee developer 
experience you're after. Is that what we want?

If not, look around. You'll see that most of our dependencies are long-dead 
upstream and that none of them provide fancy repositories we can pull stuff 
from... So we will end up hosting them, just like we do now with freenet-ext.jar

Florent
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Steve Dougherty]

We already distribute runtime dependencies with Freenet; I don't think it
would be reasonable to distribute things required only for development like
JUnit / Hamcrest.

From mobile


On Mon, Oct 12, 2015, 5:37 PM Arne Babenhauserheide @ web.de > wrote:

Am Montag, 12. Oktober 2015, 13:09:48 schrieb Ian:
> > Can we make Tor or repo-over-freenet the default for people who build
> > freenet?
>
> Since Freenet can (in theory) do it, I think it would be much better to
use
> Freenet from an "eat your own dogfood" perspective.  I think the only
> danger here is that we further complicate things for a developer trying to
> get into the project.

I agree.

Could we just ship all required libraries (jars) for building Freenet
with every freenet install, so developers could just copy them over
from their Freenet directory (or point the build tool to the freenet
install)?

That should be easier than any other way of installation, because
everyone running freenet would already have all the libraries in
place. Also it would be ensured by our auto-updater that the libraries
are always the right ones for the current version of Freenet.

Best wishes,
Arne___
Devl mailing list
Devl @ freenetproject.org

https ://

emu.freenetproject.org
/
cgi
-bin/mailman/
listinfo
/
devl

___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Ian]

On Tue, Oct 13, 2015 at 3:26 AM,  wrote:
>
> That's far from the only danger...
>
> If we make "building freenet" depend on "running freenet" to "download
> freenet over freenet", I'd like someone to explain to me how it can work
> without trusting someone else's binaries.
>

Can you explain to me how anyone can use a modern computer for anything
without trusting someone else's binaries?  Let's live in reality here.

Regardless, the top priority (as far as this part of the discussion is
concerned) is to migrate the project over to a modern build system, most
likely Gradle.  Support for downloading dependencies from Freenet (or
through Tor) would be nice but should be way down the list of priorities.

Ian.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Steve Dougherty]

It takes extra storage and bandwidth for something the vast majority of
people will never need.

On Tue, Oct 13, 2015, 5:08 PM Arne Babenhauserheide  wrote:

> Am Dienstag, 13. Oktober 2015, 14:13:00 schrieb Steve Dougherty:
> > We already distribute runtime dependencies with Freenet; I don't think it
> > would be reasonable to distribute things required only for development
> like
> > JUnit / Hamcrest.
>
> Why not? It could simplify getting into development a lot if people
> would only have to install a generic java development environment.
>
> Also I still didn’t get junit and hamcrest to work, so this isn’t
> completely theoretic ☺
>
> Best wishes,
> Arne
>
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Dienstag, 13. Oktober 2015, 14:13:00 schrieb Steve Dougherty:
> We already distribute runtime dependencies with Freenet; I don't think it
> would be reasonable to distribute things required only for development like
> JUnit / Hamcrest.

Why not? It could simplify getting into development a lot if people
would only have to install a generic java development environment.

Also I still didn’t get junit and hamcrest to work, so this isn’t
completely theoretic ☺

Best wishes,
Arne


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Dienstag, 13. Oktober 2015, 10:13:07 schrieb Ian:
> Can you explain to me how anyone can use a modern computer for anything
> without trusting someone else's binaries?  Let's live in reality here.

Gentoo GNU/Linux builds everything from source, and once gcc is
bootstrapped and the stage3-files are rebuilt, only source packages
get downloaded. For several years my jdk (icedtea 6 back then) was
built using gcj.

So this is possible right now (I use Gentoo at home and at work), and
not even inconvenient if you have a desktop system which runs anyway
(then the compile times don’t matter).

Best wishes,
Arne


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Matthew Toseland]

On 12/10/15 18:09, Ian wrote:
> On Mon, Oct 12, 2015 at 12:27 PM, Arne Babenhauserheide 
> wrote:
>
>> Am Sonntag, 11. Oktober 2015, 20:37:32 schrieb Zlatin Balevsky:
>>> Developers who care about their anonymity can force gradle or maven to
>> use
>>> a tor proxy
>> Can we make Tor or repo-over-freenet the default for people who build
>> freenet?
> Since Freenet can (in theory) do it, I think it would be much better to use
> Freenet from an "eat your own dogfood" perspective.  I think the only
> danger here is that we further complicate things for a developer trying to
> get into the project.

On 11/10/15 16:35, Steve Dougherty wrote:
> On 10/10/2015 04:14 PM, Matthew Toseland wrote:
>> On 06/10/15 15:10, Ian Clarke wrote:
>>> On Tue, Oct 6, 2015 at 4:39 AM, xor  wrote:
> ...
>> Deploying build-time dependencies via Gradle is not appropriate IMHO: It
>> means updating them is *our* responsibility, and it increases our
>> maintenance overheads as a result, and reduces the end-user's security.
>> Updating JUnit etc is the distribution's responsibility, not ours. And
>> anything that doesn't get updated is a security risk.
> In what way does it make updating them our responsibility? Checksum
> pinning does that inherently already. Charles linked to gradle-witness
> and it looks like exactly what we're looking for: transitive dependency
> checksum verification. https://github.com/WhisperSystems/gradle-witness
I made a distinction between build-time-only dependencies, like JUnit or
(probably) Mockito, versus run-time dependencies.

We need to keep checksums for run-time dependencies anyway, and we may
as well download them from Freenet if possible, and ask the user whether
to get them from the WWW if that fails. However, JUnit should be
installed via the packaging system if possible.

And as Jack pointed out, we need a build script to handle all of this:
- Download run-time dependencies and check checksums, either from
Freenet or WWW (auto-detect Tor?).
- Ask the user if this fails. Output an informative error message if we
don't have a useful stdin.
- Try to install build-time dependencies via package management (sudo
apt-get install... etc).

The second stage is this: Do we want to write a Gradle plugin to manage
dependencies.properties? Essentially this would make adding external
run-time dependencies significantly easier. The cost is we'd need to
maintain a repository, including obtaining and checking signatures etc.



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Arne Babenhauserheide]

Am Montag, 12. Oktober 2015, 13:09:48 schrieb Ian:
> > Can we make Tor or repo-over-freenet the default for people who build
> > freenet?
> 
> Since Freenet can (in theory) do it, I think it would be much better to use
> Freenet from an "eat your own dogfood" perspective.  I think the only
> danger here is that we further complicate things for a developer trying to
> get into the project.

I agree.

Could we just ship all required libraries (jars) for building Freenet
with every freenet install, so developers could just copy them over
from their Freenet directory (or point the build tool to the freenet
install)?

That should be easier than any other way of installation, because
everyone running freenet would already have all the libraries in
place. Also it would be ensured by our auto-updater that the libraries
are always the right ones for the current version of Freenet.

Best wishes,
Arne

signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Steve Dougherty]

On 10/10/2015 04:14 PM, Matthew Toseland wrote:
> On 06/10/15 15:10, Ian Clarke wrote:
>> On Tue, Oct 6, 2015 at 4:39 AM, xor  wrote:
...
> Deploying build-time dependencies via Gradle is not appropriate IMHO: It
> means updating them is *our* responsibility, and it increases our
> maintenance overheads as a result, and reduces the end-user's security.
> Updating JUnit etc is the distribution's responsibility, not ours. And
> anything that doesn't get updated is a security risk.

In what way does it make updating them our responsibility? Checksum
pinning does that inherently already. Charles linked to gradle-witness
and it looks like exactly what we're looking for: transitive dependency
checksum verification. https://github.com/WhisperSystems/gradle-witness



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Dependency management was Re: Behind the times

[Ian]

Good point.  Or we could mirror the repos in Freenet, more dogfoody.

On Sun, Oct 11, 2015 at 2:37 PM, Zlatin Balevsky  wrote:

> Developers who care about their anonymity can force gradle or maven to use
> a tor proxy
>
> On Sun, Oct 11, 2015 at 4:35 PM, Steve Dougherty 
> wrote:
>
> > On 10/10/2015 04:14 PM, Matthew Toseland wrote:
> > > On 06/10/15 15:10, Ian Clarke wrote:
> > >> On Tue, Oct 6, 2015 at 4:39 AM, xor  wrote:
> > ...
> > > Deploying build-time dependencies via Gradle is not appropriate IMHO:
> It
> > > means updating them is *our* responsibility, and it increases our
> > > maintenance overheads as a result, and reduces the end-user's security.
> > > Updating JUnit etc is the distribution's responsibility, not ours. And
> > > anything that doesn't get updated is a security risk.
> >
> > In what way does it make updating them our responsibility? Checksum
> > pinning does that inherently already. Charles linked to gradle-witness
> > and it looks like exactly what we're looking for: transitive dependency
> > checksum verification. https://github.com/WhisperSystems/gradle-witness
> >
> >
> > ___
> > Devl mailing list
> > Devl@freenetproject.org
> > https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
> ___
> Devl mailing list
> Devl@freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Dependency management was Re: Behind the times

[Matthew Toseland]

On 06/10/15 15:10, Ian Clarke wrote:
> On Tue, Oct 6, 2015 at 4:39 AM, xor  wrote:
>
 2) What can Maven do which Ant cannot do? Do we need those features?
>>> Dependency management
There is a lot of important context being missed in this discussion, in
particular what our needs are and what our current set-up actually involves.

There are 2 sets of dependencies here:
1. Run-time dependencies
2. Build-time dependencies

Run-time dependencies we have to include in the deployed Freenet, and we
now deploy them as separate jar files. Some of them (notably Bouncy
Castle) have to be signed by their upstream authors, but in any case we
keep track of versions, hashes etc in the file dependencies.properties.
We need to keep them up to date ourselves for auto-update to work. Some
of the files are included in freenet-ext.jar but the idea is to
transition to each of the dependencies being deployed separately. This
allows cheaper, faster updating, and makes packaging easier.

In other words, we *already have the infrastructure* for run-time
dependencies, and we could easily arrange to download them - the only
reason this doesn't happen automatically is that we'd like to have
anonymous developers - and traceably downloading files during a normal
build is not acceptable in that case. Note that there are indeed active
anonymous developers, notably Eleriseth, and the hope is we will have
more in future. What we could do is have the build script ask the user
whether they want us to download the files from the web, and possibly
whether they want to use Tor to do so etc - or even Freenet (we have the
CHK's!). At the moment, there is an over-long error message to this
effect, and the user needs to either download the files themselves or
edit override.properties. Arguably this is yet another case of a clash
between security and convenience. But it could be fixed with a build.sh
script (AFAICS it's better that ant itself doesn't ask questions?).

Build-time dependencies should generally be packaged on the user's Linux
distribution. We need to provide a script which searches for them and
automatically updates override.properties. And if they are not present,
it offers to install them, if possible via the package manager using
sudo. For the minority of Windows/Mac developers, we should of course
make it easy - e.g. build.sh/build.cmd could install them via Maven/Gradle.

Deploying build-time dependencies via Gradle is not appropriate IMHO: It
means updating them is *our* responsibility, and it increases our
maintenance overheads as a result, and reduces the end-user's security.
Updating JUnit etc is the distribution's responsibility, not ours. And
anything that doesn't get updated is a security risk.

Of course, we could generate the run-time dependencies and
build.properties via Gradle. That would require more work.



signature.asc
Description: OpenPGP digital signature
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl