Re: Racist statement of Ian Clarke on this list

[Zlatin Balevsky]

Oops, forgot to mention my Twitter handle is "zlatinb".

Good luck!

On Fri, Jan 27, 2023 at 10:54 AM Zlatin Balevsky  wrote:

> Hello everyone,
>
> My name is Zlatin Balevsky.  I was involved in the development of Freenet
> sometime around 2001-2003.  I've kept a relatively close eye on what's been
> going on since then.
>
> I want to say that I consider Ian Clarke / Sanity a great man, and even
> though I've never met him in person, I consider him a close friend.  He
> gave me a very good professional recommendation when I was applying for my
> job at Lime Wire LLC (you may remember there was a file-sharing P2P program
> called Limewire).
>
> Therefore, I fully support Ian's decision to look for an "offensively
> racist" domain name.  Ian, do not give up!  It is of paramount importance
> that you offend as many people as possible!  You DA MAN!
>
> Thank you for your attention.
> Zlatin Balevsky
>
> P.S.  Since I like to practice what I preach, I have a tweet on my Twitter
> timeline where I expose some seriously evil things about the Invisible
> Internet Project ("I2P" for short).  As you may already guess, I do my best
> to offend as many people as possible in the pastebin linked from that
> tweet.  I'm not going to post a direct link to the tweet just so I don't
> trip some mailing list safeguard.
>
> On Sun, Jan 22, 2023 at 5:38 PM craig mcgee <
> craig.mc...@guilt-management.org.uk> wrote:
>
>> I have to admit I was confused by the statement Ian made. Joke or not, it
>> wasn't funny.
>> but this is as far as I will go on the matter as I don't wish to get
>> involved in the freenet dispute.
>>
>> - Original Message -
>> *From:* Russell Glenn 
>> *To:* devl@freenetproject.org
>> *Sent:* Sunday, January 22, 2023 3:33 PM
>> *Subject:* Racist statement of Ian Clarke on this list
>>
>> Mr. Ian Clarke,
>>
>> when discussing your acquisition of a secondary domain,
>> "freenet.org", for your highly controversial unilateral decision
>> to rewrite the Freenet Project which has always been hosted
>> on "freenetproject.org", you said the following:
>>
>> > Yeah, all of the provocatively racist domain names were taken anyway.
>>
>> Notice that this is a VERBATIM quote, and the email of yours did
>> not contain ANY other content, nothing to reduce the gravity of
>> this statement; the gravity of which cannot be reduced anyway.
>> Proof: https://www.mail-archive.com/devl@freenetproject.org/msg55336.html
>>
>> As a black person, but not only as such, I inform you that racism
>> is NOT tolerable by any means whatsoever on this planet and will
>> NEVER be.
>>
>> And before you attempt to dismiss this as a joke:
>> Millions of people died due to racism.
>> You, especially as a white person, are not entitled to joke about that.
>> Nobody is.
>>
>> And this piles onto the provable falsehoods and other insults you have
>> produced on this list in the previous days.
>>
>> Therefore, it is my duty to join the ranks of people who request
>> you to:
>> * immediately step down as leader of the Freenet Project and
>> * leave its board and
>> * transfer ownership of the domains to the Freenet developer team and
>> * transfer ownership of all other related accounts and
>> * cease to use the name "Freenet" for any of your new projects.
>>
>> Racists, or those who joke about racism, do not belong into positions
>> where power is wielded.
>>
>> As you have failed to prove that the board of the project is still
>> constructed of active members of the community,
>> and not of those who have been inactive for decades as Mr. Daigniere
>> stated,
>> I join those who request leadership is transferred to the
>> leading developer.
>> To my knowledge, this is Mr. Babenhauserheide.
>>
>> Should he not accept the position, I would vouch for the previous lead
>> developer Mr. Dougherty, or anyone of the choice of the two.
>>
>> Finally, I urgently implore anyone who reads this email to state
>> their support. We must stand united against racism.
>>
>> - Russell Glenn
>>
>>

Re: Racist statement of Ian Clarke on this list

[Zlatin Balevsky]

Hello everyone,

My name is Zlatin Balevsky.  I was involved in the development of Freenet
sometime around 2001-2003.  I've kept a relatively close eye on what's been
going on since then.

I want to say that I consider Ian Clarke / Sanity a great man, and even
though I've never met him in person, I consider him a close friend.  He
gave me a very good professional recommendation when I was applying for my
job at Lime Wire LLC (you may remember there was a file-sharing P2P program
called Limewire).

Therefore, I fully support Ian's decision to look for an "offensively
racist" domain name.  Ian, do not give up!  It is of paramount importance
that you offend as many people as possible!  You DA MAN!

Thank you for your attention.
Zlatin Balevsky

P.S.  Since I like to practice what I preach, I have a tweet on my Twitter
timeline where I expose some seriously evil things about the Invisible
Internet Project ("I2P" for short).  As you may already guess, I do my best
to offend as many people as possible in the pastebin linked from that
tweet.  I'm not going to post a direct link to the tweet just so I don't
trip some mailing list safeguard.

On Sun, Jan 22, 2023 at 5:38 PM craig mcgee <
craig.mc...@guilt-management.org.uk> wrote:

> I have to admit I was confused by the statement Ian made. Joke or not, it
> wasn't funny.
> but this is as far as I will go on the matter as I don't wish to get
> involved in the freenet dispute.
>
> - Original Message -
> *From:* Russell Glenn 
> *To:* devl@freenetproject.org
> *Sent:* Sunday, January 22, 2023 3:33 PM
> *Subject:* Racist statement of Ian Clarke on this list
>
> Mr. Ian Clarke,
>
> when discussing your acquisition of a secondary domain,
> "freenet.org", for your highly controversial unilateral decision
> to rewrite the Freenet Project which has always been hosted
> on "freenetproject.org", you said the following:
>
> > Yeah, all of the provocatively racist domain names were taken anyway.
>
> Notice that this is a VERBATIM quote, and the email of yours did
> not contain ANY other content, nothing to reduce the gravity of
> this statement; the gravity of which cannot be reduced anyway.
> Proof: https://www.mail-archive.com/devl@freenetproject.org/msg55336.html
>
> As a black person, but not only as such, I inform you that racism
> is NOT tolerable by any means whatsoever on this planet and will
> NEVER be.
>
> And before you attempt to dismiss this as a joke:
> Millions of people died due to racism.
> You, especially as a white person, are not entitled to joke about that.
> Nobody is.
>
> And this piles onto the provable falsehoods and other insults you have
> produced on this list in the previous days.
>
> Therefore, it is my duty to join the ranks of people who request
> you to:
> * immediately step down as leader of the Freenet Project and
> * leave its board and
> * transfer ownership of the domains to the Freenet developer team and
> * transfer ownership of all other related accounts and
> * cease to use the name "Freenet" for any of your new projects.
>
> Racists, or those who joke about racism, do not belong into positions
> where power is wielded.
>
> As you have failed to prove that the board of the project is still
> constructed of active members of the community,
> and not of those who have been inactive for decades as Mr. Daigniere
> stated,
> I join those who request leadership is transferred to the
> leading developer.
> To my knowledge, this is Mr. Babenhauserheide.
>
> Should he not accept the position, I would vouch for the previous lead
> developer Mr. Dougherty, or anyone of the choice of the two.
>
> Finally, I urgently implore anyone who reads this email to state
> their support. We must stand united against racism.
>
> - Russell Glenn
>
>

Re: [freenet-dev] Project Status

[Zlatin Balevsky]

> Maybe so, but it means a full rewrite every few years, which we are
> unlikely to have the resources for. Even if we did, it would mean
> throwing out years of hard-won expertise.

That's not necessarily a bad thing.  It forces the accumulation of new
expertise, keeps bringing in new generations of developers and makes sure
the project takes advantage of whatever language advancements have taken
place.  Encourages forking and pits the different implementations against
one another so that the fittest may survive.  OTOH having a single
platform/toolset dictated discourages contributions.  Yes, java is the most
popular language by many metrics, but that popularity does not translate to
a large pool of passionate volunteers.

> But there are other trends that might favour us, e.g. cheap but powerful
router boxes, Raspberry Pi /
> Arduino hobbyist stuff etc.

Every trend can bring more volunteers passionate about it, so all venues
are worth pursuing :)

zab/topiltzin

On Thu, Oct 15, 2015 at 8:57 PM, Matthew Toseland <mj...@cam.ac.uk> wrote:

> On 15/10/15 20:40, Zlatin Balevsky wrote:
> > I first got involved in Freenet when it was 0.2.  At the time it was
> using
> > cutting edge technologies and an contributing was an opportunity to learn
> > valuable skills.  Contributing was fun and that was the driving factor.
> >
> > If Freenet was to start fresh, it should do whatever it takes to regain
> the
> > coolness factor.  That means embracing new tools and technologies even if
> > there is no strict technological advantage in doing so.  For better or
> > worse Java will never be hip with the open-source crowd, and personally,
> > after 10 hours of looking at Java code for my day job the last thing I
> want
> > is to look at more Java code in my free time.  Some exotic new language
> > like Scala or Go or whatever the $COOL_LANGUAGE_DU_JOUR is would be a
> > different story.
> Maybe so, but it means a full rewrite every few years, which we are
> unlikely to have the resources for. Even if we did, it would mean
> throwing out years of hard-won expertise.
>
> Can we make it more attractive to new devs without needing to take such
> a drastic step?
> > Yes this can lead to fragmentation as various contributors veer off each
> > into their own direction; it's the job of the leader to keep things
> > coherent and aligned with the project vision.  It's very easy to
> > underestimate how difficult the job of the leader is.
> Agreed.
> > Lastly, I'd like to point out that mobile is the future - not that I like
> > that a single bit.
> If mobile is the future, we're stuffed. Mobile simply can't do p2p. The
> networks will do everything necessary to stop it, and it drains power,
> storage and above all scarce bandwidth. The only realistic options for
> mobile are pure client nodes ("transient mode"), which is what mobile is
> designed for, or variants on Sneakernet. But there are other trends that
> might favour us, e.g. cheap but powerful router boxes, Raspberry Pi /
> Arduino hobbyist stuff etc.
> > zab/topiltzin
>
>
> ___
> Devl mailing list
> Devl@freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Behind the times

[Zlatin Balevsky]

There exists a pure java implementation of a Tor client -
https://subgraph.com/orchid/index.en.html
apologies if this has already been brought up, searching archives is hard
(TM)

On Mon, Oct 12, 2015 at 9:41 AM, xor  wrote:

> On Tuesday, October 06, 2015 09:10:28 AM Ian Clarke wrote:
> > On Tue, Oct 6, 2015 at 4:39 AM, xor  wrote:
> > > [Sorted/trimmed/amended the quotes for readability]
> > >
> > > On Monday, October 05, 2015 12:52:08 PM Ian Clarke wrote:
> > > > On Mon, Oct 5, 2015 at 12:57 AM, xor  wrote:
> > > > Right, but it appears that solutions exist for this with Gradle.
> > >
> > > "Apache Ant" = 1 320 000 Google hits
> > > "Gradle" = 957 000 Google hits
> >
> > That's a terribly unscientific way to assess the popularity of a tool.
> As
> > a professional Java developer please take my word for it when I say that
> > Apache Ant is an outdated tool, it has been replaced by Maven, and Maven
> is
> > in the process of being replaced by Gradle (although we're early in that
> > process).  If you don't believe me just ask Google, they selected Gradle
> as
> > the standard build tool for Android.  Or failing that just ask almost any
> > other professional Java developer, they'll tell you the same thing.
>
> Yes, you're right, thats a toy metric - I just couldn't think of any other
> one, sorry.
> Your observation that Google uses it sounds like a good one!
>
> I also wasn't aware that you're actively doing Java development, so now
> I'll
> trust you even more with what you say.
>
> As I've exhausted my knowledge about the stuff (which wasn't any deep
> anyway,
> I never even read a Maven/Gradle script), I think this should be my last
> reply
> to this aspect if that's OK with you.
>
> To produce a result, I filed bugtracker entries at my subprojects to adopt
> whatever the results of this thread will be:
> https://bugs.freenetproject.org/view.php?id=6697
> https://bugs.freenetproject.org/view.php?id=6698
>
> I have set the target versions in the roadmap to be the ones right after
> the
> most critical pending performance / usability fixes.
>
> > > > If someone wants to use both Freenet and Tor then they can download
> them
> > > > individually, but I see no good reason to bundle two independent
> pieces
> > >
> > > of
> > >
> > > > software just because they both solve related (but different)
> problems.
> > >
> > > Well, the question is if the user's care about the difference:
> > That seems like a very peculiar criteria with which to decide to bundle
> any
> > two projects.  If a user didn't care about the difference between Freenet
> > and Angry Birds, should we bundle it with Angry Birds?
> >
> > The only good criteria for bundling two pieces of software is that the
> > combination is dramatically more useful than either individually (eg. if
> > one depends on the other).  That wouldn't be the case here, it would just
> > be two somewhat related pieces of software glued together for no good
> > reason.
>
> I think there even is one of those anecdotal "laws" about this, which is
> something like: "Every software project converges towards becoming a full
> operating system."
> :D(if someone knows the name of that law, please tell me)
>
> The reason I recommended this is the large amount of users which come to
> the
> support chat and ask for Tor-functionality :|
> It's sad that we cannot help them easily.
>
> However, I think we can settle this for now as there is another good reason
> for me to postpone my suggestion for a while:
> The CENO project aims to provide easy mirroring of regular websites into
> Freenet: https://equalit.ie/portfolio/censorship-no/
>
> Before we consider bundling Tor, we should probably give them a year of
> time
> to get their stuff finished to the point where we can bundle it.
> With CENO we wouldn't need Tor :)
> I'm in contact with the developers (= idling in their IRC channel), and
> will
> continue to remind them to push for bundling.
>
> --
> hopstolive  (keyword for Ians spam filter)
>
> ___
> Devl mailing list
> Devl@freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Behind the times

[Zlatin Balevsky]

As a random external java developer looking to possibly contribute, I'd
much prefer dealing with a Gradle script than Ant/Maven.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] circumvention tools hackfest in NYC in July

[Zlatin Balevsky]

Anyone in NYC and nearby area check out
http://openitp.org/?q=circumvention_tools_hackfest_nyc_july_2012

I'll try to stop by

[freenet-dev] circumvention tools hackfest in NYC in July

[Zlatin Balevsky]

Anyone in NYC and nearby area check out
http://openitp.org/?q=circumvention_tools_hackfest_nyc_july_2012

I'll try to stop by
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Offer to help with Windows installer

[Zlatin Balevsky]

Hi Jeff, what is your preferred installer tool?  It would be great to
1. automatically install a recent jre  2.  use pack200 compression for
the jars.  If you plan on using NSIS I can share some script snippets
that kind of do this.

On Sat, Nov 26, 2011 at 9:45 AM, Jeff Rivett  wrote:
> Greetings. ?According to the wiki
> (http://new-wiki.freenetproject.org/Status): "Windows installer -
> We could do with a more active maintainer, and we definitely need more
> testers (with different windows versions) for it."
>
> If that's still the case, I would like to offer my assistance. ?I have a ton
> of experience writing installers for most versions of Windows. ?I also have
> access to various versions of Windows in virtual form that I can use for
> testing.
>
> Let me know if I can be of any help as a Windows installer
> developer/maintainer.
>
> Regards,
> Jeff Rivett
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Offer to help with Windows installer

[Zlatin Balevsky]

Hi Jeff, what is your preferred installer tool?  It would be great to
1. automatically install a recent jre  2.  use pack200 compression for
the jars.  If you plan on using NSIS I can share some script snippets
that kind of do this.

On Sat, Nov 26, 2011 at 9:45 AM, Jeff Rivett jriv...@shaw.ca wrote:
 Greetings.  According to the wiki
 (http://new-wiki.freenetproject.org/Status): Windows installer -
 We could do with a more active maintainer, and we definitely need more
 testers (with different windows versions) for it.

 If that's still the case, I would like to offer my assistance.  I have a ton
 of experience writing installers for most versions of Windows.  I also have
 access to various versions of Windows in virtual form that I can use for
 testing.

 Let me know if I can be of any help as a Windows installer
 developer/maintainer.

 Regards,
 Jeff Rivett

 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

>
> On 06/19/2012 04:10 PM, Matthew Toseland wrote:
> > However, DoS protection should be a little stronger than has been
> > discussed: You should limit the average number of probes on a
> > given link per unit time, like we do with swapping. This should
> > probably be an average, and should be generous enough that it isn't
> > going to be violated by accident, but it's preferable to having a
> > limit on in-flight probes, as it will quench any flood more or less
> > at source, and the attacker will be limited by the number of
> > connections he has (at least on darknet, connections are
> > expensive).
>
> The number of probes accepted per peer is limited with a counter which
> increments when a request is accepted, decrements 60 seconds later,
> and has a maximum (currently 10) above which no more requests are
> accepted from that peer. Is my understanding correct that this is an
> acceptable way to implement per-link limits?
>

You can also use probes to detect DoS (or plain old bugs in the probe
implementation) if probes report back 1 the average number of outstanding
probes across all peer nodes 2 the count or fraction of nodes which are at
the maximum 10.  That information can be used for all kinds of other
research as well.
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: [freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

 On 06/19/2012 04:10 PM, Matthew Toseland wrote:
  However, DoS protection should be a little stronger than has been
  discussed: You should limit the average number of probes on a
  given link per unit time, like we do with swapping. This should
  probably be an average, and should be generous enough that it isn't
  going to be violated by accident, but it's preferable to having a
  limit on in-flight probes, as it will quench any flood more or less
  at source, and the attacker will be limited by the number of
  connections he has (at least on darknet, connections are
  expensive).

 The number of probes accepted per peer is limited with a counter which
 increments when a request is accepted, decrements 60 seconds later,
 and has a maximum (currently 10) above which no more requests are
 accepted from that peer. Is my understanding correct that this is an
 acceptable way to implement per-link limits?


You can also use probes to detect DoS (or plain old bugs in the probe
implementation) if probes report back 1 the average number of outstanding
probes across all peer nodes 2 the count or fraction of nodes which are at
the maximum 10.  That information can be used for all kinds of other
research as well.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

>
> On 05/23/2012 10:47 PM, Zlatin Balevsky wrote:
> > On a global scale, the if the rate of new probe requests is higher
> > than the rate at which existing ones expire the number of active
> > probes at any moment will not reach balance.   Higher HTL makes a
> > ddos against the probe mechanism easier; in this scenario the
> > internal limit of 5 simultaneous probes ends up assisting the
> > attacker.
>
> Good point. I'm not sure what to do to improve that behavior though. I
> can add some rate limiting if that looks like it'll be necessary.

The way we dealt with this problem in Gnutella was to cap the max htl at
each hop.  Even if an attacker sent a message with very high htl each node
on the path would reduce it to a small value before forwarding.  Not sure
if this will work with Freenet.

>
> > Would it be possible to simulate a single-digit HTL network?  My
> > intuition suggests the graph of effectiveness of probes vs. HTL has
> > a logarithmic shape.
>
> Indeed it is possible to simulate, and that was the subject of my
> second update on this project. [1] My main findings are here, [2]
> where one can see that it's true that an HTL of 5 or 10 or so could
> provide pretty good distribution already.

can you tell what is the absolutely lowest htl value that will give "good
enough" performance?

>
> evanbd, my mentor for this project, suggested the maximum HTL of 50.
> Here's some of his reasoning from the #freenet logs:
>
> 2012-05-09:
> "So it looks to me from the graphs like HTL 20 is plenty for the new
> probes. Which I take to mean we should set the default HTL as at least
> 30, possibly 40. Because your nice simulated graphs don't have
> problematic behaviors like clustering or partitioning or whatever :)
> Basically, I think we should have a fairly high *maximum* HTL (at
> least 50), and have the actual HTL be a user-specified parameter."
>
> 2012-05-19:
> "And the plan is to send the requests at < max HTL, assuming that we
> don't need the full 50. It looked from graphs like 20 was sufficient,
> so I want to send at 30."

We should discuss this in more detail and have more people involved before
releasing these changes.  I can see evanbd's point but the side effects of
very high htl must be taken into account as well.
-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20120524/ee685047/attachment.html>

Re: [freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

 On 05/23/2012 10:47 PM, Zlatin Balevsky wrote:
  On a global scale, the if the rate of new probe requests is higher
  than the rate at which existing ones expire the number of active
  probes at any moment will not reach balance.   Higher HTL makes a
  ddos against the probe mechanism easier; in this scenario the
  internal limit of 5 simultaneous probes ends up assisting the
  attacker.

 Good point. I'm not sure what to do to improve that behavior though. I
 can add some rate limiting if that looks like it'll be necessary.

The way we dealt with this problem in Gnutella was to cap the max htl at
each hop.  Even if an attacker sent a message with very high htl each node
on the path would reduce it to a small value before forwarding.  Not sure
if this will work with Freenet.


  Would it be possible to simulate a single-digit HTL network?  My
  intuition suggests the graph of effectiveness of probes vs. HTL has
  a logarithmic shape.

 Indeed it is possible to simulate, and that was the subject of my
 second update on this project. [1] My main findings are here, [2]
 where one can see that it's true that an HTL of 5 or 10 or so could
 provide pretty good distribution already.

can you tell what is the absolutely lowest htl value that will give good
enough performance?


 evanbd, my mentor for this project, suggested the maximum HTL of 50.
 Here's some of his reasoning from the #freenet logs:

 2012-05-09:
 So it looks to me from the graphs like HTL 20 is plenty for the new
 probes. Which I take to mean we should set the default HTL as at least
 30, possibly 40. Because your nice simulated graphs don't have
 problematic behaviors like clustering or partitioning or whatever :)
 Basically, I think we should have a fairly high *maximum* HTL (at
 least 50), and have the actual HTL be a user-specified parameter.

 2012-05-19:
 And the plan is to send the requests at  max HTL, assuming that we
 don't need the full 50. It looked from graphs like 20 was sufficient,
 so I want to send at 30.

We should discuss this in more detail and have more people involved before
releasing these changes.  I can see evanbd's point but the side effects of
very high htl must be taken into account as well.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

>> IMHO maximum HTL should be single-digits or you risk flooding.
>
> I don't follow. What do you mean? At each hop a probe either stays
> locally (if a candidate is not accepted) or is routed to another
> random node.


On a global scale, the if the rate of new probe requests is higher
than the rate at which existing ones expire the number of active
probes at any moment will not reach balance.   Higher HTL makes a ddos
against the probe mechanism easier; in this scenario the internal
limit of 5 simultaneous probes ends up assisting the attacker.

Would it be possible to simulate a single-digit HTL network?  My
intuition suggests the graph of effectiveness of probes vs. HTL has a
logarithmic shape.

[freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

Are these rate-limited in any way?  Do they obey some general
rate-limiting policies together with other messages?  IMHO maximum HTL
should be single-digits or you risk flooding.  Does fred have any
metrics collection system that would allow us to detect such flooding
events?


On Sat, May 19, 2012 at 5:46 PM, Steve Dougherty  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> I've finished a proof of concept for Metropolis-Hastings corrected
> probes in Fred. If you are so inclined, please offer feedback on the
> patches. [1] Metropolis-Hastings correction was described previously,
> [2] and the probes are exposed over FCP. [3] An originating node starts
> a probe request by specifying the type of result they're looking for,
> and optionally a value for hopsToLive, which currently defaults to its
> maximum of 50 if not specified. We will in practice initially use
> something closer to 30. One can disable responding to any number of
> result types on the Core settings page.
>
> Possible types are:
>
> BANDWIDTH - outgoing bandwidth limit.
> BUILD - Freenet build number. (For instance, currently 1407.) Useful
> ? ? ? ?for measuring update propagation.
> IDENTIFIER - endpoint identifier. Useful for improving estimates of
> ? ? ? ? ? ? network size and churn.
> LINK_LENGTHS - differences in location between endpoint and each of
> ? ? ? ? ? ? ? endpoint's peers.
> STORE_SIZE - size of datastore.
> UPTIME - session uptime and the percentage of the past 48 hours during
> ? ? ? ? which the endpoint was up.
>
> +/- up to 1% noise is added to bandwidth, link lengths, store size, and
> uptime information in an attempt to make the information less
> identifiable but still useful for analysis.
>
> Bandwidth, link lengths, store size, and uptime are intended to improve
> knowledge of the network so that any problems will hopefully become
> apparent, or at the very least allow more detailed simulations which
> more accurately reflect actual network conditions. This is with the
> goal of implementing changes to improve network performance and
> reliability.
>
> Possible outcomes of the probes as implemented are:
> ? ?-non-propagated timeout
> ? ?-non-propagated disconnection
> ? ?-endpoint refusal for the result type
> ? ?-result from endpoint
>
> Based on feedback in #freenet I've built a TODO list:
>
> 1. Access MHProbe from the callbacks, which are inner classes, via
> MHProbe.this. This will mean no need to pass it in to use MHProbe as a
> ByteCounter. Also because the callbacks are inner classes there is no
> need for the horrible hack that is making pendingProbes public static.
> 2. If FOAF data for a peer is not available, ignore that peer.
> 3. Implement error messages so that resources can be freed quickly in
> the event on an error:
> ? ?- next in chain disconnected
> ? ?- timeout
> ? ?- type unrecognized
> ? ?- FOAF disabled
> 4. If FOAF is completely disabled on a node, rather than ignore every
> peer, leading to HTL decrementing until a local response, respond with
> a "FOAF disabled" error message as above.
> 5. Add a node configuration option for a random (not based on noderef
> like current probe and swap request identifier) identifier which is set
> to a random value when at its default of -1.
> 6. Use a Gaussian distribution for randomNoise() because many tests
> assume Gaussian noise, so why not?
> 7. Quantize things: instead of just adding noise, report bandwidth in
> as 64-bit integer KiB and store size as 64-bit integer GiB.
> 8. Instead of separate identifier and uptime requests, allow requesting:
> ? ?- [ identifier, quantized noisy 7-day uptime percentage ]
> ? ?- [ noisy 7-day uptime percentage ]
> ? ?- [ noisy 48-hour uptime percentage ]
> 9. Set timeout as constant1 * HTL + constant2 to account for
> probabilistic decrement at HTL = 1.
>
> Next week I plan to address the TODO list and any more issues that come
> up in the patch set, then submit it for merging into Fred once it has
> been approved. After that, I will work on FCP library support for the
> new probes, and update pyProbe to support gathering and analyzing the
> new information. In the event that I finish these things I can work on
> making the network simulator more flexible. Our target deployment date
> is Friday the 25th May or the day after.
>
> Thanks,
> operhiem1
>
> [1] https://github.com/thynix/fred-staging/tree/newProbes
> [2] https://emu.freenetproject.org/pipermail/devl/2012-April/036363.html
> [3] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
>
> iQIcBAEBAgAGBQJPuBSaAAoJECLJP19KqmFurkUQAJZbvpgctvsZxgppkRIW/ZOY
> meNolQJCutEUHl8VFSdut/6ahRwJ3gfdCgOiF/GHcbEkZqm4S+Jna58Sz6E9WBSa
> MfhjQ1dULutiE1vmXN/5v/vNwFYXlATAk2XK35IrUqbCorq7NZHQhSyn61j6fLdB
> cGIBOV8GCRHd2rhFu4+oP9tECyoBbL7B9OqjI3AHPEuDEyvFVfuxfk6HDcOam+4Z
> dzeecRUMU4T1wquOZ9l6sDTqe8m+RRuR++WHObQ/pI/n1s848HWTM0qs3Dzj3pIU
> 

Re: [freenet-dev] Statistics Project Update #4

[Zlatin Balevsky]

Are these rate-limited in any way?  Do they obey some general
rate-limiting policies together with other messages?  IMHO maximum HTL
should be single-digits or you risk flooding.  Does fred have any
metrics collection system that would allow us to detect such flooding
events?


On Sat, May 19, 2012 at 5:46 PM, Steve Dougherty st...@asksteved.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 I've finished a proof of concept for Metropolis-Hastings corrected
 probes in Fred. If you are so inclined, please offer feedback on the
 patches. [1] Metropolis-Hastings correction was described previously,
 [2] and the probes are exposed over FCP. [3] An originating node starts
 a probe request by specifying the type of result they're looking for,
 and optionally a value for hopsToLive, which currently defaults to its
 maximum of 50 if not specified. We will in practice initially use
 something closer to 30. One can disable responding to any number of
 result types on the Core settings page.

 Possible types are:

 BANDWIDTH - outgoing bandwidth limit.
 BUILD - Freenet build number. (For instance, currently 1407.) Useful
        for measuring update propagation.
 IDENTIFIER - endpoint identifier. Useful for improving estimates of
             network size and churn.
 LINK_LENGTHS - differences in location between endpoint and each of
               endpoint's peers.
 STORE_SIZE - size of datastore.
 UPTIME - session uptime and the percentage of the past 48 hours during
         which the endpoint was up.

 +/- up to 1% noise is added to bandwidth, link lengths, store size, and
 uptime information in an attempt to make the information less
 identifiable but still useful for analysis.

 Bandwidth, link lengths, store size, and uptime are intended to improve
 knowledge of the network so that any problems will hopefully become
 apparent, or at the very least allow more detailed simulations which
 more accurately reflect actual network conditions. This is with the
 goal of implementing changes to improve network performance and
 reliability.

 Possible outcomes of the probes as implemented are:
    -non-propagated timeout
    -non-propagated disconnection
    -endpoint refusal for the result type
    -result from endpoint

 Based on feedback in #freenet I've built a TODO list:

 1. Access MHProbe from the callbacks, which are inner classes, via
 MHProbe.this. This will mean no need to pass it in to use MHProbe as a
 ByteCounter. Also because the callbacks are inner classes there is no
 need for the horrible hack that is making pendingProbes public static.
 2. If FOAF data for a peer is not available, ignore that peer.
 3. Implement error messages so that resources can be freed quickly in
 the event on an error:
    - next in chain disconnected
    - timeout
    - type unrecognized
    - FOAF disabled
 4. If FOAF is completely disabled on a node, rather than ignore every
 peer, leading to HTL decrementing until a local response, respond with
 a FOAF disabled error message as above.
 5. Add a node configuration option for a random (not based on noderef
 like current probe and swap request identifier) identifier which is set
 to a random value when at its default of -1.
 6. Use a Gaussian distribution for randomNoise() because many tests
 assume Gaussian noise, so why not?
 7. Quantize things: instead of just adding noise, report bandwidth in
 as 64-bit integer KiB and store size as 64-bit integer GiB.
 8. Instead of separate identifier and uptime requests, allow requesting:
    - [ identifier, quantized noisy 7-day uptime percentage ]
    - [ noisy 7-day uptime percentage ]
    - [ noisy 48-hour uptime percentage ]
 9. Set timeout as constant1 * HTL + constant2 to account for
 probabilistic decrement at HTL = 1.

 Next week I plan to address the TODO list and any more issues that come
 up in the patch set, then submit it for merging into Fred once it has
 been approved. After that, I will work on FCP library support for the
 new probes, and update pyProbe to support gathering and analyzing the
 new information. In the event that I finish these things I can work on
 making the network simulator more flexible. Our target deployment date
 is Friday the 25th May or the day after.

 Thanks,
 operhiem1

 [1] https://github.com/thynix/fred-staging/tree/newProbes
 [2] https://emu.freenetproject.org/pipermail/devl/2012-April/036363.html
 [3] https://wiki.freenetproject.org/User:Operhiem1/MHProbeFCP
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.11 (GNU/Linux)

 iQIcBAEBAgAGBQJPuBSaAAoJECLJP19KqmFurkUQAJZbvpgctvsZxgppkRIW/ZOY
 meNolQJCutEUHl8VFSdut/6ahRwJ3gfdCgOiF/GHcbEkZqm4S+Jna58Sz6E9WBSa
 MfhjQ1dULutiE1vmXN/5v/vNwFYXlATAk2XK35IrUqbCorq7NZHQhSyn61j6fLdB
 cGIBOV8GCRHd2rhFu4+oP9tECyoBbL7B9OqjI3AHPEuDEyvFVfuxfk6HDcOam+4Z
 dzeecRUMU4T1wquOZ9l6sDTqe8m+RRuR++WHObQ/pI/n1s848HWTM0qs3Dzj3pIU
 GNcgX4dbgH8yz3veInur2i6FSkM41IdwEbV3OHro+peOIxVmCJ3c6cEbkEYLQW/R
 1mICM7+6CLNkzjrxSwiPNrhVYGGKrYy7F4YSDj2GWwwfwd5aQaxzCM5CO9+VaHWj
 

[freenet-dev] Lantern

[Zlatin Balevsky]

> "Desktop app" isn't necessarily a good thing.

Generally true, but you're missing out on a lot of positive black
swans with such position.

Anybody given a thought to some kind of mobile app?  A mobile app that
connects to a Freenet node running at home and does something
meaningful would increase the number of active nodes.

Re: [freenet-dev] Lantern

[Zlatin Balevsky]

 Desktop app isn't necessarily a good thing.

Generally true, but you're missing out on a lot of positive black
swans with such position.

Anybody given a thought to some kind of mobile app?  A mobile app that
connects to a Freenet node running at home and does something
meaningful would increase the number of active nodes.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Statistics Project Update #1

[Zlatin Balevsky]

> On 04/28/2012 06:56 PM, Zlatin Balevsky wrote:
>> In Gnutella we observed that long-lived nodes tend to be better
>> connected and that they also cluster with other high-uptime nodes.
>> If the same is true for Freenet it's a good idea to keep an eye for
>> side effects as you tweak the behavior.
>
> Good to know - I'll look for that. Are there any particular effects
> you had in mind? The Metropolis-Hastings correction in the new probes
> should produce a fairly uniform distribution of endpoints despite
> clustering and well-connected nodes, but explicitly simulating the
> effects of high uptime could be helpful.

There was a study that higher uptime correlated with the probability
of further uptime so if you shift bias towards low-uptime nodes you
could end will lower overall reliability.  It was done on a different
network with different usage patterns but imho you should definitely
treat node uptime as a parameter in any simulations.

Re: [freenet-dev] Statistics Project Update #1

[Zlatin Balevsky]

 On 04/28/2012 06:56 PM, Zlatin Balevsky wrote:
 In Gnutella we observed that long-lived nodes tend to be better
 connected and that they also cluster with other high-uptime nodes.
 If the same is true for Freenet it's a good idea to keep an eye for
 side effects as you tweak the behavior.

 Good to know - I'll look for that. Are there any particular effects
 you had in mind? The Metropolis-Hastings correction in the new probes
 should produce a fairly uniform distribution of endpoints despite
 clustering and well-connected nodes, but explicitly simulating the
 effects of high uptime could be helpful.

There was a study that higher uptime correlated with the probability
of further uptime so if you shift bias towards low-uptime nodes you
could end will lower overall reliability.  It was done on a different
network with different usage patterns but imho you should definitely
treat node uptime as a parameter in any simulations.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Statistics Project Update #1

[Zlatin Balevsky]

On Fri, Apr 27, 2012 at 10:34 PM, Steve Dougherty  
wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Here's an update on my progress on the statistics project for the
> first week:
>
> The current probes are biased towards better-connected nodes: at each
> hop they choose a random peer to pass the request to - this is a random
> walk. However, because better-connected nodes by definition have more
> connections, the requests will be passed to them more often and they
> will be over-represented in results. To address this, the new probes I
> will implement will use Metropolis-Hastings correction: unlike the
> uniform random walk which always uses the random peer it picks, it is
> less likely to pick a well-connected node, and more likely to pick a
> poorly-connected node. As there are more chances to pick a
> well-connected node than a poorly-connected one, this balances out to
> a uniform probability to pick any given node.

In Gnutella we observed that long-lived nodes tend to be better
connected and that they also cluster with other high-uptime nodes.  If
the same is true for Freenet it's a good idea to keep an eye for side
effects as you tweak the behavior.

Re: [freenet-dev] Statistics Project Update #1

[Zlatin Balevsky]

On Fri, Apr 27, 2012 at 10:34 PM, Steve Dougherty st...@asksteved.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Here's an update on my progress on the statistics project for the
 first week:

 The current probes are biased towards better-connected nodes: at each
 hop they choose a random peer to pass the request to - this is a random
 walk. However, because better-connected nodes by definition have more
 connections, the requests will be passed to them more often and they
 will be over-represented in results. To address this, the new probes I
 will implement will use Metropolis-Hastings correction: unlike the
 uniform random walk which always uses the random peer it picks, it is
 less likely to pick a well-connected node, and more likely to pick a
 poorly-connected node. As there are more chances to pick a
 well-connected node than a poorly-connected one, this balances out to
 a uniform probability to pick any given node.

In Gnutella we observed that long-lived nodes tend to be better
connected and that they also cluster with other high-uptime nodes.  If
the same is true for Freenet it's a good idea to keep an eye for side
effects as you tweak the behavior.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Improving build security against back-doors: Bytecode verification

[Zlatin Balevsky]

>
> It appears we can just compare the bytecode however. If you want to compare 
> the disassembly that's good too, but somebody should check the source.
>
> I have uploaded a basic version of a bytecode verification script called 
> verify-build to the "Maintenance scripts" repository on github. Unfortunately 
> build 1406 includes some classes that are only in my local tree because 
> cleanup occurs a little too late. Anyway if you want to use it, or improve 
> it, that would be cool.
>
> I have completed proof of concept (the bytecode is the same for two builds, 
> including when doing a clean checkout in a separate folder). Provided that 
> you use the same java compiler as the person doing the release, it should 
> work (for 1407 onwards).

If at some point in the future the installers start using pack200 jar
compression that may mangle the bytecode and would complicate the
verification process as uncompressed .class files will be different
than javac output.  If for whatever reason Freenet explodes in
popularity overnight you may not have choice - pack200 is far cheaper
than finding more hosting bandwidth.

Re: [freenet-dev] Improving build security against back-doors: Bytecode verification

[Zlatin Balevsky]

 It appears we can just compare the bytecode however. If you want to compare 
 the disassembly that's good too, but somebody should check the source.

 I have uploaded a basic version of a bytecode verification script called 
 verify-build to the Maintenance scripts repository on github. Unfortunately 
 build 1406 includes some classes that are only in my local tree because 
 cleanup occurs a little too late. Anyway if you want to use it, or improve 
 it, that would be cool.

 I have completed proof of concept (the bytecode is the same for two builds, 
 including when doing a clean checkout in a separate folder). Provided that 
 you use the same java compiler as the person doing the release, it should 
 work (for 1407 onwards).

If at some point in the future the installers start using pack200 jar
compression that may mangle the bytecode and would complicate the
verification process as uncompressed .class files will be different
than javac output.  If for whatever reason Freenet explodes in
popularity overnight you may not have choice - pack200 is far cheaper
than finding more hosting bandwidth.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

>>
>> Any patch with lazy evaluation will not cause problems in the "normal"
>> case but will affect debugging Fred because turning on debug log level
>> for a single class will cause all lazy parameters everywhere to be
>> created.

Before someone calls me out on this one I this is how you can use lazy
evaluation without garbage on hotspot jvms.  The statement becomes
getLog().log(  ) and the predicate goes in the getter.

= CallSiteJitting.java =
public class CallSiteJitting {

static interface Logger {
void log(Object ar1, Object ar2);
}

static class MuteLogger implements Logger {
public void log(Object ar1, Object ar2){}
}

static class RealLogger implements Logger {
static long sideEffect;
public void log(Object ar1, Object ar2) {
sideEffect += ar1.hashCode();
sideEffect += ar2.hashCode();
}
}

private static final Logger mute = new MuteLogger();
private static final Logger real = new RealLogger();

private static boolean once;
private static Logger getLog() {
if (!once) {
once = true;
System.out.println("real logging once");
// if you return real every time lots of
// garbage collections happen
return real;
}
return mute;
}

public static void main(String []ar) throws Exception {

for (long l = 0; l < Long.MAX_VALUE; l++) {
getLog().log(new Object(), new Object());
}

System.out.println(RealLogger.sideEffect);
}
}
= end CallSiteJitting.java

[freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

> ?If your 'solution' is trading 'jar
> ?size' and 'readability' against run-time performance (for the common
> ?case assuming logNORMAL), we won't merge it. Freenet is slow enough
> ?as is.

Any patch with lazy evaluation will not cause problems in the "normal"
case but will affect debugging Fred because turning on debug log level
for a single class will cause all lazy parameters everywhere to be
created.  Imagine you're trying to reproduce a rare bug and have
turned on one or two debug statements - because of lazy evaluation
your jvm will garbage collect much more often and you may never
reproduce the bug.  This is valid for current generation jvms.

>
> Hint: your doing it wrong, the one way to make it faster and nicer is to
> ?use dependancy injection
>

DI is generally great but it doesn't solve the problem of "ugly
predicate".  You're better off looking at some bytecode weaving
techniques like aspects.  Combined with DI they can make the code very
clean.

Re: [freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

  If your 'solution' is trading 'jar
  size' and 'readability' against run-time performance (for the common
  case assuming logNORMAL), we won't merge it. Freenet is slow enough
  as is.

Any patch with lazy evaluation will not cause problems in the normal
case but will affect debugging Fred because turning on debug log level
for a single class will cause all lazy parameters everywhere to be
created.  Imagine you're trying to reproduce a rare bug and have
turned on one or two debug statements - because of lazy evaluation
your jvm will garbage collect much more often and you may never
reproduce the bug.  This is valid for current generation jvms.


 Hint: your doing it wrong, the one way to make it faster and nicer is to
  use dependancy injection


DI is generally great but it doesn't solve the problem of ugly
predicate.  You're better off looking at some bytecode weaving
techniques like aspects.  Combined with DI they can make the code very
clean.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

 Any patch with lazy evaluation will not cause problems in the normal
 case but will affect debugging Fred because turning on debug log level
 for a single class will cause all lazy parameters everywhere to be
 created.

Before someone calls me out on this one I this is how you can use lazy
evaluation without garbage on hotspot jvms.  The statement becomes
getLog().log( lazy args ) and the predicate goes in the getter.

= CallSiteJitting.java =
public class CallSiteJitting {

static interface Logger {
void log(Object ar1, Object ar2);
}

static class MuteLogger implements Logger {
public void log(Object ar1, Object ar2){}
}

static class RealLogger implements Logger {
static long sideEffect;
public void log(Object ar1, Object ar2) {
sideEffect += ar1.hashCode();
sideEffect += ar2.hashCode();
}
}

private static final Logger mute = new MuteLogger();
private static final Logger real = new RealLogger();

private static boolean once;
private static Logger getLog() {
if (!once) {
once = true;
System.out.println(real logging once);
// if you return real every time lots of
// garbage collections happen
return real;
}
return mute;
}

public static void main(String []ar) throws Exception {

for (long l = 0; l  Long.MAX_VALUE; l++) {
getLog().log(new Object(), new Object());
}

System.out.println(RealLogger.sideEffect);
}
}
= end CallSiteJitting.java
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

The problem of abusing the predicate by performing anything other than
logging inside it.

You cannot get rid of the predicate without introducing side effects as
I've demonstrated throughout this thread.
On Apr 4, 2012 6:01 AM, "Marco Schulze"  wrote:

> Which problem is solved? There's still a predicate there.
>
> On 03-04-2012 20:49, Zlatin Balevsky wrote:
>
>> May I suggest a nice little script someone ( novice? ) could write,
>> solve the logging problem and learning a thing or two about language
>> theory in the process :
>>
>> Transform:
>> 
>> if (LOG.isLoggable(Level.DEBUG)) {
>>   
>>   
>> }
>> 
>> Into
>> 
>> private static void logComplexComputation( .. arguments! .. ) {
>><  do the stuff above>
>> }
>>
>> if (LOG.isLoggable(Level.DEBUG)) {
>> logComplexComputation( .. arguments ..);
>> }
>> 
>>
>> This gets run as pre-commit hook and the problem is solved with
>> positive run-time side effects.
>>
>>
>> On Sun, Mar 25, 2012 at 6:22 PM, Marco Schulze
>>   wrote:
>>
>>> Working (but incomplete) code is available @
>>> https://github.com/Heiral/**fred-staging/tree/logger++<https://github.com/Heiral/fred-staging/tree/logger++>
>>> __**_
>>> Devl mailing list
>>> Devl at freenetproject.org
>>> https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devl<https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl>
>>>
>> __**_
>> Devl mailing list
>> Devl at freenetproject.org
>> https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devl<https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl>
>>
> __**_
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devl<https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl>
>
-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20120404/b43b43df/attachment.html>

Re: [freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

The problem of abusing the predicate by performing anything other than
logging inside it.

You cannot get rid of the predicate without introducing side effects as
I've demonstrated throughout this thread.
On Apr 4, 2012 6:01 AM, Marco Schulze marco.c.schu...@gmail.com wrote:

 Which problem is solved? There's still a predicate there.

 On 03-04-2012 20:49, Zlatin Balevsky wrote:

 May I suggest a nice little script someone ( novice? ) could write,
 solve the logging problem and learning a thing or two about language
 theory in the process :

 Transform:
 
 if (LOG.isLoggable(Level.DEBUG)) {
   some complex computation that wouldn't happen otherwise
   log results of complex computation, maybe other stuff too
 }
 
 Into
 
 private static void logComplexComputation( .. arguments! .. ) {
  do the stuff above
 }

 if (LOG.isLoggable(Level.DEBUG)) {
 logComplexComputation( .. arguments ..);
 }
 

 This gets run as pre-commit hook and the problem is solved with
 positive run-time side effects.


 On Sun, Mar 25, 2012 at 6:22 PM, Marco Schulze
 marco.c.schu...@gmail.com  wrote:

 Working (but incomplete) code is available @
 https://github.com/Heiral/**fred-staging/tree/logger++https://github.com/Heiral/fred-staging/tree/logger++
 __**_
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devlhttps://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

 __**_
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devlhttps://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

 __**_
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devlhttps://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

May I suggest a nice little script someone ( novice? ) could write,
solve the logging problem and learning a thing or two about language
theory in the process :

Transform:

if (LOG.isLoggable(Level.DEBUG)) {
  
  
}

Into

private static void logComplexComputation( .. arguments! .. ) {
   < do the stuff above>
}

if (LOG.isLoggable(Level.DEBUG)) {
logComplexComputation( .. arguments ..);
}


This gets run as pre-commit hook and the problem is solved with
positive run-time side effects.


On Sun, Mar 25, 2012 at 6:22 PM, Marco Schulze
 wrote:
> Working (but incomplete) code is available @
> https://github.com/Heiral/fred-staging/tree/logger++
> ___
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

May I suggest a nice little script someone ( novice? ) could write,
solve the logging problem and learning a thing or two about language
theory in the process :

Transform:

if (LOG.isLoggable(Level.DEBUG)) {
  some complex computation that wouldn't happen otherwise
  log results of complex computation, maybe other stuff too
}

Into

private static void logComplexComputation( .. arguments! .. ) {
do the stuff above
}

if (LOG.isLoggable(Level.DEBUG)) {
logComplexComputation( .. arguments ..);
}


This gets run as pre-commit hook and the problem is solved with
positive run-time side effects.


On Sun, Mar 25, 2012 at 6:22 PM, Marco Schulze
marco.c.schu...@gmail.com wrote:
 Working (but incomplete) code is available @
 https://github.com/Heiral/fred-staging/tree/logger++
 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

On Mon, Apr 2, 2012 at 6:04 PM, Matthew Toseland
 wrote:
> On Saturday 31 Mar 2012 04:03:10 Zlatin Balevsky wrote:
>> >>
>> >> In 'log.info("Random message: %s", obj.toString())' evaluating 
>> >> 'obj.toString()' was what caused issues, not recycling it, or so I 
>> >> believe to remember.
>> >
>> > You don't need to call obj.toString() before calling log() - just pass the 
>> > object itself. Then the only overheads are:
>> > 1) Calling the function, including passing the arguments - which hopefully 
>> > will be optimised to be on some stack (but the code might not have been 
>> > JITted yet).
>> > 2) Looking up whether it needs to be logged.
>> >
>>
>> 3) Autoboxing because you cannot pass a primitive if the argument is
>> Object without creating a . ?Small ranges of
>> Char/Short/Integer/Long values are cached, anything outside those will
>> end up creating garbage if the shouldLog predicate evaluate true even
>> once.
>
> Will it still autobox them if they are on the stack, and never used?

If the isDebuggable predicate never evaluates true, then it's not a
problem.  But if it evaluates true even once for the lifetime of the
loaded class it will create these objects.  It's a quirk of current
hotspot jvm which will not affect production deployments but will
affect the developer: turning on debug logging for one class will
cause all primitives in all log statements to be instantiated.  I had
some sample code earlier in the thread.

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

On Mon, Apr 2, 2012 at 6:04 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Saturday 31 Mar 2012 04:03:10 Zlatin Balevsky wrote:
 
  In 'log.info(Random message: %s, obj.toString())' evaluating 
  'obj.toString()' was what caused issues, not recycling it, or so I 
  believe to remember.
 
  You don't need to call obj.toString() before calling log() - just pass the 
  object itself. Then the only overheads are:
  1) Calling the function, including passing the arguments - which hopefully 
  will be optimised to be on some stack (but the code might not have been 
  JITted yet).
  2) Looking up whether it needs to be logged.
 

 3) Autoboxing because you cannot pass a primitive if the argument is
 Object without creating a ? extends Number.  Small ranges of
 Char/Short/Integer/Long values are cached, anything outside those will
 end up creating garbage if the shouldLog predicate evaluate true even
 once.

 Will it still autobox them if they are on the stack, and never used?

If the isDebuggable predicate never evaluates true, then it's not a
problem.  But if it evaluates true even once for the lifetime of the
loaded class it will create these objects.  It's a quirk of current
hotspot jvm which will not affect production deployments but will
affect the developer: turning on debug logging for one class will
cause all primitives in all log statements to be instantiated.  I had
some sample code earlier in the thread.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

>>
>> In 'log.info("Random message: %s", obj.toString())' evaluating 
>> 'obj.toString()' was what caused issues, not recycling it, or so I believe 
>> to remember.
>
> You don't need to call obj.toString() before calling log() - just pass the 
> object itself. Then the only overheads are:
> 1) Calling the function, including passing the arguments - which hopefully 
> will be optimised to be on some stack (but the code might not have been 
> JITted yet).
> 2) Looking up whether it needs to be logged.
>

3) Autoboxing because you cannot pass a primitive if the argument is
Object without creating a .  Small ranges of
Char/Short/Integer/Long values are cached, anything outside those will
end up creating garbage if the shouldLog predicate evaluate true even
once.

... but you could get around this problem by adding many log functions
with different signatures that take primitives.  The burden then falls
on the programmer to find the appropriate function at each call site.

[freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

>
> But we use a static logging API anyway, and this works adequately well. I 
> have debugged stuff largely through logs in multi-node simulators, I 
> generally rely on the thread name to identify the node (e.g. the UDP receive 
> and send thread include the port number in the thread name).
>

It could become an issue when trying to reproduce rare concurrency
bugs that require running tests in a loop until they fail.  The
smaller the synchronization overhead of the logging subsystem, the
less noise you introduce in the simulation as you increase
parallelism.  It also runs faster and you reproduce the bug sooner.

Re: [freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

 But we use a static logging API anyway, and this works adequately well. I 
 have debugged stuff largely through logs in multi-node simulators, I 
 generally rely on the thread name to identify the node (e.g. the UDP receive 
 and send thread include the port number in the thread name).


It could become an issue when trying to reproduce rare concurrency
bugs that require running tests in a loop until they fail.  The
smaller the synchronization overhead of the logging subsystem, the
less noise you introduce in the simulation as you increase
parallelism.  It also runs faster and you reproduce the bug sooner.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

 In 'log.info(Random message: %s, obj.toString())' evaluating 
 'obj.toString()' was what caused issues, not recycling it, or so I believe 
 to remember.

 You don't need to call obj.toString() before calling log() - just pass the 
 object itself. Then the only overheads are:
 1) Calling the function, including passing the arguments - which hopefully 
 will be optimised to be on some stack (but the code might not have been 
 JITted yet).
 2) Looking up whether it needs to be logged.


3) Autoboxing because you cannot pass a primitive if the argument is
Object without creating a ? extends Number.  Small ranges of
Char/Short/Integer/Long values are cached, anything outside those will
end up creating garbage if the shouldLog predicate evaluate true even
once.

... but you could get around this problem by adding many log functions
with different signatures that take primitives.  The burden then falls
on the programmer to find the appropriate function at each call site.
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

There are many problems with
https://github.com/Heiral/fred-staging/commit/b876f8c454cb269281ffcb18d14d25b22818d130#L0R49
:


line 25: the static field log should be volatile.  It may work now and
then but it's broken.  Google "double-checked locking" for more info.
Either that or synchronize it properly everywhere

lines 49-60 - "out" will refer to the parameter, not the static field
so you will end up closing the parameter.  Also what if Log.out is
null?  Can't synchronize on null.

Line 61: bad synchronization again

lines 65-75 Level.ordinal() will compile to a vtable call if you have
more than two levels in use across the jvm.

lines 100-101: again, if out is null you can't synchronize on it.


Besides all this, adding a static dependency will create problems
if/when a multi-node simulator or multi-node black box unit tests are
written.

On Tue, Mar 27, 2012 at 2:35 PM, Marco Schulze
 wrote:
> On 27-03-2012 12:51, Martin Nyhus wrote:
>>
>> On Monday 26. March 2012 00:22:24 Marco Schulze wrote:
>>>
>>> Working (but incomplete) code is available @
>>> https://github.com/Heiral/fred-staging/tree/logger++
>>
>> Please keep in mind that simply deleting Logger will break pretty much
>> every
>> single plugin out there, so you really should rewrite Logger to call the
>> new
>> methods in your new logger class.
>
> Will do.
>
>
>>
>> I won't say much about the code since you say you aren't finished, but
>> please
>> follow the code style of the rest of the code base.
>
> Apart from the lack of braces, what violates the coding standards? I mean,
> compared to the rest of fred code, I use too many blank lines, 80 character
> lines, variables are declared at the top of the function and other minor
> details. I hope that that isn't a problem.
>
>
>>
>> Also, I'm pretty sure you don't want to close the new stream here[0].
>
> Fixed.
>
>
>> And the
>> locking when you update out isn't sufficient for visibility (either that
>> or it
>> isn't clear enough why it works IMHO).
>
> You mean, things like if(out == null) without locking (inside isLoggable())?
> In this case, it doesn't really matter as it is meant to be a very quick
> check and getting the wrong values don't matter much.
>
>
>>
>> [0] https://github.com/Heiral/fred-
>> staging/commit/b876f8c454cb269281ffcb18d14d25b22818d130#L0R49
>> ___
>> Devl mailing list
>> Devl at freenetproject.org
>> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Logging subsystem rewrite

[Zlatin Balevsky]

There are many problems with
https://github.com/Heiral/fred-staging/commit/b876f8c454cb269281ffcb18d14d25b22818d130#L0R49
:


line 25: the static field log should be volatile.  It may work now and
then but it's broken.  Google double-checked locking for more info.
Either that or synchronize it properly everywhere

lines 49-60 - out will refer to the parameter, not the static field
so you will end up closing the parameter.  Also what if Log.out is
null?  Can't synchronize on null.

Line 61: bad synchronization again

lines 65-75 Level.ordinal() will compile to a vtable call if you have
more than two levels in use across the jvm.

lines 100-101: again, if out is null you can't synchronize on it.


Besides all this, adding a static dependency will create problems
if/when a multi-node simulator or multi-node black box unit tests are
written.

On Tue, Mar 27, 2012 at 2:35 PM, Marco Schulze
marco.c.schu...@gmail.com wrote:
 On 27-03-2012 12:51, Martin Nyhus wrote:

 On Monday 26. March 2012 00:22:24 Marco Schulze wrote:

 Working (but incomplete) code is available @
 https://github.com/Heiral/fred-staging/tree/logger++

 Please keep in mind that simply deleting Logger will break pretty much
 every
 single plugin out there, so you really should rewrite Logger to call the
 new
 methods in your new logger class.

 Will do.



 I won't say much about the code since you say you aren't finished, but
 please
 follow the code style of the rest of the code base.

 Apart from the lack of braces, what violates the coding standards? I mean,
 compared to the rest of fred code, I use too many blank lines, 80 character
 lines, variables are declared at the top of the function and other minor
 details. I hope that that isn't a problem.



 Also, I'm pretty sure you don't want to close the new stream here[0].

 Fixed.


 And the
 locking when you update out isn't sufficient for visibility (either that
 or it
 isn't clear enough why it works IMHO).

 You mean, things like if(out == null) without locking (inside isLoggable())?
 In this case, it doesn't really matter as it is meant to be a very quick
 check and getting the wrong values don't matter much.



 [0] https://github.com/Heiral/fred-
 staging/commit/b876f8c454cb269281ffcb18d14d25b22818d130#L0R49
 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

It doesn't look like a big deal but if all of Fred was using lazy
evaluation for logging it would make one common use case very
annoying:

Say I'm working on a specific class and need to run that class with
debug logging enabled.  This would have the side effect of changing
the compiled LOG.debug( ... ) method globally across the JVM and would
cause every log lazily evaluated parameter for every debug statement
to be instantiated.  Production deployments would not be affected but
the developer experience would.

P.S. nice trick with the Double.MIN_VALUE, I'll use that sometime :)

On Fri, Mar 23, 2012 at 11:33 PM, Ximin Luo  wrote:
> I see.
>
> Actually we didn't even need the side effects thing - I did notice that your
> example stopped doing GC when setting "shouldLog = false" always, so I tried 
> to
> introduce your "if (shouldLog) { shouldLog = false; }" into my example, but 
> was
> still unable to reproduce GC. Then changing
>
> - System.out.println("don't optimise me out");
> + System.out.println(format + args);
>
> worked to re-introduce GC.
>
> Well, that sucks. I guess the escape analyzer isn't sophisticated enough to 
> see
> that ar2.toString() does not cause ar2 to escape - and removing that from your
> log() method does result in no GC.
>
> I still think it's negligible though :p - for my test, I'm seeing only ~8ms
> lost to GC after 20 million calls, and this appears to not change by much even
> if I increase the number of arguments (and objects created) to log_info.
>
> On 24/03/12 03:02, Zlatin Balevsky wrote:
>> Your test has nothing to do with stack allocation because you never
>> use the objects that you pass to the log_info method so JIT removes
>> them. ?Apply the following patch and see yourself create garbage in
>> both testWith and testWithout.
>>
>> --- Test2.java ? ? ? ?2012-03-23 23:01:05.540475497 -0400
>> +++ Test.java 2012-03-23 23:01:47.304477562 -0400
>> @@ -6,8 +6,11 @@
>> ? ?final Object existing = new Object();
>> ? ?Object tmp = null;
>>
>> + ?long sideEffect;
>> ? ?public void log_info(String format, Object ... args) {
>> ? ? ?if (log_level) {
>> + ? ? ?for (Object o : args)
>> + ? ? sideEffect += o.hashCode();
>> ? ? ? ?System.out.println("don't optimise me out");
>> ? ? ?}
>> ? ?}
>> @@ -88,11 +91,8 @@
>> ? ? ?printSummary("with alloc", res[0], "w/o alloc", res[1]);
>> ? ? ?printSummary("with alloc", res[2], "w/o alloc", res[3]);
>> ? ? ?printSummary("with alloc", res[4], "w/o alloc", res[5]);
>> - ? ?System.out.println(" with stack allocation and \"real use case\"");
>> - ? ?res = testWithOtherCode(z);
>> - ? ?printSummary("log only", res[0], "log+work", res[1]);
>> - ? ?printSummary("log only", res[2], "log+work", res[3]);
>> - ? ?printSummary("log only", res[4], "log+work", res[5]);
>> +
>> + ? ?System.out.println(sideEffect);
>> ? ?}
>>
>> ? ?public static void main(String[] args) {
>>
>>
>> On Fri, Mar 23, 2012 at 10:48 PM, Ximin Luo  wrote:
>>> On 24/03/12 02:44, Zlatin Balevsky wrote:
>>>> Wrong. ?Extracting the log statement into its own method and calling
>>>> that method inside the while loop still creates garbage. ?Stack frames
>>>> are different.
>>>>
>>>
>>> How do you explain the results obtained from running my test code, then? 
>>> Turn
>>> on -verbose:gc. testWithStackAllocation results in no GC,
>>> testWithoutStackAllocation results in lots of GC.
>>>
>>>> Read again exactly how Escape Analysis works. ?The objects it prevents
>>>> from creating do not appear in heap histograms. ?Take a "jmap -histo
>>>> " and you will see a huge number of java.lang.Integer objects in
>>>> the heap, not on the stack.
>>>>
>>>
>>> If you're still talking about your example, this is exactly consistent with 
>>> my
>>> explanation, i.e. that escape analysis is NOT occurring, properly anyway.
>>>
>>>> The difference in run times of your example could be due to many
>>>> things. ?How many iterations did you run? ?It may be still in
>>>> interpreted mode. ?Enable -XX:+PrintCompilation and discard any
>>>> results until JIT stops printing.
>>>
>>> Done that, JIT stops printing after the 3rd iteration and results are the 
>>> same
>>> thereafter.
>>>
>>> --
>>> GPG: 4096R/5FBBDBCE
>>> https://github.com/infinity0
>>> https://bitbucket.org/infinity0
>>> https://launchpad.net/~infinity0
>>>
>>>
>>> ___
>>> Devl mailing list
>>> Devl at freenetproject.org
>>> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>> ___
>> Devl mailing list
>> Devl at freenetproject.org
>> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
>
> --
> GPG: 4096R/5FBBDBCE
> https://github.com/infinity0
> https://bitbucket.org/infinity0
> https://launchpad.net/~infinity0
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

As to why it still creates garbage even if you do not touch the
arguments in the testWithout method, I would guess that different
inlining rules apply to large methods, small methods and methods that
are already inlined within other methods.

To give definitive answer I'll have to look through the opto assembly
which will take me a couple of hours.  There was something in the
-XX:+LogCompilation output but that's just a hint.  To get to the
truth you need the assembly code.

Back to the original issue

Some kind of automated post-processing can be set up that cause
compilation errors if anything other than LOG.log is used inside of an
if (isLoggable) predicate.  I've seen an example where AspectJ can be
set up to trigger compilation errors every time it encounters a
System.out and System.err reference.  I'm sure there are many
solutions, some even integrate with IDEs.

On Fri, Mar 23, 2012 at 11:02 PM, Zlatin Balevsky  wrote:
> Your test has nothing to do with stack allocation because you never
> use the objects that you pass to the log_info method so JIT removes
> them. ?Apply the following patch and see yourself create garbage in
> both testWith and testWithout.
>
> --- Test2.java ?2012-03-23 23:01:05.540475497 -0400
> +++ Test.java ? 2012-03-23 23:01:47.304477562 -0400
> @@ -6,8 +6,11 @@
> ? final Object existing = new Object();
> ? Object tmp = null;
>
> + ?long sideEffect;
> ? public void log_info(String format, Object ... args) {
> ? ? if (log_level) {
> + ? ? ?for (Object o : args)
> + ? ? ? sideEffect += o.hashCode();
> ? ? ? System.out.println("don't optimise me out");
> ? ? }
> ? }
> @@ -88,11 +91,8 @@
> ? ? printSummary("with alloc", res[0], "w/o alloc", res[1]);
> ? ? printSummary("with alloc", res[2], "w/o alloc", res[3]);
> ? ? printSummary("with alloc", res[4], "w/o alloc", res[5]);
> - ? ?System.out.println(" with stack allocation and \"real use case\"");
> - ? ?res = testWithOtherCode(z);
> - ? ?printSummary("log only", res[0], "log+work", res[1]);
> - ? ?printSummary("log only", res[2], "log+work", res[3]);
> - ? ?printSummary("log only", res[4], "log+work", res[5]);
> +
> + ? ?System.out.println(sideEffect);
> ? }
>
> ? public static void main(String[] args) {
>
>
> On Fri, Mar 23, 2012 at 10:48 PM, Ximin Luo  wrote:
>> On 24/03/12 02:44, Zlatin Balevsky wrote:
>>> Wrong. ?Extracting the log statement into its own method and calling
>>> that method inside the while loop still creates garbage. ?Stack frames
>>> are different.
>>>
>>
>> How do you explain the results obtained from running my test code, then? Turn
>> on -verbose:gc. testWithStackAllocation results in no GC,
>> testWithoutStackAllocation results in lots of GC.
>>
>>> Read again exactly how Escape Analysis works. ?The objects it prevents
>>> from creating do not appear in heap histograms. ?Take a "jmap -histo
>>> " and you will see a huge number of java.lang.Integer objects in
>>> the heap, not on the stack.
>>>
>>
>> If you're still talking about your example, this is exactly consistent with 
>> my
>> explanation, i.e. that escape analysis is NOT occurring, properly anyway.
>>
>>> The difference in run times of your example could be due to many
>>> things. ?How many iterations did you run? ?It may be still in
>>> interpreted mode. ?Enable -XX:+PrintCompilation and discard any
>>> results until JIT stops printing.
>>
>> Done that, JIT stops printing after the 3rd iteration and results are the 
>> same
>> thereafter.
>>
>> --
>> GPG: 4096R/5FBBDBCE
>> https://github.com/infinity0
>> https://bitbucket.org/infinity0
>> https://launchpad.net/~infinity0
>>
>>
>> ___
>> Devl mailing list
>> Devl at freenetproject.org
>> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

Your test has nothing to do with stack allocation because you never
use the objects that you pass to the log_info method so JIT removes
them.  Apply the following patch and see yourself create garbage in
both testWith and testWithout.

--- Test2.java  2012-03-23 23:01:05.540475497 -0400
+++ Test.java   2012-03-23 23:01:47.304477562 -0400
@@ -6,8 +6,11 @@
   final Object existing = new Object();
   Object tmp = null;

+  long sideEffect;
   public void log_info(String format, Object ... args) {
 if (log_level) {
+  for (Object o : args)
+   sideEffect += o.hashCode();
   System.out.println("don't optimise me out");
 }
   }
@@ -88,11 +91,8 @@
 printSummary("with alloc", res[0], "w/o alloc", res[1]);
 printSummary("with alloc", res[2], "w/o alloc", res[3]);
 printSummary("with alloc", res[4], "w/o alloc", res[5]);
-System.out.println(" with stack allocation and \"real use case\"");
-res = testWithOtherCode(z);
-printSummary("log only", res[0], "log+work", res[1]);
-printSummary("log only", res[2], "log+work", res[3]);
-printSummary("log only", res[4], "log+work", res[5]);
+
+System.out.println(sideEffect);
   }

   public static void main(String[] args) {


On Fri, Mar 23, 2012 at 10:48 PM, Ximin Luo  wrote:
> On 24/03/12 02:44, Zlatin Balevsky wrote:
>> Wrong. ?Extracting the log statement into its own method and calling
>> that method inside the while loop still creates garbage. ?Stack frames
>> are different.
>>
>
> How do you explain the results obtained from running my test code, then? Turn
> on -verbose:gc. testWithStackAllocation results in no GC,
> testWithoutStackAllocation results in lots of GC.
>
>> Read again exactly how Escape Analysis works. ?The objects it prevents
>> from creating do not appear in heap histograms. ?Take a "jmap -histo
>> " and you will see a huge number of java.lang.Integer objects in
>> the heap, not on the stack.
>>
>
> If you're still talking about your example, this is exactly consistent with my
> explanation, i.e. that escape analysis is NOT occurring, properly anyway.
>
>> The difference in run times of your example could be due to many
>> things. ?How many iterations did you run? ?It may be still in
>> interpreted mode. ?Enable -XX:+PrintCompilation and discard any
>> results until JIT stops printing.
>
> Done that, JIT stops printing after the 3rd iteration and results are the same
> thereafter.
>
> --
> GPG: 4096R/5FBBDBCE
> https://github.com/infinity0
> https://bitbucket.org/infinity0
> https://launchpad.net/~infinity0
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

Wrong.  Extracting the log statement into its own method and calling
that method inside the while loop still creates garbage.  Stack frames
are different.

Read again exactly how Escape Analysis works.  The objects it prevents
from creating do not appear in heap histograms.  Take a "jmap -histo
" and you will see a huge number of java.lang.Integer objects in
the heap, not on the stack.

The difference in run times of your example could be due to many
things.  How many iterations did you run?  It may be still in
interpreted mode.  Enable -XX:+PrintCompilation and discard any
results until JIT stops printing.

On Fri, Mar 23, 2012 at 10:28 PM, Ximin Luo  wrote:
> On 24/03/12 01:56, Zlatin Balevsky wrote:
>> Not really. ?Here, I'll change my call to allocate everything on one level:
>>
>> ? while (true) {
>> ? ? ? ? ? ? ? log(" list size is {1} ",
>> ? ? ? ? ? ? ? ? new Integer(listToLog.size()));
>> ? ? ? ? ? }
>>
>> I don't see how using this you can log primitive values without garbage.
>>
>
> This still allocates many "new Integer()" on the stack without giving them an
> opportunity to be destroyed, because this occurs all on the same stack frame.
> Eventually there is no more space for stack allocation, leading to heap
> allocation and GC. This is an uncommon situation to pop up in real code
> however, because in most cases you don't have an extremely long loop.
>
> Have a look at the difference between testWithStackAllocation and
> testWithoutStackAllocation in the example I posted, and the huge difference in
> their run times; maybe it will be clearer then.
>
> Arguably the JVM should be smart enough to destroy things at the end of a code
> block when things go out of scope, including loops, but practically that
> doesn't appear to be the case. (Unless you have another explanation for the
> behaviour encountered when running my test code?)
>
> One reason I'm opposed to the predicate form, is because it encourages things 
> like
>
> | if (log_level) {
> | ? // huge complex calculation
> | ? // that takes up many lines
> | ? // and distracts from the rest of the code
> | ? log.level(args);
> | }
>
> We could agree to restrict logging calls to be of the form
>
> | if (log_level) log.level(
> | ? ?args ... );
>
> which would at least force the complex calculations to be factored out into
> another method.
>
>> On Fri, Mar 23, 2012 at 9:33 PM, Ximin Luo  wrote:
>>> That is because in your example you're allocating all those new Object()s in
>>> the same stack frame, so the allocator runs out of space on the stack.
>>>
>>> If you put the call to log(x, new Object()) inside its only function call, 
>>> each
>>> Object will be destroyed once that call exits, and no GC occurs.
>>>
>>> Example code here:
>>> https://gist.github.com/2177070
>>>
>>> The "real use case" just adds one simple operation with each logging call
>>> (since you're never logging without doing something else as well) to show 
>>> how
>>> little of a difference it makes in real terms (<5%).
>>>
>>> X
>>>
>>> On 24/03/12 01:15, Zlatin Balevsky wrote:
>>>> You're right, the example you gave will put the arguments on the
>>>> stack. ?I wanted to clarify Escape Analysis in its current form works
>>>> only if a new object doesn't exit the stack frame where it's created.
>>>> Because of that, the moment you do something slightly more complicated
>>>> you will create garbage like this:
>>>>
>>>> ? ?private static void logListSize ( final List listToLog ) {
>>>> ? ? ? ? LOG.log(" list size is {1} ",
>>>> ? ? ? ? ? ? ? ? new Object() {
>>>> ? ? ? ? ? ? ? ? ? ? ? ? public String toString() {
>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return String.valueOf(listToLog.size());
>>>> ? ? ? ? ? ? ? ? ? ? ? ? }
>>>> ? ? ? ? ? ? ? ? });
>>>> ? ? }
>>>>
>>>> Here is the tricky part - if you can guarantee that everywhere in the
>>>> jvm the LOG.log method is disabled by the logging system, then the
>>>> entire call will compile to noop and no objects will be created.
>>>> However, if even one call site enables logging you will end up
>>>> creating the temp objects even if they don't get logged. ?Below is a
>>>> full example, compile and run passing "-verbosegc -XX:+PrintGC" and
>>>> watch it GC all the time.
>>>>
>>>> = ?LazyEvalTest.java ===
>>>> import java.util.*;
>&g

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

The only bug-vector I see in the current predicate approach is that
when used incorrectly it cause the size of methods to explode.  There
are some hard-coded thresholds for the method size in bytecodes that
turn off some or all JIT optimizations; there are also a whole host of
compilation parameters which are optimized with smaller methods in
mind.

Unless someone can offer a great syntactic improvement I recommend
against home-brew frameworks.  Aspects may be able to clean things up
nicely though, I'd be very curious to see some examples.

On Fri, Mar 23, 2012 at 9:56 PM, Zlatin Balevsky  wrote:
> Not really. ?Here, I'll change my call to allocate everything on one level:
>
> ?while (true) {
> ? ? ? ? ? ? ?log(" list size is {1} ",
> ? ? ? ? ? ? ? ?new Integer(listToLog.size()));
> ? ? ? ? ?}
>
> I don't see how using this you can log primitive values without garbage.
>
> On Fri, Mar 23, 2012 at 9:33 PM, Ximin Luo  wrote:
>> That is because in your example you're allocating all those new Object()s in
>> the same stack frame, so the allocator runs out of space on the stack.
>>
>> If you put the call to log(x, new Object()) inside its only function call, 
>> each
>> Object will be destroyed once that call exits, and no GC occurs.
>>
>> Example code here:
>> https://gist.github.com/2177070
>>
>> The "real use case" just adds one simple operation with each logging call
>> (since you're never logging without doing something else as well) to show how
>> little of a difference it makes in real terms (<5%).
>>
>> X
>>
>> On 24/03/12 01:15, Zlatin Balevsky wrote:
>>> You're right, the example you gave will put the arguments on the
>>> stack. ?I wanted to clarify Escape Analysis in its current form works
>>> only if a new object doesn't exit the stack frame where it's created.
>>> Because of that, the moment you do something slightly more complicated
>>> you will create garbage like this:
>>>
>>> ? ?private static void logListSize ( final List listToLog ) {
>>> ? ? ? ? LOG.log(" list size is {1} ",
>>> ? ? ? ? ? ? ? ? new Object() {
>>> ? ? ? ? ? ? ? ? ? ? ? ? public String toString() {
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return String.valueOf(listToLog.size());
>>> ? ? ? ? ? ? ? ? ? ? ? ? }
>>> ? ? ? ? ? ? ? ? });
>>> ? ? }
>>>
>>> Here is the tricky part - if you can guarantee that everywhere in the
>>> jvm the LOG.log method is disabled by the logging system, then the
>>> entire call will compile to noop and no objects will be created.
>>> However, if even one call site enables logging you will end up
>>> creating the temp objects even if they don't get logged. ?Below is a
>>> full example, compile and run passing "-verbosegc -XX:+PrintGC" and
>>> watch it GC all the time.
>>>
>>> = ?LazyEvalTest.java ===
>>> import java.util.*;
>>>
>>> public class LazyEvalTest {
>>>
>>> ? ? // only if this is is always false does logging produce no garbage.
>>> ? ? private static boolean shouldLog = true;
>>>
>>> ? ? private static void log(String ar1, Object ar2) {
>>> ? ? ? ? if (shouldLog)
>>> ? ? ? ? ? ? ?System.out.println(ar1 + ar2.toString());
>>> ? ? ? ? shouldLog = false;
>>> ? ? }
>>>
>>> ? ? public static void main(String []ar) {
>>>
>>> ? ? ? ? final List listToLog = new ArrayList(1000);;
>>>
>>> ? ? ? ? ? while (true) {
>>> ? ? ? ? ? ? ? log(" list size is {1} ",
>>> ? ? ? ? ? ? ? ? new Object() {
>>> ? ? ? ? ? ? ? ? ? ? ? ? public String toString() {
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return String.valueOf(listToLog.size());
>>> ? ? ? ? ? ? ? ? ? ? ? ? }
>>> ? ? ? ? ? ? ? ? });
>>> ? ? ? ? ? }
>>> ? ? }
>>> }
>>> ==
>>>
>>> As far as garbage collection times go, they are a function of the
>>> count of live objects and on how easy it is to scan them in parallel
>>> and a whole bunch of other factors. ?Others can answer better what
>>> values are common for Fred.
>>>
>>> Zlatin
>>>
>>> On Thu, Mar 22, 2012 at 11:47 PM, Ximin Luo  wrote:
>>>> LOL, are you kidding me?
>>>>
>>>> Please point to the exact lines of code that results in "double-digit
>>>> millisecond pauses"?
>>>>
>>>> Talk is cheap, show us some numbers.
>>>>
>>>> BTW, the "example" I gave is not real c

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

Not really.  Here, I'll change my call to allocate everything on one level:

  while (true) {
  log(" list size is {1} ",
new Integer(listToLog.size()));
  }

I don't see how using this you can log primitive values without garbage.

On Fri, Mar 23, 2012 at 9:33 PM, Ximin Luo  wrote:
> That is because in your example you're allocating all those new Object()s in
> the same stack frame, so the allocator runs out of space on the stack.
>
> If you put the call to log(x, new Object()) inside its only function call, 
> each
> Object will be destroyed once that call exits, and no GC occurs.
>
> Example code here:
> https://gist.github.com/2177070
>
> The "real use case" just adds one simple operation with each logging call
> (since you're never logging without doing something else as well) to show how
> little of a difference it makes in real terms (<5%).
>
> X
>
> On 24/03/12 01:15, Zlatin Balevsky wrote:
>> You're right, the example you gave will put the arguments on the
>> stack. ?I wanted to clarify Escape Analysis in its current form works
>> only if a new object doesn't exit the stack frame where it's created.
>> Because of that, the moment you do something slightly more complicated
>> you will create garbage like this:
>>
>> ? ?private static void logListSize ( final List listToLog ) {
>> ? ? ? ? LOG.log(" list size is {1} ",
>> ? ? ? ? ? ? ? ? new Object() {
>> ? ? ? ? ? ? ? ? ? ? ? ? public String toString() {
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return String.valueOf(listToLog.size());
>> ? ? ? ? ? ? ? ? ? ? ? ? }
>> ? ? ? ? ? ? ? ? });
>> ? ? }
>>
>> Here is the tricky part - if you can guarantee that everywhere in the
>> jvm the LOG.log method is disabled by the logging system, then the
>> entire call will compile to noop and no objects will be created.
>> However, if even one call site enables logging you will end up
>> creating the temp objects even if they don't get logged. ?Below is a
>> full example, compile and run passing "-verbosegc -XX:+PrintGC" and
>> watch it GC all the time.
>>
>> = ?LazyEvalTest.java ===
>> import java.util.*;
>>
>> public class LazyEvalTest {
>>
>> ? ? // only if this is is always false does logging produce no garbage.
>> ? ? private static boolean shouldLog = true;
>>
>> ? ? private static void log(String ar1, Object ar2) {
>> ? ? ? ? if (shouldLog)
>> ? ? ? ? ? ? ?System.out.println(ar1 + ar2.toString());
>> ? ? ? ? shouldLog = false;
>> ? ? }
>>
>> ? ? public static void main(String []ar) {
>>
>> ? ? ? ? final List listToLog = new ArrayList(1000);;
>>
>> ? ? ? ? ? while (true) {
>> ? ? ? ? ? ? ? log(" list size is {1} ",
>> ? ? ? ? ? ? ? ? new Object() {
>> ? ? ? ? ? ? ? ? ? ? ? ? public String toString() {
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return String.valueOf(listToLog.size());
>> ? ? ? ? ? ? ? ? ? ? ? ? }
>> ? ? ? ? ? ? ? ? });
>> ? ? ? ? ? }
>> ? ? }
>> }
>> ==
>>
>> As far as garbage collection times go, they are a function of the
>> count of live objects and on how easy it is to scan them in parallel
>> and a whole bunch of other factors. ?Others can answer better what
>> values are common for Fred.
>>
>> Zlatin
>>
>> On Thu, Mar 22, 2012 at 11:47 PM, Ximin Luo  wrote:
>>> LOL, are you kidding me?
>>>
>>> Please point to the exact lines of code that results in "double-digit
>>> millisecond pauses"?
>>>
>>> Talk is cheap, show us some numbers.
>>>
>>> BTW, the "example" I gave is not real code, and contains no variable
>>> declarations, so your challenge makes no sense. Since you apparently didn't
>>> understand my implicit argument, here it is in more detail: a typical method
>>> that computes something simply for the purpose of logging it somewhere, 
>>> usually
>>> only allocates local variables that are not stored anywhere in the long 
>>> term.
>>> Escape analysis can turn these into stack allocations, saving GC overhead. 
>>> (If
>>> they do use more long-term variables, they will need to be stored on the 
>>> heap,
>>> but then GC doesn't need to clean these up anyway.)
>>>
>>> Why are you even looking at blog entries? Escape analysis has been around 
>>> for
>>> years. http://hg.openjdk.java.net/jdk6/jdk6/hotspot/rev/357d4e2eb4dd
>>>
>>> On 23/03/12 02:06, Zlatin Balevsky wrote:
>>>> My claim is

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

You're right, the example you gave will put the arguments on the
stack.  I wanted to clarify Escape Analysis in its current form works
only if a new object doesn't exit the stack frame where it's created.
Because of that, the moment you do something slightly more complicated
you will create garbage like this:

   private static void logListSize ( final List listToLog ) {
LOG.log(" list size is {1} ",
new Object() {
public String toString() {
return String.valueOf(listToLog.size());
}
});
}

Here is the tricky part - if you can guarantee that everywhere in the
jvm the LOG.log method is disabled by the logging system, then the
entire call will compile to noop and no objects will be created.
However, if even one call site enables logging you will end up
creating the temp objects even if they don't get logged.  Below is a
full example, compile and run passing "-verbosegc -XX:+PrintGC" and
watch it GC all the time.

=  LazyEvalTest.java ===
import java.util.*;

public class LazyEvalTest {

// only if this is is always false does logging produce no garbage.
private static boolean shouldLog = true;

private static void log(String ar1, Object ar2) {
if (shouldLog)
 System.out.println(ar1 + ar2.toString());
shouldLog = false;
}

public static void main(String []ar) {

final List listToLog = new ArrayList(1000);;

  while (true) {
  log(" list size is {1} ",
new Object() {
public String toString() {
return String.valueOf(listToLog.size());
}
});
  }
}
}
==

As far as garbage collection times go, they are a function of the
count of live objects and on how easy it is to scan them in parallel
and a whole bunch of other factors.  Others can answer better what
values are common for Fred.

Zlatin

On Thu, Mar 22, 2012 at 11:47 PM, Ximin Luo  wrote:
> LOL, are you kidding me?
>
> Please point to the exact lines of code that results in "double-digit
> millisecond pauses"?
>
> Talk is cheap, show us some numbers.
>
> BTW, the "example" I gave is not real code, and contains no variable
> declarations, so your challenge makes no sense. Since you apparently didn't
> understand my implicit argument, here it is in more detail: a typical method
> that computes something simply for the purpose of logging it somewhere, 
> usually
> only allocates local variables that are not stored anywhere in the long term.
> Escape analysis can turn these into stack allocations, saving GC overhead. (If
> they do use more long-term variables, they will need to be stored on the heap,
> but then GC doesn't need to clean these up anyway.)
>
> Why are you even looking at blog entries? Escape analysis has been around for
> years. http://hg.openjdk.java.net/jdk6/jdk6/hotspot/rev/357d4e2eb4dd
>
> On 23/03/12 02:06, Zlatin Balevsky wrote:
>> My claim is based on day-to-day observations of hundreds of JVMs under 
>> various
>> load scenarios.
>>
>> Your claim that modern JVMs "do escape analysis" is worthless as it shows 
>> that
>> you have merely read some blog posts, and even those you've read only
>> partially. ?Please point to the exact lines of code in hotspot or any other
>> modern jvm that will optimize the specific lazy evaluation example you
>> presented, together with the opto-assembly that their JITs produce. ?Failing
>> that, take your attitude elsewhere.
>>
>> On Thu, Mar 22, 2012 at 8:49 PM, Ximin Luo > <mailto:infinity0 at gmx.com>> wrote:
>>
>> ? ? The "drastically cleaner syntax" is a red herring. Most of the time when 
>> you
>> ? ? are doing a complex calculation, you are not going to put it on one line
>> ? ? anyway. In such cases, the function you are using to do the calculation 
>> might
>> ? ? as well be the toString() method of some object.
>>
>> ? ? Your claim of "double-digit millisecond" pauses is worthless without some
>> ? ? benchmarking and actual numbers. Modern JVMs do escape analysis to avoid 
>> heap
>> ? ? allocation and this helps especially for transient computations like in
>> ? ? logging.
>>
>> ? ? On 22/03/12 21:59, Zlatin Balevsky wrote:
>> ? ? > Double-digit millisecond pauses are not nothing. ?They may be 
>> acceptable
>> ? ? right
>> ? ? > now but unless you can offer a drastically cleaner syntax Fred should 
>> stick
>> ? ? > with predicates as they are handled much better by the hotspot jit.
>> ? ? >
>

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

You're right, the example you gave will put the arguments on the
stack.  I wanted to clarify Escape Analysis in its current form works
only if a new object doesn't exit the stack frame where it's created.
Because of that, the moment you do something slightly more complicated
you will create garbage like this:

   private static void logListSize ( final List? listToLog ) {
LOG.log( list size is {1} ,
new Object() {
public String toString() {
return String.valueOf(listToLog.size());
}
});
}

Here is the tricky part - if you can guarantee that everywhere in the
jvm the LOG.log method is disabled by the logging system, then the
entire call will compile to noop and no objects will be created.
However, if even one call site enables logging you will end up
creating the temp objects even if they don't get logged.  Below is a
full example, compile and run passing -verbosegc -XX:+PrintGC and
watch it GC all the time.

=  LazyEvalTest.java ===
import java.util.*;

public class LazyEvalTest {

// only if this is is always false does logging produce no garbage.
private static boolean shouldLog = true;

private static void log(String ar1, Object ar2) {
if (shouldLog)
 System.out.println(ar1 + ar2.toString());
shouldLog = false;
}

public static void main(String []ar) {

final ListObject listToLog = new ArrayListObject(1000);;

  while (true) {
  log( list size is {1} ,
new Object() {
public String toString() {
return String.valueOf(listToLog.size());
}
});
  }
}
}
==

As far as garbage collection times go, they are a function of the
count of live objects and on how easy it is to scan them in parallel
and a whole bunch of other factors.  Others can answer better what
values are common for Fred.

Zlatin

On Thu, Mar 22, 2012 at 11:47 PM, Ximin Luo infini...@gmx.com wrote:
 LOL, are you kidding me?

 Please point to the exact lines of code that results in double-digit
 millisecond pauses?

 Talk is cheap, show us some numbers.

 BTW, the example I gave is not real code, and contains no variable
 declarations, so your challenge makes no sense. Since you apparently didn't
 understand my implicit argument, here it is in more detail: a typical method
 that computes something simply for the purpose of logging it somewhere, 
 usually
 only allocates local variables that are not stored anywhere in the long term.
 Escape analysis can turn these into stack allocations, saving GC overhead. (If
 they do use more long-term variables, they will need to be stored on the heap,
 but then GC doesn't need to clean these up anyway.)

 Why are you even looking at blog entries? Escape analysis has been around for
 years. http://hg.openjdk.java.net/jdk6/jdk6/hotspot/rev/357d4e2eb4dd

 On 23/03/12 02:06, Zlatin Balevsky wrote:
 My claim is based on day-to-day observations of hundreds of JVMs under 
 various
 load scenarios.

 Your claim that modern JVMs do escape analysis is worthless as it shows 
 that
 you have merely read some blog posts, and even those you've read only
 partially.  Please point to the exact lines of code in hotspot or any other
 modern jvm that will optimize the specific lazy evaluation example you
 presented, together with the opto-assembly that their JITs produce.  Failing
 that, take your attitude elsewhere.

 On Thu, Mar 22, 2012 at 8:49 PM, Ximin Luo infini...@gmx.com
 mailto:infini...@gmx.com wrote:

     The drastically cleaner syntax is a red herring. Most of the time when 
 you
     are doing a complex calculation, you are not going to put it on one line
     anyway. In such cases, the function you are using to do the calculation 
 might
     as well be the toString() method of some object.

     Your claim of double-digit millisecond pauses is worthless without some
     benchmarking and actual numbers. Modern JVMs do escape analysis to avoid 
 heap
     allocation and this helps especially for transient computations like in
     logging.

     On 22/03/12 21:59, Zlatin Balevsky wrote:
      Double-digit millisecond pauses are not nothing.  They may be 
 acceptable
     right
      now but unless you can offer a drastically cleaner syntax Fred should 
 stick
      with predicates as they are handled much better by the hotspot jit.
     
      On Mar 22, 2012 5:36 PM, Ximin Luo infini...@gmx.com
     mailto:infini...@gmx.com
      mailto:infini...@gmx.com mailto:infini...@gmx.com wrote:
     
          Lazy evaluation is trivial.
     
          Log.info({1} did {2},
           new Object(){ public String toString() { return ITEM_1; } },
           new Object(){ public String toString() { return ITEM_2; } }
          );
     
          Garbage collection with short-lived objects costs next

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

Not really.  Here, I'll change my call to allocate everything on one level:

  while (true) {
  log( list size is {1} ,
new Integer(listToLog.size()));
  }

I don't see how using this you can log primitive values without garbage.

On Fri, Mar 23, 2012 at 9:33 PM, Ximin Luo infini...@gmx.com wrote:
 That is because in your example you're allocating all those new Object()s in
 the same stack frame, so the allocator runs out of space on the stack.

 If you put the call to log(x, new Object()) inside its only function call, 
 each
 Object will be destroyed once that call exits, and no GC occurs.

 Example code here:
 https://gist.github.com/2177070

 The real use case just adds one simple operation with each logging call
 (since you're never logging without doing something else as well) to show how
 little of a difference it makes in real terms (5%).

 X

 On 24/03/12 01:15, Zlatin Balevsky wrote:
 You're right, the example you gave will put the arguments on the
 stack.  I wanted to clarify Escape Analysis in its current form works
 only if a new object doesn't exit the stack frame where it's created.
 Because of that, the moment you do something slightly more complicated
 you will create garbage like this:

    private static void logListSize ( final List? listToLog ) {
         LOG.log( list size is {1} ,
                 new Object() {
                         public String toString() {
                                 return String.valueOf(listToLog.size());
                         }
                 });
     }

 Here is the tricky part - if you can guarantee that everywhere in the
 jvm the LOG.log method is disabled by the logging system, then the
 entire call will compile to noop and no objects will be created.
 However, if even one call site enables logging you will end up
 creating the temp objects even if they don't get logged.  Below is a
 full example, compile and run passing -verbosegc -XX:+PrintGC and
 watch it GC all the time.

 =  LazyEvalTest.java ===
 import java.util.*;

 public class LazyEvalTest {

     // only if this is is always false does logging produce no garbage.
     private static boolean shouldLog = true;

     private static void log(String ar1, Object ar2) {
         if (shouldLog)
              System.out.println(ar1 + ar2.toString());
         shouldLog = false;
     }

     public static void main(String []ar) {

         final ListObject listToLog = new ArrayListObject(1000);;

           while (true) {
               log( list size is {1} ,
                 new Object() {
                         public String toString() {
                                 return String.valueOf(listToLog.size());
                         }
                 });
           }
     }
 }
 ==

 As far as garbage collection times go, they are a function of the
 count of live objects and on how easy it is to scan them in parallel
 and a whole bunch of other factors.  Others can answer better what
 values are common for Fred.

 Zlatin

 On Thu, Mar 22, 2012 at 11:47 PM, Ximin Luo infini...@gmx.com wrote:
 LOL, are you kidding me?

 Please point to the exact lines of code that results in double-digit
 millisecond pauses?

 Talk is cheap, show us some numbers.

 BTW, the example I gave is not real code, and contains no variable
 declarations, so your challenge makes no sense. Since you apparently didn't
 understand my implicit argument, here it is in more detail: a typical method
 that computes something simply for the purpose of logging it somewhere, 
 usually
 only allocates local variables that are not stored anywhere in the long 
 term.
 Escape analysis can turn these into stack allocations, saving GC overhead. 
 (If
 they do use more long-term variables, they will need to be stored on the 
 heap,
 but then GC doesn't need to clean these up anyway.)

 Why are you even looking at blog entries? Escape analysis has been around 
 for
 years. http://hg.openjdk.java.net/jdk6/jdk6/hotspot/rev/357d4e2eb4dd

 On 23/03/12 02:06, Zlatin Balevsky wrote:
 My claim is based on day-to-day observations of hundreds of JVMs under 
 various
 load scenarios.

 Your claim that modern JVMs do escape analysis is worthless as it shows 
 that
 you have merely read some blog posts, and even those you've read only
 partially.  Please point to the exact lines of code in hotspot or any other
 modern jvm that will optimize the specific lazy evaluation example you
 presented, together with the opto-assembly that their JITs produce.  
 Failing
 that, take your attitude elsewhere.

 On Thu, Mar 22, 2012 at 8:49 PM, Ximin Luo infini...@gmx.com
 mailto:infini...@gmx.com wrote:

     The drastically cleaner syntax is a red herring. Most of the time 
 when you
     are doing a complex calculation, you are not going to put it on one 
 line
     anyway. In such cases, the function you are using to do the 
 calculation might
     as well be the toString() method of some object.

     Your

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

The only bug-vector I see in the current predicate approach is that
when used incorrectly it cause the size of methods to explode.  There
are some hard-coded thresholds for the method size in bytecodes that
turn off some or all JIT optimizations; there are also a whole host of
compilation parameters which are optimized with smaller methods in
mind.

Unless someone can offer a great syntactic improvement I recommend
against home-brew frameworks.  Aspects may be able to clean things up
nicely though, I'd be very curious to see some examples.

On Fri, Mar 23, 2012 at 9:56 PM, Zlatin Balevsky zlat...@gmail.com wrote:
 Not really.  Here, I'll change my call to allocate everything on one level:

  while (true) {
              log( list size is {1} ,
                new Integer(listToLog.size()));
          }

 I don't see how using this you can log primitive values without garbage.

 On Fri, Mar 23, 2012 at 9:33 PM, Ximin Luo infini...@gmx.com wrote:
 That is because in your example you're allocating all those new Object()s in
 the same stack frame, so the allocator runs out of space on the stack.

 If you put the call to log(x, new Object()) inside its only function call, 
 each
 Object will be destroyed once that call exits, and no GC occurs.

 Example code here:
 https://gist.github.com/2177070

 The real use case just adds one simple operation with each logging call
 (since you're never logging without doing something else as well) to show how
 little of a difference it makes in real terms (5%).

 X

 On 24/03/12 01:15, Zlatin Balevsky wrote:
 You're right, the example you gave will put the arguments on the
 stack.  I wanted to clarify Escape Analysis in its current form works
 only if a new object doesn't exit the stack frame where it's created.
 Because of that, the moment you do something slightly more complicated
 you will create garbage like this:

    private static void logListSize ( final List? listToLog ) {
         LOG.log( list size is {1} ,
                 new Object() {
                         public String toString() {
                                 return String.valueOf(listToLog.size());
                         }
                 });
     }

 Here is the tricky part - if you can guarantee that everywhere in the
 jvm the LOG.log method is disabled by the logging system, then the
 entire call will compile to noop and no objects will be created.
 However, if even one call site enables logging you will end up
 creating the temp objects even if they don't get logged.  Below is a
 full example, compile and run passing -verbosegc -XX:+PrintGC and
 watch it GC all the time.

 =  LazyEvalTest.java ===
 import java.util.*;

 public class LazyEvalTest {

     // only if this is is always false does logging produce no garbage.
     private static boolean shouldLog = true;

     private static void log(String ar1, Object ar2) {
         if (shouldLog)
              System.out.println(ar1 + ar2.toString());
         shouldLog = false;
     }

     public static void main(String []ar) {

         final ListObject listToLog = new ArrayListObject(1000);;

           while (true) {
               log( list size is {1} ,
                 new Object() {
                         public String toString() {
                                 return String.valueOf(listToLog.size());
                         }
                 });
           }
     }
 }
 ==

 As far as garbage collection times go, they are a function of the
 count of live objects and on how easy it is to scan them in parallel
 and a whole bunch of other factors.  Others can answer better what
 values are common for Fred.

 Zlatin

 On Thu, Mar 22, 2012 at 11:47 PM, Ximin Luo infini...@gmx.com wrote:
 LOL, are you kidding me?

 Please point to the exact lines of code that results in double-digit
 millisecond pauses?

 Talk is cheap, show us some numbers.

 BTW, the example I gave is not real code, and contains no variable
 declarations, so your challenge makes no sense. Since you apparently didn't
 understand my implicit argument, here it is in more detail: a typical 
 method
 that computes something simply for the purpose of logging it somewhere, 
 usually
 only allocates local variables that are not stored anywhere in the long 
 term.
 Escape analysis can turn these into stack allocations, saving GC overhead. 
 (If
 they do use more long-term variables, they will need to be stored on the 
 heap,
 but then GC doesn't need to clean these up anyway.)

 Why are you even looking at blog entries? Escape analysis has been around 
 for
 years. http://hg.openjdk.java.net/jdk6/jdk6/hotspot/rev/357d4e2eb4dd

 On 23/03/12 02:06, Zlatin Balevsky wrote:
 My claim is based on day-to-day observations of hundreds of JVMs under 
 various
 load scenarios.

 Your claim that modern JVMs do escape analysis is worthless as it shows 
 that
 you have merely read some blog posts, and even those you've read only
 partially.  Please point to the exact lines of code

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

Wrong.  Extracting the log statement into its own method and calling
that method inside the while loop still creates garbage.  Stack frames
are different.

Read again exactly how Escape Analysis works.  The objects it prevents
from creating do not appear in heap histograms.  Take a jmap -histo
pid and you will see a huge number of java.lang.Integer objects in
the heap, not on the stack.

The difference in run times of your example could be due to many
things.  How many iterations did you run?  It may be still in
interpreted mode.  Enable -XX:+PrintCompilation and discard any
results until JIT stops printing.

On Fri, Mar 23, 2012 at 10:28 PM, Ximin Luo infini...@gmx.com wrote:
 On 24/03/12 01:56, Zlatin Balevsky wrote:
 Not really.  Here, I'll change my call to allocate everything on one level:

   while (true) {
               log( list size is {1} ,
                 new Integer(listToLog.size()));
           }

 I don't see how using this you can log primitive values without garbage.


 This still allocates many new Integer() on the stack without giving them an
 opportunity to be destroyed, because this occurs all on the same stack frame.
 Eventually there is no more space for stack allocation, leading to heap
 allocation and GC. This is an uncommon situation to pop up in real code
 however, because in most cases you don't have an extremely long loop.

 Have a look at the difference between testWithStackAllocation and
 testWithoutStackAllocation in the example I posted, and the huge difference in
 their run times; maybe it will be clearer then.

 Arguably the JVM should be smart enough to destroy things at the end of a code
 block when things go out of scope, including loops, but practically that
 doesn't appear to be the case. (Unless you have another explanation for the
 behaviour encountered when running my test code?)

 One reason I'm opposed to the predicate form, is because it encourages things 
 like

 | if (log_level) {
 |   // huge complex calculation
 |   // that takes up many lines
 |   // and distracts from the rest of the code
 |   log.level(args);
 | }

 We could agree to restrict logging calls to be of the form

 | if (log_level) log.level(
 |    args ... );

 which would at least force the complex calculations to be factored out into
 another method.

 On Fri, Mar 23, 2012 at 9:33 PM, Ximin Luo infini...@gmx.com wrote:
 That is because in your example you're allocating all those new Object()s in
 the same stack frame, so the allocator runs out of space on the stack.

 If you put the call to log(x, new Object()) inside its only function call, 
 each
 Object will be destroyed once that call exits, and no GC occurs.

 Example code here:
 https://gist.github.com/2177070

 The real use case just adds one simple operation with each logging call
 (since you're never logging without doing something else as well) to show 
 how
 little of a difference it makes in real terms (5%).

 X

 On 24/03/12 01:15, Zlatin Balevsky wrote:
 You're right, the example you gave will put the arguments on the
 stack.  I wanted to clarify Escape Analysis in its current form works
 only if a new object doesn't exit the stack frame where it's created.
 Because of that, the moment you do something slightly more complicated
 you will create garbage like this:

    private static void logListSize ( final List? listToLog ) {
         LOG.log( list size is {1} ,
                 new Object() {
                         public String toString() {
                                 return String.valueOf(listToLog.size());
                         }
                 });
     }

 Here is the tricky part - if you can guarantee that everywhere in the
 jvm the LOG.log method is disabled by the logging system, then the
 entire call will compile to noop and no objects will be created.
 However, if even one call site enables logging you will end up
 creating the temp objects even if they don't get logged.  Below is a
 full example, compile and run passing -verbosegc -XX:+PrintGC and
 watch it GC all the time.

 =  LazyEvalTest.java ===
 import java.util.*;

 public class LazyEvalTest {

     // only if this is is always false does logging produce no garbage.
     private static boolean shouldLog = true;

     private static void log(String ar1, Object ar2) {
         if (shouldLog)
              System.out.println(ar1 + ar2.toString());
         shouldLog = false;
     }

     public static void main(String []ar) {

         final ListObject listToLog = new ArrayListObject(1000);;

           while (true) {
               log( list size is {1} ,
                 new Object() {
                         public String toString() {
                                 return String.valueOf(listToLog.size());
                         }
                 });
           }
     }
 }
 ==

 As far as garbage collection times go, they are a function of the
 count of live objects and on how easy it is to scan them in parallel
 and a whole bunch

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

It doesn't look like a big deal but if all of Fred was using lazy
evaluation for logging it would make one common use case very
annoying:

Say I'm working on a specific class and need to run that class with
debug logging enabled.  This would have the side effect of changing
the compiled LOG.debug( ... ) method globally across the JVM and would
cause every log lazily evaluated parameter for every debug statement
to be instantiated.  Production deployments would not be affected but
the developer experience would.

P.S. nice trick with the Double.MIN_VALUE, I'll use that sometime :)

On Fri, Mar 23, 2012 at 11:33 PM, Ximin Luo infini...@gmx.com wrote:
 I see.

 Actually we didn't even need the side effects thing - I did notice that your
 example stopped doing GC when setting shouldLog = false always, so I tried 
 to
 introduce your if (shouldLog) { shouldLog = false; } into my example, but 
 was
 still unable to reproduce GC. Then changing

 - System.out.println(don't optimise me out);
 + System.out.println(format + args);

 worked to re-introduce GC.

 Well, that sucks. I guess the escape analyzer isn't sophisticated enough to 
 see
 that ar2.toString() does not cause ar2 to escape - and removing that from your
 log() method does result in no GC.

 I still think it's negligible though :p - for my test, I'm seeing only ~8ms
 lost to GC after 20 million calls, and this appears to not change by much even
 if I increase the number of arguments (and objects created) to log_info.

 On 24/03/12 03:02, Zlatin Balevsky wrote:
 Your test has nothing to do with stack allocation because you never
 use the objects that you pass to the log_info method so JIT removes
 them.  Apply the following patch and see yourself create garbage in
 both testWith and testWithout.

 --- Test2.java        2012-03-23 23:01:05.540475497 -0400
 +++ Test.java 2012-03-23 23:01:47.304477562 -0400
 @@ -6,8 +6,11 @@
    final Object existing = new Object();
    Object tmp = null;

 +  long sideEffect;
    public void log_info(String format, Object ... args) {
      if (log_level) {
 +      for (Object o : args)
 +     sideEffect += o.hashCode();
        System.out.println(don't optimise me out);
      }
    }
 @@ -88,11 +91,8 @@
      printSummary(with alloc, res[0], w/o alloc, res[1]);
      printSummary(with alloc, res[2], w/o alloc, res[3]);
      printSummary(with alloc, res[4], w/o alloc, res[5]);
 -    System.out.println( with stack allocation and \real use case\);
 -    res = testWithOtherCode(z);
 -    printSummary(log only, res[0], log+work, res[1]);
 -    printSummary(log only, res[2], log+work, res[3]);
 -    printSummary(log only, res[4], log+work, res[5]);
 +
 +    System.out.println(sideEffect);
    }

    public static void main(String[] args) {


 On Fri, Mar 23, 2012 at 10:48 PM, Ximin Luo infini...@gmx.com wrote:
 On 24/03/12 02:44, Zlatin Balevsky wrote:
 Wrong.  Extracting the log statement into its own method and calling
 that method inside the while loop still creates garbage.  Stack frames
 are different.


 How do you explain the results obtained from running my test code, then? 
 Turn
 on -verbose:gc. testWithStackAllocation results in no GC,
 testWithoutStackAllocation results in lots of GC.

 Read again exactly how Escape Analysis works.  The objects it prevents
 from creating do not appear in heap histograms.  Take a jmap -histo
 pid and you will see a huge number of java.lang.Integer objects in
 the heap, not on the stack.


 If you're still talking about your example, this is exactly consistent with 
 my
 explanation, i.e. that escape analysis is NOT occurring, properly anyway.

 The difference in run times of your example could be due to many
 things.  How many iterations did you run?  It may be still in
 interpreted mode.  Enable -XX:+PrintCompilation and discard any
 results until JIT stops printing.

 Done that, JIT stops printing after the 3rd iteration and results are the 
 same
 thereafter.

 --
 GPG: 4096R/5FBBDBCE
 https://github.com/infinity0
 https://bitbucket.org/infinity0
 https://launchpad.net/~infinity0


 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl


 --
 GPG: 4096R/5FBBDBCE
 https://github.com/infinity0
 https://bitbucket.org/infinity0
 https://launchpad.net/~infinity0


 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

The example you provide is fast because it does not do any object
allocation.  Please don't take my word for it - download (or better
yet, build one yourself) a "fastdebug" build from openjdk.org and look
at the opto-assembly that gets generated when you pass the
"-XX:+PrintOptoAssembly" switch.

Benchmarking JIT's is a fascinating topic but beyond the scope of this
list.  Furthermore, anything I say applies only to the Hotspot JVM.

On Thu, Mar 22, 2012 at 8:18 PM, Marco Schulze
 wrote:
>
> I already have all but log rotation and async ready, and haven't yet found a 
> single benchmark supporting the use of a branch as the performance holy 
> grail. For example (outputting to /dev/null):
>
> public static void main (String[] args) {
> ??? ??? for (int i = 0; i < 100; i++) {
> ??? ?? ? ?? ??? Log.fatal (Log.class, Log.class, "akd\n\n", i, '\n', out, ' 
> ');
> ?? ? ?? ??? ??? Log.trace (Log.class, Log.class, "akd\n\n", i, '\n', out, ' 
> ');
> ? ?? ?? }
> }
>
> Every call means, minimally, varargs boxing, another call (since fatal() and 
> trace() are simple convenience methods) and an isLoggable() check composed by 
> a ConcurrentHashMap lookup against the class name and (possibly) a 
> synchronized read on the global threshold. trace() is filtered but fatal() is 
> not.
>
> This snipped ran in an average 6.482 seconds. If the call to trace() is 
> commented out (thus removing the filtering overhead), the average falls to 
> 6.366 seconds. Disabling JIT, the figures became 1:37.952 and 1:35.880, 
> respectively. Over a million calls, checking costs only a few milliseconds.
>
> To be sure, this is a fairly simple example: it all runs on a single thread, 
> the hash table is empty and the pressure on the GC is low. Still, differences 
> are very small. Plus, there's no overhead due to a dedicated logging thread.
>
>
> On 22-03-2012 18:59, Zlatin Balevsky wrote:
>
> Double-digit millisecond pauses are not nothing.? They may be acceptable 
> right now but unless you can offer a drastically cleaner syntax Fred should 
> stick with predicates as they are handled much better by the hotspot jit.
>
> On Mar 22, 2012 5:36 PM, "Ximin Luo"  wrote:
>>
>> Lazy evaluation is trivial.
>>
>> Log.info("{1} did {2}",
>> ?new Object(){ public String toString() { return ITEM_1; } },
>> ?new Object(){ public String toString() { return ITEM_2; } }
>> );
>>
>> Garbage collection with short-lived objects costs next to nothing.
>>
>> On 22/03/12 21:15, Zlatin Balevsky wrote:
>> > Constructing the logging strings is half of the problem. ?The amount of 
>> > garbage
>> > they will generate will result in significantly more time in garbage 
>> > collection
>> > pauses.
>> >
>> > Unless you figure out a way to mimic lazy evaluation you have to live with 
>> > the
>> > isLoggable predicates. ?varargs are not an option either because they also
>> > create garbage.
>> >
>> > On Mar 22, 2012 8:11 AM, "Marco Schulze" > > <mailto:marco.c.schulze at gmail.com>> wrote:
>> >
>> >
>> >
>> > ? ? On 22-03-2012 08:50, Matthew Toseland wrote:
>> >
>> > ? ? ? ? On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
>> >
>> > ? ? ? ? ? ? There are basically two big concerns regarding logging in fred:
>> >
>> > ? ? ? ? ? ? - Readability and code clutter, which was my original 
>> > questioning;
>> > ? ? ? ? ? ? - Raw throughput, as raised by toad.
>> >
>> > ? ? ? ? ? ? Point 1 could mostly be solved by removing any traces of 
>> > logMINOR and
>> > ? ? ? ? ? ? logDEBUG on all but the few places where generating messages 
>> > to be
>> > ? ? ? ? ? ? logged brings noticeable slowdown. That'd be enough, but, 
>> > personally,
>> > ? ? ? ? ? ? the mess that the logging backend is does warrant a 
>> > replacement.
>> > ? ? ? ? ? ? According to toad, the current system needs log{MINOR,DEBUG} to
>> > ? ? ? ? ? ? function
>> > ? ? ? ? ? ? in a timely manner. Based on this, I think we all agree a
>> > ? ? ? ? ? ? replacement is
>> > ? ? ? ? ? ? desirable.
>> >
>> > ? ? ? ? ? ? Logging has a few additional requirements:
>> >
>> > ? ? ? ? ? ? - Log rotation (possibly live);
>> > ? ? ? ? ? ? - Reentrant;
>> > ? ? ? ? ? ? - Per-class filtering;
>> > ? ? ? ? ? ? - Specific information in log (class-name, for example).
>> >
>> > ? ? ? ? ? ? Now, _any_ library whic

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

My claim is based on day-to-day observations of hundreds of JVMs under
various load scenarios.

Your claim that modern JVMs "do escape analysis" is worthless as it shows
that you have merely read some blog posts, and even those you've read only
partially.  Please point to the exact lines of code in hotspot or any other
modern jvm that will optimize the specific lazy evaluation example you
presented, together with the opto-assembly that their JITs produce.
Failing that, take your attitude elsewhere.

On Thu, Mar 22, 2012 at 8:49 PM, Ximin Luo  wrote:

> The "drastically cleaner syntax" is a red herring. Most of the time when
> you
> are doing a complex calculation, you are not going to put it on one line
> anyway. In such cases, the function you are using to do the calculation
> might
> as well be the toString() method of some object.
>
> Your claim of "double-digit millisecond" pauses is worthless without some
> benchmarking and actual numbers. Modern JVMs do escape analysis to avoid
> heap
> allocation and this helps especially for transient computations like in
> logging.
>
> On 22/03/12 21:59, Zlatin Balevsky wrote:
> > Double-digit millisecond pauses are not nothing.  They may be acceptable
> right
> > now but unless you can offer a drastically cleaner syntax Fred should
> stick
> > with predicates as they are handled much better by the hotspot jit.
> >
> > On Mar 22, 2012 5:36 PM, "Ximin Luo"  > <mailto:infinity0 at gmx.com>> wrote:
> >
> > Lazy evaluation is trivial.
> >
> > Log.info("{1} did {2}",
> >  new Object(){ public String toString() { return ITEM_1; } },
> >  new Object(){ public String toString() { return ITEM_2; } }
> > );
> >
> > Garbage collection with short-lived objects costs next to nothing.
> >
> > On 22/03/12 21:15, Zlatin Balevsky wrote:
> > > Constructing the logging strings is half of the problem.  The
> amount of
> > garbage
> > > they will generate will result in significantly more time in
> garbage
> > collection
> > > pauses.
> > >
> > > Unless you figure out a way to mimic lazy evaluation you have to
> live
> > with the
> > > isLoggable predicates.  varargs are not an option either because
> they also
> > > create garbage.
> > >
> > > On Mar 22, 2012 8:11 AM, "Marco Schulze" <
> marco.c.schulze at gmail.com
> > <mailto:marco.c.schulze at gmail.com>
> > > <mailto:marco.c.schulze at gmail.com  marco.c.schulze at gmail.com>>> wrote:
> > >
> > >
> > >
> > > On 22-03-2012 08:50, Matthew Toseland wrote:
> > >
> > > On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
> > >
> > > There are basically two big concerns regarding logging
> in fred:
> > >
> > > - Readability and code clutter, which was my original
> > questioning;
> > > - Raw throughput, as raised by toad.
> > >
> > > Point 1 could mostly be solved by removing any traces
> of
> > logMINOR and
> > > logDEBUG on all but the few places where generating
> messages
> > to be
> > > logged brings noticeable slowdown. That'd be enough,
> but,
> > personally,
> > > the mess that the logging backend is does warrant a
> replacement.
> > > According to toad, the current system needs
> log{MINOR,DEBUG} to
> > > function
> > > in a timely manner. Based on this, I think we all
> agree a
> > > replacement is
> > > desirable.
> > >
> > > Logging has a few additional requirements:
> > >
> > > - Log rotation (possibly live);
> > > - Reentrant;
> > > - Per-class filtering;
> > > - Specific information in log (class-name, for
> example).
> > >
> > > Now, _any_ library which fits would make me happy, as
> long as
> > they
> > > agree
> > > to two points:
> > >
> > > - Either lightweight or with optional features. Else,
> it
> > would only
> > > transfer bloat to freenet-ext.jar. For example

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

Double-digit millisecond pauses are not nothing.  They may be acceptable
right now but unless you can offer a drastically cleaner syntax Fred should
stick with predicates as they are handled much better by the hotspot jit.
On Mar 22, 2012 5:36 PM, "Ximin Luo"  wrote:

> Lazy evaluation is trivial.
>
> Log.info("{1} did {2}",
>  new Object(){ public String toString() { return ITEM_1; } },
>  new Object(){ public String toString() { return ITEM_2; } }
> );
>
> Garbage collection with short-lived objects costs next to nothing.
>
> On 22/03/12 21:15, Zlatin Balevsky wrote:
> > Constructing the logging strings is half of the problem.  The amount of
> garbage
> > they will generate will result in significantly more time in garbage
> collection
> > pauses.
> >
> > Unless you figure out a way to mimic lazy evaluation you have to live
> with the
> > isLoggable predicates.  varargs are not an option either because they
> also
> > create garbage.
> >
> > On Mar 22, 2012 8:11 AM, "Marco Schulze"  > <mailto:marco.c.schulze at gmail.com>> wrote:
> >
> >
> >
> > On 22-03-2012 08:50, Matthew Toseland wrote:
> >
> > On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
> >
> > There are basically two big concerns regarding logging in
> fred:
> >
> > - Readability and code clutter, which was my original
> questioning;
> > - Raw throughput, as raised by toad.
> >
> > Point 1 could mostly be solved by removing any traces of
> logMINOR and
> > logDEBUG on all but the few places where generating messages
> to be
> > logged brings noticeable slowdown. That'd be enough, but,
> personally,
> > the mess that the logging backend is does warrant a
> replacement.
> > According to toad, the current system needs log{MINOR,DEBUG}
> to
> > function
> > in a timely manner. Based on this, I think we all agree a
> > replacement is
> > desirable.
> >
> > Logging has a few additional requirements:
> >
> > - Log rotation (possibly live);
> > - Reentrant;
> > - Per-class filtering;
> > - Specific information in log (class-name, for example).
> >
> > Now, _any_ library which fits would make me happy, as long
> as they
> > agree
> > to two points:
> >
> > - Either lightweight or with optional features. Else, it
> would only
> > transfer bloat to freenet-ext.jar. For example: log2socket,
> config
> > management and multiple logging instances;
> > - Implementable in a few LoC. Specially, it shouldn't need
> specialized
> > Formatter and Writer.
> >
> > Plus, it should be fast.
> >
> >  From the quick research I made (yep, too many lists):
> >
> > - SLF4J already fails on point one: it is simply a wrapper;
> > - The Java logging API fails on point two: specialized
> classes would
> > have to be written to deal with log rotation, per-class
> filtering and
> > formatting, plus a wrapper for Logger.{info,warning,...}()
> methods.
> > Exactly the same as a custom logger, with one more
> dependency and using
> > more LoC;
> >
> > No dependancies, it's part of the JDK, isn't it?
> >
> > More classes need to be loaded at startup. It's just me thinking too
> much.
> >
> >
> > However, if it's not a clearer/simpler API, it probably doesn't
> make
> > much sense.
> >
> > - Log4J seems to fail on point one - it only lacks a button
> that brings
> > back the dead. It seems interesting, and I haven't dropped
> this yet.
> >
> > In either case (custom or external), log* would be banished.
> Forever.
> >
> > I don't follow. You object to using a separate logs folder?
> >
> > log* == log{MINOR,DEBUG}, not the logs folder.
> > _
> > Devl mailing list
> > Devl at freenetproject.org <mailto:Devl at freenetproject.org>
> > https://emu.freenetproject.__org/cgi-bin/mailman/listinfo/__devl
> > <https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl>
> >
> >
> >
> > ___
> > Devl mailing list
> > Devl at freenetproject.org
> > https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
>
> --
> GPG: 4096R/5FBBDBCE
> https://github.com/infinity0
> https://bitbucket.org/infinity0
> https://launchpad.net/~infinity0
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20120322/66a50f58/attachment.html>

[freenet-dev] Logging in Fred

[Zlatin Balevsky]

Constructing the logging strings is half of the problem.  The amount of
garbage they will generate will result in significantly more time in
garbage collection pauses.

Unless you figure out a way to mimic lazy evaluation you have to live with
the isLoggable predicates.  varargs are not an option either because they
also create garbage.
On Mar 22, 2012 8:11 AM, "Marco Schulze"  wrote:

>
>
> On 22-03-2012 08:50, Matthew Toseland wrote:
>
>> On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
>>
>>> There are basically two big concerns regarding logging in fred:
>>>
>>> - Readability and code clutter, which was my original questioning;
>>> - Raw throughput, as raised by toad.
>>>
>>> Point 1 could mostly be solved by removing any traces of logMINOR and
>>> logDEBUG on all but the few places where generating messages to be
>>> logged brings noticeable slowdown. That'd be enough, but, personally,
>>> the mess that the logging backend is does warrant a replacement.
>>> According to toad, the current system needs log{MINOR,DEBUG} to function
>>> in a timely manner. Based on this, I think we all agree a replacement is
>>> desirable.
>>>
>>> Logging has a few additional requirements:
>>>
>>> - Log rotation (possibly live);
>>> - Reentrant;
>>> - Per-class filtering;
>>> - Specific information in log (class-name, for example).
>>>
>>> Now, _any_ library which fits would make me happy, as long as they agree
>>> to two points:
>>>
>>> - Either lightweight or with optional features. Else, it would only
>>> transfer bloat to freenet-ext.jar. For example: log2socket, config
>>> management and multiple logging instances;
>>> - Implementable in a few LoC. Specially, it shouldn't need specialized
>>> Formatter and Writer.
>>>
>>> Plus, it should be fast.
>>>
>>>  From the quick research I made (yep, too many lists):
>>>
>>> - SLF4J already fails on point one: it is simply a wrapper;
>>> - The Java logging API fails on point two: specialized classes would
>>> have to be written to deal with log rotation, per-class filtering and
>>> formatting, plus a wrapper for Logger.{info,warning,...}() methods.
>>> Exactly the same as a custom logger, with one more dependency and using
>>> more LoC;
>>>
>> No dependancies, it's part of the JDK, isn't it?
>>
> More classes need to be loaded at startup. It's just me thinking too much.
>
>
>> However, if it's not a clearer/simpler API, it probably doesn't make much
>> sense.
>>
>>  - Log4J seems to fail on point one - it only lacks a button that brings
>>> back the dead. It seems interesting, and I haven't dropped this yet.
>>>
>>> In either case (custom or external), log* would be banished. Forever.
>>>
>> I don't follow. You object to using a separate logs folder?
>>
> log* == log{MINOR,DEBUG}, not the logs folder.
> __**_
> Devl mailing list
> Devl at freenetproject.org
> https://emu.freenetproject.**org/cgi-bin/mailman/listinfo/**devl
>
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

Double-digit millisecond pauses are not nothing.  They may be acceptable
right now but unless you can offer a drastically cleaner syntax Fred should
stick with predicates as they are handled much better by the hotspot jit.
On Mar 22, 2012 5:36 PM, Ximin Luo infini...@gmx.com wrote:

 Lazy evaluation is trivial.

 Log.info({1} did {2},
  new Object(){ public String toString() { return ITEM_1; } },
  new Object(){ public String toString() { return ITEM_2; } }
 );

 Garbage collection with short-lived objects costs next to nothing.

 On 22/03/12 21:15, Zlatin Balevsky wrote:
  Constructing the logging strings is half of the problem.  The amount of
 garbage
  they will generate will result in significantly more time in garbage
 collection
  pauses.
 
  Unless you figure out a way to mimic lazy evaluation you have to live
 with the
  isLoggable predicates.  varargs are not an option either because they
 also
  create garbage.
 
  On Mar 22, 2012 8:11 AM, Marco Schulze marco.c.schu...@gmail.com
  mailto:marco.c.schu...@gmail.com wrote:
 
 
 
  On 22-03-2012 08:50, Matthew Toseland wrote:
 
  On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
 
  There are basically two big concerns regarding logging in
 fred:
 
  - Readability and code clutter, which was my original
 questioning;
  - Raw throughput, as raised by toad.
 
  Point 1 could mostly be solved by removing any traces of
 logMINOR and
  logDEBUG on all but the few places where generating messages
 to be
  logged brings noticeable slowdown. That'd be enough, but,
 personally,
  the mess that the logging backend is does warrant a
 replacement.
  According to toad, the current system needs log{MINOR,DEBUG}
 to
  function
  in a timely manner. Based on this, I think we all agree a
  replacement is
  desirable.
 
  Logging has a few additional requirements:
 
  - Log rotation (possibly live);
  - Reentrant;
  - Per-class filtering;
  - Specific information in log (class-name, for example).
 
  Now, _any_ library which fits would make me happy, as long
 as they
  agree
  to two points:
 
  - Either lightweight or with optional features. Else, it
 would only
  transfer bloat to freenet-ext.jar. For example: log2socket,
 config
  management and multiple logging instances;
  - Implementable in a few LoC. Specially, it shouldn't need
 specialized
  Formatter and Writer.
 
  Plus, it should be fast.
 
   From the quick research I made (yep, too many lists):
 
  - SLF4J already fails on point one: it is simply a wrapper;
  - The Java logging API fails on point two: specialized
 classes would
  have to be written to deal with log rotation, per-class
 filtering and
  formatting, plus a wrapper for Logger.{info,warning,...}()
 methods.
  Exactly the same as a custom logger, with one more
 dependency and using
  more LoC;
 
  No dependancies, it's part of the JDK, isn't it?
 
  More classes need to be loaded at startup. It's just me thinking too
 much.
 
 
  However, if it's not a clearer/simpler API, it probably doesn't
 make
  much sense.
 
  - Log4J seems to fail on point one - it only lacks a button
 that brings
  back the dead. It seems interesting, and I haven't dropped
 this yet.
 
  In either case (custom or external), log* would be banished.
 Forever.
 
  I don't follow. You object to using a separate logs folder?
 
  log* == log{MINOR,DEBUG}, not the logs folder.
  _
  Devl mailing list
  Devl@freenetproject.org mailto:Devl@freenetproject.org
  https://emu.freenetproject.__org/cgi-bin/mailman/listinfo/__devl
  https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
 
 
 
  ___
  Devl mailing list
  Devl@freenetproject.org
  https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl


 --
 GPG: 4096R/5FBBDBCE
 https://github.com/infinity0
 https://bitbucket.org/infinity0
 https://launchpad.net/~infinity0


 ___
 Devl mailing list
 Devl@freenetproject.org
 https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

My claim is based on day-to-day observations of hundreds of JVMs under
various load scenarios.

Your claim that modern JVMs do escape analysis is worthless as it shows
that you have merely read some blog posts, and even those you've read only
partially.  Please point to the exact lines of code in hotspot or any other
modern jvm that will optimize the specific lazy evaluation example you
presented, together with the opto-assembly that their JITs produce.
Failing that, take your attitude elsewhere.

On Thu, Mar 22, 2012 at 8:49 PM, Ximin Luo infini...@gmx.com wrote:

 The drastically cleaner syntax is a red herring. Most of the time when
 you
 are doing a complex calculation, you are not going to put it on one line
 anyway. In such cases, the function you are using to do the calculation
 might
 as well be the toString() method of some object.

 Your claim of double-digit millisecond pauses is worthless without some
 benchmarking and actual numbers. Modern JVMs do escape analysis to avoid
 heap
 allocation and this helps especially for transient computations like in
 logging.

 On 22/03/12 21:59, Zlatin Balevsky wrote:
  Double-digit millisecond pauses are not nothing.  They may be acceptable
 right
  now but unless you can offer a drastically cleaner syntax Fred should
 stick
  with predicates as they are handled much better by the hotspot jit.
 
  On Mar 22, 2012 5:36 PM, Ximin Luo infini...@gmx.com
  mailto:infini...@gmx.com wrote:
 
  Lazy evaluation is trivial.
 
  Log.info({1} did {2},
   new Object(){ public String toString() { return ITEM_1; } },
   new Object(){ public String toString() { return ITEM_2; } }
  );
 
  Garbage collection with short-lived objects costs next to nothing.
 
  On 22/03/12 21:15, Zlatin Balevsky wrote:
   Constructing the logging strings is half of the problem.  The
 amount of
  garbage
   they will generate will result in significantly more time in
 garbage
  collection
   pauses.
  
   Unless you figure out a way to mimic lazy evaluation you have to
 live
  with the
   isLoggable predicates.  varargs are not an option either because
 they also
   create garbage.
  
   On Mar 22, 2012 8:11 AM, Marco Schulze 
 marco.c.schu...@gmail.com
  mailto:marco.c.schu...@gmail.com
   mailto:marco.c.schu...@gmail.com mailto:
 marco.c.schu...@gmail.com wrote:
  
  
  
   On 22-03-2012 08:50, Matthew Toseland wrote:
  
   On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
  
   There are basically two big concerns regarding logging
 in fred:
  
   - Readability and code clutter, which was my original
  questioning;
   - Raw throughput, as raised by toad.
  
   Point 1 could mostly be solved by removing any traces
 of
  logMINOR and
   logDEBUG on all but the few places where generating
 messages
  to be
   logged brings noticeable slowdown. That'd be enough,
 but,
  personally,
   the mess that the logging backend is does warrant a
 replacement.
   According to toad, the current system needs
 log{MINOR,DEBUG} to
   function
   in a timely manner. Based on this, I think we all
 agree a
   replacement is
   desirable.
  
   Logging has a few additional requirements:
  
   - Log rotation (possibly live);
   - Reentrant;
   - Per-class filtering;
   - Specific information in log (class-name, for
 example).
  
   Now, _any_ library which fits would make me happy, as
 long as
  they
   agree
   to two points:
  
   - Either lightweight or with optional features. Else,
 it
  would only
   transfer bloat to freenet-ext.jar. For example:
 log2socket,
  config
   management and multiple logging instances;
   - Implementable in a few LoC. Specially, it shouldn't
 need
  specialized
   Formatter and Writer.
  
   Plus, it should be fast.
  
From the quick research I made (yep, too many lists):
  
   - SLF4J already fails on point one: it is simply a
 wrapper;
   - The Java logging API fails on point two: specialized
  classes would
   have to be written to deal with log rotation, per-class
  filtering and
   formatting, plus a wrapper for
 Logger.{info,warning,...}()
  methods.
   Exactly the same as a custom logger, with one more
 dependency
  and using
   more LoC;
  
   No dependancies, it's part of the JDK, isn't it?
  
   More classes

Re: [freenet-dev] Logging in Fred

[Zlatin Balevsky]

The example you provide is fast because it does not do any object
allocation.  Please don't take my word for it - download (or better
yet, build one yourself) a fastdebug build from openjdk.org and look
at the opto-assembly that gets generated when you pass the
-XX:+PrintOptoAssembly switch.

Benchmarking JIT's is a fascinating topic but beyond the scope of this
list.  Furthermore, anything I say applies only to the Hotspot JVM.

On Thu, Mar 22, 2012 at 8:18 PM, Marco Schulze
marco.c.schu...@gmail.com wrote:

 I already have all but log rotation and async ready, and haven't yet found a 
 single benchmark supporting the use of a branch as the performance holy 
 grail. For example (outputting to /dev/null):

 public static void main (String[] args) {
         for (int i = 0; i  100; i++) {
                 Log.fatal (Log.class, Log.class, akd\n\n, i, '\n', out, ' 
 ');
                 Log.trace (Log.class, Log.class, akd\n\n, i, '\n', out, ' 
 ');
         }
 }

 Every call means, minimally, varargs boxing, another call (since fatal() and 
 trace() are simple convenience methods) and an isLoggable() check composed by 
 a ConcurrentHashMap lookup against the class name and (possibly) a 
 synchronized read on the global threshold. trace() is filtered but fatal() is 
 not.

 This snipped ran in an average 6.482 seconds. If the call to trace() is 
 commented out (thus removing the filtering overhead), the average falls to 
 6.366 seconds. Disabling JIT, the figures became 1:37.952 and 1:35.880, 
 respectively. Over a million calls, checking costs only a few milliseconds.

 To be sure, this is a fairly simple example: it all runs on a single thread, 
 the hash table is empty and the pressure on the GC is low. Still, differences 
 are very small. Plus, there's no overhead due to a dedicated logging thread.


 On 22-03-2012 18:59, Zlatin Balevsky wrote:

 Double-digit millisecond pauses are not nothing.  They may be acceptable 
 right now but unless you can offer a drastically cleaner syntax Fred should 
 stick with predicates as they are handled much better by the hotspot jit.

 On Mar 22, 2012 5:36 PM, Ximin Luo infini...@gmx.com wrote:

 Lazy evaluation is trivial.

 Log.info({1} did {2},
  new Object(){ public String toString() { return ITEM_1; } },
  new Object(){ public String toString() { return ITEM_2; } }
 );

 Garbage collection with short-lived objects costs next to nothing.

 On 22/03/12 21:15, Zlatin Balevsky wrote:
  Constructing the logging strings is half of the problem.  The amount of 
  garbage
  they will generate will result in significantly more time in garbage 
  collection
  pauses.
 
  Unless you figure out a way to mimic lazy evaluation you have to live with 
  the
  isLoggable predicates.  varargs are not an option either because they also
  create garbage.
 
  On Mar 22, 2012 8:11 AM, Marco Schulze marco.c.schu...@gmail.com
  mailto:marco.c.schu...@gmail.com wrote:
 
 
 
      On 22-03-2012 08:50, Matthew Toseland wrote:
 
          On Wednesday 21 Mar 2012 21:18:37 Marco Schulze wrote:
 
              There are basically two big concerns regarding logging in fred:
 
              - Readability and code clutter, which was my original 
  questioning;
              - Raw throughput, as raised by toad.
 
              Point 1 could mostly be solved by removing any traces of 
  logMINOR and
              logDEBUG on all but the few places where generating messages 
  to be
              logged brings noticeable slowdown. That'd be enough, but, 
  personally,
              the mess that the logging backend is does warrant a 
  replacement.
              According to toad, the current system needs log{MINOR,DEBUG} to
              function
              in a timely manner. Based on this, I think we all agree a
              replacement is
              desirable.
 
              Logging has a few additional requirements:
 
              - Log rotation (possibly live);
              - Reentrant;
              - Per-class filtering;
              - Specific information in log (class-name, for example).
 
              Now, _any_ library which fits would make me happy, as long as 
  they
              agree
              to two points:
 
              - Either lightweight or with optional features. Else, it would 
  only
              transfer bloat to freenet-ext.jar. For example: log2socket, 
  config
              management and multiple logging instances;
              - Implementable in a few LoC. Specially, it shouldn't need 
  specialized
              Formatter and Writer.
 
              Plus, it should be fast.
 
               From the quick research I made (yep, too many lists):
 
              - SLF4J already fails on point one: it is simply a wrapper;
              - The Java logging API fails on point two: specialized classes 
  would
              have to be written to deal with log rotation, per-class 
  filtering and
              formatting, plus a wrapper for Logger.{info

[freenet-dev] webui and accessibility

[Zlatin Balevsky]

Just want to add JRuby to the list.  Rails has lots of fans and third-party
tools and it's under active development and supposedly on java 1.7 it runs
faster than the C implementation.

On Wed, Mar 7, 2012 at 11:23 AM, Nicolas Hernandez <
nicolas.hernandez at aleph-networks.com> wrote:

> Hello Ian,
>
> We are still looking for the best choice between running in jetty
> a. Accessibility for devs (nice eclipse plugin for example)
> b. Accessibility for user
> c. Light weight
> d. Performance
>
> In our side the priority are like that a>b>d>c (+eclipse plugin)
> Four some freenet devs it loks like c>d>a>b (+velocity for templating)
>
> Not so easy to make the good choice. We have three mains ideas
> A- using Apache Wickets http://wicket.apache.org/
> B- gwt
> C- struts+extjs
> *D-  freenet devs ideas*
>
> This frameworks have to;
> 1. be active and old enough
> 2. including web n.m fionctionnalities
> 3. replace the Toad controller :-)
> 4. using jetty
> 5. be fully toolled for Eclipse
>
> We hope to have freenet devs ideas on it before starting anything :-)
>
> Rgds
>
> - Nicolas Hernandez
> a-n - aleph-networks
> *associ?*
> http://www.aleph-networks.com
>
>
>
>
> On Wed, Mar 7, 2012 at 3:33 PM, Ian Clarke  wrote:
>
>> Hi Nicolas,
>>
>> Did you ever make any progress on this?
>>
>> Ian.
>>
>> On Tue, Feb 21, 2012 at 11:23 AM, Nicolas Hernandez <
>> nicolas.hernandez at aleph-networks.com> wrote:
>>
>>> Hello,
>>>
>>> This is a sort of personnal introduction:
>>>
>>> We want to work on freenet accessibility. The first step could be the
>>> webui. In the case of a total rework of the webui. Wich framework could be
>>> the 'good' one in the freenet philosophy ? Wich software architecture ?
>>>
>>> Is GWT and his model a good idea ? Could it be possible to have help
>>> from GSoc 2012 - (i can be the man with the Umbrella) ?
>>>
>>> ... lots of questions ...
>>>
>>> Rgds
>>>
>>> - Nicolas Hernandez
>>> a-n - aleph-networks
>>> *associ?*
>>> http://www.aleph-networks.com
>>>
>>>
>>>
>>> ___
>>> Devl mailing list
>>> Devl at freenetproject.org
>>> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
>>>
>>
>>
>>
>> --
>> Ian Clarke
>> Personal blog: http://blog.locut.us/
>>
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
>
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: [freenet-dev] webui and accessibility

[Zlatin Balevsky]

Just want to add JRuby to the list.  Rails has lots of fans and third-party
tools and it's under active development and supposedly on java 1.7 it runs
faster than the C implementation.

On Wed, Mar 7, 2012 at 11:23 AM, Nicolas Hernandez 
nicolas.hernan...@aleph-networks.com wrote:

 Hello Ian,

 We are still looking for the best choice between running in jetty
 a. Accessibility for devs (nice eclipse plugin for example)
 b. Accessibility for user
 c. Light weight
 d. Performance

 In our side the priority are like that abdc (+eclipse plugin)
 Four some freenet devs it loks like cdab (+velocity for templating)

 Not so easy to make the good choice. We have three mains ideas
 A- using Apache Wickets http://wicket.apache.org/
 B- gwt
 C- struts+extjs
 *D-  freenet devs ideas*

 This frameworks have to;
 1. be active and old enough
 2. including web n.m fionctionnalities
 3. replace the Toad controller :-)
 4. using jetty
 5. be fully toolled for Eclipse

 We hope to have freenet devs ideas on it before starting anything :-)

 Rgds

 - Nicolas Hernandez
 a-n - aleph-networks
 *associé*
 http://www.aleph-networks.com




 On Wed, Mar 7, 2012 at 3:33 PM, Ian Clarke i...@locut.us wrote:

 Hi Nicolas,

 Did you ever make any progress on this?

 Ian.

 On Tue, Feb 21, 2012 at 11:23 AM, Nicolas Hernandez 
 nicolas.hernan...@aleph-networks.com wrote:

 Hello,

 This is a sort of personnal introduction:

 We want to work on freenet accessibility. The first step could be the
 webui. In the case of a total rework of the webui. Wich framework could be
 the 'good' one in the freenet philosophy ? Wich software architecture ?

 Is GWT and his model a good idea ? Could it be possible to have help
 from GSoc 2012 - (i can be the man with the Umbrella) ?

 ... lots of questions ...

 Rgds

 - Nicolas Hernandez
 a-n - aleph-networks
 *associé*
 http://www.aleph-networks.com



 ___
 Devl mailing list
 Devl@freenetproject.org
 http://freenetproject.org/cgi-bin/mailman/listinfo/devl




 --
 Ian Clarke
 Personal blog: http://blog.locut.us/



 ___
 Devl mailing list
 Devl@freenetproject.org
 http://freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] GSoC 2012

[Zlatin Balevsky]

+1 you might just have a winner

On Thu, Feb 16, 2012 at 1:32 PM, Ian Clarke  wrote:

> I'm not sure whether people would object to this, but I would quite like
> to mentor a student to work on Tahrir under the umbrella of the Freenet
> project.
>
> Thoughts?  Opinions?  Insults?
>
> Ian.
>
>
> On Thu, Feb 16, 2012 at 12:12 PM, Florent Daigniere <
> nextgens at freenetproject.org> wrote:
>
>> Hi!
>>
>> Should Freenet take part to Google Summer of Code 2012?
>>
>> Is anyone motivated to mentor candidates?
>>
>> I'd like to mentor but probably won't do it if I'm the only one.
>>
>> Florent
>> ___
>> Devl mailing list
>> Devl at freenetproject.org
>> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
>>
>
>
>
> --
> Ian Clarke
> Founder, The Freenet Project
> Email: ian at freenetproject.org
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
>
-- next part --
An HTML attachment was scrubbed...
URL: 

[freenet-dev] GSoC 2012

[Zlatin Balevsky]

I'm not up to speed with specifics of the current Freenet codebase but I'd
enjoy mentoring for general java/jvm development best practices.  More so,
I'd hate to see Freenet miss out on fresh talent.

On Thu, Feb 16, 2012 at 1:12 PM, Florent Daigniere <
nextgens at freenetproject.org> wrote:

> Hi!
>
> Should Freenet take part to Google Summer of Code 2012?
>
> Is anyone motivated to mentor candidates?
>
> I'd like to mentor but probably won't do it if I'm the only one.
>
> Florent
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://freenetproject.org/cgi-bin/mailman/listinfo/devl
>
-- next part --
An HTML attachment was scrubbed...
URL: 

Re: [freenet-dev] GSoC 2012

[Zlatin Balevsky]

I'm not up to speed with specifics of the current Freenet codebase but I'd
enjoy mentoring for general java/jvm development best practices.  More so,
I'd hate to see Freenet miss out on fresh talent.

On Thu, Feb 16, 2012 at 1:12 PM, Florent Daigniere 
nextg...@freenetproject.org wrote:

 Hi!

 Should Freenet take part to Google Summer of Code 2012?

 Is anyone motivated to mentor candidates?

 I'd like to mentor but probably won't do it if I'm the only one.

 Florent
 ___
 Devl mailing list
 Devl@freenetproject.org
 http://freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] GSoC 2012

[Zlatin Balevsky]

+1 you might just have a winner

On Thu, Feb 16, 2012 at 1:32 PM, Ian Clarke i...@freenetproject.org wrote:

 I'm not sure whether people would object to this, but I would quite like
 to mentor a student to work on Tahrir under the umbrella of the Freenet
 project.

 Thoughts?  Opinions?  Insults?

 Ian.


 On Thu, Feb 16, 2012 at 12:12 PM, Florent Daigniere 
 nextg...@freenetproject.org wrote:

 Hi!

 Should Freenet take part to Google Summer of Code 2012?

 Is anyone motivated to mentor candidates?

 I'd like to mentor but probably won't do it if I'm the only one.

 Florent
 ___
 Devl mailing list
 Devl@freenetproject.org
 http://freenetproject.org/cgi-bin/mailman/listinfo/devl




 --
 Ian Clarke
 Founder, The Freenet Project
 Email: i...@freenetproject.org

 ___
 Devl mailing list
 Devl@freenetproject.org
 http://freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] some observations on linux priorities & NativeThread

[Zlatin Balevsky]

While chasing a 100% cpu usage bug on latest ubuntu with linux kernel
3.0.0-14, ( https://bugs.freenetproject.org/view.php?id=5336 ) Nextgens and
myself verified that passing -XX:ThreadPriorityPolicy=42 to the jvm makes
thread priorities work even without the NativeThread library.  I'll report
in a few days if I have observed the busyloop again while running this
way.

Thanks,
Zlatin
-- next part --
An HTML attachment was scrubbed...
URL: 

[freenet-dev] some observations on linux priorities NativeThread

[Zlatin Balevsky]

While chasing a 100% cpu usage bug on latest ubuntu with linux kernel
3.0.0-14, ( https://bugs.freenetproject.org/view.php?id=5336 ) Nextgens and
myself verified that passing -XX:ThreadPriorityPolicy=42 to the jvm makes
thread priorities work even without the NativeThread library.  I'll report
in a few days if I have observed the busyloop again while running this
way.

Thanks,
Zlatin
___
Devl mailing list
Devl@freenetproject.org
http://freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-CVS] freenet/src/freenet/node/states/FNP NewDataRequest.java.BlackHole, 1.1, 1.2

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/FNP
In directory sc8-pr-cvs1:/tmp/cvs-serv23667/src/freenet/node/states/FNP

Modified Files:
NewDataRequest.java.BlackHole 
Log Message:
slightly smarter black hole, more work to follow

Index: NewDataRequest.java.BlackHole
===
RCS file: 
/cvsroot/freenet/freenet/src/freenet/node/states/FNP/NewDataRequest.java.BlackHole,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -w -r1.1 -r1.2
--- NewDataRequest.java.BlackHole   29 Nov 2003 04:30:38 -  1.1
+++ NewDataRequest.java.BlackHole   4 Dec 2003 19:49:17 -   1.2
@@ -13,6 +13,12 @@
 
 public class NewDataRequest extends NewRequest {
 
+   //the probability with which to DNF incoming requests.
+   //the current implementation simply stalls queries which
+   //are not dnf'd; it is trivial to make it route them
+   //(just copy the code from the original ;-))
+   final static double pDNF=1.0; 
+
 public NewDataRequest(long id) {
 super(id);
 }
@@ -36,9 +42,11 @@
Core.diagnostics.occurrenceCounting(incomingRequests, 1); 
//send back a DNF immediately, on same thread, without callback
 
+   if (Math.random() = pDNF) {
Message m = new DataNotFound(drmo.id());
n.sendMessageAsync(m, drmo.getSource(), PeerHandler.EXPENDABLE,
   null);
+   }
return null;
 } catch (RequestAbortException e) {
 return e.state;

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-dev] what would YOU do to help NGR?

[Zlatin Balevsky]

Hello users,

here is a small poll to let us know how willing you are to help us get 
NGR working.  Please get in touch with us somehow (directly to amphibian 
@ dyndns.org, irc, this list, the support list, frost, etc.) to let us 
know which of the following best describes your willingness to help ngr:

a) I would run a node without anonimity and let the devs get all info 
they want, and would stick around to assist them

b) I would run a node without anonimity and let the devs get all info 
they want as long as I'm not bothered

c) I would not run a node without anonimity but would let the devs get 
all other info they need

d) I would not run a node without anonimity and would only give 
information as I please

e) I would only file bug reports

f) screw you, you should be glad I even run a node!

Thank you in advance for your participation!



pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Re: Drone nodes was Re: what would YOU do to help NGR?

[Zlatin Balevsky]

Remote access to everything on the web interface, AND LOGS. Remote
setting of logLevel, logLevelDetail and perhaps other options. 
Automatic upgrading, perhaps with remote triggering, or perhaps on a 
timer. Basically we want a couple of hundred 'drone' nodes that we can
experiment on, to try to speed up the debugging cycle, and to enable us
to track what is happening between nodes at the network level as well 
as on our local node

keep in mind that this is a novel and cool idea and for the average 
slashdotter the very fact that it has not been done before may very well 
outweigh the potential risks, i.e. it may win us some new users! :)


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] restoring qr due to bandwith on stable

[Zlatin Balevsky]

stable isn't limiting its outgoing bandwith very well.. I've  set it to 
15k but it spikes over 30 often, and it also has hundreds of connections 
transmitting data, and the number of successfully retrieved keys is very 
small compared to the number of requested keys.  Perhaps we could try 
restoring qr-ing based on bandwith in stable?


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Re: Where now?

[Zlatin Balevsky]

We already know that old stuff is subject to that attack
No, we don't know if classic routing is.  We suspect it might be to some degree, but we are not sure.



pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Re: Where now?

[Zlatin Balevsky]

Zlatin Balevsky wrote:

We already know that old stuff is subject to that attack


No, we don't know if classic routing is.  We suspect it might be to some 
degree, but we are not sure.


Fine, but that's not the point.  The point is to make Joe User happy 
with stable so we don't kill off donations.  Experiments with black 
holes in stable might make Joe mad.

-Martin
IF (-- big if) classical routing is vulnerable to black holes, 
then making it immune would be much easier than making NGR immune, 
because the only strength a black hole has in classical routing is 
that it never QRs.  So all it will take to make classical immune 
is some tweaking of the backoff parameters.

and yes, joe user will be unhappy if we do it, but won't be happy
either if someone else ([RI|MP]AA | script kiddie) does it for us. 



pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] stable net kicking ass

[Zlatin Balevsky]

Ok, yesterday it didn't work - nodes didn't learn about each other, it 
was busy, etc.

Today however, the request success ratio was more than 10% for a brand 
new node (18 out of 124 externally requested keys successful), and frost 
is chugging along (recorded a download speed of 100k at some point). 
And an insert of a small file with htl 21 took just few seconds (hope 
nobody is running a black hole).  And nodes learn about each other all 
of a sudden...

I still think its a good idea to invest some toadtime into polishing it 
before 0.5.3, but so far its kicking ass.


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Errors in formulas

[Zlatin Balevsky]

The very idea of using a formula for making decisions about routing has 
one major flaw and that is the innacuracy of the estimators.  Unless a 
perfect estimator is developed which will give the exact value of a 
given variable, any formula will produce humonguous margin of error. 

As you know, if every variable in the formula was 90% accurate (which is 
extremely optimistic in our case), the accuracy of the formula would be 
at most 90% if only additions and substractions are used and much worse 
if there are other operations.  The accuracy of each estimator in a live 
network is anywhere between 0 and 100%, and the older the last 
measurement is, the bigger the error.  If the formula uses 3 or more 
variables from estimators, if a single of those measurements goes under 
80% accuracy you can dump the result altogether.

To judge different formulas it is not enough to just plug them in and 
see which one produces least error; if that was the case I can tell you 
right now that the formula that uses the least # of variables and only 
additions and substractions will be the best ;).  We need to be able to 
know which estimator tends to produce the biggest error and to have some 
known range for that error.  What if estimator A consistently produced 
10 times more error than estimator B?  Wouln't you want to decrease the 
weight or totally get rid of pA in that case?

The only way to find the error of the various estimators is to know what 
the actual values are.  And the only way to have that knowledge is to 
run the tests in controlled environment - either sim or watchme network.

P.s. personally I'm against using a formula alone to make the most 
important decision a node has to make, but if we're going to do it lets 
at least do it the right way.


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] something we haven't seen for a long time...

[Zlatin Balevsky]

Histogram of requested keys.
This count has nothing to do with keys in your datastore
Dec 1, 2003 1:39:01 AM
keys: 478
scale factor: 0.4383561611175537 (This is used to keep lines  64 characters)
  0 |===
  1 |==
  2 |
  3 |==
  4 |===
  5 |
  6 |=
  7 |==
  8 |==
  9 |
  a |===
  b |===
  c |=
  d |=
  e |
  f |=
peaks (count/mean)
0 -- (0.30125523)
4 -- (4.887029)
b -- (0.53556484)
f -- (0.43514645)
:)))



pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] weird crash with 6348

[Zlatin Balevsky]

After several hours of uptime but no interaction with the user (i.e. 
node was left overnight)  when I decided to check out the web interface 
in the morning fred suddenly hang.  No cpu whatsoever was used, and when 
I tried to get a stack dump I got:

Fatal: Stack size too small. Use 'java -Xss' to increase default stack size.

kernel is 2.4 nptl enabled (fc1) and LD_ASSUME_KERNEL=2.2.5


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] 6348 - sharp specialization

[Zlatin Balevsky]

After starting a brand new node with 6348 the outbound requests become 
sharply specialized in a matter of minutes.  The incoming requests are 
of course in disaray, but hopefully when more nodes start running this 
code we'll finally see the coveted specialization  *coughmergecough*


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] rateless codes

[Zlatin Balevsky]

This idea may be rather unpopular, but too bad... :))


Rateless codes are like FEC except that they support infinite number of 
checkblocks, and the file becomes a stream.  Simple implementation is 
just to wrap the checkblocks from fec in a repeating sequence and insert 
them in a channel like frost boards, etc.  More sophisticated 
implementations don't exist afaik, the theory is at

http://www.scs.cs.nyu.edu/~petar/oncodes.pdf
http://www.scs.cs.nyu.edu/~petar/msdtalk.pdf
and if you have postscript,
http://www.scs.cs.nyu.edu/~petar/msd.ps


I believe this will provide a better performance than fec for large 
files than the current FEC algorithm as it will ensure that as long as 
someone is inserting the file stream it will be retrievable and 
reconstructable.  Of course mass adoption may have disastrous effects on 
the network (but then again the same has been said for the polling 
mechanism frost uses).  It has different uses than the standard FEC, 
most notably a large rare file that is not going to be very popular. 
When the file-stream stops being requested the inserter will effectively 
become a ddos-er, but freenet can handle that ;)  It may be integrated 
with the frost feedback mechanism (i.e. turn the source off when x 
succesful downloads complete)

As soon as I hear back from the author of the paper I will start 
implementing a client which does that.  Forward all hatemail to 
/dev/null :-pp


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Re: rateless codes

[Zlatin Balevsky]

 
This idea may be rather unpopular, but too bad... :))


Rateless codes are like FEC except that they support infinite number of 
checkblocks, and the file becomes a stream.  Simple implementation is 
just to wrap the checkblocks from fec in a repeating sequence and insert 
them in a channel like frost boards, etc.  More sophisticated 
implementations don't exist afaik, the theory is at

http://www.scs.cs.nyu.edu/~petar/oncodes.pdf
http://www.scs.cs.nyu.edu/~petar/msdtalk.pdf

and if you have postscript,
http://www.scs.cs.nyu.edu/~petar/msd.ps



I believe this will provide a better performance than fec for large 
files than the current FEC algorithm as it will ensure that as long as 
someone is inserting the file stream it will be retrievable and 
reconstructable.  Of course mass adoption may have disastrous effects on 
the network (but then again the same has been said for the polling 
mechanism frost uses).  It has different uses than the standard FEC, 
most notably a large rare file that is not going to be very popular. 
When the file-stream stops being requested the inserter will effectively 
become a ddos-er, but freenet can handle that ;)  It may be integrated 
with the frost feedback mechanism (i.e. turn the source off when x 
succesful downloads complete)
 
 
 Why is it better to insert an infinite number of different blocks than
 to insert the original N blocks? If they cease to be reachable, reinsert
 them with skipDS, or pull them through with skipDS, or pull them from
 another node with skipDS. I don't see the advantage.

1. reachability can be measured only from those nodes to which the 
inserter has fcp access; in a properly functioning routing the blocks 
should be reachable from the inserting node most of the time even with 
skipDS.
2.  pulling them after some have ceased to be reachable will only 
refresh them on the path from the inserting node(s) to wherever the 
blocks were stored.

This mechanism is intended for large files that are not likely to be 
popular and ensures that a node which is very distant topologically 
reconstructs the entire file.  As long as the network is not totally 
disjoint, any node will be able to get the entire file exactly because 
the number of chuncks is infinite and they're likely to get routed 
evenly across the network.

If a finite number of chuncks is inserted, and then re-inserted with the 
same CHKs, then if the requesting node cannot reach some of the 
necessary chuncks due to crappy rt it won't be able to reach them ever 
unless the topology of the nodes around the inserter changes (but with 
good routing we expect the same key to go to similar place no matter how 
dynamic the network is)

Having infinite number of chucks ends up being a broadcast of sorts; 
however freenet is designed to handle exactly this kind of ddos.  As 
long as the requesting node has a chance greater than 0 of getting a 
single chunk from the file stream, its guaranteed to get the entire file 
  as long as it is being seeded.  (infinity * anything over 0 = infinity)

 
As soon as I hear back from the author of the paper I will start 
implementing a client which does that.  Forward all hatemail to 
/dev/null :-pp
 
 
 Hehe, we haven't got to that stage yet, first we have to argue about
 it, then it degenerates into hatemail. :)

and what happens after its implemented and totally screws everybody's 
freesite browsing experience?  death threats ? ;



pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Re: QueryReject patterns and NGRouting

[Zlatin Balevsky]

One radical solution:
Remove the code to reject queries when the bandwidth limit is exceeded!
which returns us in the state 5010-5018 where the node has accepted 
umpteen transfers, each going at snail speed.  Making that prevalent 
accross the network will be catastrophic.

Another possibility is to go even further back in 5007-5010 where the 
node would still accept everything but wouldn't round-robin to the 
senders which effectively meant the queries were served in a fifo queue.

NGRouting can figure out when nodes are slow due to long term overload 
a lot more easily and less alchemically than it can deal with query
rejections. Or so the theory goes. Am I smoking crack here?

So far NGRouting hasn't demonstrated ability to figure out its head from 
its arse (pardon my french).  Why not remove all code but NGRouting and 
let it figure out everything on its own, including world hunger and 
price of fish /sarcasm


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] polling sharing estimator data

[Zlatin Balevsky]

This is an extention to the idea of keeping estimator data in the .refs 
which aims to bring more consistent worldview between the nodes.  It 
works best if there is trust implemented between nodes, but will also 
bring some limited results without trust.

At random interval the node asks a random subset of the nodes in the rt 
for the estimator data they have of another node/subset of nodes (the 
two subsets can overlap).  After that the node merges their results 
with its own estimator in a manner that would be as cancer-proof as 
possible in a network without trust.  (that means removing outliers, 
averaging, etc.)

Drawbacks:
* increased network traffic.  The timing will have to be chosen not to 
create a cascade effect.
* powerful attacker with many nodes totally screws up our estimates, or 
even worse - provides such data which would make the requesting node 
favor his cancer nodes.

Benefits:
* Nodes have a more consistent picture of each other's strengths and 
weaknesses.
* Newer nodes find their place faster
* The network does not become any more static
* Since topological neighbors know similar things about each other, each 
request from within the neighborhood will go directly to the best node.

Neighborhood is not rigidly defined; if a network map is drawn, there 
will be no rigid boundaries between shared estimator information; rather 
it will flow gradually, providing optimal routing paths.  (ok this 
sounds psycho but its much easier to visualise than explain - will try 
and get a pic/graph soon)


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] packet-level compression

[Zlatin Balevsky]

We are often sending 10s of messages in a single packet; most of these 
messages are very similar (i.e. QRs).  While each message in itself 
isn't well compressible because of the nature of the strings it 
contains, a packet with 20 messages could be compressed much better, 
resulting in reduced bandwith usage for high-traffic nodes.

This compression will of course happen before encryption and will be 
completely transparent to the upper layers of the code.


pgp0.pgp
Description: PGP signature
___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-CVS] freenet/src/freenet/node/states/request Pending.java, 1.80, 1.81

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/request
In directory sc8-pr-cvs1:/tmp/cvs-serv15769/src/freenet/node/states/request

Modified Files:
Pending.java 
Log Message:
get rid of some yellowies

Index: Pending.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/request/Pending.java,v
retrieving revision 1.80
retrieving revision 1.81
diff -u -w -r1.80 -r1.81
--- Pending.java31 Oct 2003 19:21:17 -  1.80
+++ Pending.java31 Oct 2003 19:26:52 -  1.81
@@ -271,7 +271,7 @@
receivedTime = 0;
} else {
long x = gotTime - receivedTime;
-   n.diagnostics.occurrenceContinuous(preRoutingTime, x);
+   Core.diagnostics.occurrenceContinuous(preRoutingTime, x);
if(x  500  logDEBUG)
Core.logger.log(this, Got long preRoutingTime (+x+
 = +gotTime+-+receivedTime+
@@ -279,7 +279,7 @@
Logger.DEBUG);
}
long regotTime = System.currentTimeMillis();
-   n.diagnostics.occurrenceContinuous(regotTime, regotTime - gotTime);
+   Core.diagnostics.occurrenceContinuous(regotTime, regotTime - gotTime);
 // is this the first time?
 if (routes == null) {
if(logDEBUG)
@@ -302,7 +302,7 @@
searchKey.getExpectedTransmissionLength(),
false, isInsert());
gotRouteTime = System.currentTimeMillis();
-   n.diagnostics.occurrenceContinuous(getRouteTime, gotRouteTime - 
searchDataTime);
+   Core.diagnostics.occurrenceContinuous(getRouteTime, gotRouteTime - 
searchDataTime);
if((gotRouteTime - searchDataTime)  1000)
if(logMINOR)
Core.logger.log(this, getting routing object took +
@@ -324,7 +324,7 @@
ri.initException, Logger.DEBUG);
}

-n.diagnostics.occurrenceBinomial(restartedRequestAccepted,
+Core.diagnostics.occurrenceBinomial(restartedRequestAccepted,
  1, accepted ? 1 : 0);

 
@@ -547,7 +547,7 @@
  Logger.DEBUG);
long stillInSendOnTime = System.currentTimeMillis();
if(isFirst  attemptCount == 0) {
-   n.diagnostics.occurrenceContinuous(stillInSendOnTime, 
+   Core.diagnostics.occurrenceContinuous(stillInSendOnTime, 
   stillInSendOnTime - 
   gotRouteTime);
if(logDEBUG) Core.logger.log(this, stillInSendOnTime=+
@@ -574,7 +574,7 @@
  Logger.DEBUG);
long stillStillInSendOnTime = System.currentTimeMillis();
if(isFirst  attemptCount == 0) {
-   n.diagnostics.occurrenceContinuous(stillStillInSendOnTime, 
+   Core.diagnostics.occurrenceContinuous(stillStillInSendOnTime, 
   stillStillInSendOnTime - 
   stillInSendOnTime);
}
@@ -609,16 +609,16 @@
if (attemptCount == 0  receivedTime  0 
receivedTime  routedTime) { // sanity
long rt = routedTime - receivedTime;
-   n.diagnostics.occurrenceContinuous(routingTime, rt);
+   Core.diagnostics.occurrenceContinuous(routingTime, rt);
if(rt  1000  logDEBUG)
Core.logger.log(this, routingTime +rt+ for +
 this, new Exception(debug), 
 Logger.DEBUG);
-   n.diagnostics.occurrenceContinuous(subRoutingTime,
+   Core.diagnostics.occurrenceContinuous(subRoutingTime,
   routedTime - 
   stillStillInSendOnTime);
if(searchDataRoutingTime  0)
-   n.diagnostics.occurrenceContinuous(searchDataRoutingTime, 
+   Core.diagnostics.occurrenceContinuous(searchDataRoutingTime, 
   searchDataRoutingTime);
receivedTime = -2; // make sure we aren't called twice
}
@@ -859,7 +859,7 @@
doc = null;
thrownTime = System.currentTimeMillis();
if(origPeer != null)
-   n.diagnostics.occurrenceContinuous(sendingReplyHTL, 
+   Core.diagnostics.occurrenceContinuous(sendingReplyHTL, 
   hopsToLive);
throw new 

[freenet-CVS] freenet/src/freenet/node/states/request AwaitingStoreData.java, 1.27, 1.28 SendingReply.java, 1.25, 1.26 SendFinished.java, 1.12, 1.13 ReceivingInsert.java, 1.25, 1.26 RequestDone.java, 1.7, 1.8

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/request
In directory sc8-pr-cvs1:/tmp/cvs-serv16347/src/freenet/node/states/request

Modified Files:
AwaitingStoreData.java SendingReply.java SendFinished.java 
ReceivingInsert.java RequestDone.java 
Log Message:
get rid of some yellowies

Index: AwaitingStoreData.java
===
RCS file: 
/cvsroot/freenet/freenet/src/freenet/node/states/request/AwaitingStoreData.java,v
retrieving revision 1.27
retrieving revision 1.28
diff -u -w -r1.27 -r1.28
--- AwaitingStoreData.java  31 Oct 2003 19:21:17 -  1.27
+++ AwaitingStoreData.java  31 Oct 2003 19:29:21 -  1.28
@@ -102,7 +102,7 @@
 }
 NodeReference nr = (sd == null ? null : sd.dataSource());
 if (nr != null  !nr.checkAddresses(n.transports)) {
-if (n.logger.shouldLog(Logger.MINOR))
+if (Core.logger.shouldLog(Logger.MINOR))
 Core.logger.log(this,
  Not referencing because addresses 
  + of DataSource wrong:  
@@ -111,31 +111,31 @@
 }
 long rate = -1;
if(sd != null)
-   n.diagnostics.occurrenceContinuous(incomingHopsSinceReset, 
+   Core.diagnostics.occurrenceContinuous(incomingHopsSinceReset, 
   sd.hopsSinceReset());
 if (n.shouldReference(nr, sd)) {
-   n.diagnostics.occurrenceCounting(prefAccepted, 1);
+   Core.diagnostics.occurrenceCounting(prefAccepted, 1);
Core.logger.log(this, PCaching: Referencing +searchKey, Logger.DEBUG);
 n.reference(searchKey, nr);
 rate = sd.requestsPerHour();
 } else if(nr != null) {
-   n.diagnostics.occurrenceCounting(prefRejected, 1);
+   Core.diagnostics.occurrenceCounting(prefRejected, 1);
Core.logger.log(this, PCaching: Not Referencing +searchKey,
 Logger.DEBUG);
}
if(n.shouldCache(sd)) {
Core.logger.log(this, PCaching: Keeping +searchKey, Logger.DEBUG);
-   n.diagnostics.occurrenceCounting(pcacheAccepted, 1);
+   Core.diagnostics.occurrenceCounting(pcacheAccepted, 1);
// Keep it
} else {
// Try to delete it
Core.logger.log(this, PCaching: Removing +searchKey, Logger.DEBUG);
-   n.diagnostics.occurrenceCounting(pcacheRejected, 1);
+   Core.diagnostics.occurrenceCounting(pcacheRejected, 1);
if(!n.ds.demote(searchKey))
-   n.diagnostics.occurrenceCounting(pcacheFailedDelete, 1);
+   Core.diagnostics.occurrenceCounting(pcacheFailedDelete, 1);
}
 try {
-   n.diagnostics.occurrenceCounting(storeDataAwaitingStoreData,1);
+   Core.diagnostics.occurrenceCounting(storeDataAwaitingStoreData,1);
 ft.storeData(n, nr, rate, sd == null ? 0 : sd.hopsSinceReset(),
 new RequestSendCallback(StoreData, n, this));
 } catch (CommunicationException e) {
@@ -146,16 +146,16 @@
Core.logger.log(this, Logging external success on success*Distribution +
 from +origPeer+ on +this, Logger.DEBUG);
if (isInsert) {
-   if (n.successInsertDistribution != null) {
-n.successInsertDistribution.add(searchKey.getVal());
+   if (Node.successInsertDistribution != null) {
+Node.successInsertDistribution.add(searchKey.getVal());
}
} else {
-   if (n.successDataDistribution != null) {
-n.successDataDistribution.add(searchKey.getVal());
+   if (Node.successDataDistribution != null) {
+Node.successDataDistribution.add(searchKey.getVal());
}
}
-   if (n.inboundRequests != null)
-   n.inboundRequests.incActive(origPeer.getAddress().toString());
+   if (Node.inboundRequests != null)
+   Node.inboundRequests.incActive(origPeer.getAddress().toString());
}
 }
 }

Index: SendingReply.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/request/SendingReply.java,v
retrieving revision 1.25
retrieving revision 1.26
diff -u -w -r1.25 -r1.26
--- SendingReply.java   31 Oct 2003 19:21:17 -  1.25
+++ SendingReply.java   31 Oct 2003 19:29:21 -  1.26
@@ -75,11 +75,11 @@
Core.logger.log(this, Logging external success on +
 success*Distribution from +origPeer+ on +
 this, Logger.DEBUG);
-   if (n.successDataDistribution != null) {
-n.successDataDistribution.add(searchKey.getVal());
+

[freenet-CVS] freenet/src/freenet/node/states/FNP NewDataRequest.java, 1.7, 1.8 NewInsertRequest.java, 1.7, 1.8

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/FNP
In directory sc8-pr-cvs1:/tmp/cvs-serv16939/src/freenet/node/states/FNP

Modified Files:
NewDataRequest.java NewInsertRequest.java 
Log Message:
get rid of some yellowies

Index: NewDataRequest.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/FNP/NewDataRequest.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -u -w -r1.7 -r1.8
--- NewDataRequest.java 17 Oct 2003 23:02:00 -  1.7
+++ NewDataRequest.java 31 Oct 2003 19:32:17 -  1.8
@@ -33,8 +33,8 @@
 genReceived(n, drmo, true);
// Run the request even if we cannot send the accepted back
// This is not an attack IMHO because of the overhead of connecting
-if (n.requestDataDistribution != null) {
-n.requestDataDistribution.add(drmo.searchKey.getVal());
+if (Core.requestDataDistribution != null) {
+Core.requestDataDistribution.add(drmo.searchKey.getVal());
 }
 
 FeedbackToken ft= new FNPFeedbackToken(id, origRec, 
@@ -43,7 +43,7 @@
drmo.getReceivedTime());
 Pending p = new DataPending(id, (int) drmo.hopsToLive,
 drmo.searchKey, origRec, ft, ri);
-   n.diagnostics.occurrenceCounting(incomingRequests, 1);
+   Core.diagnostics.occurrenceCounting(incomingRequests, 1);
 return p.received(n, ri);
 } catch (RequestAbortException e) {
 return e.state;

Index: NewInsertRequest.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/FNP/NewInsertRequest.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -u -w -r1.7 -r1.8
--- NewInsertRequest.java   17 Oct 2003 23:02:00 -  1.7
+++ NewInsertRequest.java   31 Oct 2003 19:32:17 -  1.8
@@ -32,8 +32,8 @@
 try {
 genReceived(n, irmo, false); 
// no point continuing if can't send Accepted back
-if (n.requestInsertDistribution != null) {
-n.requestInsertDistribution.add(irmo.searchKey.getVal());
+if (Core.requestInsertDistribution != null) {
+Core.requestInsertDistribution.add(irmo.searchKey.getVal());
 }
 
 FeedbackToken ft= new FNPFeedbackToken(id, origRec, 
@@ -42,7 +42,7 @@
irmo.getReceivedTime());
 Pending p = new InsertPending(id, (int) irmo.hopsToLive,
   irmo.searchKey, origRec, ft, ri);
-   n.diagnostics.occurrenceCounting(incomingInserts, 1);
+   Core.diagnostics.occurrenceCounting(incomingInserts, 1);
 return p.received(n, ri);
 } catch (RequestAbortException rae) {
 return rae.state;

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/support FileLoggerHook.java, 1.36, 1.37

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/support
In directory sc8-pr-cvs1:/tmp/cvs-serv16635/src/freenet/support

Modified Files:
FileLoggerHook.java 
Log Message:
get rid of some yellowies

Index: FileLoggerHook.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/support/FileLoggerHook.java,v
retrieving revision 1.36
retrieving revision 1.37
diff -u -w -r1.36 -r1.37
--- FileLoggerHook.java 30 Oct 2003 01:34:01 -  1.36
+++ FileLoggerHook.java 31 Oct 2003 19:30:38 -  1.37
@@ -117,9 +117,9 @@
 }
 
 protected String getHourLogName(Calendar c) {
-   return baseFilename + - + c.get(c.YEAR) + - + pad2(c.get(c.MONTH)) +
-   - + pad2(c.get(c.DAY_OF_MONTH)) + - + pad2(c.get(c.HOUR_OF_DAY)) +
-   (INTERVAL == c.MINUTE ? (- + pad2(c.get(c.MINUTE))) : );
+   return baseFilename + - + c.get(Calendar.YEAR) + - + 
pad2(c.get(Calendar.MONTH)) +
+   - + pad2(c.get(Calendar.DAY_OF_MONTH)) + - + 
pad2(c.get(Calendar.HOUR_OF_DAY)) +
+   (INTERVAL == Calendar.MINUTE ? (- + pad2(c.get(Calendar.MINUTE))) : );
 }
 
 protected String pad2(int x) {

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node/states/FCP NewClientGet.java, 1.10, 1.11 ReturnDiagnostics.java, 1.5, 1.6 NewClientPut.java, 1.14, 1.15

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/FCP
In directory sc8-pr-cvs1:/tmp/cvs-serv17214/src/freenet/node/states/FCP

Modified Files:
NewClientGet.java ReturnDiagnostics.java NewClientPut.java 
Log Message:
get rid of some yellowies

Index: NewClientGet.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/FCP/NewClientGet.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -u -w -r1.10 -r1.11
--- NewClientGet.java   31 Oct 2003 18:13:15 -  1.10
+++ NewClientGet.java   31 Oct 2003 19:33:08 -  1.11
@@ -42,7 +42,7 @@
 throw new BadStateException(expecting ClientGet);
 }
 
-n.diagnostics.occurrenceCounting(inboundClientRequests, 1);
+Core.diagnostics.occurrenceCounting(inboundClientRequests, 1);
 
 ClientGet cgmo = (ClientGet) mo;
 

Index: ReturnDiagnostics.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/FCP/ReturnDiagnostics.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -u -w -r1.5 -r1.6
--- ReturnDiagnostics.java  31 Oct 2003 19:21:19 -  1.5
+++ ReturnDiagnostics.java  31 Oct 2003 19:33:08 -  1.6
@@ -30,7 +30,7 @@
 ArrayBucket bucket = new ArrayBucket();
 PrintWriter bout = 
 new PrintWriter(bucket.getOutputStream());
-n.diagnostics.writeVars(bout, new FieldSetFormat());
+Core.diagnostics.writeVars(bout, new FieldSetFormat());
 bout.close();
 
 TrailerWriter tw = 

Index: NewClientPut.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/FCP/NewClientPut.java,v
retrieving revision 1.14
retrieving revision 1.15
diff -u -w -r1.14 -r1.15
--- NewClientPut.java   31 Oct 2003 19:21:19 -  1.14
+++ NewClientPut.java   31 Oct 2003 19:33:08 -  1.15
@@ -54,7 +54,7 @@
 throw new BadStateException(expecting ClientPut);
 }
 
-n.diagnostics.occurrenceCounting(inboundClientRequests, 1);
+Core.diagnostics.occurrenceCounting(inboundClientRequests, 1);
 
 ClientPut cpmo = (ClientPut) mo;
 

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node/states/data ReceiveData.java, 1.24, 1.25 SendData.java, 1.32, 1.33

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/data
In directory sc8-pr-cvs1:/tmp/cvs-serv17682/src/freenet/node/states/data

Modified Files:
ReceiveData.java SendData.java 
Log Message:
get rid of some yellowies

Index: ReceiveData.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/data/ReceiveData.java,v
retrieving revision 1.24
retrieving revision 1.25
diff -u -w -r1.24 -r1.25
--- ReceiveData.java31 Oct 2003 19:21:21 -  1.24
+++ ReceiveData.java31 Oct 2003 19:35:40 -  1.25
@@ -103,7 +103,7 @@
 byte[] buffer = new byte[Core.blockSize];
 long moved = 0;
 
-   boolean logDEBUG = n.logger.shouldLog(Logger.DEBUG,this);
+   boolean logDEBUG = Core.logger.shouldLog(Logger.DEBUG,this);

 try {
 while (moved  length) {
@@ -170,7 +170,7 @@
result = Presentation.CB_CACHE_FAILED; // umm, kinda
 } finally {

-n.diagnostics.occurrenceBinomial(receivedData, 1, 
+Core.diagnostics.occurrenceBinomial(receivedData, 1, 
  result == Presentation.CB_OK ?
  1 : 0);
 

Index: SendData.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/data/SendData.java,v
retrieving revision 1.32
retrieving revision 1.33
diff -u -w -r1.32 -r1.33
--- SendData.java   31 Oct 2003 19:21:21 -  1.32
+++ SendData.java   31 Oct 2003 19:35:40 -  1.33
@@ -67,7 +67,7 @@
 this.partSize = partSize;
this.n= n;
myTWCM = new TrailerWriteCallbackMessage(id, n, this);
-   logDEBUG = n.logger.shouldLog(Logger.DEBUG,this);
+   logDEBUG = Core.logger.shouldLog(Logger.DEBUG,this);
if(logDEBUG)
Core.logger.log(this, Creating SendData(+this+), Logger.DEBUG);
 }
@@ -165,7 +165,7 @@
m = 0;
}
 
-   logDEBUG = n.logger.shouldLog(Logger.DEBUG, this);
+   logDEBUG = Core.logger.shouldLog(Logger.DEBUG, this);
 
if (isTWCM) {
waitingForWriteNotify = false;
@@ -271,7 +271,7 @@
result = Presentation.CB_CACHE_FAILED; // well, sorta
}

-   n.diagnostics.occurrenceBinomial(sentData, 1, 0);
+   Core.diagnostics.occurrenceBinomial(sentData, 1, 0);
try {
in.close();
closedIn = true;
@@ -385,7 +385,7 @@
}
if (moved == length)
result = Presentation.CB_OK;
-   n.diagnostics.occurrenceBinomial(sentData, 1, result == 
Presentation.CB_OK ? 1 : 0);
+   Core.diagnostics.occurrenceBinomial(sentData, 1, result == 
Presentation.CB_OK ? 1 : 0);
if (!silent)
n.schedule(new DataSent(this));
return null;

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node/states/announcing CompleteAnnouncement.java, 1.11, 1.12 Announcing.java, 1.37, 1.38

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/announcing
In directory sc8-pr-cvs1:/tmp/cvs-serv17793/src/freenet/node/states/announcing

Modified Files:
CompleteAnnouncement.java Announcing.java 
Log Message:
get rid of some yellowies

Index: CompleteAnnouncement.java
===
RCS file: 
/cvsroot/freenet/freenet/src/freenet/node/states/announcing/CompleteAnnouncement.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -u -w -r1.11 -r1.12
--- CompleteAnnouncement.java   31 Oct 2003 19:21:21 -  1.11
+++ CompleteAnnouncement.java   31 Oct 2003 19:36:14 -  1.12
@@ -69,7 +69,7 @@
 
 nc.cancel();
 
-n.diagnostics.occurrenceCounting(announcedTo, hopsToLive);
+Core.diagnostics.occurrenceCounting(announcedTo, hopsToLive);
 signalSuccess(n);
 
 try {

Index: Announcing.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/states/announcing/Announcing.java,v
retrieving revision 1.37
retrieving revision 1.38
diff -u -w -r1.37 -r1.38
--- Announcing.java 31 Oct 2003 19:21:21 -  1.37
+++ Announcing.java 31 Oct 2003 19:36:15 -  1.38
@@ -180,11 +180,11 @@
 }
 } else if (mo instanceof ScheduleAnnouncing) {
 try {
-double traffic = n.diagnostics.getValue(localQueryTraffic, 
+double traffic = Core.diagnostics.getValue(localQueryTraffic, 
 Diagnostics.HOUR, 
 Diagnostics.NUMBER_OF_EVENTS);
 
-double connections = n.diagnostics.getValue(connectingTime,
+double connections = Core.diagnostics.getValue(connectingTime,
Diagnostics.HOUR,

Diagnostics.NUMBER_OF_EVENTS);
 

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node/states/announcement ReplyPending.java, 1.13, 1.14 NewAnnouncement.java, 1.17, 1.18

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node/states/announcement
In directory sc8-pr-cvs1:/tmp/cvs-serv17945/src/freenet/node/states/announcement

Modified Files:
ReplyPending.java NewAnnouncement.java 
Log Message:
get rid of some yellowies

Index: ReplyPending.java
===
RCS file: 
/cvsroot/freenet/freenet/src/freenet/node/states/announcement/ReplyPending.java,v
retrieving revision 1.13
retrieving revision 1.14
diff -u -w -r1.13 -r1.14
--- ReplyPending.java   31 Oct 2003 19:21:16 -  1.13
+++ ReplyPending.java   31 Oct 2003 19:37:16 -  1.14
@@ -120,7 +120,7 @@
 throws BadStateException {
 checkReply(qr);
 
-   if(n.logger.shouldLog(Logger.DEBUG,this))
+   if(Core.logger.shouldLog(Logger.DEBUG,this))
Core.logger.log(this, Received QueryRejected on +Fields.longToHex(id)+
  (+qr.hopsToLive+,+qr.reason+), Logger.DEBUG);
 

Index: NewAnnouncement.java
===
RCS file: 
/cvsroot/freenet/freenet/src/freenet/node/states/announcement/NewAnnouncement.java,v
retrieving revision 1.17
retrieving revision 1.18
diff -u -w -r1.17 -r1.18
--- NewAnnouncement.java31 Oct 2003 19:21:16 -  1.17
+++ NewAnnouncement.java31 Oct 2003 19:37:16 -  1.18
@@ -77,7 +77,7 @@
 }
 
 if (!announcee.checkAddresses(n.transports)) {
-if (n.logger.shouldLog(Logger.MINOR))
+if (Core.logger.shouldLog(Logger.MINOR))
 Core.logger.log(this,
  Rejecting announcement because addresses 
  + of announcee wrong:  
@@ -89,7 +89,7 @@
 
 }
 if (n.rt.references(announcee.getIdentity())) {
-   if(n.logger.shouldLog(Logger.DEBUG,this))
+   if(Core.logger.shouldLog(Logger.DEBUG,this))
Core.logger.log(this, Previously knew Announcee, rejecting.,
 Logger.DEBUG);
 n.sendMessage(new AnnouncementFailed(id, 
AnnouncementFailed.KNOWN_ANNOUNCEE),
@@ -97,7 +97,7 @@

 return null;
 } else if ((depth + hopsToLive)  Node.maxHopsToLive) {
-   if(n.logger.shouldLog(Logger.DEBUG,this))
+   if(Core.logger.shouldLog(Logger.DEBUG,this))
Core.logger.log(this, Too high HTL on Announcement, rejecting.,
 Logger.DEBUG);
 n.sendMessage(
@@ -176,7 +176,7 @@
 routed++;

 while (routed  MAX_ROUTING_TIMES) { // routed does not change in loop
-   if(n.logger.shouldLog(Logger.DEBUG,this))
+   if(Core.logger.shouldLog(Logger.DEBUG,this))
Core.logger.log(this, Trying to route +Fields.longToHex(id)+
  - iteration +routed+ of +MAX_ROUTING_TIMES,
 Logger.DEBUG);

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node SmartFailureTable.java, NONE, 1.1

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node
In directory sc8-pr-cvs1:/tmp/cvs-serv23686/src/freenet/node

Added Files:
SmartFailureTable.java 
Log Message:
initial commit

--- NEW FILE: SmartFailureTable.java ---
package freenet.node;

import freenet.Core;
import freenet.Key;
import freenet.support.Checkpointed;
import freenet.support.Heap;
import freenet.support.Heap.Element;
import freenet.support.sort.ArraySorter;
import freenet.support.sort.QuickSorter;


import java.util.Hashtable;
import java.util.Date;

import java.util.Enumeration;
import java.util.Random;

import java.io.PrintWriter;

/**
 *  Keeps track of more keys than the standart failure table as discussed
 *  on devl.  (yeah I know such description is lame --zab)
 */
public class SmartFailureTable extends FailureTable {
final int treshold;

public SmartFailureTable(int size, int treshold){
super(size,-1); //keys don't expire, do they?
this.treshold = treshold;
}

public synchronized void checkpoint() {
//override and do nothing, keys don't expire
}
}
___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node SmartFailureTable.java, 1.1, 1.2 FailureTable.java, 1.11, 1.12

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node
In directory sc8-pr-cvs1:/tmp/cvs-serv25916/src/freenet/node

Modified Files:
SmartFailureTable.java FailureTable.java 
Log Message:
initial commit

Index: SmartFailureTable.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/SmartFailureTable.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -w -r1.1 -r1.2
--- SmartFailureTable.java  28 Oct 2003 17:25:40 -  1.1
+++ SmartFailureTable.java  28 Oct 2003 17:35:04 -  1.2
@@ -9,11 +9,7 @@
 import freenet.support.sort.QuickSorter;
 
 
-import java.util.Hashtable;
-import java.util.Date;
-
-import java.util.Enumeration;
-import java.util.Random;
+import java.util.*;
 
 import java.io.PrintWriter;
 
@@ -24,6 +20,11 @@
 public class SmartFailureTable extends FailureTable {
final int treshold;

+   /**
+* reuse the constructor
+* @param treshold - the number of hits before the key
+* is marked irrelevant to pDNF calculation
+*/
public SmartFailureTable(int size, int treshold){
super(size,-1); //keys don't expire, do they?
this.treshold = treshold;
@@ -32,4 +33,16 @@
public synchronized void checkpoint() {
//override and do nothing, keys don't expire
}
+   
+   //contains a list of FailureEntries, doesn't extend it, aggregates it instead.
+   private class ExtendedFailureEntry {
+   Key key;
+   LinkedList history;
+   public ExtendedFailureEntry(Key key, int hopsToLive, long time) {
+   this.key = key;
+   history = new LinkedList();
+   history.add(new FailureEntry(key, hopsToLive, time));
+   }
+   }
+   
 }

Index: FailureTable.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/FailureTable.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -u -w -r1.11 -r1.12
--- FailureTable.java   1 Jan 2003 09:54:36 -   1.11
+++ FailureTable.java   28 Oct 2003 17:35:04 -  1.12
@@ -196,7 +196,7 @@
 }
 
 
-private class FailureEntry extends Element {
+protected class FailureEntry extends Element {
 private int hopsToLive;
 private Key key;
 private long time;

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-CVS] freenet/src/freenet/node SmartFailureTable.java, 1.2, 1.3 FailureTable.java, 1.12, 1.13

[Zlatin Balevsky]

Update of /cvsroot/freenet/freenet/src/freenet/node
In directory sc8-pr-cvs1:/tmp/cvs-serv28319/src/freenet/node

Modified Files:
SmartFailureTable.java FailureTable.java 
Log Message:
have fun toad :)

Index: SmartFailureTable.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/SmartFailureTable.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -w -r1.2 -r1.3
--- SmartFailureTable.java  28 Oct 2003 17:35:04 -  1.2
+++ SmartFailureTable.java  28 Oct 2003 17:48:37 -  1.3
@@ -31,6 +31,7 @@
}

public synchronized void checkpoint() {
+lastCp = System.currentTimeMillis();
//override and do nothing, keys don't expire
}

@@ -43,6 +44,31 @@
history = new LinkedList();
history.add(new FailureEntry(key, hopsToLive, time));
}
+   public FailureEntry getOldestEntry() {
+   return (FailureEntry) history.getFirst();
}

+   public FailureEntry getNewestEntry() {
+   return (FailureEntry) history.getLast();
+   }
+   
+   public void update(int hopsToLive, long time) {
+   history.addLast(new FailureEntry(key,hopsToLive,time));
+   }
+   }
+   
+   /**
+* override...
+*/
+   public synchronized void failedToFind(Key k, int hopsToLive, long time) {
+   ExtendedFailureEntry efe = (ExtendedFailureEntry) failedKeys.get(k);
+   if (efe==null){
+   efe = new ExtendedFailureEntry(k, hopsToLive, time);
+   failedKeys.put(k,efe);
+   } else 
+   if (hopsToLive = efe.getNewestEntry().hopsToLive)  //FIXME:  or =???
+   efe.update(hopsToLive,time);
+   
+   //FINISHUP: decide if the heap is needed.
+   }
 }

Index: FailureTable.java
===
RCS file: /cvsroot/freenet/freenet/src/freenet/node/FailureTable.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -u -w -r1.12 -r1.13
--- FailureTable.java   28 Oct 2003 17:35:04 -  1.12
+++ FailureTable.java   28 Oct 2003 17:48:37 -  1.13
@@ -54,12 +54,12 @@
 //  } 
 //  }
 
-private int maxSize;
-private long maxMillis;
-private long lastCp = 0;
+protected int maxSize;
+protected long maxMillis;
+protected long lastCp = 0;
 
-private Hashtable failedKeys;
-private Heap queue;
+protected Hashtable failedKeys;
+protected Heap queue;
 
 /**
  * Creates a new.
@@ -197,7 +197,7 @@
 
 
 protected class FailureEntry extends Element {
-private int hopsToLive;
+public int hopsToLive;  //screw getters...
 private Key key;
 private long time;
 

___
cvs mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/cvs

[freenet-dev] xsl in fproxy

[Zlatin Balevsky]

Can anyone familiar with xsl list the differences between filtering html 
for anonimity-compromising content and filtering xsl transformations?  
If an xml file is filtered agains the current rules (no off-freenet 
links, no actions) and its corresponding xsl is made sure not to contain 
any such transformations will there be any additional issues freenet 
users should be concerned about?

___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] null stream from buffer is back! ;-)

[Zlatin Balevsky]

To all fans of this eternal error message, here's some more:  (build 6273)

Oct 22, 2003 9:55:17 PM (freenet.node.Node, QThread-174, ERROR): Error 
while receiving message freenet.Message: Ac
cepted @[EMAIL PROTECTED] for tcp/connection: 
55930209.210.55.3:11902,freenet.transport.tcpConnec
[EMAIL PROTECTED],[EMAIL PROTECTED], sending null:135 for 
[EMAIL PROTECTED] (DSA(eb02 aaf1 f4fa
0965 4cbf  f1a2 0d57 ffa3 9687 3f08),tcp/209.210.55.3:11902, sessions=1, 
presentations=1, ID=DSA(eb02 aaf1 f4fa 09
65 4cbf  f1a2 0d57 ffa3 9687 3f08)): outbound attempts=20:6/26 @ 
aacd70e1cb5b0d2b in state freenet.node.states.req
uest.TransferInsertPending: 
key=0026c303abc0581c39bee469953029f35aebf515120302, hopsToLive=10, 
id=aacd70e1cb5b0d2b
, [EMAIL PROTECTED] 
(0026c303abc0581c39bee469953029f35aebf515120302,insert),ft=freenet.node
[EMAIL PROTECTED]@1066874117179, 
routedTime=1066874116934, replyTime=-1, outwardSender=null, no
t-approved, insertReplyTime=-1: java.lang.IllegalStateException: null 
stream from buffer freenet.fs.dir.NativeFSDi
[EMAIL PROTECTED]:null
java.lang.IllegalStateException: null stream from buffer 
[EMAIL PROTECTED]
7e78:null
   at 
freenet.node.ds.FSDataStoreElement$KeyInputStreamImpl.init(FSDataStoreElement.java:238)
   at 
freenet.node.ds.FSDataStoreElement.getKeyInputStream(FSDataStoreElement.java:47)
   at 
freenet.node.ds.FSDataStoreElement$KeyOutputStreamImpl.getKeyInputStream(FSDataStoreElement.java:138)
   at 
freenet.node.states.data.ReceiveData.getKeyInputStream(ReceiveData.java:69)
   at 
freenet.node.states.request.InsertPending.relayInsert(InsertPending.java:345)
   at 
freenet.node.states.request.TransferInsertPending.receivedMessage(TransferInsertPending.java:213)
   at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:324)
   at freenet.node.State.received(State.java:126)
   at freenet.node.StateChain.received(StateChain.java:195)
   at freenet.node.StateChain.received(StateChain.java:71)
   at 
freenet.node.StandardMessageHandler$Ticket.run(StandardMessageHandler.java:234)
   at 
freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.java:172)
   at 
freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler.java:124)
   at 
freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:72)
   at freenet.Ticker$Event.run(Ticker.java:323)
   at 
freenet.thread.QThreadFactory$QThread.run(QThreadFactory.java:234)

and other fun messages on loglevel Normal:

Oct 22, 2003 9:55:25 PM (freenet.node.rt.NGRouting, Finalizer, NORMAL): 
Did not terminate freenet.node.rt.NGRoutin
...skipping...
ata @ 6f6c3256e8f67a72: 6f6c3256e8f67a72/88f7301665083cf6: 
[EMAIL PROTECTED],
in=Key: a5a60dcb9db9c72d9863926fab7f686cc64f7730140302 Buffer: 
freenet.fs.dir.NativeFSDirectory$ExternalNativeBuff
[EMAIL PROTECTED]:null New: true ( 106496 of 1049900 read), moved=98304/1049900, 
partSize=16384,result=1,lastPacketLength=
8192,sentPadding=8192/126)

and of course tons of:

Oct 22, 2003 10:19:42 PM (freenet.node.states.request.TransferReply, 
QThread-327, NORMAL): Upstream node connectio
n died for freenet.node.states.request.TransferReply: 
key=3ae9be8c71fbb199ea49e6087c5d0c17ee9daede140302, hopsToLi
ve=12, id=680e77db5dfb8cd4, [EMAIL PROTECTED] 
(3ae9be8c71fbb199ea49e6087c5d0c17ee9daede1403
02,request),ft=freenet.client.InternalClient$InternalGetToken:[EMAIL PROTECTED],key=free
net:[EMAIL PROTECTED],yn4VAw~hNslsZ9KMmZK6Rw,[EMAIL PROTECTED], 
routedTime=1066874986543
, replyTime=1066874987597, outwardSender=null



___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] cache broken too

[Zlatin Balevsky]

lets not forget this other favorite...

Oct 22, 2003 10:32:31 PM (freenet.node.states.request.TransferReply, 
QThread-260, ERROR): Send failed, cache broke
n! for freenet.node.states.request.TransferReply: 
key=92b8d318833e6d7e5e8a8d18388dbbe9ba94df00140302, hopsToLive=1
3, id=bfac2582bdef4813, [EMAIL PROTECTED] 
(92b8d318833e6d7e5e8a8d18388dbbe9ba94df00140302,re
quest),ft=freenet.client.InternalClient$InternalGetToken:[EMAIL PROTECTED],key=freenet:C
[EMAIL PROTECTED],hpwXNWarhRfvWVwg5lN2Lg,[EMAIL PROTECTED], 
routedTime=1066875074121, rep
lyTime=1066875075708, outwardSender=null
Oct 22, 2003 10:37:11 PM (freenet.node.states.data.SendData, 
QThread-349, ERROR): Cache failed signalled after exc
eption after 1047280 of 1049900 bytes: java.io.IOException: read failed: 
Sending Data @ 4a544f1c5b4ef007: 4a544f1c
5b4ef007/b75c0086dd0524be: 
[EMAIL PROTECTED], in=Key: 
1a2990946b4f0525feabf9ec456f2176
51f4f1da140302 Buffer: 
[EMAIL PROTECTED]:0x1 : 
1a2990946b4f0525feabf9ec
456f217651f4f1da140302:temp:1050081:717c0d63c791fe6f New: true ( 1049900 
of 1049900 read), moved=1047280/1049900,
partSize=16384,result=-1,lastPacketLength=1490 for 4a544f1c5b4ef007 
(b75c0086dd0524be.
java.io.IOException: read failed: Sending Data @ 4a544f1c5b4ef007: 
4a544f1c5b4ef007/b75c0086dd0524be: send=freenet
[EMAIL PROTECTED], in=Key: 
1a2990946b4f0525feabf9ec456f217651f4f1da140302 Buffer: freenet.fs.dir.N
[EMAIL PROTECTED]:0x1 : 
1a2990946b4f0525feabf9ec456f217651f4f1da140302:temp:1050081:717
c0d63c791fe6f New: true ( 1049900 of 1049900 read), 
moved=1047280/1049900, partSize=16384,result=-1,lastPacketLeng
th=1490
   at freenet.node.states.data.SendData.received(SendData.java:188)
   at freenet.node.StateChain.received(StateChain.java:195)
   at freenet.node.StateChain.received(StateChain.java:71)
   at 
freenet.node.StandardMessageHandler$Ticket.run(StandardMessageHandler.java:234)
   at 
freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.java:172)
   at 
freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler.java:124)
   at 
freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:72)
   at freenet.Ticker$Event.run(Ticker.java:323)
   at 
freenet.thread.QThreadFactory$QThread.run(QThreadFactory.java:234)

___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] exception in 6264

[Zlatin Balevsky]

java.lang.NullPointerException
   at 
freenet.interfaces.BaseLocalNIOInterface.intAddress(BaseLocalNIOInterface.java:37)
   at 
freenet.interfaces.BaseLocalNIOInterface.hostAllowed(BaseLocalNIOInterface.java:189)
   at 
freenet.interfaces.BaseLocalNIOInterface.dispatch(BaseLocalNIOInterface.java:230)
   at 
freenet.interfaces.NIOInterface.acceptConnection(NIOInterface.java:98)
   at freenet.transport.tcpNIOListener.accept(tcpNIOListener.java:103)
   at 
freenet.transport.ListenSelectorLoop.processConnections(ListenSelectorLoop.java:106)
   at 
freenet.transport.AbstractSelectorLoop.loop(AbstractSelectorLoop.java:681)
   at 
freenet.transport.ListenSelectorLoop.run(ListenSelectorLoop.java:146)
   at java.lang.Thread.run(Thread.java:534)

___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] using reflection in fred

[Zlatin Balevsky]

The state chain code uses the reflection api heavily.  Official 
tutorials from Sun recommend reflection to be used only as a last 
resort, i.e. if the problem cannot be solved any other way.  RTTI in 
win32 adds tons of overhead.  insert other examples here

Is anyone out there 100% positive that reflection in java is cheap, 
lightweight, efficient?  If not I suggest we put getting rid of it on 
the TODO list

___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] incoming at port -2 with build 6235

[Zlatin Balevsky]

sorry for the html, images were removed.

OCM lists some connections as incoming at port -2, -1...

-1

208.39.42.23:35634  0   2,493 Bytes -   None0 s 1 s
35713
207.32.9.50:11975   366 299 KiB -   294 KiB 0 s
 6:46
-2
	207.32.9.50:11975 	0 	None 	- 	None 	1:50 	1:50

and here's one that has had some data on it:
-1
	67.80.241.230:2366 	4 	541 Bytes 	- 	None 	0 s 	10:36



___
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl