[freenet-support] Load
Hello everyone. I started a node on a machine with lots of bandwidth and a very lousy I/O subsystem. Not much else is going on on the machine, so without freenet the load is steadily between 0.01 and 0.10. When freenet runs, the load is constantly around 3.50, with peaks reaching well above 5.00. The system latency caused by these loads gets other stuff to malfunction; for instance, mail server queries to LDAP time out and result in service temporarily unavailable errors. Obviously, such things become a show stopper for freenet. The machine is a Celeron 2.4 GHz with 512 MB RAM running RHEL3 with Sun java 1.4.2_05. I reduced maxNodeConnections to 85 from the default 200 in the hope to reduce the number of java instances, but that didn't help much. I also set diagnosticsPath=/dev/null to stop the constant writing on disk (is there a better way to say disable stats, I don't need them?) and that didn't help much either. So I'm asking for advice from those more experienced with freenet: what can I do to reduce the load? More specifically, how can I reduce the number of java instances running? With my current settings (most else at default), I have 96 java processes eating away everything on the machine and the machine itself too. Z -- Framtiden r som en babianrv, frggrann och full av skit. Arne Anka ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Monday 19 July 2004 15:14, Zenon Panoussis wrote: Hello everyone. I started a node on a machine with lots of bandwidth and a very lousy I/O subsystem. Not much else is going on on the machine, so without freenet the load is steadily between 0.01 and 0.10. When freenet runs, the load is constantly around 3.50, with peaks reaching well above 5.00. The system latency caused by these loads gets other stuff to malfunction; for instance, mail server queries to LDAP time out and result in service temporarily unavailable errors. Obviously, such things become a show stopper for freenet. The machine is a Celeron 2.4 GHz with 512 MB RAM running RHEL3 with Sun java 1.4.2_05. I reduced maxNodeConnections to 85 from the default 200 in the hope to reduce the number of java instances, but that didn't help much. I also set diagnosticsPath=/dev/null to stop the constant writing on disk (is there a better way to say disable stats, I don't need them?) and that didn't help much either. So I'm asking for advice from those more experienced with freenet: what can I do to reduce the load? More specifically, how can I reduce the number of java instances running? With my current settings (most else at default), I have 96 java processes eating away everything on the machine and the machine itself too. I run freenet niced at +10 on a 2x500MHz computer, load stays at 2-3 all the time. I suspect the problem you have lies in the fact that freenet will eat ALL available bandwidth that you give it, which will lead to starvation, so adjust the following settings: inputBandwidthLimit= Your input limit outputBandwidthLimit= Your output limit Its in bytes/s. You should not allocate your whole bandwidth to freenet, leave at least some 10% to other traffic. You could also limit the threads used by adjusting the maximumThreads setting. - -- Roger Oksanen [EMAIL PROTECTED] +358 50 355 1990 CS Student at Helsinki UniversityPGP id 1B125A3E Homepage http://www.cs.helsinki.fi/u/raoksane/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQFA+8CV78OZUBsSWj4RAmrtAJ9hcwJfIScktZbf224djNidhETolACgk/AY 2D594laB1rVZI7oOlwGm1ug= =5ffn -END PGP SIGNATURE- ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
Roger Oksanen wrote: I run freenet niced at +10 on a 2x500MHz computer, load stays at 2-3 all the time. Ah yes, I forgot to mention that. It's niced at 19. Beats me how something that's niced 19 can bring the load to 5.00, but that's a different issue. I suspect the problem you have lies in the fact that freenet will eat ALL available bandwidth that you give it, which will lead to starvation, so adjust the following settings: inputBandwidthLimit= Your input limit outputBandwidthLimit= Your output limit That's done already, it's not where the problem lies. Both these settings are at 10240, calculated for a monthly consumption of about 50 GB. The machine has a 100 Mbit connection to the net, so starvation is out of the question. You could also limit the threads used by adjusting the maximumThreads setting. Reducing maximumThreads from default 120 to 60 had very little positive impact on the load. However, while I was there I noticed the overLoadlow parametre, which I had missed earlier. I set it to 0.8 but it dosn't work as advertised. After 35 minutes with this setting in effect, I'm looking at 9:57, 1 user, load average: 1.13, 1.74, 1.04 9:58, 1 user, load average: 1.53, 1.71, 1.07 10:00, 1 user, load average: 1.84, 1.77, 1.17 10:01, 1 user, load average: 3.04, 2.05, 1.31 10:05, 1 user, load average: 2.37, 2.39, 1.61 10:17, 1 user, load average: 5.49, 4.00, 2.69 10:26, 1 user, load average: 4.27, 4.20, 3.39 ./stop-freenet.sh 10:30, 1 user, load average: 0.15, 1.99, 2.66 If all averages are constantly above the overLoadlow limit and the one-minute average keeps increasing, then this setting is simply not being obeyed. Duh. I don't remember running a more aggressive piece of software, ever. A note to the developers: RAM is cheap. Working software is very expensive. Freedom is horrendously expensive. Sadly, this is an over-simplification and reality is more complex than that. The people who have money can buy freedom and don't need more RAM. The people who mostly need more RAM in order to have freedom are mainly those who can't afford the RAM. This is true on a national level, comparing the degree of repression and the financial situation of the average citizen in, say, China or Egypt to those in the US or Europe, and it is also true on the personal level; he who can pay a good team of lawyers will seldom need to fiddle with freenet. In my case, I rent a server somewhere for 39 euro per month. It's crappy hardware, but it's fully sufficient for all my needs and it's all I can afford anyway. To get better hardware where I have the bandwidth I'd have to double my expense. At home, where I have better hardware, I pay the traffic at the tune of 3 euro/GB. The sum of this equation is, unfortunately, one freenet node less. I do think that resource management would be a worthy priority for the project. Z -- Framtiden r som en babianrv, frggrann och full av skit. Arne Anka ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] A severe freenet exploit?? - or just FUD?
Hello, yesterday, there was a post on the Frost-board freenet where the anonymous poster claims to have found a severe freenet exploit. He explains that he could determine anyones IP address, no matter how many hops the person is away from his own node. (for details see the attached message). I personally think that this is just an attempt to spread FUD (Fear, Uncertainty, Doubt) among the people using freenet. The poster (or the governmental organisation behind him) can't get any hold of the freenet users and thus tries to make them not to use freenet in the first place. However, it would really put my mind at rest, if one of the developers could confirm that this claim has no substance. I attach the message from the Frost-board below: start of attached message Have I found an exploit in freenet?!? With a modified freenet java client I'm able to trace IPs and hop count of posts. example: bebe's post in CHK_Dragon - [EMAIL PROTECTED] - 2004.07.09 - 21:16:35GMT - sage auch mal wieder hallo und grüsse alle neuzugänge !! bis die tage bebe - [EMAIL PROTECTED] - 2004.07.10 - 11:12:57GMT - baba bebe :) -- [EMAIL PROTECTED],yVzuyL1y1y7LbMG4OH7KoQ/naphtala.asc [EMAIL PROTECTED] IP 80.134.xxx.xx (german DSL dial-in) 8 hops away from me (on posting time 9/7/2004) naphtala resides in Germany too. I'll not go into details here but some technical background: I 'mark' the IP headers and request a retransmit of this packet from hop next to me. If it's not the originator of the packet the generation of his answer causes (due the corrupted IP packet header I send for retransmit request) an exploit and forces a retransmit to _its_ packet source hop. This answer is routed to me and now I have the IP of the freenet client 2(!!!) hops away from me. I do this recursively for all other hops up to the originator. The originator doesn't have a hop to request retransmit and the exploit does nothing (no return packet). The only condition I need is the hop chain (IP connections) must be still in connect state. So the trace must be done in a very short interval after receiving the original post. It's very hard to catch a intact IP connection cain. In my current implementation there is a success rate of about 0.5%. Due low level nature of the expliot it is not neccesary to crack any encryption. !!! I'll never ever compromize all of your anonymity !!! But is someone willing to support me? Please make a post subject called 'freenet exploit #1' and let me try to catch posters IP address. I'll anwser the IP so the original poster can say yes/no. Thanks for your support 130303 ** bebe, naphtala and all, I hope so I'm wrong!!! *** end of attached message ___ Gesendet von Yahoo! Mail - Jetzt mit 100MB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
On Mon, Jul 19, 2004 at 02:14:38PM +0200, Zenon Panoussis wrote: Hello everyone. I started a node on a machine with lots of bandwidth and a very lousy I/O subsystem. Not much else is going on on the machine, so without freenet the load is steadily between 0.01 and 0.10. When freenet runs, the load is constantly around 3.50, with peaks reaching well above 5.00. The system latency caused by these loads gets other stuff to malfunction; for instance, mail server queries to LDAP time out and result in service temporarily unavailable errors. Obviously, such things become a show stopper for freenet. Strange. What is your logLevel ? The machine is a Celeron 2.4 GHz with 512 MB RAM running RHEL3 with Sun java 1.4.2_05. I reduced maxNodeConnections to 85 from the default 200 in the hope to reduce the number of java instances, but that didn't help much. Won't make much difference. The setting you want is maximumThreads. I also set diagnosticsPath=/dev/null to stop the constant writing on disk (is there a better way to say disable stats, I don't need them?) and that didn't help much either. You do. The node uses them for estimating load. My long-lived unstable node's stats dir is 3MB, as is my long-lived stable node's stats dir. It's not a problem. So I'm asking for advice from those more experienced with freenet: what can I do to reduce the load? More specifically, how can I reduce the number of java instances running? With my current settings (most else at default), I have 96 java processes eating away everything on the machine and the machine itself too. Set maximumThreads=60 (remove the leading %), logLevel=error, and doCPULoad=true (is your node pegged on CPU, or just on I/O? what's the idle % typically?). -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
On Mon, Jul 19, 2004 at 03:37:41PM +0300, Roger Oksanen wrote: I run freenet niced at +10 on a 2x500MHz computer, load stays at 2-3 all the time. I suspect the problem you have lies in the fact that freenet will eat ALL available bandwidth that you give it, which will lead to starvation, so adjust the following settings: inputBandwidthLimit= Your input limit outputBandwidthLimit= Your output limit Its in bytes/s. You should not allocate your whole bandwidth to freenet, leave at least some 10% to other traffic. I recommend allocating no more than half. Thus if you have an uplink of 256kbps, set: outputBytes=16000 I doubt that inputBandwidthLimit matters that much.. but if you have a symmetric connection, set it. You could also limit the threads used by adjusting the maximumThreads setting. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
On Mon, Jul 19, 2004 at 03:54:09PM +0200, Zenon Panoussis wrote: Roger Oksanen wrote: I run freenet niced at +10 on a 2x500MHz computer, load stays at 2-3 all the time. Ah yes, I forgot to mention that. It's niced at 19. Beats me how something that's niced 19 can bring the load to 5.00, but that's a different issue. I suspect the problem you have lies in the fact that freenet will eat ALL available bandwidth that you give it, which will lead to starvation, so adjust the following settings: inputBandwidthLimit= Your input limit outputBandwidthLimit= Your output limit That's done already, it's not where the problem lies. Both these settings are at 10240, calculated for a monthly consumption of about 50 GB. The machine has a 100 Mbit connection to the net, so starvation is out of the question. You could also limit the threads used by adjusting the maximumThreads setting. Reducing maximumThreads from default 120 to 60 had very little positive impact on the load. However, while I was there I noticed the overLoadlow parametre, which I had missed earlier. I set it to 0.8 but it dosn't work as advertised. After 35 minutes with this setting in effect, I'm looking at Please leave overloadLow alone. It messes up rate limiting. 9:57, 1 user, load average: 1.13, 1.74, 1.04 9:58, 1 user, load average: 1.53, 1.71, 1.07 10:00, 1 user, load average: 1.84, 1.77, 1.17 10:01, 1 user, load average: 3.04, 2.05, 1.31 10:05, 1 user, load average: 2.37, 2.39, 1.61 10:17, 1 user, load average: 5.49, 4.00, 2.69 10:26, 1 user, load average: 4.27, 4.20, 3.39 ./stop-freenet.sh 10:30, 1 user, load average: 0.15, 1.99, 2.66 If all averages are constantly above the overLoadlow limit and the one-minute average keeps increasing, then this setting is simply not being obeyed. Duh. I don't remember running a more aggressive piece of software, ever. A note to the developers: RAM is cheap. Working software is very expensive. Freedom is horrendously expensive. Sadly, this is an over-simplification and reality is more complex than that. The people who have money can buy freedom and don't need more RAM. The people who mostly need more RAM in order to have freedom are mainly those who can't afford the RAM. This is true on a national level, comparing the degree of repression and the financial situation of the average citizen in, say, China or Egypt to those in the US or Europe, and it is also true on the personal level; he who can pay a good team of lawyers will seldom need to fiddle with freenet. Yeah, whatever. IMHO we need to make freenet work before we make it work fast. And the timescale on making it work may be a timescale of years. In which case, RAM really is not an issue, for the time being. Also, we HAVE done significant work on reducing memory consumption in the last year or so. In my case, I rent a server somewhere for 39 euro per month. It's crappy hardware, but it's fully sufficient for all my needs and it's all I can afford anyway. To get better hardware where I have the bandwidth I'd have to double my expense. At home, where I have better hardware, I pay the traffic at the tune of 3 euro/GB. Yikes. Where do you live? Poland? Spain? Greece? Lithuania? :) The sum of this equation is, unfortunately, one freenet node less. I do think that resource management would be a worthy priority for the project. Along with the other 300 worthy priorities! -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
Toad wrote: Strange. What is your logLevel ? Well, that's relative. The log level is set to debug, but the log file is a FIFO, where a simple perl script greps for URIs and dumps the rest. My idea was to feed those URIs to mnogosearch and create a non-anonymous search engine fo freenet. Won't make much difference. The setting you want is maximumThreads. I took them down from 120 to 60, saw hardly any difference at all. I also set diagnosticsPath=/dev/null to stop the constant writing on disk (is there a better way to say disable stats, I don't need them?) and that didn't help much either. You do. The node uses them for estimating load. My long-lived unstable node's stats dir is 3MB, as is my long-lived stable node's stats dir. It's not a problem. It's not the size of the stats on disk I want to avoid, but the extra I/O that comes from keeping them. Set maximumThreads=60 (remove the leading %), logLevel=error, and doCPULoad=true (is your node pegged on CPU, or just on I/O? what's the idle % typically?). OK, I'll give it one more try to see if it's the logging that does it. Of course, the logging is required for what I want to do, so I don't know how to get around the problem, if that's where it is. The normal load of the machine is around 0.10 an I/O is its big problem in general. Z -- Framtiden r som en babianrv, frggrann och full av skit. Arne Anka ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] A severe freenet exploit?? - or just FUD?
On Mon, Jul 19, 2004 at 10:33:09AM +0200, Arnold Weizendrescher wrote: Hello, yesterday, there was a post on the Frost-board freenet where the anonymous poster claims to have found a severe freenet exploit. He explains that he could determine anyones IP address, no matter how many hops the person is away from his own node. (for details see the attached message). Take him up on it. There ARE possibilities to fake it (harvest all nodes, insert on each node Your IP address is this node's IP address, fool! Tremble before me!... but with some collaboration we can probably defeat such an attack i.e. just publish the IP here and see if other Frost users get the same. I personally think that this is just an attempt to spread FUD (Fear, Uncertainty, Doubt) among the people using freenet. Very likely. The poster (or the governmental organisation behind him) can't get any hold of the freenet users and thus tries to make them not to use freenet in the first place. Well, he can get hold of Freenet node operators, fairly easily. That's not what you're talking about of course. However, it would really put my mind at rest, if one of the developers could confirm that this claim has no substance. I'll have a look. I attach the message from the Frost-board below: start of attached message Have I found an exploit in freenet?!? With a modified freenet java client I'm able to trace IPs and hop count of posts. example: bebe's post in CHK_Dragon - [EMAIL PROTECTED] - 2004.07.09 - 21:16:35GMT - sage auch mal wieder hallo und gr??sse alle neuzug??nge !! bis die tage bebe - [EMAIL PROTECTED] - 2004.07.10 - 11:12:57GMT - baba bebe :) -- [EMAIL PROTECTED],yVzuyL1y1y7LbMG4OH7KoQ/naphtala.asc [EMAIL PROTECTED] IP 80.134.xxx.xx (german DSL dial-in) 8 hops away from me (on posting time 9/7/2004) naphtala resides in Germany too. I'll not go into details here but some technical background: I 'mark' the IP headers and request a retransmit of this packet from hop next to me. Why bother? You know full well where the node that relayed the message is. If it's not the originator of the packet the generation of his answer causes (due the corrupted IP packet header I send for retransmit request) an exploit and forces a retransmit to _its_ packet source hop. Uh, I have no idea what you are talking about here. Freenet does not deal with IP packets directly. It uses sockets. If there is an exploit in the operating system or the JVM, then OF COURSE you can trace packets back to source. That's one of the more depressing things about the prevalance of insecure OSs, and the current dependance on a proprietary JVM. :( Now, if he means an exploit in Freenet, then I'd love to see it, but it seems unlikely as as I said we don't deal with IP anyway. This answer is routed to me and now I have the IP of the freenet client 2(!!!) hops away from me. For THAT request? Doubtful. Possible, but unlikely especially when framed as he has. I do this recursively for all other hops up to the originator. The originator doesn't have a hop to request retransmit and the exploit does nothing (no return packet). The only condition I need is the hop chain (IP connections) must be still in connect state. So the trace must be done in a very short interval after receiving the original post. It's very hard to catch a intact IP connection cain. This seems unlikely, our connection flux isn't _THAT_ bad. In my current implementation there is a success rate of about 0.5%. LOL. Due low level nature of the expliot it is not neccesary to crack any encryption. If the exploit is that low level, I don't really see how it could work, except perhaps by exploiting a remote execution hole in the host operating system (or JVM). !!! I'll never ever compromize all of your anonymity !!! But is someone willing to support me? Please make a post subject called 'freenet exploit #1' and let me try to catch posters IP address. I'll anwser the IP so the original poster can say yes/no. Thanks for your support 130303 -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
On Mon, Jul 19, 2004 at 05:02:42PM +0200, Zenon Panoussis wrote: Toad wrote: Strange. What is your logLevel ? Well, that's relative. The log level is set to debug, but the log file is a FIFO, where a simple perl script greps for URIs and dumps the rest. My idea was to feed those URIs to mnogosearch and create a non-anonymous search engine fo freenet. Okay, this is your basic problem. I don't even use logLevel=debug any more. It produces crazy amounts of data, uses a lot of CPU, and the system cannot keep up. I doubt that feeding it through a FIFO will make that much difference - much of it is probably the generation side (as well as the syscalls). I recommend you set the following: logLevelDetail=freenet.client:debug That should catch all the URIs, probably. You don't need debug log level on everything else, and having it will really mess stuff up. Won't make much difference. The setting you want is maximumThreads. I took them down from 120 to 60, saw hardly any difference at all. I also set diagnosticsPath=/dev/null to stop the constant writing on disk (is there a better way to say disable stats, I don't need them?) and that didn't help much either. You do. The node uses them for estimating load. My long-lived unstable node's stats dir is 3MB, as is my long-lived stable node's stats dir. It's not a problem. It's not the size of the stats on disk I want to avoid, but the extra I/O that comes from keeping them. Which is minimal, if the OS has enough memory to batch the writes into one medium sized write every 30 seconds. Set maximumThreads=60 (remove the leading %), logLevel=error, and doCPULoad=true (is your node pegged on CPU, or just on I/O? what's the idle % typically?). OK, I'll give it one more try to see if it's the logging that does it. Of course, the logging is required for what I want to do, so I don't know how to get around the problem, if that's where it is. The normal load of the machine is around 0.10 an I/O is its big problem in general. I have seen similar problems when running logLevel=debug on my own nodes. The solution is to set logLevel=error and logLevelDetails for the individual subsystems you want to monitor. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
On Mon, Jul 19, 2004 at 09:03:47PM +0200, Zenon Panoussis wrote: Toad wrote: I recommend you set the following: logLevelDetail=freenet.client:debug You did uncomment it, right? Of course :) ...that now the URIs don't get logged. ' That's strange. What URIs were you after? Those of requests in transit and inserts. Because of the island-like nature of freenet publishing, traditional spidering won't get very far; you can't seed a search engine with a few sites and assume that you will find the entire network by following links. Monitoring requests and inserts musters the collective URI knowledge of one's peers and of their peers, so it could go a long way, especially if you can put together a mesh of URI-grabbing nodes in different places. In a word, NO. You cannot monitor the URIs of requests going through your node. If you could, you could decrypt the data. Then you could search it, sure. However you'd also be liable for it. It would probably make tracking authors down easier too. We have no way of knowing how big the other islands out there are... we can only spider out from known sites, and from publicly visible Frost traffic. The thing is, the lack of search capabilities reduces the useability of freenet Of course. There are ways to implement search, however. Sooner or later somebody will implement a good spider based anonymous search. This would probably have two components: 1. A spider, which would spider out from known freesites, scan NIMs, and Frost traffic, and insert index files. 2. A client, probably integrated into fproxy, which would fetch the index files that are appropriate to the search given. and, indirectly, compromises anonymity too. Not really. I can publish stuff anonymously all I want but, unless I post a URL somewhere, nobody is going to find my publications. Indeed. Thus we have NIMs, FreeMail and Frost within Freenet, and outside it we have Mixmaster remailers, IIP, I2P, various kinds of proxies and so on. Sadly some people use hushmail too, which is not exactly the safest option. But there are many possibilities. And conversely, if I'm looking for a piece of information that might well be on freenet, I won't find it without asking. No. You will search for it. Just like on the real Internet. You may use a search engine, or you may follow links. Either way, there are issues of trust which are ironically much more readily solved by hypertext than by just making everything searchable. A spammable search system is of little practical use. Especially for someone who's new to freenet and doesn't already have a set of bookmarks and starting points, the threshold for getting anywhere is pretty high despite the proxy bookmarks. How do you propose to protect against spam, and plain malicious content? A non-anonymous search engine on the web could solve part of both these problems and at the same time function as an invitation to freenet for non-freenet users. So spider! We're not stopping you. Freenet does not know the URIs of data that passes through the node, only those requested locally. It does know the requests that pass through the node. Nope. It doesn't. It only knows the routing keys, which are insufficient to decrypt the actual data. Any other URIs in the logs will be locally originated. Example: CHK@routing key,decrypt key/human readable key The node only knows the routing key. It does not know the decrypt key and therefore cannot decrypt the data. If it could, it could guess the human readable key. Last night, all of freenet for me was the few URIs that are published on freenetproject.org. This morning I had a whole long list in my logs, and through that I was able to start finding my way around. Would be interested to see some of this list. Are you running a public gateway? Are you fetching lots of stuff locally? That's how this idea of a search engine popped up and turned into a small project in itself. Most of these would go through freenet.client... some might go through freenet.node.states.FCP, and there are a few internal ones. I'll look. I'm grateful for any tips you might have. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
Toad wrote: The thing is, the lack of search capabilities reduces the useability of freenet Of course. There are ways to implement search, however. Sooner or later somebody will implement a good spider based anonymous search. I searched a bit on the web. At http://conferences.oreillynet.com/cs/p2pweb2001/view/e_sess/1669 I found someone claiming that searching freenet would be possible real soon, to quote: right about now. That was in 2001. At http://www.freenet.org.nz/search/ I found a totally defunct search engine, obviously based on the same principle I'm trying to apply now. I fully agree with you that anonymous search is much better than a non-anonymous. However, as I mentioned, the problem of anonymity has two sides: that of the publisher and that of the user. If a non-anonymous search solves one part without affecting the other, what's the harm of it? This would probably have two components: 1. A spider, which would spider out from known freesites, scan NIMs, and Frost traffic, and insert index files. 2. A client, probably integrated into fproxy, which would fetch the index files that are appropriate to the search given. You mean creating index files before a search has been made? Wouldn't that be highly inaccurate and/or produce massive volumes of indices? I can publish stuff anonymously all I want but, unless I post a URL somewhere, nobody is going to find my publications. Indeed. Thus we have NIMs, FreeMail and Frost within Freenet, and outside it we have Mixmaster remailers, IIP, I2P, various kinds of proxies and so on. Sadly some people use hushmail too, which is not exactly the safest option. But there are many possibilities. All this put together is still a *very* small world. If I'd find and publish, say, the Bush administration's plans to invade Cuba, or detailed information on Israel's chemical and biological weapons, I don't want this information to to reach the users of freenet and hushmail; I want it to reach the huge and clueless masses who watch CNN and use hotmail. And I also want to protect my anonymity damn well. The way to go? Publish on freenet and let automation, i.e. nobody, make the bridge to the web. How do you propose to protect against spam, and plain malicious content? I don't. I'm not Google. As you have already gathered, my financial capacity is enough to run a 39-euro server, but not a 78-euro one. Because of that, things get very simple: if I make a freenet search, it will be just as well or ill protected from spam and malicious content as freenet itself is. Freenet does not know the URIs of data that passes through the node, only those requested locally. It does know the requests that pass through the node. Nope. It doesn't. It only knows the routing keys, which are insufficient to decrypt the actual data. Any other URIs in the logs will be locally originated. Example: CHK@routing key,decrypt key/human readable key Uhm, there's something eluding me here. You know freenet's internals; I don't. If you say so, then so it is. Yet I stuck some of those URIs I found in my logs into my browser and got sites to which I had never been before. Taking what you say here for granted, the entire discussion up to this point is probably a meaningless exchange based on some misunderstanding on my part. But what? [URIs from logs] Would be interested to see some of this list. Duh. So am I by now, but with all the messing around today I deleted them. I can try again though. Are you running a public gateway? Are you fetching lots of stuff locally? Neither. Z -- Framtiden r som en babianrv, frggrann och full av skit. Arne Anka ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Load
I wrote: Taking what you say here for granted, the entire discussion up to this point is probably a meaningless exchange based on some misunderstanding on my part. But what? [URIs from logs] Would be interested to see some of this list. Duh. So am I by now, but with all the messing around today I deleted them. I can try again though. Now I know what the misunderstanding was. The working URIs I found in my logs come from the default bookmarks in the interface servlet. I had never visited them before, but they had passed my client anyway. Z -- Framtiden r som en babianrv, frggrann och full av skit. Arne Anka ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
