On Friday, October 18, 2013 04:45:40 PM Matthew Toseland wrote: > On 12/10/13 23:37, xor wrote: > > [Thanks for providing feedback!] > > > > On Saturday, October 12, 2013 03:26:02 PM you wrote: > >> p0s's report for last week talks about synchronous FCP calls and IMHO > >> goes in completely the wrong direction: > >> > >> The stuff on synchronous in-fred notifications is hideous and > >> non-network-transparent. > > > > What do you mean by "network transparent"? > >
It would be good if you answered to that so we are talking about the same thing. I thought about it again and I understand it as: The synchronous API should be implementable even if the communicating plugin are connected via network, i.e. if the two of them are NOT running in the same node. > >> Having said that I haven't read the code, but > >> it sounds hideous. You should have some sort of request/response using > >> an identifier, like the main fred FCP does. Or a sequence number or > >> something. > > > > There are two possible code paths: > > (1) Both plugins are running in the same node. > > (2) Inter-node synchronous FCP-calls. Both parties are NOT within the same > > node. > > > > I've only implemented (1). I've implemented it by running a second thread > > to execute the FCP call at the target AND having the original calling > > thread wait for it to finish. > > > > (2) should be implemented like you suggested above. > > No, they should BOTH wait for a response using the Identifier field. > Plugins may well answer asynchronously - "when it returns" is an > implementation detail. Well obviously you have to wait if you want to know a result of an FCP call, you cant have the result if it is not finished yet. My point is to wrap this waiting part within an API of fred, thats what I mean by "synchronous": A send() function which does wait for the result. I have seend at least 3 different plugins invent wrappers around PluginTalker which implement this, and they are all bugged (one of them is in Freetalk so its my own fault that it is bugged). > >> IMHO mechanisms that only work internally should be avoided > >> given that everything else we do with FCP is, if not actually > >> network-transparent, then could easily be made so. But certainly you > >> could provide library code to wait for a single response using a known > >> identifier field; ideally the code for using plugins would be network > >> transparent via an official java library so it could use the hack you're > >> doing if it's internal... in fact, that's wrong: it's a different > >> concept. What we should do is wait for a reply to the specific question > >> we asked (i.e. wait for a message with the correct label), rather than > >> "wait for the function to return" - I mean, what function? Well written > >> code only hogs threads when there is a clear benefit in code simplicity. > > > > I suppose your criticism is that the implementation of (1) cannot be > > externalized because there is no additional message sent which signals > > that > > the call is finished? > > There doesn't need to be. In ALL cases, FCP works like this: > I send a message with Identifier. > I get back a message (possibly an error message) with the same Identifier. > > At least, that's how node-level FCP works. That's how plugins SHOULD work. Well where exactly is the "identifier" field you are talking about? Do you mean the one we pass to PluginRespirator like this: mConnection = mPluginRespirator.getPluginTalker(mFCPMessageReceiver, WOT_FCP_NAME, mConnectionIdentifier); - Thats the client-side code to open a connection to a plugin, i.e. WOT IMHO this identifier is suitable for being used as a marker for synchronous calls: 1) There isn't any checking on the uniqueness of it. They are chosen freely by the client. Client authors might use something like their pluginname, which is constantly NOT unique. So the "ACK"s would always hit the identifier, no matter whether to which sendSynchronous() they belonged. 2) There is nothing which prevents a client from re-using the same PluginTalker for sending multiple messages to WOT. And it is very likely that clients will re-use it. For example the reference-client which I wrote does that and I do think it is the right design. It would be ineffecient to re- create the whole connection at every send. This makes the identifier not unique as well because there will be multiple sends() with the same talker. Arguably we could mark a talker invalid after a single sendSynchronous() and prevent future use with that flag. But do we really want people to constantly create fresh PluginTalker objects using getPluginTalker? 3) What WOT specifically needs is to send something *back* at a future point in time - event-notifications always happen in the far future. For this, it keeps a WeakReference to the "PluginReplySender" of the PluginTalker which was used to talk to it. I.e. the PluginReplySender allows to send something back to the plugin which originally sent something to WOT with the associated PluginTalker. This is another usecase of continuing to use the same PluginTalker. Therefore I would rather impement sendSynchronous() to be network transparent by encapsulating a special random value in the SimpleFieldSet which identifies the particular call. > >> Feel free to ignore me though, I don't have time to code a proper > >> solution. But there was a strong belief at the conference that we should > >> try to limit the further accumulation of layers of dirty hacks, and try > >> to get rid of some of the existing ones. Especially as the FCP API for > >> WoT is necessarily in flux at the moment. > > > > I agree. In fact that is what I was trying to do with implementing > > synchronous FCP calls: There are currently 3 plugins which emulate them, > > each by having its own wrapper around PluginTalker (Freetalk, FlogHelper, > > ContentCuration) and I've probably forgotten about one. And all of those > > wrapper- > > implementations are buggy IMHO. This signals that there is a need for a > > reference implementation. > > Sure, but it should do it the right way. > > > The algorithm of event-notifications is mostly asynchronous but some areas > > would be a PITA if I had do do them asynchronously. > > And I do think that it is a fundamental property of algorithms that some > > algorithms will need synchronous communication while others work better > > asynchronously. That's probably why most communication frameworks provide > > both types of communication. > > "Synchronous communication" does not exist at the FCP level. What does > exist is messages - requests and responses. This is a perfectly > acceptable means of synchronizing! Yes of course it is possible to implement them on top of it, but its annoyance that every plugin has to re-invent this wheel. > > > So my conclusion is: I vote strongly for making Fred provide ways to do > > synchronous FCP calls. I am OK with changing the way I've implemented > > them > > though. > > It already does. We should be consistent between plugins' usage of FCP > and fred's usage of FCP. In the fred client layer case, it is very > clear: there is ALWAYS an Identifier field on the reply. Sometimes it's > more complex e.g. the global queue etc. But that's how we listen/wait > for a response. If the identifier is acceptable to be changed during a single connection, I would suggest to change PluginTalker to allow that. It would also need a mechanism for enforcing its uniqueness, which might even result in disallowing the client to chose the identifier at all. > > Anyway, thanks for still reviewing my work even though you are off to > > University :) > > > > PS: Storing an event-notification takes around 1ms so your fear of things > > being slowed down maybe is not fullfilled :) > > I'm more concerned about the uglification here. The FCPClientReferenceImplementation actually isn't ugly IMHO, its quite symmetric and robust against failure of various mechanisms. And the DebugFCPClient, which is a reference user of the reference client, is only ~200 lines - it manages to obtain a full copy of the WOT database with that. If I stripped the debug code which tests whether the data matches WOT's database, it would probably only be 100 lines. Providing synchronous calls also de-uglifies the code. Consider their following usecase of deploying event notifications: WOT sends out N notifications to a client. For each notification, it needs to know whether it was successfully delivered to the client before it can send the next: We cannot send them in parallel because it doesn't make sense to create a trust value before the identities were created which constitute the pair between which the trust exists. If there were no synchronous calls, the sending loop would have to send out 1 notification and then quit. Then there would be a receiving loop, which receives the confirmation of arrival of the notification, and it would have to query the according object from the database so it can mark it as delivered. Then the sending loop would have to query the database again for the next notification to send. So for N notifications there would be 2*N database queries. With synchronous calls, there is ONE database query which obtains all N notfications and sends them out. _______________________________________________ Devl mailing list [email protected] https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
