Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Sep 9, 2011, at 7:35 PM, Stephan Beal wrote: On Fri, Sep 9, 2011 at 7:04 PM, Twylite twyl...@crypt.co.za wrote: e.g. /timeline produces HTML by default, but /timeline.json would return the same information in JSON. i like that idea. i hadn't thought of simply using an extension. i don't have a strong opinion as to whether, e.g. /json/stat or /stat.json is better. Anyone want to give me their own strong opinion? (We could probably support both - they're stored as string-to-function mappings, but the current path/arg-handling code would need to be slightly different for each case.) My strong opinion: whatever.json is better than json/whatever. It's a matter of semantics, if we want to render whatever in some representation, or we want a representation containing a whatever... I'd say the former. Kind regards, Remigiusz Modrzejewski ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 2:46 PM, Martin S. Weber martin.we...@nist.gov wrote: ... and c) a separation of concerns (you, fossil library, take care of the fossil stuff, I take care of invoking hooks, presenting ... I think that invoking hooks would be an intermediary between the client and a pure Fossil server. Keep in mind that a Fossil instance is a client of another Fossil instance during a sync/pull/push operation and both instances could have their own hooks to invoke. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 7:38 AM, Martin S. Weber ephae...@gmx.net wrote: are talking about REST and JSON, but you really, really mean JSON over HTTP. What I'm trying to say is that you seem more concerned Yes, i am very possibly abusing some terminology. :/ Feel free to set me straight. preference of using POST to fetch multiple resources, where I'd rather expect a matrix parameter in the URL (values separated by semicolons). Old habits die hard. I'm wondering about the rationale behind establishing a set of return codes stuck into the JSON. If you are getting this via a REST request, we do have the HTTP request status codes. Again, it's my abuse of terminology. We could certainly add an option/mode which uses different HTTP result codes. My planned uses would be mostly from JS code which wants a nice, clean HTTP 200 (because i've already got a lot of code which uses this model and it clean separates transport- and app-level errors). of return codes buried inside the JSON. In fact, in an error condition, to avoid leaking information about how you might actually attack the server (DOS or breaking in), you should not be too specific about the error cause to the user. All that matters to the user is something like: wrong request, not authenticated, transient error, permanent error, resource moved, ... etc. The current error codes model takes that into account by allowing the codes to optionally be dumbed down by moduloing them with 1000 or 100 (depending on the level of dumb-downedness). All of this can be (and has been already) modeled via HTTP status codes. Have you got some links for me? FOSSIL-0: We have a successful http status code and are getting data. Yes the request has been successful. It's not necessary to duplicate this information. Actually, my intention is to remove from responses (as i do in my other code), since it is redundant (_except_ that it verifies that the response is-a fossil response, but that's not strictly necessary). FOSSIL-1001: We don't need a request envelope, and we shouldn't require one. The majority of the requests should be HTTP GETs, and all their data should be in the URL. No JSON at all in the request part. The envelope is only used for POST, and will (IMO) be needed when dealing with, e.g. the authentication info (which should not be passed in via GET because that lands in my apache log). We can't rely on cookies for auth info because not all clients can use them. (But we can piggyback on/integrate with the current fossil cookie mechanism, i think.) FOSSIL-1002: unknown command = URL not found. There's a status code for that (404). My client-side AJAX code will fail in ways its not programmed to if a non-200 comes back, which is why i modeled it this way. i.e. it treats non-200 as a transport-level error, whereas with this model a 200 says that HTTP-side everything was okay but the error code reveals an app (not transport) level error. FOSSIL-1005: If a server side timeout occurs, the client making the connection will / would timeout on its own. Otherwise, use 408. fossil doesn't do any timeouts itself, AFAIK (maybe under high concurrency?), so i think this one can go away entirely. i basically just copied the list of error codes from a different project of mine. ...FOSSIL-: We definitely don't want a (evil) user to know that. Generic error it is. Good point. FOSSIL-6667: Actually, 500 (or 400) is a good response on that. i also can't argue against that. Loose thoughts: For someone who is used to using REST services, the necessity for a JSON-based POST when you really want a GET seems awkward. POST is only needed for some requests. We can't save a 12kb wiki page using GET only because GET is often limited to 8kb or less (and some web servers segfault if sent more). Also while we're there (and I'm thinking about shunning now), HTTP DELETE should probably be on the list of supported transactions. Definitely HTTP PUT for editing stuff should be there (to keep things natural wrt REST). I don't like the requests that pass parameters (list?page=, wiki/get?name=). They make the URL unnecessarily longer IMO. my personal preference is /path/style. json/wiki/save should be a PUT/UPDATE. The Pun: Was man ueblicherweise mit fossilien macht, ist sie auszugraben und auszustellen. Ausgrabung waer daher ein passender Name (Excavation). i was hoping for something catchier/more obviously a pun, but excavation isn't bad. Since i currently have no name for it, Fossil Excavator is the first on the list so far. Hmm all the thoughts before I call it a day. All very valid, and i apologize profusely for my blatant abuse of the term REST (borne of my ignorance of its details). i'll start removing that word for the time being :). i'm not well-versed in the various semantics of HTTP codes other than 200, 404, and 500, and i have tons of code which currently uses the HTTP 200 + response
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/08/2011 09:27 PM, Stephan Beal wrote: On Thu, Sep 8, 2011 at 10:11 PM, Martin S. Weber martin.we...@nist.govwrote: On 09/08/11 16:01, Stephan Beal wrote: 1) A split between library and app. i.e. libfossil vs. the fossil server/cgi/shell app(s) I couldn't agree more. One of my goals (luckily no milestone I'll ever be evaluated against) is to do that. i can promise you that herculean effort was not a hyperbole! That said, i would be happy to assist you in this, and i think it can be done incrementally, taking a long time to do but having no outwardly effect on the apps. What will this mean for the famed single binary, just copy it into your PATH effect? Will said single binary still exist and statically link libfossil into itself, but also have a libfossil.so available alongside? ABS - -- Alaric Snell-Pym http://www.snell-pym.org.uk/alaric/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5pz4sACgkQRgz/WHNxCGrFLACcCz9R9FPxtRvjJOxlX8FtkL9q 3C8An2ZHKw8QupPr2NUxZxcB+LcuY0uu =acGv -END PGP SIGNATURE- ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
Great idea! Please consider that when using cgi a complete REST over http style interface is not supported: cgi defines only http GET and POST DELETE and PUT are not in the cgi spec and handled differently by different httpd's. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 10:34 AM, Alaric Snell-Pym ala...@snell-pym.org.ukwrote: What will this mean for the famed single binary, just copy it into your PATH effect? Will said single binary still exist and statically link libfossil into itself, but also have a libfossil.so available alongside? A single-binary build would still be required, i believe, if only due to Fossil's long and glorious history of being buildable that way (much to our benefit). Such a restructure would be more to enable other apps rather than hinder the main app. But in any case we're a long way from such a split (if it ever materializes at all - it would require a very large undertaking). -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On 09:59 PM, fossil-m...@h-rd.org wrote: Great idea! Please consider that when using cgi a complete REST over http style interface is not supported: cgi defines only http GET and POST DELETE and PUT are not in the cgi spec and handled differently by different httpd's. A common convention is to allow a _method query parameter in the URL to turn a POST into a PUT/DELETE. See http://microformats.org/wiki/rest/urls . It strikes me that Fossil is already quite RESTful and its URL structure is quite conducive to a REST+JSON interface, using the convention of file extension implies representation. e.g. /timeline produces HTML by default, but /timeline.json would return the same information in JSON. Similarly: - /wcontent.json produces a list of all Wiki pages - /wiki.json?name=foo is used to GET, POST, PUT or DELETE a specific Wiki page - /info/HASH.json produces a json representation of an check-in - /finfo.json?name=crypt_wp1.sql produces a json history of a single file - /artifact/HASH.json produces a json representation of a specific artifact I would imagine that it is not terribly difficult to set the default representation to .html, and to detect requests for .json representation at quite a high level (it's always the extension on the last component of the URL), and simply render the output differently for those resources that support a JSON representation. With some carefully designed helpers the same logic could be used to produce JSON or XML or whatever next year's favorite happens to be (along the lines of rep = json /* or whatever */; s = new StructuredOutput(rep); s.put(name, type, value); output = s.render(); ) Regards, Twylite ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
For me, making it easier to create an IDE plug-in for Fossil would be the greatest benefit. Also, I agree that HTTP status codes should be for transport rather than application errors. On Thu, Sep 8, 2011 at 4:01 PM, Stephan Beal sgb...@googlemail.com wrote: ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 7:11 PM, Ron Wilson ronw.m...@gmail.com wrote: Also, I agree that HTTP status codes should be for transport rather than application errors. :-D. Okay, so there's 2 votes for that. In my experience (and i've written boatloads of JS- and Java-based Ajax the past 2 years), i find it more intuitive to write code where i have a clear separation of transport-vs-app errors. e.g. if we use jQuery for the Ajax then an HTTP XYZ (any error code) will trigger the error() AJAX callback, and from there we have no recovery strategy (and can't say much about what happened to the user). If, OTOH, the app returns: { resultCode:123, resultText:broken code} then success() AJAX callback is called and the client can interrogate resultCode. In some cases he might be able to retry, whereas others he has no choice but to fail (e.g. don't retry a failed login, but maybe retry a failed wiki page fetch 2 or 3 times). Even if he can't retry, assuming the resultCode is a well-defined value then he can tell the user what the problem was (which he can't do all that accurately with HTTP codes). The REST model breaks that pretty badly, IMO. That said, i recognize the benefit of vague error codes, but during the development/debugging i want the codes to be as specific as possible so that i can track down errors. It also helps when supporting users. But for production servers i agree that the error codes should be vaguified a bit. The current model supports that but there is no code yet to dumb the codes down (and the list of codes will still be in flux for a while). -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 7:42 PM, Stephan Beal sgb...@googlemail.com wrote: On Fri, Sep 9, 2011 at 7:11 PM, Ron Wilson ronw.m...@gmail.com wrote: Also, I agree that HTTP status codes should be for transport rather than application errors. :-D. Okay, so there's 2 votes for that. In my experience (and i've written boatloads of JS- and Java-based Ajax the past 2 years), i find it more intuitive to write code where i have a clear separation of transport-vs-app errors. To stress this point a bit using a real-world example, please take a look at: http://whiki.wanderinghorse.net Poke around the wiki there - the content's not important, just try the various buttons, the editor, and whatnot. Try swapping between pages while editing (by following wiki links, NOT using the browser back button!). And keep in mind that that site's UI is 100% JavaScript/HTML/CSS and the data is 100% JSON served by a CGI. Even the wiki rendering is done on the client. i think that app would have been more troublesome to code if i had had to be able to distinguish REST-style HTTP errors from real HTTP errors. The overall goal of this effort is that that type of thing can be done for fossil. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On 09/09/11 13:52, Stephan Beal wrote: (...) While there is arguably little use for JSON in CLI mode, i'm trying to keep it all structured so that i can use the same code/commands in both CLI and CGI/server modes (...) Actually, if the complete CLI functionality was available as JSON output, we'd automatically have our library/frontend model. Think about it, a library is a backend that you communicate with. Typically you call functions in the same address space. The benefit of course is tight integration, compile time safety and what not. The drawback is becoming eminent when you are trying to access it from multiple callers / threads. Do you synchronize, what about locked DBs etc. etc. Now, if your library is actually a server that you communicate with, and the CLI frontend-client is doing the same, anybody who's capable of sending a HTTP request to localhost can use all of fossils capabilities and program on top of it. What would then be lacking, of course, would be the simple steps that e.g. sync, commit, list, mv, etc. are being composed of. That might be a not-so-herculian effort in the end to make fossil a library. I haven't given this a lot of thought, just saying: in the end what we would want from a fossil library would be that a) the cli uses the same library so that we are able to be first-class citizens and b) finer grained control and c) a separation of concerns (you, fossil library, take care of the fossil stuff, I take care of invoking hooks, presenting a GUI and slapping my users if they try to do something stupid). a) to c) can be achieved with IPC, too, of course (just look at the COM/CORBA model, remove all the bloat and voila, you have data-format over protocol). Btw, I suggest starting a new thread, called: FOSSILS JSON-RPC INTERFACE ;) Just to get REST out of people minds... Regards, -Martin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 8:46 PM, Martin S. Weber martin.we...@nist.govwrote: Actually, if the complete CLI functionality was available as JSON output, we'd automatically have our library/frontend model. Think about it, a library is a backend that you communicate with. i agree with that, but actually using it that way from client scripts seems tedious and inefficient. What would then be lacking, of course, would be the simple steps that e.g. sync, commit, list, mv, etc. are being composed of. That might be a not-so-herculian effort in the end to make fossil a library. Agreed. In a mail to Richard on the topic i mentioned that once we have a JSON API, we get the lib/app split for free. In a sense, anway. I haven't given this a lot of thought, just saying: in the end what we would want ...a GUI and slapping my users if they try to do something stupid). a) to c) can be achieved with IPC, too, of course (just look at the COM/CORBA model, remove all the bloat and voila, you have data-format over protocol). CORBA *shudder*!!! That's all true, though. Btw, I suggest starting a new thread, called: FOSSILS JSON-RPC INTERFACE ;) Just to get REST out of people minds... LOL! It took me all day to get it out of my mind. So are we agreed that REST is not the goal (at least not initially)? Since CGI doesn't support the full range of REST features (PUT/DELETE, apparently), REST support would take me into areas i'm not yet fit to code myself. Regarding your earlier comments about error code leakage to clients: i just added some code to set a paranoia level for error reporting. e.g. code 1234 can be returned as any of (1234, 1230, 1200, or 1000). Once again, i appreciate your continued feedback. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Thu, Sep 8, 2011 at 10:11 PM, Martin S. Weber martin.we...@nist.govwrote: On 09/08/11 16:01, Stephan Beal wrote: 1) A split between library and app. i.e. libfossil vs. the fossil server/cgi/shell app(s) I couldn't agree more. One of my goals (luckily no milestone I'll ever be evaluated against) is to do that. i can promise you that herculean effort was not a hyperbole! That said, i would be happy to assist you in this, and i think it can be done incrementally, taking a long time to do but having no outwardly effect on the apps. 2) A JSON interface. I agree with the REST interface, choosing JSON seems natural, then. But then again, you only need the JSON for the replies, not the requests themselves, as the URL will contain the query (update, drop, ...) already. In case you have data to accompany the request, I'm not sure that JSON is the right choice (in contrast to multipart/forms), but I don't have a strong opinion on that one. See the draft doc - SOME requests (commit) would benefit from POST, but yes, the majority can get by with GET. i recommend JSON mainly for portability. It can be generated (with some limitations) even from shell scripts and posted via wget or curl. Form-encoding is more expensive and (with JSON) unnecessary. That said, i have code in place (in another tree, but it could be ported) for translating form-encoded data into a JSON object so that the application can treat both forms identically. i.e. with that in place form encoding can also be used and the app doesn't need to change except for the initial parsing code (which type to parse depends on the CONTENT_TYPE header). That being said, what about the CLI? If you're using client-side UI, then the CLI should do that, too, shouldn't it? May I point to the wonderful jansson C library (even though it's maybe too late already?), an easy to use, lightweight, BSD licensed json library while we're at it. jansson is very nice (i only discovered it last week), and the library i'm using is actually (and 100% coincidentally) _very_ similar to jansson. (i emailed jansson's author last week about that, actually, and we discussed a few implementation differences.) The current code (called cson) is Public Domain, so there's no licensing constraint. See: http://whiki.wanderinghorse.net/wikis/cson/ My fossil fork contains a trimmed-down copy of that project's amalgamation build. Or is the CLI still going to be a special case ? i would also like for the CLI to be able to spit out JSON (and possibly even read requests via stdin, which can internally be treated as POSTed JSON), but that's a secondary goal for me. The JSON-emitting code is actually very minimal - it uses the existing SQL queries and blindly (with no interpretation other than figuring out each field's data type) translates the results to JSON. Because of that, it's especially easy to add to CLI commands (i have it added to tickets and timeline in my copy). Adding an HTTP interface will require more internal refactoring and internal infrastructure, as opposed to only function-local changes like the CLI JSON integration typically requires. i recently had to implement all of this for another app (also a CGI in C), so it's clear to me what infrastructure we generically need, it's now just a matter of doing it. The infrastructure is largely independent of the public JSON interface, so i can continue to implement that in peace while we all bat around the important parts - the public interfaces. Here's an example from the CLI (pedantic note: it doesn't conform to what the doc i posted suggests because i'm not yet far enough along, at the fossil/json infrastructure level, to do that. That doc represents the goal/ideal, not the current state.): stephan@tiny:~/cvs/fossil/fossil-sgb$ ./fossil time -n 3 -j {columns:[rid, uuid, mDateTime, comment, primPlinkCount, plinkCount, mtime], rows:[{rid:13466, uuid:9c74371928779b08704116581cd79476e3e1eb0f, mDateTime:2011-09-06 21:27:38, comment:timeline json: now only add LIMIT clause if n!=0. Added a FIXME regarding showfilesFlag support. (user: stephan tags: sgb-cson), primPlinkCount:0, plinkCount:1, mtime:2455811.394196}, ... (i _really_ hope your mail client wrapped that!) Anyways, thumbs up! :-D Thank you for your feedback - keep it coming! -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Thu, Sep 8, 2011 at 10:27 PM, Stephan Beal sgb...@googlemail.com wrote: week about that, actually, and we discussed a few implementation differences.) The current code (called cson) is Public Domain, so there's no licensing constraint. See: http://whiki.wanderinghorse.net/wikis/cson/ Actually... the underlying JSON push parser (originally based off of Doug Crockford's parser, i believe), has a BSD-like leave this copyright here license with a do no evil clause. Which begs the question... is a do no evil clause allowable in sources imported into the fossil project? Yes, i'm serious. If it is a potential problem, i can almost certainly get the author to drop that clause for this purpose, but if the clause is not a problem i'd rather not pursue it. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
A JSON/REST interface sounds very promising. A few weeks ago I was playing around and tried creating a client-side file-browser treeview in HTML. What I ended up doing was getting the raw artifact (/raw/) and parsing it clientside. Not exactly perfect so I dropped it. A well designed JSON-interface would certainly have made my experiment easier. One thought: when the interface would rely on GET/POST it might create cross-domain problems when used from a web-browser. Therefore my vote goes to JSON(P) Jos ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Thu, Sep 8, 2011 at 11:05 PM, Jos Groot Lipman donts...@home.nl wrote: ** A well designed JSON-interface would certainly have made my experiment easier. Then keep the ideas coming :). One thought: when the interface would rely on GET/POST it might create cross- domain problems when used from a web-browser. Therefore my vote goes to JSON(P) That's an excellent point which i had completely forgotten about. JSONP should definitely be an option, and i'll be sure to account for that while coding. Note that the so-called Same Origin Policy is going away with HTML5 and XMLHttpRequest2 (the _server_ will decide instead who requests can come from, and publishes that via a new header), which will likely make JSONP a dying technology (since it's just a workaround for the Same Origin Policy). That said, HTML pages are _not_ the only target for a JSON interface. Arbitrary languages can pull JSON for various purposes. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Thu, Sep 8, 2011 at 10:27 PM, Stephan Beal sgb...@googlemail.com wrote: i recommend JSON mainly for portability. It can be generated (with some limitations) even from shell scripts and posted via wget or curl. Form-encoding is more expensive and (with JSON) unnecessary. That said, i have code in place (in another tree, but it could be ported) for translating form-encoded data into a JSON object so that the application can treat both forms identically. i.e. with that in place form encoding can ... It just occurred to me (and got me out of bed)... form-decoding is not _quite_ the same. If we have a form with: a=1 b=2 and convert it to JSON: { a: 1, b:2 } note that the values are strings (because we won't have enough type information in the parsing code and don't want to scan every token to determine its type). When we get JSON as input, the type information is there and is retained throughout the app. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users