Re: [fossil-users] Draft doc for JSON/REST Fossil interface
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/08/2011 09:27 PM, Stephan Beal wrote: > On Thu, Sep 8, 2011 at 10:11 PM, Martin S. Weber wrote: > >> On 09/08/11 16:01, Stephan Beal wrote: >> >>> 1) A split between library and app. i.e. libfossil vs. the fossil >>> server/cgi/shell app(s) >>> >> >> I couldn't agree more. One of my goals (luckily no milestone I'll ever be >> evaluated against) is to do that. >> > > i can promise you that "herculean effort" was not a hyperbole! That said, i > would be happy to assist you in this, and i think it can be done > incrementally, taking a long time to do but having no outwardly effect on > the apps. What will this mean for the famed "single binary, just copy it into your PATH" effect? Will said single binary still exist and statically link libfossil into itself, but also have a libfossil.so available alongside? ABS - -- Alaric Snell-Pym http://www.snell-pym.org.uk/alaric/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5pz4sACgkQRgz/WHNxCGrFLACcCz9R9FPxtRvjJOxlX8FtkL9q 3C8An2ZHKw8QupPr2NUxZxcB+LcuY0uu =acGv -END PGP SIGNATURE- ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
Great idea! Please consider that when using cgi a complete REST over http style interface is not supported: cgi defines only http GET and POST DELETE and PUT are not in the cgi spec and handled differently by different httpd's. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 10:34 AM, Alaric Snell-Pym wrote: > What will this mean for the famed "single binary, just copy it into your > PATH" effect? Will said single binary still exist and statically link > libfossil into itself, but also have a libfossil.so available alongside? > A single-binary build would still be required, i believe, if only due to Fossil's long and glorious history of being buildable that way (much to our benefit). Such a restructure would be more to enable other apps rather than hinder the main app. But in any case we're a long way from such a split (if it ever materializes at all - it would require a very large undertaking). -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] sqlite3_prepare_v2() is generating output to stderr?
Maybe this is a bug, maybe not, but i've certainly never seen it before...
i'm intentionally introducing SQL errors to test my JSON-side error
handling, and i found that this:
rc = sqlite3_prepare_v2(g.db, zSql, -1, &pStmt->pStmt, 0);
outputs to stderr when preparation fails.
i've never seen sqlite3 do that before. Is that normal? It interferes with
the JSON output:
stephan@tiny:~/cvs/fossil/fossil-sgb$ ./fossil json stat
./fossil: SQLITE_ERROR: near "(": syntax error
{
"fossil":"fa2d6af481b1a27db0206cef717fc9edeab59201",
"timestamp":1315581772,
"resultCode":"FOSSIL-1003",
"resultText":"near \"(\": syntax error\nSELECT x count(*) FROM blob"
}
that first output line "shouldn't be there".
Is this a debug mode thing? (In which case i don't mind so much.)
--
- stephan beal
http://wanderinghorse.net/home/stephan/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] sqlite3_prepare_v2() is generating output to stderr?
On 09/09/11 12:31, Stephan Beal wrote: Maybe this is a bug, maybe not, but i've certainly never seen it before... i'm intentionally introducing SQL errors to test my JSON-side error handling, and i found that this: rc = sqlite3_prepare_v2(g.db, zSql, -1, &pStmt->pStmt, 0); outputs to stderr when preparation fails. i've never seen sqlite3 do that before. Is that normal? It interferes with the JSON output: Really? I'm assuming the JSON output goes to stdout, and the error message goes to stderr. In that case it wouldn't interfere with the JSON output at all. Just pipe the output to your json reader without a 2>&1...there'll be no problem. -Martin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] sqlite3_prepare_v2() is generating output to stderr?
On 09/09/11 12:34, Martin S. Weber wrote: Really? I'm assuming the JSON output goes to stdout, and the error message goes to stderr. In that case it wouldn't interfere with the JSON output at all. Just pipe the output to your json reader without a 2>&1...there'll be no problem. That being said, of course a library should never write to stderr directly. -Martin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] sqlite3_prepare_v2() is generating output to stderr?
On Fri, Sep 9, 2011 at 6:35 PM, Martin S. Weber wrote:
> That being said, of course a library should never write to stderr directly.
>
>
i just found the reason for it:
/* Error logs from SQLite */
void fossil_sqlite_log(void *notUsed, int iCode, const char *zErrmsg){
fossil_warning("%s: %s", sqlite_error_code_name(iCode), zErrmsg);
}
main.c: sqlite3_config(SQLITE_CONFIG_LOG, fossil_sqlite_log, 0);
which means i can disable it in JSON mode :).
--
- stephan beal
http://wanderinghorse.net/home/stephan/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] sqlite3_prepare_v2() is generating output to stderr?
On Fri, Sep 9, 2011 at 6:34 PM, Martin S. Weber wrote: > Really? I'm assuming the JSON output goes to stdout, and the error message > goes to stderr. In that case it wouldn't interfere with the JSON output at > all. Just pipe the output to your json reader without a 2>&1...there'll be > no problem. > Sorry, i wasn't clear - "interfere" meaning "aesthetically ugly." The JSON went where i told it to (stdout) but stderr was getting unexpected output. But i just found the reason, so i'm happy now :). -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On 09:59 PM, fossil-m...@h-rd.org wrote:
Great idea!
Please consider that when using cgi a complete REST over http style
interface is not supported: cgi defines only http GET and POST
DELETE and PUT are not in the cgi spec and handled differently by
different
httpd's.
A common convention is to allow a "_method" query parameter in the URL
to turn a POST into a PUT/DELETE. See
http://microformats.org/wiki/rest/urls .
It strikes me that Fossil is already quite RESTful and its URL structure
is quite conducive to a REST+JSON interface, using the convention of
"file extension implies representation".
e.g. "/timeline" produces HTML by default, but "/timeline.json" would
return the same information in JSON.
Similarly:
- "/wcontent.json" produces a list of all Wiki pages
- "/wiki.json?name=foo" is used to GET, POST, PUT or DELETE a specific
Wiki page
- "/info/HASH.json" produces a json representation of an check-in
- "/finfo.json?name=crypt_wp1.sql" produces a json history of a single file
- "/artifact/HASH.json" produces a json representation of a specific
artifact
I would imagine that it is not terribly difficult to set the default
representation to .html, and to detect requests for .json representation
at quite a high level (it's always the extension on the last component
of the URL), and simply render the output differently for those
resources that support a JSON representation.
With some carefully designed helpers the same logic could be used to
produce JSON or XML or whatever next year's favorite happens to be
(along the lines of rep = "json" /* or whatever */; s = new
StructuredOutput(rep); s.put("name", type, "value"); output = s.render(); )
Regards,
Twylite
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
For me, making it easier to create an IDE plug-in for Fossil would be the greatest benefit. Also, I agree that HTTP status codes should be for transport rather than application errors. On Thu, Sep 8, 2011 at 4:01 PM, Stephan Beal wrote: ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 7:04 PM, Twylite wrote:
> e.g. "/timeline" produces HTML by default, but "/timeline.json" would
> return the same information in JSON.
>
i like that idea. i hadn't thought of simply using an extension. i don't
have a strong opinion as to whether, e.g. /json/stat or /stat.json is
better. Anyone want to give me their own strong opinion? (We could probably
support both - they're stored as string-to-function mappings, but the
current path/arg-handling code would need to be slightly different for each
case.)
Similarly:
> - "/wcontent.json" produces a list of all Wiki pages
>
That's what i'm working on at the moment, actually.
- "/wiki.json?name=foo" is used to GET, POST, PUT or DELETE a specific Wiki
> page
>
i can't say i'm comfortable with handling PUT/DELETE requests, but i say
that because in some 15 years of programming web pages i've never once had
the need for them (and thus don't know off-hand what they're for, their
semantics, etc.).
i know i shot myself in the foot by incorrectly using the term REST. REST is
not truly what i meant, but (as Martin pointed out), "JSON over HTTP." It's
simple, straightforward, and doesn't rely on any
tricky/uncommon/non-standard semantics.
- "/info/HASH.json" produces a json representation of an check-in
> - "/finfo.json?name=crypt_wp1.**sql" produces a json history of a single
> file
> - "/artifact/HASH.json" produces a json representation of a specific
> artifact
>
All good ideas.
I would imagine that it is not terribly difficult to set the default
> representation to .html, and to detect requests for .json representation at
> quite a high level (it's always the extension on the last component of the
> URL), and simply render the output differently for those resources that
> support a JSON representation.
>
Because JSON output has to be "pure JSON" and fossil has a long history of
relying on HTML/console output, i'm having to be somewhat careful in where i
can add/generate output. In particular, error handling can be problematic
because db_err() and friends assume HTML/console modes (which i've just
patched in my copy to generate JSON error responses).
_In general_, but not quite 100%, the various existing commands/web pages
contain one or more SQL commands which we can use as-is for generating json
payloads. In some cases i can merge the two without much work. In some cases
i have to factor the SQL prep into a common routine so that both the
original command and the JSON variant can use it. In one case (/json/stat,
which will now likely be renamed to /stat.json) i ended up copy/pasting the
SQL because for that particular case (with many queries) it's easier to have
two separate impls than to combine them.
With some carefully designed helpers the same logic could be used to produce
> JSON or XML or whatever next year's favorite happens to be (along the lines
> of rep = "json" /* or whatever */; s = new StructuredOutput(rep);
> s.put("name", type, "value"); output = s.render(); )
>
Absolutely. In fact, the JSON layer uses a DOM-like approach. You create a
JSON array/object tree then format it to whatever output channel you want
(it streams input/output via callbacks). Because it's DOM-like, we can
transform it to XML instead of JSON.
FYI:
http://fossil.wanderinghorse.net/repos/fossil-sgb/index.cgi/json/stat
--
- stephan beal
http://wanderinghorse.net/home/stephan/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 7:11 PM, Ron Wilson wrote:
> Also, I agree that HTTP status codes should be for transport rather
> than application errors.
>
:-D. Okay, so there's 2 votes for that.
In my experience (and i've written boatloads of JS- and Java-based Ajax the
past 2 years), i find it more intuitive to write code where i have a clear
separation of transport-vs-app errors.
e.g. if we use jQuery for the Ajax then an HTTP XYZ (any error code) will
trigger the error() AJAX callback, and from there we have no recovery
strategy (and can't say much about what happened to the user). If, OTOH, the
app returns:
{ resultCode:123, resultText:"broken code"}
then success() AJAX callback is called and the client can interrogate
resultCode. In some cases he might be able to retry, whereas others he has
no choice but to fail (e.g. don't retry a failed login, but maybe retry a
failed wiki page fetch 2 or 3 times). Even if he can't retry, assuming the
resultCode is a well-defined value then he can tell the user what the
problem was (which he can't do all that accurately with HTTP codes).
The REST model breaks that pretty badly, IMO.
That said, i recognize the benefit of "vague error codes", but during the
development/debugging i want the codes to be as specific as possible so that
i can track down errors. It also helps when supporting users. But for
production servers i agree that the error codes should be "vaguified" a bit.
The current model supports that but there is no code yet to dumb the codes
down (and the list of codes will still be in flux for a while).
--
- stephan beal
http://wanderinghorse.net/home/stephan/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 11:18 AM, fossil-m...@h-rd.org
wrote:
> Please consider that when using cgi a complete REST over http style
> interface is not supported: cgi defines only http GET and POST
>
> DELETE and PUT are not in the cgi spec and handled differently by different
> httpd's.
fossil server is just a wrapper around fossil cgi, so that would also apply
to server mode.
Personally, i use CGI mode 95% of the time (server mode only locally, and
only occasionally), so CGI is my primary concern here. While there is
arguably little use for JSON in CLI mode, i'm trying to keep it all
structured so that i can use the same code/commands in both CLI and
CGI/server modes (mainly because testing in CLI mode is easier). e.g.:
http://fossil.wanderinghorse.net/repos/fossil-sgb/index.cgi/json/stat
is equivalent to:
stephan@tiny:~/cvs/fossil/fossil-sgb$ ./fossil json stat
{
"fossil":"20ff808f9809541d2eca6c49a17d5cbd16e1b93f",
"timestamp":1315587046,
"payload":{
"projectName":"Fossil",
"repositorySize":21855232,
"blobCount":13620,
"deltaCount":9361,
"uncompressedArtifactSize":589418504,
"averageArtifactSize":43282,
"maxArtifactSize":4620758,
"compressionRatio":"26:1",
"checkinCount":3153,
"fileCount":456,
"wikiPageCount":23,
"ticketCount":940,
"ageDays":1512,
"ageYears":4.139744,
"projectCode":"CE59BB9F186226D80E49D1FA2DB29F935CCA0333",
"serverCode":"2cacd7c3ff350a4cede890b1be494dc3e1cf1b39",
"compiler":"gcc-4.4.5",
"sqlite":{
"version":"2011-09-04 01:27:00 [6b657ae750] (3.7.8)",
"pageCount":21343,
"pageSize":1024,
"freeList":778,
"encoding":"UTF-8",
"journalMode":"delete"
}
}
}
--
- stephan beal
http://wanderinghorse.net/home/stephan/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 7:42 PM, Stephan Beal wrote: > On Fri, Sep 9, 2011 at 7:11 PM, Ron Wilson wrote: > >> Also, I agree that HTTP status codes should be for transport rather >> than application errors. >> > > :-D. Okay, so there's 2 votes for that. > > In my experience (and i've written boatloads of JS- and Java-based Ajax the > past 2 years), i find it more intuitive to write code where i have a clear > separation of transport-vs-app errors. > To stress this point a bit using a real-world example, please take a look at: http://whiki.wanderinghorse.net Poke around the wiki there - the content's not important, just try the various buttons, the editor, and whatnot. Try swapping between pages while editing (by following wiki links, NOT using the browser back button!). And keep in mind that that site's UI is 100% JavaScript/HTML/CSS and the data is 100% JSON served by a CGI. Even the wiki rendering is done on the client. i think that app would have been more troublesome to code if i had had to be able to distinguish REST-style HTTP errors from "real" HTTP errors. The overall goal of this effort is that that type of thing can be done for fossil. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On 09/09/11 13:52, Stephan Beal wrote: (...) While there is arguably little use for JSON in CLI mode, i'm trying to keep it all structured so that i can use the same code/commands in both CLI and CGI/server modes (...) Actually, if the complete CLI functionality was available as JSON output, we'd automatically have our library/frontend model. Think about it, a library is a backend that you communicate with. Typically you call functions in the same address space. The benefit of course is tight integration, compile time safety and what not. The drawback is becoming eminent when you are trying to access it from multiple callers / threads. Do you synchronize, what about locked DBs etc. etc. Now, if your "library" is actually a server that you communicate with, and the CLI frontend-client is doing the same, anybody who's capable of sending a HTTP request to localhost can use all of fossils capabilities and program on top of it. What would then be lacking, of course, would be the simple steps that e.g. sync, commit, list, mv, etc. are being composed of. That might be a not-so-herculian effort in the end to make fossil a library. I haven't given this a lot of thought, just saying: in the end what we would want from a "fossil library" would be that a) the cli uses the same library so that we are able to be first-class citizens and b) finer grained control and c) a separation of concerns (you, fossil library, take care of the fossil stuff, I take care of invoking hooks, presenting a GUI and slapping my users if they try to do something stupid). a) to c) can be achieved with IPC, too, of course (just look at the COM/CORBA model, remove all the bloat and voila, you have over ). Btw, I suggest starting a new thread, called: FOSSILS JSON-RPC INTERFACE ;) Just to get "REST" out of people minds... Regards, -Martin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On 09/09/11 14:15, Stephan Beal wrote: (...) Even the wiki rendering is done on the client. Beautiful. Having this the CLI could finally output text markup (aka wiki markup) on the console instead of outputting HTML! Regards, -Martin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Draft doc for JSON/REST Fossil interface
On Fri, Sep 9, 2011 at 8:46 PM, Martin S. Weber wrote: > Actually, if the complete CLI functionality was available as JSON output, > we'd automatically have our library/frontend model. Think about it, a > library is a backend that you communicate with. i agree with that, but actually using it that way from client scripts seems tedious and inefficient. > What would then be lacking, of course, would be the simple steps that e.g. > sync, commit, list, mv, etc. are being composed of. That might be a > not-so-herculian effort in the end to make fossil a library. > Agreed. In a mail to Richard on the topic i mentioned that once we have a JSON API, we get the lib/app split for free. In a sense, anway. > I haven't given this a lot of thought, just saying: in the end what we > would want ...a GUI and slapping my users if they try to do something > stupid). a) to c) can be achieved with IPC, too, of course (just look at the > COM/CORBA model, remove all the bloat and voila, you have over > ). > CORBA *shudder*!!! That's all true, though. Btw, I suggest starting a new thread, called: FOSSILS JSON-RPC INTERFACE ;) > Just to get "REST" out of people minds... > LOL! It took me all day to get it out of my mind. So are we agreed that REST is not the goal (at least not initially)? Since CGI doesn't support the full range of REST features (PUT/DELETE, apparently), REST support would take me into areas i'm not yet fit to code myself. Regarding your earlier comments about error code leakage to clients: i just added some code to set a "paranoia level" for error reporting. e.g. code 1234 can be returned as any of (1234, 1230, 1200, or 1000). Once again, i appreciate your continued feedback. -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] looking for a fossil wiki query which does...
Hi, all,
i'm looking for a fossil query which will give me the following data for a
wiki page:
{
"name":"PageName1",
"timestamp":int, /* Unix Epoch GMT of check-in */
"artifactId":"...",
"savedBy": "user name", /* can we do this easily? */
"tags": ["tag1", ..."tagN"], /* can a page have tags??? */
"content": "raw or wiki-parsed content, depending on a flag"
}
Could i humbly ask one of the gurus to provide a tip (or SQL) for that? It
needn't be a single statement. Right now i'm only interested in the head
version, but if it's just as much/little work to pull a client-provided
version, that would be cool, too.
:-?
--
- stephan beal
http://wanderinghorse.net/home/stephan/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] JSON API demo app online
Hi, all! http://fossil.wanderinghorse.net/repos/fossil-sgb/json/ That's my test app for the JSON API. As new features/request types are added you'll see them pop up there. Happy Hacking! -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
