Grant, Thanks for the feedback!
I have a couple quesitons. 1. How are annotations used for judgments obtained? Separate file, specifed by the user? 2. Can you provide me with a direct link to the TREC format? --Dan On Mon, Sep 20, 2010 at 9:31 AM, Grant Ingersoll <[email protected]>wrote: > > On Sep 15, 2010, at 10:24 PM, Dan Cardin wrote: > > > Hello All, > > > > I am trying to flush out the data importing component. So if you have any > > ideas or feed back go to the wiki or respond to this email. > > > > 1. Should the Open Relevance viewer be capable of importing text and > > images? > > 2. What are some standard formats used for corporas and their > annotation > > sets? > > I don't think there are any. Corpora, you can assume, are already indexed > by the engine. TREC is probably the standard for judgments, but there are > other ways. > > > > > Is the objective of the Open Relevance Viewer to provide a crowd sourcing > > tool that can have its data annotated and then to use the annotated data > for > > determining the performance of machine learning techniques/algorithms? > Or, > > is it to provide a generic crowd souring tool for academics, government, > and > > industry to annotate data with? Or am I missing the point? > > Here's my view of what we need: > > Tool that does a couple of things: > > 1. User can enter queries and then judge the results (as deep as they want, > but at a minimum top 10). All aspects of what they do is captured (the > query, the results, the judgments) > 2. User can give a whole set of queries (i.e. the TREC ones) and provide > judgments. Capture info as always > 3. System should be search engine agnostic with a well defined interface > that allows people to plug in an implementation for their search engine. In > other words, it should be just as easy to judge Google as it is Solr/Lucene. > 4. System should be able to give metrics on the results of both an > individual user run and also, if others have done the run, inter-annotator > runs. Metrics, at a minimum, are: precision, recall, MAP, and for multiple > user setup, inter-annotator agreement. Potentially also mean-reciprocal > rank for known-item searches (user could specify up front) > 5. Should be able to export judgments, etc. to TREC format and other > formats (CSV, XML) > 6. Presumably, there should be an admin-only area which restricts access to > the configuration, etc. It's likely the case that the metrics should be for > admins only, since you don't want end users to be influenced by the results > of others > > Longer term, if we had something to support things like HCIR and other > tests, that would be great. > > For now, 1-6 is a good start, IMO. > > -Grant
