[
https://issues.apache.org/jira/browse/STANBOL-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401389#comment-13401389
]
Rupert Westenthaler commented on STANBOL-652:
---------------------------------------------
Hi Pablo
I think the best think to start would be to refactor the benchmarking module so
that is can return Benchmark results as RDF
I would even suggest to convert the Benchmarks to RDF (but still supporting the
Text based format currently used)
urn:benchmark1 a sbc:Benchmark
sbc:content "The INPUT text"
sbc:expects urn:annotation1
sbc:expects urn:annotation2
sbc:complain urn:annotation3
urn:annotation1 a sbc:Annotation
entity <entity-URI>
NOTE: we might add different sbc:Annotation types in future (e.g. to validate
Named Entities detection by NER tools or the detection of the correct language)
Results of a Benchmark execution
urn:benchmarkExecution1 a BenchmarkExecution
sbc:benchmark urn:benchmark1
sbc:num-expects 2^^xsd:int
sbc:succeeded-expects 1^^xsd:int
sbc:num-complains 1^^xsd:int
sbc:succeeded-expects 1^^xsd:int
sbc:content-item <urn:contentitem-{UUID}>
dc:created ... and other metadata
and the detailed results for Annotations
urn:result a sbc:BenchmarkResult
sbc:result-of urn:benchmarkExecution1
sbc:about urn:annotation1
sbc:state sbc:benchmark-state-succeeded (or sbc:benchmark-state-failed)
sbc:enhancement <urn:enhancement-{UUID}>, <urn:enhancement-{UUID}>
Note that a BenchmarkResult may refer to multiple enhancements as a
expected/complained Entity could be suggested by several EntityAnnotation.
The Idea would be that the "BenchmarkExecution" provides most of the
information needed for statistical analyses and the BenchmarkResults allow to
directly go to the Enhancements and look into details.
Batch processing an Analyses could than be easily added - both on the client
side and the server side.
WDYT
Rupert
> Benchmark should report evaluation summary
> ------------------------------------------
>
> Key: STANBOL-652
> URL: https://issues.apache.org/jira/browse/STANBOL-652
> Project: Stanbol
> Issue Type: Improvement
> Components: Testing
> Reporter: Pablo Mendes
> Priority: Minor
> Labels: benchmark, evaluation
>
> The SBC is a nice way to perform manual inspection of the behavior of the
> enhancement chain for different examples in the evaluation dataset. However,
> for evaluations with several hundreds of examples, it would be interesting to
> have scores that summarize the performance for the entire
> dataset. For example, precision, recall and F1. An evaluation dataset is
> available here in BDL: http://spotlight.dbpedia.org/download/stanbol/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
