Re: [topbraid-users] Reading and mapping JSON files

[Irene Polikoff]

Hi Jack,

We continuously evolve and improve our technologies based on the feedback from 
the marketplace and our users. 

SPARQLMotion and SPINMap are cool, but, for one reason or another, they did not 
get broadly used by our customers. In response to this, we have developed our 
next generation of technology - it is heavily based on SHACL and includes 
GraphQL and Active Data Shapes (ADS). To some extent, SWP as well, but we 
recommend ADS over SWP - it has less of a learning curve and better tooling. 
SPARQLMotion modules themselves continue to get used, but the scripts do not.

The original e-mail below from 2 years ago describes how the ConvertJSONToRDF 
module works. This module does not produce the same output as the 
ConvertXMLToRDF module, so you would need to change your next steps.

I understand that you would prefer to continue to use what you have been using 
starting with 6 or 7 years ago (or may be even longer). However, our advise for 
any new work is to chose what is being currently broadly used, developed and 
supported.

Our current recommendation for converting JSON into RDF is to use ADS 
scripting: 
https://www.topquadrant.com/configuring-topbraid-edg-with-javascript/. The 
other options described in the e-mail below are still applicable, but we 
believe that ADS will be the most optimal in dealing with transformations that 
you may need. It was not yet released in the summer of 2019.

ADS scripts can be developed directly in EDG which you, presumably, could 
launch on the localhost, single user model from TBC - if you kept up to date 
with the current releases. 7.0 release introduces Files option so that you 
could even do this in a file. If you are dealing with large scale data, though, 
we recommend creation of proper EDG ontologies that are separate from the asset 
collections that contain instance data. As I recall, you are in process of 
purchasing EDG server license to upgrade your outdate server version of 
TopBraid Live. Given this, you should be able to readily run these scripts on 
the server.

You will find development information, examples and videos here 
http://www.datashapes.org/active/. If you have some experience with JavaScript, 
it should be easy for you to get started with ADS.

Irene

> On Jun 10, 2021, at 7:41 PM, Jack Hodges  wrote:
> 
> I am doing the same thing and want to convert a SPIN Map mapping that I am 
> using with XML and have been asked to use JSON instead.
> 
> TBCME has a ConvertXMLToRDF SPARQL Motion module that takes plain (but valid) 
> XML and produces a default RDF graph. I then use a SPIN Map to map from that 
> generic RDF to my target schema. This works great.
> 
> I expected the ConvertJSONToRDF module to work the same way; that TBCME would 
> read the JSON, convert it to a generic RDF that would look the same as the 
> previous one (since I converted the XML to JSON to test everything). All I 
> expected to have to do was to swap the ConvertXMLToRDF module with the 
> ConvertJSONToRDF module. If I stop the execution after this module there are 
> a number of instances of the http://topbraid.org/json namespace (which 
> doesn't appear to exist) but the next conversion doesn't work.
> 
> I do not want to use GraphQL or SHACL or SWP to do this, just SPARQL Motion, 
> SPIN Map, and the published modules. Please advise.
> 
> Jack
> 
> On Tuesday, August 20, 2019 at 1:23:43 PM UTC-7 Irene Polikoff wrote:
> Hi Daniel,
> 
> The key piece is to have SHACL Shapes that correspond to the structure of 
> your JSON. They are used to guide the conversion.
> 
> If you already have a GraphQL schema for your JSON, TopBraid EDG can create 
> SHACL Shapes from it. 
> 
> For example, in TBC, you could use Import:
> 
> 
> 
> Technical details behind how this works are here: 
> https://www.topquadrant.com/graphql/graphql-shacl.html 
> 
> 
> 
> sml:ConvertJSONToRDF described below by Holger, has the following arguments:
> 
> sml:keepRootObject (xsd:boolean): [Optional] In GraphQL mode (sml:service is 
> provided) then the root object is typically just a container derived from the 
> Query. By default, the triples of this root object will not be kept. Set to 
> true to keep these triples in the result graph.
> sm:outputVariable (xsd:string): [Optional] The name of the result variable 
> (defaults to "root") that will contain the root object of the converted JSON 
> code.
> sml:service (graphql:Schema): [Optional] A GraphQL service object providing 
> information on how to map the JSON to RDF using SHACL shapes.
> sml:text (xsd:string): The JSON input text, either a JSON object or array.
> 
> If you do not have a GraphQL Schema for your JSON, then you could start by 
> creating a SHACL model that correspond to your JSON. Internally, TopBraid 
> will automatically generate a GraphQL Schema from SHACL that will be used to 
> guide the import and transformation of JSON.
> 
> Technical details of how GraphQL 

Re: [topbraid-users] Reading and mapping JSON files

[Jack Hodges]

I am doing the same thing and want to convert a SPIN Map mapping that I am 
using with XML and have been asked to use JSON instead.

TBCME has a ConvertXMLToRDF SPARQL Motion module that takes plain (but 
valid) XML and produces a default RDF graph. I then use a SPIN Map to map 
from that generic RDF to my target schema. This works great.

I expected the ConvertJSONToRDF module to work the same way; that TBCME 
would read the JSON, convert it to a generic RDF that would look the same 
as the previous one (since I converted the XML to JSON to test everything). 
All I expected to have to do was to swap the ConvertXMLToRDF module with 
the ConvertJSONToRDF module. If I stop the execution after this module 
there are a number of instances of the http://topbraid.org/json namespace 
(which doesn't appear to exist) but the next conversion doesn't work.

I do not want to use GraphQL or SHACL or SWP to do this, just SPARQL 
Motion, SPIN Map, and the published modules. Please advise.

Jack

On Tuesday, August 20, 2019 at 1:23:43 PM UTC-7 Irene Polikoff wrote:

> Hi Daniel,
>
> The key piece is to have SHACL Shapes that correspond to the structure of 
> your JSON. They are used to guide the conversion.
>
> If you already have a GraphQL schema for your JSON, TopBraid EDG can 
> create SHACL Shapes from it. 
>
> For example, in TBC, you could use Import:
>
>
> Technical details behind how this works are here: 
> https://www.topquadrant.com/graphql/graphql-shacl.html
>
>
> sml:ConvertJSONToRDF described below by Holger, has the following 
> arguments:
>
>
>- sml:keepRootObject (xsd:boolean): [Optional] In GraphQL mode 
>(sml:service is provided) then the root object is typically just a 
>container derived from the Query. By default, the triples of this root 
>object will not be kept. Set to true to keep these triples in the result 
>graph.
>- sm:outputVariable (xsd:string): [Optional] The name of the result 
>variable (defaults to "root") that will contain the root object of the 
>converted JSON code.
>- sml:service (graphql:Schema): [Optional] A GraphQL service object 
>providing information on how to map the JSON to RDF using SHACL shapes.
>- sml:text (xsd:string): The JSON input text, either a JSON object or 
>array.
>
>
> If you do not have a GraphQL Schema for your JSON, then you could start by 
> creating a SHACL model that correspond to your JSON. Internally, TopBraid 
> will automatically generate a GraphQL Schema from SHACL that will be used 
> to guide the import and transformation of JSON.
>
> Technical details of how GraphQL Schemas are generated from SHACL are here 
> https://www.topquadrant.com/graphql/shacl-graphql.html. Yes, this is used 
> to query RDF with GraphQL and get JSON out. But it is also used in the 
> other direction, to convert JSON to RDF.
>
> Also see answers below
>
> On Aug 20, 2019, at 1:46 PM, Daniel Lavoie  wrote:
>
> And:
>
>- Can we use TBC Maestro for the development?
>
> Yes. You can develop the import process using the suggested module/API. 
> You can test the script in TBC. However, TBC is a single user, not 
> networked tool. For doing the scheduled processing (assuming this is not a 
> one-of and your JSON comes in on a regular basis), you will need a server.
>
>
>- Can we implement the result in a triple store? 
>
> Yes, sure
>
>
>- Does it have to be EDG triple store or can we use another one like 
>Stardog?
>
>
> The transformation will create RDF from JSON, you can decide what to do 
> with it e.g., write it into EDG’s triple store, export it as file into one 
> of the serialized formats, etc. 
>
> Thanks again!
> Daniel
>
> On Tuesday, August 20, 2019 at 10:49:15 AM UTC-4, Daniel Lavoie wrote:
>>
>> Thank you very much Holger for your answer. I have listened to the 
>> webminar you have suggested, but most of it is about producing JSON from 
>> RDF. However, I want to do the opposite.  I am very interested of learning 
>> more about the JSON to RDF conversion using SHACL.  In fact, I would want 
>> to:
>>
>>- Generate the IRIs as we want (its seems that is possible using 
>>GRAPH QL templates)
>>- Map the JSON field names to Property IRIs.
>>- Validate the JSON fields and values.
>>
>> Where can I find more information about the JSON to RDF conversion using 
>> GraphQL and SHACL?
>>
>> Regards
>> Daniel
>> On Monday, August 19, 2019 at 8:15:16 PM UTC-4, Holger Knublauch wrote:
>>>
>>> Hi Daniel,
>>>
>>> there is a variety of options here.
>>>
>>> Obviously, if the file is JSON-LD then it can be opened like any other 
>>> RDF graph, assuming it ends with .jsonld. JSON-LD has been designed with a 
>>> notion of Contexts that provide some flexibility as an on-the-fly transform 
>>> from many JSON tree structures into JSON-based graph structure.
>>>
>>> If the file is not JSON-LD and you cannot define a suitable JSON-LD 
>>> context then you could use our generic JSON importer that can transform