Serializing Model worked for me using Spark Dataset API. Model was not
associated with any database.
Models were constructed from turtle string and then transformed to
another serializable form.
...
private final Encoder<Model> MODEL_ENCODER = Encoders.bean(Model.class);
...
.mapPartitions((Iterator<String> it) -> {...}, Encoders.STRING())
.map(ttl -> {
Model m =
ModelFactory.createDefaultModel().read(IOUtils.toInputStream(ttl,
"UTF-8"), null, "TTL");
m = getProperties(m);
m = populateLabels(m);
return m;
}, MODEL_ENCODER)
.flatMap(this::extractRelationalSentences, RELATION_ENCODER)
...
Full code here [1]
Siddhesh
[1]
https://github.com/SiddheshRane/sparkl/blob/0bd5b267ffaffdc2dc2d7e59a5b07f09706be8d2/src/main/java/siddhesh/sparkl/TagArticlesFromFile.java#L149
On Sat, Jun 8, 2019 at 5:05 PM Andy Seaborne <a...@apache.org> wrote:
>
> Hi Jason
>
> On 08/06/2019 11:49, Scarlet Remilia wrote:
> > Hello everyone,
> >
> >
> >
> > I changed model to triples in RDD/Dataset, but there is a question.
> >
> > I have triples in Dataset of Spark now, and I need to put them into a Model
> > or something else ,then output them into a file or TDB or somewhere else.
> >
> > As Dan mentioned before, is there any binary syntax for RDF?
>
> https://jena.apache.org/documentation/io/rdf-binary.html
> org.apache.jena.riot.thrift.*
>
> > Or Is Jena supported distributed model to handling billions
> > triples?(supporting parsing triples into a RDF file is OK).TDB’s MRSW is a
> > quite problem for me.
>
> (It's MR+SW - multiple reader AND single writer)
>
> Are you wanting to load smallish units of triples from multiple sources?
>
> Maybe you want to have all the streams send their output to a queue (in
> blocks, not triple by triple) and have TDB load from that queue.
> Multiple StreamRDF to a single StreamRDF, load the StreamTDB.
>
> There is the TDB2 parallel loader - that is, loading from a single
> source using internal parallelism, not loading from parallel inputs.
> (It's 5 threads for triples, more for quads). It load from a StreamRDF.
>
> NB - it can consume all the server's I/O bandwidth and a lot of CPU to
> make the machine unusable for anything else. It is quite hardware dependent.
>
> Andy
>
> >
> >
> >
> > Thank you very much!
> >
> > Jason
> >
> >
> >
> > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows
> > 10
> >
> >
> >
> > ________________________________
> > From: Andy Seaborne <a...@apache.org>
> > Sent: Thursday, June 6, 2019 6:35:41 PM
> > To: users@jena.apache.org
> > Subject: Re: Jena Model is serializable in Java?
> >
> >
> >
> > On 06/06/2019 08:57, Scarlet Remilia wrote:
> >> Hello everyone,
> >>
> >> My use case is a r2rml implementation, which could support millions or
> >> billions rows from RDBMS and distributed parse them into RDF.
> >> For now, We try to setup some small models in different spark executors to
> >> parse individually, and finally union them all.
> >
> > That sounds more like a stream usage.
> >
> > Jena's StreamRDF and collect to a set (model or graph don't sound like
> > they do anything for your application - sound like you are just using
> > them as container of triples to move around.
> >
> >> I think RDD[Triple] is a good idea, but I need to review exist code to
> >> change model into triples.
> >>
> >> an RDF syntax and write-then-read the RDF is also a resolution but is too
> >> loose. It’s very hard to manage these files, especially there are too many
> >> small models mentioned above.
> >>
> >> Thanks,
> >> Jason
> >>
> >> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows
> >> 10
> >>
> >> From: Lorenz B.<mailto:buehm...@informatik.uni-leipzig.de>
> >> Sent: Thursday, June 6, 2019 15:32
> >> To: users@jena.apache.org<mailto:users@jena.apache.org>
> >> Subject: Re: Jena Model is serializable in Java?
> >>
> >> I don't see why one would want to share Model instances via Spark. I
> >> mean, it's possible via wrapping it inside an object which is
> >> serializable or some other wrapper method:
> >>
> >> object ModelWrapper extends Serializable {
> >> lazy val model = ...
> >> }
> >>
> >> rdd.map(s => ModelWrapper.model. ... )
> >>
> >>
> >> This makes the model being attached to some static code that can't be
> >> changed during runtime and that's what Spark needs.
> >>
> >> Ideally, you'd use some broadcast variable, but indeed those are just
> >> use to share smaller entities among the different Spark workers. For
> >> smaller models like a schema this would work and is supposed to be more
> >> efficient than having joins etc. (yes, there are also broadcast joins in
> >> Spark, still data would be distributed during processing) - but it
> >> depends ...
> >>
> >> I don't know your use-case nor why you need a Model, but what we did
> >> when using Jena on Spark was to use RDD (or Dataset) of Triple objects,
> >> i.e. RDD[Triple]. RDD is the fundamental shared datastructure of Spark
> >> and this is the only way to scale when using very large datasets.
> >> Parsing RDF triples from e.g. N-Triples directly into RDD[Triple] is
> >> pretty easy. For Dataset you have to define a custom encoder (Kryo
> >> encoder works though).
> >>
> >> But as already mentioned, your use-case or application would be needed
> >> to give further advice if necessary.
> >>
> >>> Jason,
> >>>
> >>> I would argue that you should exchange a Set of triples, so you can take
> >>> advantage of Spark's distributed nature. Your logic can materialize that
> >>> list into a Graph or Model when needed to operate on it. Andy is right
> >>> about being careful about the size - you may want to build a specialized
> >>> set that throws if the set is too large, and you may want to experiment
> >>> with it.
> >>>
> >>> Andy,
> >>>
> >>> Does Jena Riot (or contrib) provide a binary syntax for RDF that is
> >>> optimal
> >>> for fast parse? I'm recalling Michael Stonebraker's response to the
> >>> BigTable paper -
> >>> https://pdfs.semanticscholar.org/08d1/2e771d811bcd0d4bc81fa3993563efbaeadb.pdf,
> >>> and also gSOAP and other binary XML formats. To this paper, the Google
> >>> BigTable authors then responded that they don't use loose serializations
> >>> such as provided by HDFS, but instead use structured data.
> >>>
> >>> This is hugely important to Jason's question because this is one of the
> >>> benefits of using Spark instead of HDFS - Spark will handle distributing a
> >>> huge dataset to multiple systems so that algorithm authors can operate on
> >>> a
> >>> vector (of Jena models?) far too large to fit in one machine.
> >>>
> >>> On Wed, Jun 5, 2019 at 4:40 PM Andy Seaborne <a...@apache.org> wrote:
> >>>
> >>>> Hi Jason,
> >>>>
> >>>> Models aren't serializable, nor are Graphs (the more system oriented
> >>>> view of RDF) through Triples, Quads and Node are serializable. You can
> >>>> send a list of triples.
> >>>>
> >>>> Or use an RDF syntax and write-then-read the RDF.
> >>>>
> >>>> But are the models small? RDF graph aren't always small so moving them
> >>>> around may be expensive.
> >>>>
> >>>> Andy
> >>>>
> >>>> On 05/06/2019 17:59, Scarlet Remilia wrote:
> >>>>> Hello everyone,
> >>>>> I get a problem about Jena and Spark.
> >>>>> I use Jena Model to handle some RDF models in my spark executor, but I
> >>>> get a error:
> >>>>> java.io.NotSerializableException:
> >>>> org.apache.jena.rdf.model.impl.ModelCom
> >>>>> Serialization stack:
> >>>>> - object not serializable (class:
> >>>> org.apache.jena.rdf.model.impl.ModelCom)
> >>>>> - field (class: org.nari.r2rml.entities.Template, name:
> >>>>> model,
> >>>> type: interface org.apache.jena.rdf.model.Model)
> >>>>> - object (class org.nari.r2rml.entities.Template,
> >>>> org.nari.r2rml.entities.Template@23dc70c1)
> >>>>> - field (class: org.nari.r2rml.entities.PredicateObjectMap,
> >>>> name: objectTemplate, type: class org.nari.r2rml.entities.Template)
> >>>>> - object (class org.nari.r2rml.entities.PredicateObjectMap,
> >>>> org.nari.r2rml.entities.PredicateObjectMap@2de96eba)
> >>>>> - writeObject data (class: java.util.ArrayList)
> >>>>> - object (class java.util.ArrayList,
> >>>> [org.nari.r2rml.entities.PredicateObjectMap@2de96eba])
> >>>>> - field (class: org.nari.r2rml.entities.LogicalTableMapping,
> >>>> name: predicateObjectMaps, type: class java.util.ArrayList)
> >>>>> - object (class org.nari.r2rml.entities.LogicalTableMapping,
> >>>> org.nari.r2rml.entities.LogicalTableMapping@8e00c02)
> >>>>> - field (class:
> >>>>> org.nari.r2rml.beans.Impl.EachPartitonFunction,
> >>>> name: logicalTableMapping, type: class
> >>>> org.nari.r2rml.entities.LogicalTableMapping)
> >>>>> - object (class
> >>>>> org.nari.r2rml.beans.Impl.EachPartitonFunction,
> >>>> org.nari.r2rml.beans.Impl.EachPartitonFunction@1e14b269)
> >>>>> - field (class:
> >>>> org.apache.spark.sql.Dataset$$anonfun$foreachPartition$2, name: func$4,
> >>>> type: interface
> >>>> org.apache.spark.api.java.function.ForeachPartitionFunction)
> >>>>> - object (class
> >>>> org.apache.spark.sql.Dataset$$anonfun$foreachPartition$2, <function1>)
> >>>>> at
> >>>> org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
> >>>>> at
> >>>> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
> >>>>> at
> >>>> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> >>>>> at
> >>>> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:400)
> >>>>> ... 33 more
> >>>>>
> >>>>> All these classes implement serializable interface.
> >>>>> So how could I serialize Jena model java object?
> >>>>>
> >>>>> Thanks very much!
> >>>>>
> >>>>>
> >>>>> Jason
> >>>>>
> >>>>> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> >>>> Windows 10
> >>>>>
> >> --
> >> Lorenz Bühmann
> >> AKSW group, University of Leipzig
> >> Group: http://aksw.org - semantic web research center
> >>
> >>
> >
--
Your greatest regret is the email ID you choose in 8th grade
