I’ve implemented my first pass at persisting my “where used” map into a database to serve as an index.
My approach is to create a separate element for each index entry, rather than creating a single element that then contains all the index entries as shown in the index construction example in the docs. My reasoning is that the number of index entries could be quite large so it seemed dangerous to try to create a very large element in memory before doing db:create() or db:replace(). So my question is: Should I be worrying about creating a single very large element before creating a database and adding it to it, as opposed to either creating a very long list of nodes and passing that to db:create() or, as I’m doing now, doing a separate db:replace() for each item, having previously created the database? My main runtime concern is RAM requirements as I’ll be running in a relatively small VM (16GB). My where-used index database takes about 70MB so maybe it’s not concern but I’ve not used BaseX at this level of scale. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kim...@servicenow.com> Date: Friday, January 21, 2022 at 12:31 PM To: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Strategy for Persisting Maps that Contain Nodes: db:node-id() So just create a new database and gin up the appropriate XML structure for what’s in my map (or what I would have put I the map) and then query against that, using node-id() to record the nodes from the database I’m indexing against. That definitely makes sense—I think maybe I have a little bit of ElasticSearch brain damage from a previous job. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Christian Grün <christian.gr...@gmail.com> Date: Friday, January 21, 2022 at 11:53 AM To: Eliot Kimber <eliot.kim...@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Strategy for Persisting Maps that Contain Nodes: db:node-id() [External Email] Hi Eliot, XQuery Maps are very efficient, but they are mostly useful for ad-hoc queries. If you want to create an additional persistent data structure, I would probably create a database in order to benefit from the standard index structures. db:node-id can be helpful indeed. See e.g. [1] for a little example. Best, Christian [1] https://urldefense.com/v3/__https://docs.basex.org/wiki/Indexes*Custom_Index_Structures__;Iw!!N4vogdjhuJM!XPtZc9sLkuddFj8NOxscyQ2d1RxM5t4s6KvswXztMuJgrdZ3bgKUOvmqNZKTNZd82-1ahg$<https://urldefense.com/v3/__https:/docs.basex.org/wiki/Indexes*Custom_Index_Structures__;Iw!!N4vogdjhuJM!XPtZc9sLkuddFj8NOxscyQ2d1RxM5t4s6KvswXztMuJgrdZ3bgKUOvmqNZKTNZd82-1ahg$> > I have large maps that include nodes as entry values. I want to persist these > maps in the DB for subsequent retrieval. > > I believe the best/only strategy is: > > Construct a new map where each node in a value is replaced by its node-id() > value. > Serialize the map to JSON and store as a blob > > To reconstitute it, do the reverse (parse JSON into map, replace node-ids() > with nodes). > > Is my analysis correct? Have I missed any detail or better approach? > > Thanks, > > Eliot
