This would need a lot of details, so bear with me below:
Briefly my XML files look like:
<A name="">
<B name="">
<C name="">
<D name="">
<E name=""/>
<A> can contain <B>, <C> or <D> and B, C or D can contain E. We have 1000s
(currently 3000 in my test data set) of such xml files, of size 50MB on an
average. Its tons of data ! Currently, my database is of ~18GB in size.
Query: /A/*//E/@name/string()
This query, was going OOM, within few mins.
I tried a few ways, of whitelisting, with contain clause, to truncate the
result set. That didn't help too. So, now I am out of ideas. This is giving
JVM 10GB of dedicated memory.
Once, above query works and doesn't go Out Of Memory, I also need
corresponding file names too:
XYZ.xml //E/@name
PQR.xml //E/@name
Let me know if you would need more details, to appreciate the issue ?
- Mansi
On Thu, Nov 6, 2014 at 8:48 AM, Christian Grün <[email protected]>
wrote:
> Hi Mansi,
>
> I think we need more information on the queries that are causing the
> problems.
>
> Best,
> Christian
>
>
>
> On Wed, Nov 5, 2014 at 8:48 PM, Mansi Sheth <[email protected]> wrote:
> > Hello,
> >
> > I have a use case, where I have to extract lots in information from each
> XML
> > in each DB. Something like, attribute values of most of the nodes in an
> XML.
> > For such, queries based goes Out Of Memory with below exception. I am
> giving
> > it ~12GB of RAM on i7 processor. Well I can't complain here since I am
> most
> > definitely asking for loads of data, but is there any way I can get these
> > kinds of data successfully ?
> >
> > mansi-veracode:BigData mansiadmin$ ~/Downloads/basex/bin/basexhttp
> > BaseX 8.0 beta b45c1e2 [Server]
> > Server was started (port: 1984)
> > HTTP Server was started (port: 8984)
> > Exception in thread "qtp2068921630-18" java.lang.OutOfMemoryError: Java
> heap
> > space
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073)
> > at
> >
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > at java.lang.Thread.run(Thread.java:744)
> >
> >
> > --
> > - Mansi
>
--
- Mansi
