Re: [basex-talk] schema validation
Hi Paul, In BaseX, validation is done on XQuery level. If you want to validate a document before updating it in the database, you can e.g. do something like.. let $db := 'database' let $path := 'collection/doc.xml' let $schema := 'path/to/schema.xsd' let $doc := db:open($db, $path) update ( delete node //text() ) return ( validate:xsd($doc, $schema), db:replace($db, $path, $doc) ) Hope this helps, Christian On Fri, Feb 20, 2015 at 3:33 AM, wrote: > Hi all. > > Is it possible to enforce schema validation on update in BaseX? > > Regards, > Paul Mensonides
[basex-talk] Fwd: Re: schema validation
After Christians mail I realized I pressed the wrong reply button, so I sent I forward my response to the mailing list. But good to see Christian says basically the same :) Cheers, Dirk Forwarded Message Subject: Re: [basex-talk] schema validation Date: Fri, 20 Feb 2015 11:06:53 +0100 From: Dirk Kirsten To: pmens...@comcast.net Hi Paul, no, not directly. You would have to update via your own update functions by using our validation module. You could invoke all updates through such functions, e.g. by using RESTXQ. BaseX itself is not schema-aware. Cheers, Dirk On 02/20/2015 03:33 AM, pmens...@comcast.net wrote: > Hi all. > > Is it possible to enforce schema validation on update in BaseX? > > Regards, > Paul Mensonides > -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22
Re: [basex-talk] Design pattern ...
Hi Marco,
Finally some feedback. As you may already have guessed, there is no
obvious answer to your question. A straightforward query could look as
follows..
let $docs-a := db:open('A1')[ ..condition.. ]
let $paths := $docs-a ! db:path(.)
let $docs-b := $paths ! db:open('A1', .)
return (
db:delete('A1', $paths),
db:add('A2', $docs-a),
db:delete('B1', $paths),
db:add('B2', $docs-b)
)
..but it probably leads to out of memory exceptions. An alternative
could be to serialize the documents to disk and try to add it in a
second step. If this doesn't work out neither, I would indeed propose
to move the documents in chunks. A BaseX command script for that could
be generated via XQuery.
I will think about the implications of db:copy-doc and db:move-doc
functions. It's right that such scripts may not be obvious candidates
for OOM errors, because we already have database representations for
the documents to be moved/copied. Another solution to avoid OOM would
be to write the data structures of copied documents to disk instead of
keeping them in main memory. Mostly a matter of when we find enough
time to make it happen..
Cheers,
Christian
> - 2 dbs (let's say A and B) with around 200K documents each
> - every document in B refers to exactly one document in A (many to one is
> possible)
>
> - I have to apply a filter on every document of A and depending on the
> boolean outcome "move" the document to anoter db A'.
> - According to this the correlated documents of B need to be "moved" to a
> database B'.
> - The solution should be made automatic in the mid term
>
> What is the best design pattern and tool (I use the GUI by now) for this
> finding the right compromise for limiting "Out of memory (huge PUL)", "too
> much time", "complexity introduced by chunking" and "absence of a
> _db:move-document_ operation for moving one document in one single
> operation?
>
> Every suggestion is more then warmly welcome!
>
> Thanks and best regards,
> Marco
[basex-talk] Jetty / HTTP exception
Hi to all, when looking into the logs for all sort of things we noticed that we get a lot of exceptions with the stacktrace shown in [1]. We are heavily using RestXQ to insert documents at a rather high pace. It doesn't compromise the functioning of the system which looks to be behaving correctly. Nevertheless we have the feeling that when it happens we get a slowing down of the delivery of the response to the client. I say feeling because this has yet to be proved. Anyone knows what this exception is about and whether it can be avoided somehow? Maybe tuning Jetty configuration? Thanks a lot, Marco. [1] java.io.IOException: Closed at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:117) at org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:104) at org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) at org.basex.http.restxq.RestXqFunction.process(RestXqFunction.java:104) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:29) at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) java.lang.IllegalStateException: Committed at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1130) at org.basex.http.HTTPContext.status(HTTPContext.java:254) at org.basex.http.BaseXServlet.service(BaseXServlet.java:71) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.par
Re: [basex-talk] Design pattern ...
Thank you Christian,
my plans were to serialize to disk but I'll give a try to the COMMAND
generation approach.
Separating the work in two phases could anyway improve the process from
a safety point of view ...
Will keep you ligned up.
Regards,
Marco.
On 20/02/2015 12:18, Christian Grün wrote:
Hi Marco,
Finally some feedback. As you may already have guessed, there is no
obvious answer to your question. A straightforward query could look as
follows..
let $docs-a := db:open('A1')[ ..condition.. ]
let $paths := $docs-a ! db:path(.)
let $docs-b := $paths ! db:open('A1', .)
return (
db:delete('A1', $paths),
db:add('A2', $docs-a),
db:delete('B1', $paths),
db:add('B2', $docs-b)
)
..but it probably leads to out of memory exceptions. An alternative
could be to serialize the documents to disk and try to add it in a
second step. If this doesn't work out neither, I would indeed propose
to move the documents in chunks. A BaseX command script for that could
be generated via XQuery.
I will think about the implications of db:copy-doc and db:move-doc
functions. It's right that such scripts may not be obvious candidates
for OOM errors, because we already have database representations for
the documents to be moved/copied. Another solution to avoid OOM would
be to write the data structures of copied documents to disk instead of
keeping them in main memory. Mostly a matter of when we find enough
time to make it happen..
Cheers,
Christian
- 2 dbs (let's say A and B) with around 200K documents each
- every document in B refers to exactly one document in A (many to one is
possible)
- I have to apply a filter on every document of A and depending on the
boolean outcome "move" the document to anoter db A'.
- According to this the correlated documents of B need to be "moved" to a
database B'.
- The solution should be made automatic in the mid term
What is the best design pattern and tool (I use the GUI by now) for this
finding the right compromise for limiting "Out of memory (huge PUL)", "too
much time", "complexity introduced by chunking" and "absence of a
_db:move-document_ operation for moving one document in one single
operation?
Every suggestion is more then warmly welcome!
Thanks and best regards,
Marco
Re: [basex-talk] Jetty / HTTP exception
Hi Marco, If you manage to make it reproducible, please give us a note! Thanks, Christian On Fri, Feb 20, 2015 at 12:49 PM, Marco Lettere wrote: > Hi to all, > when looking into the logs for all sort of things we noticed that we get a > lot of exceptions with the stacktrace shown in [1]. > We are heavily using RestXQ to insert documents at a rather high pace. > It doesn't compromise the functioning of the system which looks to be > behaving correctly. Nevertheless we have the feeling that when it happens we > get a slowing down of the delivery of the response to the client. I say > feeling because this has yet to be proved. > Anyone knows what this exception is about and whether it can be avoided > somehow? Maybe tuning Jetty configuration? > Thanks a lot, > Marco. > > [1] > java.io.IOException: Closed > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140) > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:117) > at org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:104) > at org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) > at org.basex.http.restxq.RestXqFunction.process(RestXqFunction.java:104) > at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:29) > at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > java.lang.IllegalStateException: Committed > at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1130) > at org.basex.http.HTTPContext.status(HTTPContext.java:254) > at org.basex.http.BaseXServlet.service(BaseXServlet.java:71) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnecti
[basex-talk] Performance of PUL commit
It's me again sorry. ;-)
One question that puzzles me in these last hours ..
This is the scenario on 2 DBs (same server) with the following sizes:
136746 docs ~1 Gb
194608 docs~0.5 Gb
declare
%rest:path('{$project}/some/url/')
%rest:POST('{$body}')
%rest:consumes("application/xml")
function _:insert($project, $body) {
prof:time(
__:insert_impl($project, $body),
false(),
"POST ACCOUNT")
};
The function insert() is an RestXQ entry point for document insertion.
It delegates to the actual implementing function.
Insertion occurs continuosly.
The implementing function performs queries for checking uniqueness of
what is inserted and then posts the actual update which is a db:add or
db:replace (BTW does this make a difference from a performance point of
view?).
Insertion might occur either on one or on both the DBs.
The time T output by the profile:time is a steady amount of time. This
means that queries and update schedule is working in constant time.
The time reported by Jetty (and perceived at the client) is a T + d.
The issue is that this d may vary from something around few hundreds of
milliseconds to something which is 4 or 5 times T itself.
And this happens in waves. When it's slow, it's slow for few minutes and
then it gets better for a few minutes and then back again ...
When staring at the tcpdump it's evident that all the time is spent
between the ack of the request and the sending of the response. Thus
network overhead is negligible.
This makes me think that the overhead is in the commit of the PUL. Is
this possible?
I'm using basex 8.0 official. Openjdk 7 and monitoring through JMX
doesn't show anything particularly critical.
Any clues on what I have to better look into?
Thanks and sorry for bothering ...
M.
Re: [basex-talk] Performance of PUL commit
Just one more hint, if we parallelly to the ongoing insertions post one
directly with basexclient, we don't experience this lag ...
M.
On 20/02/2015 15:53, Marco Lettere wrote:
It's me again sorry. ;-)
One question that puzzles me in these last hours ..
This is the scenario on 2 DBs (same server) with the following sizes:
136746 docs ~1 Gb
194608 docs~0.5 Gb
declare
%rest:path('{$project}/some/url/')
%rest:POST('{$body}')
%rest:consumes("application/xml")
function _:insert($project, $body) {
prof:time(
__:insert_impl($project, $body),
false(),
"POST ACCOUNT")
};
The function insert() is an RestXQ entry point for document insertion.
It delegates to the actual implementing function.
Insertion occurs continuosly.
The implementing function performs queries for checking uniqueness of
what is inserted and then posts the actual update which is a db:add or
db:replace (BTW does this make a difference from a performance point
of view?).
Insertion might occur either on one or on both the DBs.
The time T output by the profile:time is a steady amount of time. This
means that queries and update schedule is working in constant time.
The time reported by Jetty (and perceived at the client) is a T + d.
The issue is that this d may vary from something around few hundreds
of milliseconds to something which is 4 or 5 times T itself.
And this happens in waves. When it's slow, it's slow for few minutes
and then it gets better for a few minutes and then back again ...
When staring at the tcpdump it's evident that all the time is spent
between the ack of the request and the sending of the response. Thus
network overhead is negligible.
This makes me think that the overhead is in the commit of the PUL. Is
this possible?
I'm using basex 8.0 official. Openjdk 7 and monitoring through JMX
doesn't show anything particularly critical.
Any clues on what I have to better look into?
Thanks and sorry for bothering ...
M.
Re: [basex-talk] BaseX-Talk Digest, Vol 62, Issue 31
Hi Christian, I just query the root element like "/root" from the xml(database). When I was getting the out of memory, the thread did hold more than ~200MB in size. but actual xml size is ~1.2 MB. And that 200MB file have some encrypted data like "òª©" and the same code got repeated many times(probably this would cause the 200 MB). Is there any issue with it? Also I am using the same context for read and write. That is single context is used for read and write operation parallely and read and write operation for the same database is synchronized. please let me know if you need more information in this regard. Thanks,Jothivel On Friday, 20 February 2015 6:58 PM, Christian Grün wrote: Hi Jothivel, > I am getting out of memory error when i do the execute query operation. Sorry, I need more information on your particular query. Unfortunately, we cannot provide free support for older versions. What happens with Version 8.0? > 2. without migrating the basex 7.6 to 8.0, is it possible to avoid the > increase of size. Yes, as I said in my previous mail, we have worked on that in the meanwhile. Hope this helps, Christian PS: Please always write to the basex-talk mailing list; this way, others can benefit from the feedback as well.
Re: [basex-talk] Lock issue while editing a .xql script over WebDAV
And finally fixed. Somewhere, a timeout was set to Long.MAX_VALUE, leading to an out of bounds exception. A new snapshot is available. Thanks, Christian On Tue, Feb 17, 2015 at 9:42 PM, Christian Grün wrote: > …work in progress, thanks (still figuring out if the problem is due to > BaseX or the WebDAV Milton driver). > > > On Sun, Feb 15, 2015 at 6:01 PM, Andreas Jung wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Hi there, >> >> the following bug occurs while re-editing a .xql script. >> >> - - start cadaver against the WebDAV port >> - - inside cadaver: edit some.xsql and save an empty file >> - - re-edit the same file again and this LOCK related issue pops up >> >> Cheers >> Andreas >> >> [qtp43256131-15] INFO com.bradmcevoy.http.HttpManager - PROPFIND :: >> http://localhost:8984/webdav/scripts/foo.xql - >> http://localhost:8984/webdav/scripts/foo.xql >> [qtp43256131-15] INFO com.bradmcevoy.http.HttpManager - PROPFIND :: >> http://localhost:8984/webdav/scripts/foo.xql - >> http://localhost:8984/webdav/scripts/foo.xql >> [qtp43256131-15] INFO com.bradmcevoy.http.HttpManager - OPTIONS :: >> http://localhost:8984/webdav/scripts/foo.xql - >> http://localhost:8984/webdav/scripts/foo.xql >> [qtp43256131-20] INFO com.bradmcevoy.http.HttpManager - LOCK :: >> http://localhost:8984/webdav/scripts/foo.xql - >> http://localhost:8984/webdav/scripts/foo.xql >> com.bradmcevoy.http.exceptions.BadRequestException >> [qtp43256131-20] ERROR com.bradmcevoy.http.StandardFilter - process >> java.lang.NullPointerException >> at >> com.bradmcevoy.http.webdav.LockHandler.processNewLock(LockHandler.java:234) >> at >> com.bradmcevoy.http.webdav.LockHandler.processExistingResource(LockHandler.java:110) >> at >> com.bradmcevoy.http.webdav.LockHandler.process(LockHandler.java:86) >> at com.bradmcevoy.http.StandardFilter.process(StandardFilter.java:52) >> at com.bradmcevoy.http.FilterChain.process(FilterChain.java:40) >> at com.bradmcevoy.http.HttpManager.process(HttpManager.java:228) >> at org.basex.http.webdav.WebDAVServlet.run(WebDAVServlet.java:43) >> at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >> at >> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) >> at >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) >> at >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) >> at >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >> at org.eclipse.jetty.server.Server.handle(Server.java:370) >> at >> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) >> at >> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) >> at >> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) >> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) >> at >> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) >> at >> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >> at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) >> at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >> at java.lang.Thread.run(Unknown Source) >> -BEGIN PGP SIGNATURE- >> Version: GnuPG v1.4.11 (Darwin) >> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ >> >> iQGUBAEBAgAGBQJU4NDtAAoJEADcfz7u4AZjrYYLvA5ssGdFWLFhxjcH+h+2HzUd >> Q/GXKDuHoz1cpzBxYahaur2VbAuVysTWbb1Cr5PkOQMtXgqoNh0IbP/FDprk8VZW >> mqtpIVfPqSEhYtsxzr+5lo2iuppco+K5WOo16UV4+R2tRKi9yOtq5vwhp8mNOBZ1 >> Ottv6Q2iBON6G1DBg6qStNGBJWl1AD+HWEgvc6p0SKL8IWUnp+iaalowK00AFb6y >> Ie2xXZd/5iF9ukSKEs2TeSXyXF6Hx6dZqKRDUFjDZSdvIqcR4w3Z/3AQdseM3oh4 >> gdPMTM7HVen5mqX0DNookE4OPQY2yurNMHWrhAiHNdAisU0PkjdqZ9wjN
Re: [basex-talk] BaseX-Talk Digest, Vol 62, Issue 31
Dear Jothivel, > please let me know if you need more information in this regard. Sure… Just read my previous e-mail again ;) Thanks, Christian > On Friday, 20 February 2015 6:58 PM, Christian Grün > wrote: > > > Hi Jothivel, > >> I am getting out of memory error when i do the execute query operation. > > Sorry, I need more information on your particular query. > > Unfortunately, we cannot provide free support for older versions. What > happens with Version 8.0? > >> 2. without migrating the basex 7.6 to 8.0, is it possible to avoid the >> increase of size. > > Yes, as I said in my previous mail, we have worked on that in the meanwhile. > > > Hope this helps, > Christian > > > PS: Please always write to the basex-talk mailing list; this way, > others can benefit from the feedback as well. > > >
[basex-talk] http:send-request / Digest - problem
Dear BaseX team,
I have problems to get the http:send-request going when using Digest
authentication. I get a response as if the password were wrong, although I
think it isn't.
Request:
{$useMsg}
Response:
http://expath.org/ns/http-client"; status="401"
message="Unauthorized">
Interestingly, a JMeter test using the same URL, user name and password works
correctly. (I used copy & paste to transfer the credentials from the JMeter GUI
into the XQuery code, and the credentials are just what some documentation
tells me; tried of course also typing them in manually.)
Using WireShark, I observed the messages sent by JMeter and BaseX,
respectively. BaseX does send two requests, as it is supposed to do; but the
authorization data sent by the second one are in two respects different from
the data sent by JMeter: JMeter sends a field: algorithm="MD5", which BaseX
does not, and the cnonce is much longer. (See below for the authorization
strings.)
I use Java 8, and I suspect the problem is connected with that. I heard rumours
that Digest authentication with Java 8 might require a little adaptation of the
code. Did you try the module using Java8?
Cheers,Hans-Juergen
PS: (1) Authentication sent by BaseX, failing:
User-Agent:Java/1.8.0_31
Authorization:Digest username=foofoo,
realm=traveltainment,nonce=1424448664069:b6ed9add48830631ae90ad27cfcb5c5e,uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,qop=auth,nc=0001,cnonce=9b2ff1ce6900217dd6be667aa6f99e12,response=4e35b40dd4ba7d62cd6123d9adebc046,opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
Authorization:Digest
username=foofoo,realm=traveltainment,nonce=1424449896135:5fa18cec34de1a15d8ce2a36df77bd6a,uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,qop=auth,nc=0001,cnonce=5be8fce766d843e8ea29936b73ed94c7,response=3757d8fb6cfc4c997030541c58e72d61,opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
(2) Authentication sent by JMeter, successfully (please note the "algorithm"
field; also note the much shorter cnonce):
User-Agent:Apache-HttpClient/4.2.6 (java 1.5)
Autorization:Digest username="foofoo",
realm="traveltainment",nonce="1424448476861:032a9011541d271a429d737844ec860a",uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",response="df6416855e49a0f28cff8020c30ad3a7",qop=auth,nc=0001,cnonce="c4b43ae817866fb5",algorithm="MD5",opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
Authorization:Digest username="foofoo",
realm="traveltainment",
nonce="1424450034667:82c9d9977a208442a7926a948e163e45",
uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",response="74712ad74209c954bfb6e545b5f8670b",qop=auth,nc=0001,cnonce="788dbfb1ed7a77ef",algorithm="MD5",opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
Re: [basex-talk] Lock issue while editing a .xql script over WebDAV
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Grün wrote: > And finally fixed. Somewhere, a timeout was set to Long.MAX_VALUE, > leading to an out of bounds exception. A new snapshot is available. > > confirmed, works now. Thanks Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJU52dWAAoJEADcfz7u4AZjbBoLvRHB/o5NLt2nvvKY9AcnKdN7 pGHFH3+ltQQogiH7Fv0aCT5WnKL+kCobQDIlf1DX2QXYBFNQKMFIxiwJyvfTVeHm rbKoRXMWwJ3TQG2apP/yCqneWQXDDUDXq12YnezkftZNhJAnrqwOBNUghutuXoAB a96nedhKs2zHNGwMgMhdGFAAwGZRsH30Rhjue3cC8YINPSCCUz1TVuau5dXHM7q2 hj5Bq/WHkzGovbi3/bhrCLG6lCLUQTO+CNO1xcTzOjv2h+vuoDEOPB8t/Cnyt8n1 DWf7e0G4VuwwhG3KjuH7ggsG328oher8yHDUU0v5CDlhVb+6B0IRPwqgc73yLwge dO1ngh+B7SKoBcyVCQrmRgftITEJapL/t6+qf4QS5KQBJNfVM7WDHks1os1PMNMA hGishu/onFZHjC+MUufrwzJkLT2KW0AFKYSBfu1GrgAIfTvZdBX+7uZy0kSSe5Zi 5PBa3G5oqy/iul2iGD49vQDbaERZ7JA= =9cVQ -END PGP SIGNATURE-
Re: [basex-talk] http:send-request / Digest - problem
Hi Hans-Jürgen,
Thanks for the in-depth feedback on our client-side digest
authentication. I have now included the algorithm in the client
request (I thought the value was optional [1], but this may well be
wrong). The length of the cnonce string shouldn't matter (..I
think..).
Could you please try the latest snapshot [2] and see if your server
accepts the sent requests? If it doesn't, it may be the missing
quotes..
Thanks in advance!
Christian
[1] http://en.wikipedia.org/wiki/Digest_access_authentication#Overview
[2] http://files.basex.org/releases/latest
On Fri, Feb 20, 2015 at 5:40 PM, Hans-Juergen Rennau wrote:
> Dear BaseX team,
>
> I have problems to get the http:send-request going when using Digest
> authentication. I get a response as if the password were wrong, although I
> think it isn't.
>
> Request:
> username='foofoo' password='secret' auth-method="Digest">
> method='xml'>{$useMsg}
>
>
> Response:
> http://expath.org/ns/http-client";
> status="401" message="Unauthorized">
>
>
>
>
>
>
>
>
>
> Interestingly, a JMeter test using the same URL, user name and password
> works correctly. (I used copy & paste to transfer the credentials from the
> JMeter GUI into the XQuery code, and the credentials are just what some
> documentation tells me; tried of course also typing them in manually.)
>
> Using WireShark, I observed the messages sent by JMeter and BaseX,
> respectively. BaseX does send two requests, as it is supposed to do; but the
> authorization data sent by the second one are in two respects different from
> the data sent by JMeter: JMeter sends a field: algorithm="MD5", which BaseX
> does not, and the cnonce is much longer. (See below for the authorization
> strings.)
>
> I use Java 8, and I suspect the problem is connected with that. I heard
> rumours that Digest authentication with Java 8 might require a little
> adaptation of the code. Did you try the module using Java8?
>
> Cheers,
> Hans-Juergen
>
> PS:
> (1) Authentication sent by BaseX, failing:
>
> User-Agent: Java/1.8.0_31
>
> Authorization: Digest username=foofoo,
> realm=traveltainment,
> nonce=1424448664069:b6ed9add48830631ae90ad27cfcb5c5e,
> uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,
> qop=auth,
> nc=0001,
> cnonce=9b2ff1ce6900217dd6be667aa6f99e12,
> response=4e35b40dd4ba7d62cd6123d9adebc046,
> opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
>
>
> Authorization: Digest username=foofoo,
> realm=traveltainment,
> nonce=1424449896135:5fa18cec34de1a15d8ce2a36df77bd6a,
> uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,
> qop=auth,
> nc=0001,
> cnonce=5be8fce766d843e8ea29936b73ed94c7,
> response=3757d8fb6cfc4c997030541c58e72d61,
> opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
>
> (2) Authentication sent by JMeter, successfully (please note the "algorithm"
> field; also note the much shorter cnonce):
>
> User-Agent: Apache-HttpClient/4.2.6 (java 1.5)
>
> Autorization: Digest username="foofoo",
> realm="traveltainment",
> nonce="1424448476861:032a9011541d271a429d737844ec860a",
> uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",
> response="df6416855e49a0f28cff8020c30ad3a7",
> qop=auth,
> nc=0001,
> cnonce="c4b43ae817866fb5",
> algorithm="MD5",
> opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
>
> Authorization: Digest username="foofoo",
> realm="traveltainment",
> nonce="1424450034667:82c9d9977a208442a7926a948e163e45",
> uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",
> response="74712ad74209c954bfb6e545b5f8670b",
> qop=auth,
> nc=0001,
> cnonce="788dbfb1ed7a77ef",
> algorithm="MD5",
> opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
>
>
Re: [basex-talk] http:send-request / Digest - problem
Yet another update: I added the missing quotes in the client request.
It didn't make a difference with the browsers I tried, but it might
make a difference with other applications..
The new snapshot [1] is available since appr. four seconds,
Christian
[1] http://files.basex.org/releases/latest/
On Fri, Feb 20, 2015 at 6:43 PM, Christian Grün
wrote:
> Hi Hans-Jürgen,
>
> Thanks for the in-depth feedback on our client-side digest
> authentication. I have now included the algorithm in the client
> request (I thought the value was optional [1], but this may well be
> wrong). The length of the cnonce string shouldn't matter (..I
> think..).
>
> Could you please try the latest snapshot [2] and see if your server
> accepts the sent requests? If it doesn't, it may be the missing
> quotes..
>
> Thanks in advance!
> Christian
>
> [1] http://en.wikipedia.org/wiki/Digest_access_authentication#Overview
> [2] http://files.basex.org/releases/latest
>
>
>
> On Fri, Feb 20, 2015 at 5:40 PM, Hans-Juergen Rennau wrote:
>> Dear BaseX team,
>>
>> I have problems to get the http:send-request going when using Digest
>> authentication. I get a response as if the password were wrong, although I
>> think it isn't.
>>
>> Request:
>> > username='foofoo' password='secret' auth-method="Digest">
>> > method='xml'>{$useMsg}
>>
>>
>> Response:
>> http://expath.org/ns/http-client";
>> status="401" message="Unauthorized">
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Interestingly, a JMeter test using the same URL, user name and password
>> works correctly. (I used copy & paste to transfer the credentials from the
>> JMeter GUI into the XQuery code, and the credentials are just what some
>> documentation tells me; tried of course also typing them in manually.)
>>
>> Using WireShark, I observed the messages sent by JMeter and BaseX,
>> respectively. BaseX does send two requests, as it is supposed to do; but the
>> authorization data sent by the second one are in two respects different from
>> the data sent by JMeter: JMeter sends a field: algorithm="MD5", which BaseX
>> does not, and the cnonce is much longer. (See below for the authorization
>> strings.)
>>
>> I use Java 8, and I suspect the problem is connected with that. I heard
>> rumours that Digest authentication with Java 8 might require a little
>> adaptation of the code. Did you try the module using Java8?
>>
>> Cheers,
>> Hans-Juergen
>>
>> PS:
>> (1) Authentication sent by BaseX, failing:
>>
>> User-Agent: Java/1.8.0_31
>>
>> Authorization: Digest username=foofoo,
>> realm=traveltainment,
>> nonce=1424448664069:b6ed9add48830631ae90ad27cfcb5c5e,
>> uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,
>> qop=auth,
>> nc=0001,
>> cnonce=9b2ff1ce6900217dd6be667aa6f99e12,
>> response=4e35b40dd4ba7d62cd6123d9adebc046,
>> opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
>>
>>
>> Authorization: Digest username=foofoo,
>> realm=traveltainment,
>> nonce=1424449896135:5fa18cec34de1a15d8ce2a36df77bd6a,
>> uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,
>> qop=auth,
>> nc=0001,
>> cnonce=5be8fce766d843e8ea29936b73ed94c7,
>> response=3757d8fb6cfc4c997030541c58e72d61,
>> opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
>>
>> (2) Authentication sent by JMeter, successfully (please note the "algorithm"
>> field; also note the much shorter cnonce):
>>
>> User-Agent: Apache-HttpClient/4.2.6 (java 1.5)
>>
>> Autorization: Digest username="foofoo",
>> realm="traveltainment",
>> nonce="1424448476861:032a9011541d271a429d737844ec860a",
>> uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",
>> response="df6416855e49a0f28cff8020c30ad3a7",
>> qop=auth,
>> nc=0001,
>> cnonce="c4b43ae817866fb5",
>> algorithm="MD5",
>> opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
>>
>> Authorization: Digest username="foofoo",
>> realm="traveltainment",
>> nonce="1424450034667:82c9d9977a208442a7926a948e163e45",
>> uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",
>> response="74712ad74209c954bfb6e545b5f8670b",
>> qop=auth,
>> nc=0001,
>> cnonce="788dbfb1ed7a77ef",
>> algorithm="MD5",
>> opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
>>
>>
Re: [basex-talk] http:send-request / Digest - problem
Cordial thanks, Christian, for the prompt reaction! It is a pity that I cannot
make the test now, as I am at home where I have no access to the service in
question. The test will be the first thing to do Monday morning - looking
forward to it, and I shall report immediately!
Cheers,Hans-Juergen
Christian Grün schrieb am 19:17 Freitag,
20.Februar 2015:
Yet another update: I added the missing quotes in the client request.
It didn't make a difference with the browsers I tried, but it might
make a difference with other applications..
The new snapshot [1] is available since appr. four seconds,
Christian
[1] http://files.basex.org/releases/latest/
On Fri, Feb 20, 2015 at 6:43 PM, Christian Grün
wrote:
> Hi Hans-Jürgen,
>
> Thanks for the in-depth feedback on our client-side digest
> authentication. I have now included the algorithm in the client
> request (I thought the value was optional [1], but this may well be
> wrong). The length of the cnonce string shouldn't matter (..I
> think..).
>
> Could you please try the latest snapshot [2] and see if your server
> accepts the sent requests? If it doesn't, it may be the missing
> quotes..
>
> Thanks in advance!
> Christian
>
> [1] http://en.wikipedia.org/wiki/Digest_access_authentication#Overview
> [2] http://files.basex.org/releases/latest
>
>
>
> On Fri, Feb 20, 2015 at 5:40 PM, Hans-Juergen Rennau wrote:
>> Dear BaseX team,
>>
>> I have problems to get the http:send-request going when using Digest
>> authentication. I get a response as if the password were wrong, although I
>> think it isn't.
>>
>> Request:
>> > username='foofoo' password='secret' auth-method="Digest">
>> > method='xml'>{$useMsg}
>>
>>
>> Response:
>> http://expath.org/ns/http-client";
>> status="401" message="Unauthorized">
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Interestingly, a JMeter test using the same URL, user name and password
>> works correctly. (I used copy & paste to transfer the credentials from the
>> JMeter GUI into the XQuery code, and the credentials are just what some
>> documentation tells me; tried of course also typing them in manually.)
>>
>> Using WireShark, I observed the messages sent by JMeter and BaseX,
>> respectively. BaseX does send two requests, as it is supposed to do; but the
>> authorization data sent by the second one are in two respects different from
>> the data sent by JMeter: JMeter sends a field: algorithm="MD5", which BaseX
>> does not, and the cnonce is much longer. (See below for the authorization
>> strings.)
>>
>> I use Java 8, and I suspect the problem is connected with that. I heard
>> rumours that Digest authentication with Java 8 might require a little
>> adaptation of the code. Did you try the module using Java8?
>>
>> Cheers,
>> Hans-Juergen
>>
>> PS:
>> (1) Authentication sent by BaseX, failing:
>>
>> User-Agent: Java/1.8.0_31
>>
>> Authorization: Digest username=foofoo,
>> realm=traveltainment,
>> nonce=1424448664069:b6ed9add48830631ae90ad27cfcb5c5e,
>> uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,
>> qop=auth,
>> nc=0001,
>> cnonce=9b2ff1ce6900217dd6be667aa6f99e12,
>> response=4e35b40dd4ba7d62cd6123d9adebc046,
>> opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
>>
>>
>> Authorization: Digest username=foofoo,
>> realm=traveltainment,
>> nonce=1424449896135:5fa18cec34de1a15d8ce2a36df77bd6a,
>> uri=/TTXml-1.6/Dispatcher/Search/Package/RegionList,
>> qop=auth,
>> nc=0001,
>> cnonce=5be8fce766d843e8ea29936b73ed94c7,
>> response=3757d8fb6cfc4c997030541c58e72d61,
>> opaque=pcyalPBREdBtyQHIHRxOQPgGANZiOjeW
>>
>> (2) Authentication sent by JMeter, successfully (please note the "algorithm"
>> field; also note the much shorter cnonce):
>>
>> User-Agent: Apache-HttpClient/4.2.6 (java 1.5)
>>
>> Autorization: Digest username="foofoo",
>> realm="traveltainment",
>> nonce="1424448476861:032a9011541d271a429d737844ec860a",
>> uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",
>> response="df6416855e49a0f28cff8020c30ad3a7",
>> qop=auth,
>> nc=0001,
>> cnonce="c4b43ae817866fb5",
>> algorithm="MD5",
>> opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
>>
>> Authorization: Digest username="foofoo",
>> realm="traveltainment",
>> nonce="1424450034667:82c9d9977a208442a7926a948e163e45",
>> uri="/TTXml-1.6/Dispatcher/Search/Package/RegionList",
>> response="74712ad74209c954bfb6e545b5f8670b",
>> qop=auth,
>> nc=0001,
>> cnonce="788dbfb1ed7a77ef",
>> algorithm="MD5",
>> opaque="pcyalPBREdBtyQHIHRxOQPgGANZiOjeW"
>>
>>
Re: [basex-talk] Fwd: Re: schema validation
Thanks for the replies, guys. I need the schema enforcement up the hierarchy from what is being inserted to enforce uniqueness constraints (customer orders) as well as referential integrity (references to workcenters, suppliers, customers, etc.). Pulling the entire document and then validating would essentially be pulling the entire database--which would be ever-growing and won't scale. Schema-on-read is not sufficient for this application (the data in the database must be valid as it is used in realtime contexts), so I believe I will have to store the data in a relational database instead. :( I can imagine there would be difficulties in implementing such a thing in the database, so I am not complaining. :) Regards, Paul Mensonides - Original Message - From: "Dirk Kirsten" To: basex-talk@mailman.uni-konstanz.de Sent: Friday, February 20, 2015 3:15:52 AM Subject: [basex-talk] Fwd: Re: schema validation After Christians mail I realized I pressed the wrong reply button, so I sent I forward my response to the mailing list. But good to see Christian says basically the same :) Cheers, Dirk Forwarded Message Subject: Re: [basex-talk] schema validation Date: Fri, 20 Feb 2015 11:06:53 +0100 From: Dirk Kirsten To: pmens...@comcast.net Hi Paul, no, not directly. You would have to update via your own update functions by using our validation module. You could invoke all updates through such functions, e.g. by using RESTXQ. BaseX itself is not schema-aware. Cheers, Dirk On 02/20/2015 03:33 AM, pmens...@comcast.net wrote: > Hi all. > > Is it possible to enforce schema validation on update in BaseX? > > Regards, > Paul Mensonides > -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22
