Re: [basex-talk] limiting time or space resources in calls to xquery:eval() ?
> I'm contemplating the construction of an interface for advanced or
> dedicated users of a database, with a text box in which they type
> their queries as XQuery modules. (Non-advanced and non-dedicated
> users will make do with a variety of pre-defined queries; this interface
> is intended to provide an open-ended query interface for the few
> users who will need it.)
If possible, xquery:eval() should be avoided for such operations (we
may eventually rename it to evil()). The solution which you find on
our homepage [1] is based on our REST interface, and a user whose
permissions are restricted to reading the example databases. This way,
queries like "file:list('.')" will be rejected. The query timeout
(which doesn’t apply to admin queries [2]) has been set to 10 seconds.
There is currently no way to restrict memory resources in this demo,
because the query will run in the same virtual machine as the server
instance. One solution could be to start a new BaseX (server) instance
with limited memory (-Xmx).
Feedback from other users is welcome.
Christian
[1] http://basex.org/products/live-demo/
[2] http://docs.basex.org/wiki/Options#TIMEOUT
___
BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
[basex-talk] limiting time or space resources in calls to xquery:eval() ?
I'm contemplating the construction of an interface for advanced or dedicated users of a database, with a text box in which they type their queries as XQuery modules. (Non-advanced and non-dedicated users will make do with a variety of pre-defined queries; this interface is intended to provide an open-ended query interface for the few users who will need it.) One issue that arises is security: any use of an eval() function opens the door to code injection attacks. For now, I'm inclined to think that BaseX's rule that xquery:eval() does not accept updating expressions suffices to protect the database from harm. Parsing the query and declining potentially harmful queries is also a possibility, if I can persuade myself I can distinguish harmful from harmless queries. (Comments and advice on this topic welcome, even though it's not the focus of this question.) A second issue is resource usage. Imagine a hostile user who writes a query designed to tie up the server for a long time and consume a lot of memory. Or imagine a non-hostile but naive user who unintentionally concocts an extremely expensive query. Is there a way to put a call to eval() (or any call to the BaseX HTTP server) into some kind of box and specify limits on the amount of CPU time, the amount of clock time, and/or the amount of storage to be available for its evaluation? Michael -- * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ___ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Re: [basex-talk] Out of Main Memory; bytea type
Hi Christian,
after further reasearch I found one way it works:
> let $prepared := sql:prepare($conn, "INSERT INTO imagetest VALUES (?,
> decode(?, 'base64'))")
> let $params :=
> {$id}
> type='string'>{$image}
>
> return sql:execute-prepared($prepared, $params)
where $image contains a xs:base64Binary. This way PostgreSQL itself handles its
type bytea and there is no need for a special parameter type. To receive your
data from the table again, you may use
> SELECT i.id, encode(i.image, 'base64') FROM imagetest i;
Nonetheless, thank you very much for your support.
Best,
Alex
--
| Alexander von Bernuth
| alexander.von-bern...@student.uni-tuebingen.de
Am 24.06.2013 um 16:58 schrieb Christian Grün:
> Hi Alex,
>
> the mapping of types is defined in the BaseX FNSql class [1]. "bytea"
> seems to be a PostgreSQL-specific data type, so I’m not sure which
> mapping would be appropriate here. Could you do some research for us
> and try to find out which SQL types may give satifying results (see
> [2] for the existing setters)?
>
> Thanks,
> Christian
>
> [1]
> https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/query/func/FNSql.java
> [2] http://docs.oracle.com/javase/6/docs/api/java/sql/PreparedStatement.html
> ___
>
>> thank you very much, I am going to test this as soon as I am at home. I
>> think this is going to fix my issue.
>> However, do you happen to know which sql:parameter type I have to use when I
>> try to insert xs:base64binary into my bytea-column in postgres?
>>
>> Thank you again,
>> Alex
>>
>> --
>> | Alexander von Bernuth
>> | alexander.von-bern...@student.uni-tuebingen.de
>>
>> Am 24.06.2013 um 11:48 schrieb Christian Grün:
>>
>> Hi Alexander,
>>
>> how does your XQuery/BaseX script look like? If you use the XQuery
>> doc() function, you could try to replace it with
>> parse-xml(fetch:text(...)), because the latter approach will close
>> your documents and free memory if the processed document is not
>> required anymore.
>>
>> Best,
>> Christian
>> ___
>>
>> 2013/6/24 Alexander von Bernuth
>> :
>>
>> Hello all,
>>
>>
>> my basex-script should fetch 10.000something XML-files automatically from a
>>
>> website and insert their content into a external PostgreSQL-database. After
>>
>> about 8.000 files my script stops and I get "Out of Main Memory".
>>
>> I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm
>>
>> not sure whether these options apply to my problem - I do not build a
>>
>> basex-database but an external one. Will autoflush=false and flushing by
>>
>> myself help with this?
>>
>>
>> Second, I want to insert some xs:base64Binary into my PostgreSQL database,
>>
>> but I cannot find the correct sql:parameter type for the bytea-column.
>>
>>
>> Could you please help me with my issues?
>>
>>
>> Thank you very much,
>>
>> Alexander
>>
>>
>>
>>
>> [1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540
>>
>>
>>
>>
>> --
>>
>> | Alexander von Bernuth
>>
>> | alexander.von-bern...@student.uni-tuebingen.de
>>
>>
>>
>> ___
>>
>> BaseX-Talk mailing list
>>
>> BaseX-Talk@mailman.uni-konstanz.de
>>
>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
>>
>>
>>
___
BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Re: [basex-talk] Out of Main Memory; bytea type
Hi Alex, the mapping of types is defined in the BaseX FNSql class [1]. "bytea" seems to be a PostgreSQL-specific data type, so I’m not sure which mapping would be appropriate here. Could you do some research for us and try to find out which SQL types may give satifying results (see [2] for the existing setters)? Thanks, Christian [1] https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/query/func/FNSql.java [2] http://docs.oracle.com/javase/6/docs/api/java/sql/PreparedStatement.html ___ > thank you very much, I am going to test this as soon as I am at home. I > think this is going to fix my issue. > However, do you happen to know which sql:parameter type I have to use when I > try to insert xs:base64binary into my bytea-column in postgres? > > Thank you again, > Alex > > -- > | Alexander von Bernuth > | alexander.von-bern...@student.uni-tuebingen.de > > Am 24.06.2013 um 11:48 schrieb Christian Grün: > > Hi Alexander, > > how does your XQuery/BaseX script look like? If you use the XQuery > doc() function, you could try to replace it with > parse-xml(fetch:text(...)), because the latter approach will close > your documents and free memory if the processed document is not > required anymore. > > Best, > Christian > ___ > > 2013/6/24 Alexander von Bernuth > : > > Hello all, > > > my basex-script should fetch 10.000something XML-files automatically from a > > website and insert their content into a external PostgreSQL-database. After > > about 8.000 files my script stops and I get "Out of Main Memory". > > I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm > > not sure whether these options apply to my problem - I do not build a > > basex-database but an external one. Will autoflush=false and flushing by > > myself help with this? > > > Second, I want to insert some xs:base64Binary into my PostgreSQL database, > > but I cannot find the correct sql:parameter type for the bytea-column. > > > Could you please help me with my issues? > > > Thank you very much, > > Alexander > > > > > [1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540 > > > > > -- > > | Alexander von Bernuth > > | alexander.von-bern...@student.uni-tuebingen.de > > > > ___ > > BaseX-Talk mailing list > > BaseX-Talk@mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > ___ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Re: [basex-talk] Out of Main Memory; bytea type
Hi Christian, thank you very much, I am going to test this as soon as I am at home. I think this is going to fix my issue. However, do you happen to know which sql:parameter type I have to use when I try to insert xs:base64binary into my bytea-column in postgres? Thank you again, Alex -- | Alexander von Bernuth | alexander.von-bern...@student.uni-tuebingen.de Am 24.06.2013 um 11:48 schrieb Christian Grün: > Hi Alexander, > > how does your XQuery/BaseX script look like? If you use the XQuery > doc() function, you could try to replace it with > parse-xml(fetch:text(...)), because the latter approach will close > your documents and free memory if the processed document is not > required anymore. > > Best, > Christian > ___ > > 2013/6/24 Alexander von Bernuth > : >> Hello all, >> >> my basex-script should fetch 10.000something XML-files automatically from a >> website and insert their content into a external PostgreSQL-database. After >> about 8.000 files my script stops and I get "Out of Main Memory". >> I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm >> not sure whether these options apply to my problem - I do not build a >> basex-database but an external one. Will autoflush=false and flushing by >> myself help with this? >> >> Second, I want to insert some xs:base64Binary into my PostgreSQL database, >> but I cannot find the correct sql:parameter type for the bytea-column. >> >> Could you please help me with my issues? >> >> Thank you very much, >> Alexander >> >> >> >> [1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540 >> >> >> >> -- >> | Alexander von Bernuth >> | alexander.von-bern...@student.uni-tuebingen.de >> >> >> ___ >> BaseX-Talk mailing list >> BaseX-Talk@mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> ___ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Re: [basex-talk] Out of Main Memory; bytea type
Hi Alexander, how does your XQuery/BaseX script look like? If you use the XQuery doc() function, you could try to replace it with parse-xml(fetch:text(...)), because the latter approach will close your documents and free memory if the processed document is not required anymore. Best, Christian ___ 2013/6/24 Alexander von Bernuth : > Hello all, > > my basex-script should fetch 10.000something XML-files automatically from a > website and insert their content into a external PostgreSQL-database. After > about 8.000 files my script stops and I get "Out of Main Memory". > I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm > not sure whether these options apply to my problem - I do not build a > basex-database but an external one. Will autoflush=false and flushing by > myself help with this? > > Second, I want to insert some xs:base64Binary into my PostgreSQL database, > but I cannot find the correct sql:parameter type for the bytea-column. > > Could you please help me with my issues? > > Thank you very much, > Alexander > > > > [1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540 > > > > -- > | Alexander von Bernuth > | alexander.von-bern...@student.uni-tuebingen.de > > > ___ > BaseX-Talk mailing list > BaseX-Talk@mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > ___ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
[basex-talk] Out of Main Memory; bytea type
Hello all, my basex-script should fetch 10.000something XML-files automatically from a website and insert their content into a external PostgreSQL-database. After about 8.000 files my script stops and I get "Out of Main Memory". I found your discussion with "kgfhjjgrn" [1] regarding this issue, but I'm not sure whether these options apply to my problem - I do not build a basex-database but an external one. Will autoflush=false and flushing by myself help with this? Second, I want to insert some xs:base64Binary into my PostgreSQL database, but I cannot find the correct sql:parameter type for the bytea-column. Could you please help me with my issues? Thank you very much, Alexander [1] http://comments.gmane.org/gmane.text.xml.basex.talk/2540 -- | Alexander von Bernuth | alexander.von-bern...@student.uni-tuebingen.de ___ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Re: [basex-talk] BaseX and XML 1.0/1.1 names (and other implementation-defined items)
Dear Michael, > Which version of XML NCNames does BaseX use? The one given > in XML 1.0 5th edition and in XML 1.1? or the one given in earlier > versions of XML? BaseX supports XML 1.0 5th Edition. > And while I'm looking for implementation-defined properties -- is there a > place in the documentation that lists BaseX's decisions on the various > implementation-defined items listed in > http://www.w3.org/TR/xpath20/#id-impl-defined-items and > http://www.w3.org/TR/xquery/#id-impl-defined-items ? Currently no; instead, BaseX-specific features are mentioned on different pages in the Wiki. Some more details (I may add them to the Wiki if I find some time): 1. The Unicode version depends on the used Java version. 2. We have recently added support for additional collations [1]. 3. The implicit timezone depends on the user location. 4. Warnings are output the same way as compilation information. 5. The way errors are reported completely depends on the way how BaseX is used. 6. We provide support for XML 1.0.5. 7. Various module namespaces are statically bound [2] 8. All XPath axes are supported. 9. The default is "empty least" 10. Options can be bound via pragmas [3] 11. Options can be bound via option declarations [3] 12. Not sure about this, but "Java Bindings" may be the answer [4] 13. Our Repository page explains the procedure [5] 14. No static typing extensions are supported 15. The Serialization page gives details [6] 16. Same here [6] 17. Leads to a dynamic error in XQuery 18. Integers are limited to long values; limits for other values usually depend on Java 19. Syntactic extenstions: "using fuzzy" [7], map extension [8] Hope this helps, Christian [1] http://docs.basex.org/wiki/Full-Text#Collations [2] http://docs.basex.org/wiki/Module_Library [3] http://docs.basex.org/wiki/Options [4] http://docs.basex.org/wiki/Java_Bindings [5] http://docs.basex.org/wiki/Repository [6] http://docs.basex.org/wiki/Serialization [7] http://docs.basex.org/wiki/Full-Text#Fuzzy_Querying [8] http://docs.basex.org/wiki/Map_Module ___ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
