Re: [basex-talk] Constructing and querying map structures in a dynamic way
> Gaps? First, to avoid a misunderstanding: whereas map keys must not be the > empty sequence, map values may. Special data sources might contain challenges for further clarification of the desired referential integrity. Regards, Markus
Re: [basex-talk] Constructing and querying map structures in a dynamic way
> although I do not quite understand your description
Which wording variant would you have found clearer for the mentioned use case?
> (why should I care for the representation of the map I construct?),
* Selection of a general data structure for the handling of key/value pairs
* I guess that you would like to put customised data into keys and associated
values.
> it may be that one piece of information is useful to you,
Thanks for your constructive feedback.
> as it is not obvious from the spec: it's about how to construct a map
> dynamically,
Can such a view trigger any further consequences?
> that is,
>
> (1) when the keys are not known beforehand,
> (2) and/or when the entry values are assigned in "iterations" of a FLWOR
> expression
>
> (Especially (2) might be what puzzles you.)
I thought also about this data processing area in special ways for a moment.
> (1) and (2) can't be accomplished when using the map constructor expression,
> like this: basex "map{'x':1, 'y': false()}"
Will any information sources help to understand this technical detail better
(besides the XPath function library)?
> The trick consists of using a combination of the functions map:merge() [1]
> and map:entry() [2].
I became curious how application considerations will evolve further.
> This code may for example be (most often is) a FLWOR expression.
> In this code, each entry is constructed by a call of function map:entry().
Now I stumble on the error message “[XPTY0004] Item expected, empty sequence
found.”
from my software test according to a special data source.
How would you determine where unexpected gaps occur?
Regards,
Markus
Re: [basex-talk] Constructing and querying map structures in a dynamic way
> Maps are a derivative of the function type and highly general. I got informed that the data structure “map” got a clear meaning according to XQuery programming interfaces. > You're in effect asking for examples of using a function with no other > information provided. I came along another clarification desire according to the mentioned standard data structure. > It's just about impossible to answer usefully. The standard specification just shows examples for topics like “days” and “books” with map constructors based on literal data. How would the XQuery code look like if a detailed FLWOR expression would be applied instead? The subsequent CSV result output should show finally that keys and associated values were properly determined before. Regards, Markus
[basex-talk] Constructing and querying map structures in a dynamic way
Hello, I would like to construct a map structure based on a dynamic selection by the means of a FLWOR expression. https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps The keys and associated values should be presented as a CSV-like file by a subsequent data processing step. Can it become easier to find existing XQuery script examples for such an use case? Regards, Markus
Re: [basex-talk] ODBC support for BaseX?
> Maybe I am missing the point, so it would be interesting to know more about > your motivation 😊 Report generators are available which can handle selected formats as data sources. https://en.wikipedia.org/wiki/List_of_reporting_software The chosen formats influence how challenging or convenient the desired data processing would be for advanced reports and further data analyses. Regards, Markus
Re: [basex-talk] ODBC support for BaseX?
> > Is any support available for data management by interfaces like the > > following? > > > > * Open Database Connectivity > > * Java Database Connectivity > > Are you just looking for https://docs.basex.org/wiki/SQL_Module? No. (Not for related use cases at the moment.) This module is useful for accessing relational databases from XQuery using SQL. I became curious also for data exchange in the other direction according to such a programming interface. Can any BaseX databases be represented as ODBC (or JDBC) data sources? Regards, Markus
[basex-talk] ODBC support for BaseX?
Hello, The software “BaseX” is providing direct support for document databases. Is any support available for data management by interfaces like the following? * Open Database Connectivity * Java Database Connectivity How are corresponding drivers evolving? Regards, Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> > Would you like to improve error reporting any more? > > Maybe. We’d be thankful for: > > a) a concrete query example; My examples were questionable also because of some reasons. > b) the error you got; and > c) the error you’d have expected. The user guidance can hopefully be adjusted. Regards, Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> > I became curious if this development situation will be improved further. > Who would you expect to improve that? Various contributors. The communication on this mailing list (for example) will provide helpful information. > Are you thinking of the developers of the official W3 recommendations Some items are waiting on corresponding clarifications. https://github.com/w3c/qtspecs/issues > or the developers of BaseX? They hopefully continue to share remarkable insights. Would you like to improve error reporting any more? > > How will the available software evolve further also according to information > > like “BaseX offers a complete implementation of the XQuery Update Facility > > (XQUF).”? > > Version 1 is fully supported. We might add additional features of Version 3, > depending on requirements of our users and clients. Thanks for this information. Regards, Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> The support for the invocation of dynamic functions is another feature > that is only supported since version 3 of the specification. To be > strict, the finalized version of the spec woud require the "updating" > keyword to be prepended by an additional "invoke" keyword [4]. Thanks for your detailed feedback. How will the available software evolve further also according to information like “BaseX offers a complete implementation of the XQuery Update Facility (XQUF).”? Regards, Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> There are a couple of syntax issues with your code shown below:
How often would you dare to pass a detailed FLWOR expression directly
as a function parameter instead of specifying an extra variable name
or function call?
db:add("Test_DB", Another document example:
{$X}, "my_test.xml")
Regards,
Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> …, however the rules here can be hard to follow at first. I became curious if this development situation will be improved further. > The other XQuery rule that applies to your examples is that: > if there is only one expression in main or a function body then the keyword > "return" is omitted Which specification from the standard does support this implementation detail? https://www.w3.org/TR/2017/REC-xquery-31-20170321/#FunctionDeclns > In your second example, you can define the updating function with let, This is a general possibility. Should the shown annotation be replaced by a standard keyword for the function declaration? https://www.w3.org/TR/2011/REC-xquery-update-10-20110317/#id-function-declaration > but calling it requires the use of the updating keyword as described in … Is this a tool-specific setting? https://docs.basex.org/wiki/XQuery_Update#User-Defined_Functions https://www.w3.org/TR/2011/REC-xquery-update-10-20110317/#id-function-call Regards, Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> This aspect of XQuery syntax (when to use commas, when to construct explicit > sequences with wrapping parens) can be a bit challenging to internalize. Thanks for your constructive feedback. I hope that further clarifications will adjust known construction challenges. >This rework runs for me in the BaseX GUI: I observed also software behaviour for this adjusted script variant which is acceptable finally. * How much were the error messages (which I mentioned for the discussed use case) misleading? * Should the desired document addition work also by the application of a corresponding inline function expression? https://docs.basex.org/wiki/XQuery_Update#User-Defined_Functions Regards, Markus
Re: [basex-talk] Adding another document from XQuery evaluation
> You can add documents to the database without first writing them to the file
> system[1]:
Thanks for such information.
> This needs to be done in an updating function or …
I am looking for further advices also according to XQuery script variants
like the following.
declare %updating function local:add_document($items as item()*)
as empty-sequence()
{
let $result := db:add("Test_DB", $items, "Test.xml")
return ()
};
let $my_data as element() := My document,
$result := local:add_document($my_data)
return ()
Test result:
[XUST0001] let: no updating expression allowed.
let $my_data as element() := My document,
$add_document := %updating function($items) {db:add("Test_DB", $items,
"Test.xml")}
return ($add_document($my_data))
Test result:
[XPTY0004] Function is updating: $add_document.
https://docs.basex.org/wiki/XQuery_Update#User-Defined_Functions
How should the affected components be adjusted further according to
the development software “BaseX 9.7.2”?
Regards,
Markus
[basex-talk] Adding another document from XQuery evaluation
Hello, The software “BaseX” is providing direct support for file formats like “XML” and “CSV”. Corresponding data can be generated by customised XQuery scripts. I got the impression that additional documents can be integrated into selected databases so far by specifying file names. Does this mean that data (which would be determined by a script evaluation) need to be exported into a file for a subsequent file import? I imagine that another function combination for the available modules can make such an operation more convenient. Regards, Markus
[basex-talk] Checking run time characteristics according to the handling of customised indexes
Hello, Some indexes are generally supported also by the software “BaseX”. https://docs.basex.org/wiki/Indexes Some data processing approaches would usually depend on indexes for primary and foreign keys. https://en.wikipedia.org/wiki/Primary_key How much do run time characteristics influence data transfers (and conversion of corresponding data structures) for the available indexing technologies? Regards, Markus
Re: [basex-talk] Repeating data based on varying items in an array
> To me that kind of array with a string literal followed by a an array of > integers seemed kind of an odd data structure to start with I am curious how such views will evolve further. > so I would think about just adapting the original FLOWR expression I am trying this for a while. > to perhaps directly output XML or a sequences of maps from string > to an array of integers. I would prefer other data structures for the discussed use case at the moment. I would like to achieve that output data can be used for the construction of N:M relationships within a database. https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model#Cardinalities > But I don't see any difference in XQuery in general or for that sample > you had, it doesn't really matter whether you have a literal or > construct the data from another FLOWR expression. I got the impression that the clarification of implementation details would get more challenging the more expressions will be involved finally. > Did you run into any problems/errors using the data from a FLOWR expression? The answer depends on corresponding views. I am struggling also with the XQuery processor in the way that there are further issues to consider until this development tool will agree to my data processing intentions according to evolving script variants for some use cases. I would appreciate a bit more guidance for some error messages. > If you need help with that perhaps show a sample and the error you got. Further test examples might follow for advanced expression combinations. Regards, Markus
Re: [basex-talk] Repeating data based on varying items in an array
> declare function local:foo($item as array(*)){
> let $f:=$item(1)
> let $v:=$item(2)
> return array:for-each($v, function($q){ [$f,$q] }) ?*
> };
>
> (["f", [1, 2, 3]], ["g", [4, 5]]) ! local:foo(.)
> declare function local:foo($item as array(*)){
> let $f:=$item(1)
> let $v:=$item(2)
> return array:for-each($v, function($q){ [$f,$q] }) ?*
> };
>
> (["f", [1, 2, 3]], ["g", [4, 5]]) ! local:foo(.)
I am still looking for further clarification according to affected
implementation details.
* I would appreciate also if the declaration of such callback functions
can be described better in published software documentation for
the programming language “XQuery”.
* How would you convert the constructed array to a CSV file format?
* How should similar for-each operations be performed for other data structures
(besides arrays)?
* Would you like to compare run time characteristics any more for such
data processing?
Regards,
Markus
Re: [basex-talk] Repeating data based on varying items in an array
> Can you show us the input structure? A sequence of arrays with a string > literal as the first array item but an element or document node as the > second array item? I stumbled on the design challenge if the sequence of nested arrays would be constructed by another FLWOR expression (instead of the mentioned test data literal). Would such an XQuery script variant become more interesting? Are any more information sources available for similar use cases? Regards, Markus
Re: [basex-talk] Repeating data based on varying items in an array
> Your result asked for in the original question, if it also was meant to
> indicate some CSV representation, could be achieved using
>
>
> declare option output:method "csv";
> declare option output:csv "header=yes";
>
>
> {
> (["f", [1, 2, 3]], ["g", [4, 5]]) !
> (let $key := ?1, $values := ?2?* return $values!
>
> {$key}
> {.}
>
> )
> }
>
Can such an XQuery script variant be adapted also for the handling of items
from a subtree instead of the shown number array?
Regards,
Markus
Re: [basex-talk] Repeating data based on varying items in an array
> Possibly the below is helpful to you Thanks for your suggestion. Regards, Markus
Re: [basex-talk] Repeating data based on varying items in an array
> That is not even syntactically close to a sequence of two nested arrays,
> I am afraid.
How do you think about to extend an XQuery sccript variant like the following
for the desired transformation?
declare option output:method "csv";
declare option output:csv "header=yes";
for $x in (["f", [1, 2, 3]], ["g", [4, 5]])
return
{$x(1)}
{$x(2)}
Test result:
key,values
f,1 2 3
g,4 5
Regards,
Markus
[basex-talk] Repeating data based on varying items in an array
Hello, It seems that I am still not familiar enough with the programming language “XQuery” for the quick specification of a bit of code for a general data processing task. Thus I hope that the following use case can be clarified better. A sequence like “(["f" [1 2 3]], ["g" [4, 5]])” might be an interesting test example. It contains two nested arrays. * One part refers to fixed data. * Another part refers to some items with a varying number. I would like to convert the varying items into a data representation like the following. f,1 f,2 f,3 g,4 g,5 Can any customised functions help for the desired transformation? Regards, Markus
[basex-talk] Developments for address books
Hello, Various data models and corresponding applications can be constructed also with the help of the software “BaseX”. One popular data management tool is the address book (which can be supported by customised hardware and software), isn't it? Would you like to point any development repositories and further information sources out for such an application domain? Regards, Markus
[basex-talk] Performing extra data conversion for double values (by XQuery)
Hello, I would like to perform a sanity check for some input data which represent numbers that are using the comma as the decimal separator. I stumbled on the message “[FORG0001] Cannot convert to xs:double: …” then according to the evaluation of another XQuery script by the software “BaseX 9.7”. https://docs.basex.org/wiki/XQuery_Errors#Functions_Errors Can any functions help with the safe handling of currency values for my use case? https://docs.basex.org/wiki/Conversion_Module Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> Here's one thing you may be asking - do you want to know how to specify a
> join for n sources?
Yes. ‒ Your enquiry points into a direction for which I am looking also for
further solution ideas.
> for $e in $eng
> for $d in $deu
> for $u in $ukr
> for $h in $heb
> where $e/@n = $d/@n
> and $e/@n = $u/@n
> and $e/@n = $h/@n
> return
> { $e, $d, $u, $h }
Will any improvements become relevant for the specification of such join
conditions?
Regards,
Markus
[basex-talk] Improving the understanding for counting of entries in data groups
Hello,
I constructed the following XML file for another test of the software “BaseX
9.7”.
12
Demo1
Demo2
23
Demo1
Demo2
34
Test1
Test2
Test3
45
Test1
Test2
Test3
56
Test1
Test2
Test3
67
Probe1
I tried the following XQuery script out accordingly.
declare option output:method "csv";
declare option output:csv "header=yes, separator=|";
for $x in //test_data/info
group by $topics := string-join($x/topics/topic/data(), "*")
let $incidence := count($topics)
order by $incidence descending
return
{$topics}
{$incidence}
Corresponding test result:
topic_combination|incidence
Demo1*Demo2|1
Test1*Test2*Test3|1
Probe1|1
I would like to see the numbers “2” and “3” instead at the end of two rows
for such a data analysis approach.
I would appreciate further advices for this use case.
Regards,
Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> Have you read the XQuery specifications? Yes. > This section on joins, written by one of the inventors of SQL, may be a > helpful starting point: > > https://www.w3.org/TR/xquery-31/#id-joins It seems that I stumble on communication difficulties for this application area. Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> I was asking for your personal uses case or examples, My use cases can be similar to the example application from the XQuery specification. > and I think this is what others have been asking you for as well. I am trying to point further development possibilities out according to varying and growing numbers of entities which would be referenced in for clauses. Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> Could you please get specific and provide real use cases and examples? An example application was published already with the XQuery 3.1 specification for the combination of information from three documents. https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-joins Would you occasionally like to join significantly more entities (with for clauses according to known relationships)? Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> > I propose once more to take another look at specification efforts for > > join conditions (or constraints). > > What specific problem is it that you're trying to solve? What thing do > you want to do that you do not believe you can do in XQuery? I imagine that join parameters can be passed to customised functions so that for clauses would dynamically be constructed for further data processing. Will a function like “xquery:eval” be called finally? Regards, Markkus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> To join something, you minimally need at least two expressions which find
> the things to be joined,
I am still trying to clarify corresponding development possibilities according
to
bigger numbers of entities together with for clauses.
(I hope that other meanings can be better distinguished from related
applications
of a function like “string-join”.)
> some kind of rule for how to perform the join,
> and a destination for the result of the join.
This is usual.
> [predicates aren't joins]
How does this feedback fit to other documentations which describe the role of
predicates for join operations?
> A join requires the results of two expressions to be combined.
>
> (db:open('thing1'),db:open('thing2'))/descendant::patienti-identifier[local:check-range($interesting,.)]
>
> can be interpreted as a performing a join, you can re-write it as
>
> (db:open('thing1')/descendant::patient-identifier[local:check-range($interesting,.)],db:open('thing2')/descendant::patient-identifier[local:check-range($interesting,.)])
>
> but the predicate isn't doing the joining, the predicate is narrowing
> the selection of the XPath expressions.
This can be a desirable effect.
> In this case, the comma operator is doing the joining.
This example refers to another variant of a join operation for the construction
of a sequence.
(Such a XQuery code fragment does not use for clauses.)
> > Do you care for the number of involved items here?
>
> No.
Would you like to adjust this view according to the usage of for clauses?
https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-joins
Regards,
Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> > Would you like to check the influence of numbers according to joinable > items > once more on data processing efforts? > > > Are you asking about how well XQuery implementations perform compared to SQL > databases? Not directly. > For a particular kind of query? Yes. I propose once more to take another look at specification efforts for join conditions (or constraints). Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> This makes "source of information" too complex to define a standard > operation you can call a join; … I find that this view will need further clarification. >> I guess that you prefer to refer to them as “predicates within steps” so far >> (according to path expressions). >> https://www.w3.org/TR/xquery-31/#id-predicate > Predicates are part of path expressions, Such functionality can be helpful. > but still are not a join, Under which circumstances would you interpret the specification of special conditions (or constraints) as a join operation? > because predicates reduce the sequence produced by either the infix > expression (a function) or an axis step in this step of the path expression. > > Finding the thing you want to join and performing the join > -- producing a result that contains this thing and some other things > -- are conceptually distinct, or at least I think so. Will this view trigger further considerations? >> Would you like to extend programming interfaces for the management of >> relationships with various entities? > I need an example here, because I'm not following what you mean. I guess that you are used to some approaches for the handling of joinable items. Do you care for the number of involved items here? Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> We were looking at joining and combining diverse datasets as a key use case > for XQuery. This technical requirement is generally fine. > It's possible that we missed something, Do preferences matter if combinations of information sources are performed with a key word like “JOIN” (or not)? > but it would be extremely helpful to have a concrete use case to consider, > with real data. Would you like to check the influence of numbers according to joinable items once more on data processing efforts? Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
>> See also: >> https://www.w3.org/TR/xquery-31/#id-joins >> > You can do join _operations_, I would appreciate further clarification for the distinction which you present here. > but you aren't doing them on tables > (unless you did extra work to represent the tables hierarchically) Some “tables” can be transformed into XQuery sequences, can't they? > and there's absolutely no need for the keywords because the existing > more general mechanisms work fine. I see further development challenges in this area for the safe and convenient application of join conditions (or constraints). I guess that you prefer to refer to them as “predicates within steps” so far (according to path expressions). https://www.w3.org/TR/xquery-31/#id-predicate > Use your functions to create maps where the keys come from that id > element's string value. Customised data structures can be created together with XQuery maps and arrays. But I find that a join operation would be needed before based on available identification data. > (: bind to the sequence of id values :) > for $id in $interesting_stuff1 > return > (: run the function per-id :) > my_fun:do_something($id,$interesting_stuff2($id),$interesting_stuff3($id)) > > You could decide to skip the for clause and use > > return $interesting_stuff1 ! > my_fun:do_something(.,$interesting_stuff2(.),$interesting_stuff3(.)) > > instead. How do you think about to work without an extra identification sequence variable? >> Will any further comparisons evolve for the provided functionality? > Don't think so. I find the trick with XQuery is to not fight with it > about being some other language. > > Internalizing the sequence concept takes work; … Would you like to extend programming interfaces for the management of relationships with various entities? Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> There are neither join conditions nor record sets in XQuery.
I suggest to compare this view to the situation before the key word “JOIN”
was added to the SQL standard.
https://en.wikipedia.org/wiki/Join_(SQL)
See also:
https://www.w3.org/TR/xquery-31/#id-joins
> I've beens upposing you're interested in how you do something conceptually
> similar.
>
> Predicates filter; you can't add anything to a sequence with a
> predicate, you can only remove. ("these things, except if the prediate
> is true")
How do you think about the following XQuery script sketch?
let $interesting_stuff1 as item()* := my_fn:get_data("some expression"),
$interesting_stuff2 as item()* := my_fn:determine_further_data(),
$interesting_stuff3 as item()* := my_fn:evaluate_another_expression()
for $this1 in $interesting_stuff1,
$this2 in $interesting_stuff2,
$this3 in $interesting_stuff3
where $this1/id = $this2/id and $this2/id = $this3/id
return do_something($this1/id, $this2/description, $this3/comment)
> XQuery and SQL are not similar languages; they're both query languages,
> but SQL is built on set theory while XQuery is built on graph theory
> (XPath) and the idea of a tuple stream processor (FLOWR expressions).
> The underlying math for XQuery is younger. For example, the data
> structure under maps (finger trees) was first published _as math_ in
> 2006. You can't use either to understand the other one.
Will any further comparisons evolve for the provided functionality?
Regards,
Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
> Get the "issn" attribute from every document in a sequence of arbitrary > length: > > for $source in $sequence-of-URIs > return > $doc($source)/*/*:@issn > > So you can have an arbitrary number. It seems that your imaginations are evolving in other directions than the general data processing area I am trying to clarify here. > Do you have a specific use case? The corresponding understanding is still evolving, isn't it? The structured query language supports key words like the following. https://en.wikipedia.org/wiki/SQL_syntax#Queries * … JOIN … ON … * … NATURAL JOIN … I got the impression that required join conditions need to be specified as predicates in the where clause of the programming language “XQuery” instead. https://www.w3.org/TR/xquery-31/#id-where Did you occasionally specify join conditions for the desired combination of more than three (or ten) binding sequences (or “tables”) according to a for clause? Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
>> Did you stumble on bigger numbers of binding sequences >> for which you would like to join some information? > Generally the pattern for that is: > > 1 process each XPath expression into a sequence of maps; ideally there's > a common function you pass a sequence of nodes, but … Where did you mention join conditions for selected record sets in your approach for an algorithm description? Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
>> How do you think about to take another look at the development >> challenges according to growing numbers of binding sequences and >> corresponding join conditions? > I think there's usually four steps: Did you stumble on bigger numbers of binding sequences for which you would like to join some information? >> Would you like to share any ideas for further extensions of the >> involved programming interfaces? > Have you got a specific problem? I am looking for another bit of clarification according to a general data processing task like joining information from several sources (when their size and number would become remarkable). > Declarative languages don't do interfaces in the sense I think you might > mean; it's all functions, all the way down. Software libraries (or modules) are provided accordingly. I imagine that corresponding adjustments will become more helpful. Regards, Markus
Re: [basex-talk] Joining a varying number of information sources (with XQuery)?
>> I imagine that the selection of for bindings can occasionally vary so >> much and might become so big that it would be needed to construct >> FLWOR expressions by more sophisticated data structures. > A for clause will take any sequence as the binding sequence, How do you think about to take another look at the development challenges according to growing numbers of binding sequences and corresponding join conditions? > and the order of clauses in a FLOWR expression comes down to > "ends with a return clause" for most practical purposes. Thanks for your feedback. Would you like to share any ideas for further extensions of the involved programming interfaces? Regards, Markus
[basex-talk] Joining a varying number of information sources (with XQuery)?
Hello, It is supported to specify a fixed number of information sources for a for clause. https://www.w3.org/TR/xquery-31/#id-xquery-for-clause I imagine that the selection of for bindings can occasionally vary so much and might become so big that it would be needed to construct FLWOR expressions by more sophisticated data structures. I guess that the specification of consistent join conditions (and related constraints) would become more challenging. Can any programming interfaces help further to work with such use cases? https://docs.basex.org/wiki/XQuery_Module Regards, Markus
Re: [basex-talk] Checking relevance of the display for the section “Query” in the info view
>> How much do you care that the display for the item “Query” would be >> equivalent to >> the original XQuery script (including special characters like line breaks)? > Not at all. Interesting. > U+000A LINE FEED is one of three control characters allowed in XML. I would occasionally like to work with multi-line information which will be provided by data structures that are not XML formats. > It's inherently special, but it's also white space, and white space is not > important. Can any more special characters become relevant for further data processing? > The query itself is not in and must not be in a presentation > format where white space has semantic meaning. The presentation variant “Optimized Query” can preserve mentioned details. > I've been doing mostly document-representation XML processing for > twenty-odd years now. Thanks for such background information. > There is no business case for pretty, stable indents, > or semantic white space in XML or XQuery. I got other impressions and expectations for some use cases. Regards, Markus
Re: [basex-talk] Checking relevance of the display for the section “Query” in the info view
>> Do you find the item “Query” still relevant for further displays? > Sure! > > The teaching use is to compare the Query, … How much do care that the display for the item “Query” would be equivalent to the original XQuery script (including special characters like line breaks)? > If the Query return has been rendered on a single line, well, > XML doesn't consider white space significant, and a few reminders of that do > no harm. I suggest to reconsider such a view also a bit more. Regards, Markus
Re: [basex-talk] Checking relevance of the display for the section “Query” in the info view
> This would be the "simplify FLWOR expression:" returned just above the > "Optimized Query" section of the info window? I suggest to take another look at the items in the drop-down list of the info view. Do you find the item “Query” still relevant for further displays? Regards, Markus
Re: [basex-talk] Checking the display of line breaks in the info view
> What do you mean with space character? I suggest to take a closer look at the differences from a display like “``["line 1 line 2 line 3"]``” in comparison to the mentioned test example. Would you find any other representation more appropriate here? Regards, Markus
Re: [basex-talk] Checking the display of line breaks in the info view
> I find the shown query display improvable. > > > Feel free ;) How challenging would it become to replace space characters by line breaks at this place? Regards, Markus
Re: [basex-talk] Checking the display of line breaks in the info view
> May I expect that original line breaks will be preserved in the shown > query display? > > > Yes, you can. Thanks for your positive feedback. Another XQuery example: ``["line 1 line 2 line 3"]`` Corresponding info view: “… Optimized Query: """line 1 line 2 line 3""" Query: ``["line 1 line 2 line 3"]`` …” I find the shown query display improvable. Regards, Markus
[basex-talk] Checking the display of line breaks in the info view
Hello,
I tried the following XQuery script out with the software “BaseX 9.7”.
let $t1 := ``[line 1
line 2]``,
$t2 := ``[line 3
line 4]``
return ('"' || $t1 || '"', "---", '"' || $t2 || '"')
I observed the following display then in the info view.
“…
Optimized Query:
("""line 1
line 2""", "---", """line 3
line 4""")
Query:
let $t1 := ``[line 1 line 2]``, $t2 := ``[line 3 line 4]`` return ('"' || $t1
|| '"', "---", '"' || $t2 || '"')
…”
May I expect that original line breaks will be preserved in the shown query
display?
Regards,
Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
Dear Tamara, Thanks for your detailed feedback. > In the Microsoft documentation you linked, the name "text qualifier" is > misleading. Can this view trigger any further consequences? > It doesn't mean the value will always be formatted as text by the program. This aspect is interesting, isn't it? > "01" became the integer 1, … Which data types can handle “numbers” so that leading zeros would be preserved? > So adding double quotes around all values to designate a "text" datatype will > not work for Excel; This can be fine in some use cases. > users still need to manually change the formats of the cells. This can happen also. > The "Format quoted field as text" is a LibreOffice-specific option Would you like to take related program variants better into account? > that does format all of the values as text, Do you like such a software functionality? > not numbers or dates. I find such a setting occasionally useful. > To accomplish the same in Excel requires several complicated steps. Do any advanced users require special CSV data import configurations? > Examples: > https://www.winhelponline.com/blog/stop-excel-convert-text-to-number-date-format-csv-file/ Will the attention grow for further clarification also according to information from the article “Stop Excel from Converting Text to Number or Date format when Opening a CSV file”? >…, and even if it were it wouldn't accomplish what you want. I suggest to reconsider this view a bit more. Do I eventually need to reimplement CSV export functionality by XQuery means of customised string concatenations? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> My opinion of these results is that they are correct - they meet my > expectations. I find this view interesting. > Do you feel like these results are missing something? Yes. > If so, what specifically? I am missing text quoting for the fields “ID”, “T1”, “T2” and “T4”. > Is there something in the documentation that could change to make things more > explicit or provide better clarity? I got such an impression. Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> > Test result: > > T3|T4 > > "line 1 > > line 2?"|line 3\nline 4? > > > > > > Will any more clarification help here? > > …, but if you're asking "Why are there double quotes around my `T3` field?", Partly, yes. (But not really.) > then the answer appears to be in the CSV specification[1]. Which impressions did you get from the other BaseX information sources which I pointed out? > So, the behavior you're seeing in your sample script matches the CSV RFC. What is your opinion for the data processing of the fields “T1”, “T2” and “T4”? Regards, Markus
[basex-talk] Support for format “XQuery” by CSV parser configuration?
Hello, I would like to create another database from a CSV file with the software “BaseX 9.7”. It seems that I can not choose the format option “XQuery” on the tab “Parsing” of the dialogue “Create Database”. https://docs.basex.org/wiki/CSV_Module#Conversion Will any further configuration parameters become relevant for the CSV data import? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> No quotes will be added, as your delimiter does not occur in the text value. How does this feedback fit to previously provided information? A) https://docs.basex.org/wiki/CSV_Module#Options “… Serialization: If the option is enabled, the value will be wrapped with quotes if it contains characters that might be treated as control characters. …” B) https://mailman.uni-konstanz.de/pipermail/basex-talk/2022-April/017038.html “… The double quote is the text qualifier that’s used for serializing fields with spaces. …” Another XQuery script example: declare option output:method "csv"; declare option output:csv "header=yes, quotes=yes, separator=|"; line 1 line 2? line 3\nline 4? Test result: T3|T4 "line 1 line 2?"|line 3\nline 4? Will any more clarification help here? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> > I did not observe text quotation during my software tests.
>
> See [1] for some instruction on how to create a reproducible example.
I created a BaseX database from a single XML file which contains the following
data
for another test approach.
01
X Y
Y, X
I tried the following XQuery script out.
declare option output:method "csv";
declare option output:csv "header=yes, quotes=yes, separator=|";
for $x in //test_data/row
return
{$x/ID/data()}
{$x/T1/data()}
{$x/T2/data()}
Will the corresponding test result need any further clarification?
ID|T1|T2
01|X Y|Y, X
How good does such a data display fit to expectations on desirable software
behaviour?
https://docs.basex.org/wiki/CSV_Module#Options
Regards,
Markus
Re: [basex-talk] Support for performing queries on query results
> So when you attempt to print out {$incidence}, XQuery
> finds a sequence (1, 1, 1) instead of the value you want, 3.
Dear Tamara,
I found this test result surprising.
> To get the count of $incidence, you could return
> {count($incidence)} instead.
My understanding is still evolving also for implementation details according to
FLWOR expressions.
> But since you're not using the IDs at all, placing them in a sequence of
> arrays first isn't necessary.
I imagined that I would need a subquery.
Thus I found the specification of an identification relevant (for a moment).
> You can skip that part like:
>
> for $x in //test_data/product
> let $count := count($x/contributor)
> let $id := $x/id/data()
> group by $count
> let $incidence := count($id)
My knowledge was improvable also for desirable variable bindings.
> order by $incidence descending
> return
>
>
>{$count}
>{$incidence}
>
>
I thank you very much for the demonstration of a report approach which is
working
as expected finally.
Will similar use cases become interesting for further clarification?
Regards,
Markus
Re: [basex-talk] Support for performing queries on query results
> As I said, consider to provide an input/output sample of what you have > and want to achieve, … I did that a moment ago. https://mailman.uni-konstanz.de/pipermail/basex-talk/2022-April/017045.html A simple query can provide the following data display. id|contributor_count 123|1 45|1 67|1 89|2 These data should be taken for another analysis. contributor_count|incidence 1|3 2|1 This would be the expected output for such a test case. I became curious if such a report result could be achieved also without the generation of the first table. Regards, Markus
Re: [basex-talk] Support for performing queries on query results
> for $x in $results > > group by $count := $x(1) I tried the code variant “for $r in $results let $count := $r(1)” also out. But I wonder that the incidence “1” (only one) would be determined then for all record set counters. An other simple query is working as expected. id|contributor_count 123|1 45|1 67|1 89|2 Thus I would expect a data display like the following for the discussed query variant. contributor_count|incidence 1|3 2|1 Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> That’s what already happens. The double quote is the text qualifier > that’s used for serializing fields with spaces. I did not observe text quotation during my software tests. > There’s currently no feature to choose an alternative character This can be fine. > (such as the apostrophe), Applications can occasionally need additional data type indicators, can't they? > but I think that’s not what you are looking for anyway. I would like to enclose field data which correspond to text string data types by double quotes. Regards, Markus
Re: [basex-talk] Support for performing queries on query results
> > for $x in $results > > let $count := $x[1] > > Perhaps you want > > $x(1) > > here to access the first item in the array $x? Yes. ‒ Thanks that you pointed a typo out. Thus I adjusted the member access specification. But I stumble on the message “[XPTY0004] Item expected, sequence found: (1, 1, 1)” from the query evaluation. Which script fine-tuning will help then? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> Double quotes are only required for parsing, I would like to know a bit more about corresponding requirements. > they are not used to judge if the input is a string or number. I got other impressions (or expectations). > A little example: > 1;"1" > > If you save this one-liner as CSV file and open it with Excel, > but values will be interpreted and formatted as numbers. I suggest to take another look at available information sources. * Text Import Wizard by Microsoft Excel https://support.microsoft.com/en-gb/office/text-import-wizard-c5b02af6-fda1-4440-899f-f78bafe41857#ID0EBBJ-supTabControlContent-1 Step 2 of 3 (Delimited data) … Text qualifier … * Text Import by LibreOffice https://help.libreoffice.org/7.3/en-GB/text/shared/00/0208.html#hd_id314847411 Regards, Markus
Re: [basex-talk] Support for performing queries on query results
> With XQuery 3.0, a group by clause was introduced [1, 2]. I can be
> applied to all data structures including arrays (“sequences” are the
> most basic data structure in XQuery, though).
I am looking for further hints in this software design area.
The following script got parsed.
declare option output:method "csv";
declare option output:csv "header=yes, separator=|";
let $results :=
for $x in //test_data/product
return array { fn:count($x/contributor), $x/id/data() }
for $x in $results
let $count := $x[1]
let $incidence := fn:count($count)
group by $count
order by $incidence descending
return
{$count}
{$incidence}
But I stumble on the message “[XPTY0004] Item expected, sequence found: (1,
"123").”
from the query evaluation.
https://docs.basex.org/wiki/XQuery_Errors#Type_Errors
Do you find the presented data processing approach reasonable (in principle)?
Regards,
Markus
Re: [basex-talk] Support for performing queries on query results
> > Can any XQuery scripts be referenced by known file names at such places? > > Yes, you can e.g. use xquery:eval for that purpose [1]. The result can > again be bound to a variable and further processed. It can also be > written to a file, sent to any remote source, an SQL database, > or wherever you need it. Thanks for your hints. > You can regard XQuery as full information processing language, > which includes database core features that resemble SQL. I am trying also to become more familiar with the capabilities of the programming language “XQuery” (and corresponding software libraries). I achieved some data processing results together with mutable data structures. Now I am looking more at the construction of data structures by the means of functional algorithms. I would appreciate further advices in this design area. How do you think about to perform queries by grouping data from arrays (or maps) besides the extraction of contents from XML subtrees? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> > https://docs.basex.org/wiki/CSV_Module#Options … > 2. What’s the name of the software applications that you use that >requires double quotes? Text qualifiers can accordingly be configured for data exports and imports. Examples: * LibreOffice Calc * Microsoft Excel > 3. What would be the determining factor for using double quotes? A specific data type indication should occasionally be applied for texts. >Is it the existence of whitespaces (spaces, tabs, nl, cr, …?), This can happen. I observed that content from different XML elements can be joined by using space characters. >or would all non-numeric values need to be quoted? Probably, not. I am unsure under which circumstances additional delimiters would be requested for more fine-grained data type distinctions. Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> > Further data type indications can eventually be omitted, can't they? > > A custom XQuery function may be the best option if you require a > data-type specific output of CSV data. I hope that such code can be avoided if the BaseX CSV module could be adjusted accordingly. https://docs.basex.org/wiki/CSV_Module#Options Regards, Markus
Re: [basex-talk] Support for performing queries on query results
> > I would prefer to avoid the creation of another BaseX database > > for each query result. > > Just fine as well. If you use XQuery, you can bind your query results > to variables for further processing. I guess that it is expected that this kind of query result binding would work only with XML data structures. Can any XQuery scripts be referenced by known file names at such places? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> > I came along software applications which would need the usage of double > > quotes > > for the distinction that provided data should be handled as text strings. > > So you would probably require all non-numeric values to be enclosed by > quotes, no matter if it contains spaces? The answer depends on the configuration for data import tools. Example: spreadsheet applications > What about other data types (booleans, etc.)? Further data type indications can eventually be omitted, can't they? Regards, Markus
Re: [basex-talk] Support for performing queries on query results
> > But I would like to reuse query results for further queries directly. > > One common approach is to create volatile database instances from from > query results (databases in BaseX are prety lightweight). I would prefer to avoid the creation of another BaseX database for each query result. > How does your script look like? I suggest to clarify software design consequences better according to the execution of queries which determine values for subsequent queries. I guess that a popular use case is to count items also in XML data structures. The computed numbers would be passed then to aggregate functions for further data analyses, wouldn't they? Regards, Markus
Re: [basex-talk] Enclosing strings by double quotes for CSV data output
> Double quotes in CSV files do not give any information on the data > type of a value. I came along software applications which would need the usage of double quotes for the distinction that provided data should be handled as text strings. > Instead, they merely ensure that the data can be correctly parsed. I suggest to reconsider this view a bit more for some data imports. > If the pipe symbol is used as separator, there is no need to enclosed > whitespaced string by quotes. How do you think about to avoid the interpretation of a column delimiter character within strings as a CSV separator? > Do you possibly have a processor that interprets the syntax of the > incoming CSV data to detect and assign data types? Occasionally, yes. Regards, Markus
[basex-talk] Enclosing strings by double quotes for CSV data output
Hello, I constructed a small XQuery script which contains the following specifications. declare option output:method "csv"; declare option output:csv "header=yes, separator=|, quotes=yes"; … The query execution succeeded (in principle) with the software “BaseX 9.7”. I observed data processing results where displayed columns contained space characters. I would expect that such data should be handled as text strings then. Thus I would expect also that these data should be enclosed by double quotes (according to the safe handling of CSV files). But this did not happen for my software test. Will any additional settings become relevant for this use case? Regards, Markus
[basex-talk] Support for performing queries on query results
Hello, XQuery scripts can be used also to compute values from selected XML data structures. Query results can be exported then into other file formats. But I would like to reuse query results for further queries directly. The SQL standard is providing named views for this purpose. How will the support evolve for such data processing requirements with XML tools? Regards, Markus
