Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/skip-items into lp:zorba
I've looked into the index probe iterators but they already optimize any skipping. They take a Skip parameter and then they push it into an underlying iterator e.g. ProbeValueTreeIndexIterator which handles the skip internally. So there is nothing to be done there. I've pushed the pending fixes and the branch is ready for merging. -- https://code.launchpad.net/~zorba-coders/zorba/skip-items/+merge/174723 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/dataguide into lp:zorba
I've done some additional testing, and these are the results: For the xray query, the largest that we have in the testsuite, compilation time with --compile-only is pretty much the same with and without the dataguide computaiton, at around ~0.08 sec. With a specially constructed query that looks like this: (see dataguide-29.jq test) let $col := dml:collection() let $col2 := ($col.cat1, $col.cat2, ... , $col.cat10) return $col2.category.category.category ... category (repeated ~2000 times) the compilation time goes from ~0.7s without the dataguide to ~10s with the dataguide enabled, so it is significant. But this is a worst-case scenario. The resulting dataguide is an object 2000-levels deep. The compilation can be improved significantly by: - keeping track of the leaves nodes in the dataguide tree - rewriting a bit the dataguide structure to store the trees incrementally instead of cloning them - adding a depth cutoff What do you think? -- https://code.launchpad.net/~zorba-coders/zorba/dataguide/+merge/173026 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue starting for the following merge proposals: https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Progress dashboard at http://jenkins.lambda.nu/view/ValidationQueue -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Review: Approve -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue result for https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Stage BuildZorbaUbuntu failed. Check compiler output at http://jenkins.lambda.nu/job/BuildZorbaUbuntu/122/parsed_console to view the results. -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue starting for the following merge proposals: https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Progress dashboard at http://jenkins.lambda.nu/view/ValidationQueue -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue starting for the following merge proposals: https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Progress dashboard at http://jenkins.lambda.nu/view/ValidationQueue -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Voting criteria failed for the following merge proposals:
https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 :
Votes: {'Approve': 1, 'Pending': 1}
--
https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue result for https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Stage CommitZorba failed. Check console output at http://jenkins.lambda.nu/job/CommitZorba/47/console to view the results. -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Paul J. Lucas has proposed merging lp:~paul-lucas/zorba/pjl-misc into lp:zorba.
Commit message:
Added operator for enums; moved to .cpp.
Requested reviews:
Paul J. Lucas (paul-lucas)
For more details, see:
https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445
Added operator for enums; moved to .cpp.
--
https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'include/zorba/store_consts.h'
--- include/zorba/store_consts.h 2013-06-29 08:38:53 +
+++ include/zorba/store_consts.h 2013-07-23 17:37:32 +
@@ -20,7 +20,10 @@
#include string
#include zorba/config.h
-namespace zorba { namespace store {
+namespace zorba {
+namespace store {
+
+///
/***
!!! ATTENTION: The order of the enum values within SchemaTypeCode is important.
@@ -91,8 +94,10 @@
XS_LAST
};
+ZORBA_DLL_PUBLIC
std::ostream operator( std::ostream, SchemaTypeCode );
+///
class ZORBA_DLL_PUBLIC StoreConsts
{
@@ -135,74 +140,10 @@
namespaceNode = 7
};
- static std::string toString(NodeKind k)
- {
-switch(k)
-{
-case anyNode:
- return anyNode;
-
-case documentNode:
- return documentNode;
-
-case elementNode:
- return elementNode;
-
-case attributeNode:
- return attributeNode;
-
-case textNode:
- return textNode;
-
-case piNode:
- return piNode;
-
-case commentNode:
- return commentNode;
-
-case namespaceNode:
- return namespaceNode;
-
-default:
- return unknown NodeKind;
-}
- }
-
- static std::string toSchemaString(NodeKind k)
- {
-switch(k)
-{
-case anyNode:
- return node;
-
-case documentNode:
- return document-node;
-
-case elementNode:
- return element;
-
-case attributeNode:
- return attribute;
-
-case textNode:
- return text;
-
-case piNode:
- return processing-instruction;
-
-case commentNode:
- return comment;
-
-case namespaceNode:
- return namespace-node;
-
-default:
- return unknown NodeKind;
-}
- }
-
-
- /* ATTENTION: the ordering of the enum values is important. Do NOT change it! */
+ static std::string toString(NodeKind);
+ static std::string toSchemaString(NodeKind);
+
+ // ATTENTION: the ordering of the enum values is important. Do NOT change it!
enum JSONItemKind
{
jsonItem = 0,
@@ -210,26 +151,16 @@
jsonArray = 2
};
-
- static std::string toString(JSONItemKind k)
- {
-switch(k)
-{
- case jsonItem:
-return json-item;
-
- case jsonObject:
-return object;
-
- case jsonArray:
-return array;
-
- default:
-return unknown JSONItemKind;
-}
- }
};
+ZORBA_DLL_PUBLIC
+std::ostream operator( std::ostream, StoreConsts::NodeKind );
+
+ZORBA_DLL_PUBLIC
+std::ostream operator( std::ostream, StoreConsts::JSONItemKind );
+
+///
+
} // namespace store
} // namespace zorba
#endif
=== modified file 'src/api/store_consts.cpp'
--- src/api/store_consts.cpp 2013-02-26 20:56:33 +
+++ src/api/store_consts.cpp 2013-07-23 17:37:32 +
@@ -15,6 +15,9 @@
*/
#include stdafx.h
+
+#include sstream
+
#include zorba/store_consts.h
namespace zorba {
@@ -76,13 +79,74 @@
if ( c = 0 c XS_LAST )
o s[ c ];
else
-o [illegal type code: (int)c ']';
-
+o unknown SchemaTypeCode: (int)c '';
return o;
};
///
+std::ostream operator( std::ostream o, StoreConsts::NodeKind k ) {
+ static char const *const s[] = {
+anyNode,// 0
+documentNode, // 1
+elementNode,// 2
+attributeNode, // 3
+textNode, // 4
+piNode, // 5
+commentNode,// 6
+namespaceNode // 7
+ };
+
+ if ( k = 0 k = StoreConsts::namespaceNode )
+o s[ k ];
+ else
+o unknown NodeKind: (int)k '';
+ return o;
+}
+
+std::string StoreConsts::toString( NodeKind k ) {
+ std::ostringstream oss;
+ oss k;
+ return oss.str();
+}
+
+std::string StoreConsts::toSchemaString( NodeKind k ) {
+ static char const *const s[] = {
+node, // 0
+document-node, // 1
+element,// 2
+attribute, // 3
+text, // 4
+processing-instruction, // 5
+comment,// 6
+namespace-node // 7
+ };
+
+ if ( k = 0 k = namespaceNode )
+return s[ k ];
+ else {
+std::ostringstream oss;
+oss unknown NodeKind: (int)k '';
+return oss.str();
+ }
+}
+
+std::ostream operator(
Re: [Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Review: Approve -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue starting for the following merge proposals: https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Progress dashboard at http://jenkins.lambda.nu/view/ValidationQueue -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
The proposal to merge lp:~paul-lucas/zorba/pjl-misc into lp:zorba has been updated. Status: Needs review = Approved For more details, see: https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Review: Approve -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Review: Approve -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Validation queue starting for the following merge proposals: https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Progress dashboard at http://jenkins.lambda.nu/view/ValidationQueue -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Validation queue result for https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Stage CommitZorba failed. Check console output at http://jenkins.lambda.nu/job/CommitZorba/48/console to view the results. -- https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Voting criteria failed for the following merge proposals:
https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385 :
Votes: {'Approve': 1, 'Pending': 1}
--
https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Validation queue succeeded - proposal merged! -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
The proposal to merge lp:~paul-lucas/zorba/pjl-misc into lp:zorba has been updated. Status: Approved = Merged For more details, see: https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 -- https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/176445 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/bug-1188033 into lp:zorba/archive-module
Review: Needs Fixing In archive_module.cpp: 1. Please don't leave commented-out code. Just delete it; it's in the version history if we ever need to refer back to it. For example, remove these lines: +//Item lNameItem = theFactory-createString(lName); +//lMemberName = theFactory-createString(name); 2. As discussed, if possible, create single zorba::Items for all the JSON object key strings, eg. name, size, etc. Possibly create those in the ArchiveModule constructor. Use those constants when creating JSON objects in EntriesIterator::next(), as well as comparing input values in ArchiveEntry::setValues() (and anywhere else it happens to be appropriate). Also don't forget to assign a bug to yourself to create a JSON schema for this module, and to go back and update the module and test cases accordingly. -- https://code.launchpad.net/~zorba-coders/zorba/bug-1188033/+merge/176523 Your team Zorba Coders is subscribed to branch lp:zorba/archive-module. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/bug-1188033 into lp:zorba/archive-module
Review: Needs Fixing Also, you have at least one /p typo in archive.xq. Always remember to run ctest -R moduleDocumentation before committing, to be sure that your XQDoc is syntactically correct and that you didn't forget to document any parameters, etc. -- https://code.launchpad.net/~zorba-coders/zorba/bug-1188033/+merge/176523 Your team Zorba Coders is subscribed to branch lp:zorba/archive-module. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/skip-items into lp:zorba
Review: Needs Information Looks good to me. Only two issues. - The thing with the probe iterators is that there are additional functions called probe-*-skip that allow you to skip explicitly. However, subsequence is not rewritten into these functions. We will need to override the skip function in all the probe iterators (similar to count) such that subsequence(probe-*) will be executed efficiently. We can do this is a second step though. I'll create a follow up story for this. - In ZorbaCollectionIterator::initCollection lines 307 and 326, you eventually add skipCount twice. Something smells spooky here. -- https://code.launchpad.net/~zorba-coders/zorba/skip-items/+merge/174723 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/use-dataguide into lp:zorba
Review: Needs Fixing
I have tried some basic queries and there is a huge performance improvement.
This is great.
However, I detected a memory leak and have one question.
- memory leak in translator.cpp:4823
==20325== 88 (32 direct, 56 indirect) bytes in 1 blocks are definitely lost in
loss record 4 of 4
==20325==at 0x4C2B1C7: operator new(unsigned long) (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20325==by 0x55B8897:
zorba::TranslatorImpl::begin_visit(zorba::AnnotationListParsenode const)
(translator.cpp:4823)
==20325==by 0x556447D:
zorba::AnnotationListParsenode::accept(zorba::parsenode_visitor) const
(parsenodes.cpp:903)
==20325==by 0x55B18A8:
zorba::TranslatorImpl::preprocessVFOList(zorba::VFO_DeclList const)
(translator.cpp:3904)
==20325==by 0x55A9951: zorba::TranslatorImpl::begin_visit(zorba::Prolog
const) (translator.cpp:2889)
==20325==by 0x5562201: zorba::Prolog::accept(zorba::parsenode_visitor)
const (parsenodes.cpp:262)
==20325==by 0x5561E7A:
zorba::LibraryModule::accept(zorba::parsenode_visitor) const
(parsenodes.cpp:174)
==20325==by 0x559CCD8: zorba::translate_aux(zorba::TranslatorImpl*,
zorba::parsenode const, zorba::static_context*, unsigned long,
zorba::ModulesInfo*,
std::mapzorba::rstringzorba::rstring_classes::repzorba::atomic_int,
std::char_traitschar, std::allocatorchar ,
zorba::rstringzorba::rstring_classes::repzorba::atomic_int,
std::char_traitschar, std::allocatorchar ,
std::lesszorba::rstringzorba::rstring_classes::repzorba::atomic_int,
std::char_traitschar, std::allocatorchar ,
std::allocatorstd::pairzorba::rstringzorba::rstring_classes::repzorba::atomic_int,
std::char_traitschar, std::allocatorchar const,
zorba::rstringzorba::rstring_classes::repzorba::atomic_int,
std::char_traitschar, std::allocatorchar const, bool,
zorba::StaticContextConsts::xquery_version_t) (translator.cpp:16343)
==20325==by 0x55AF5C4: zorba::TranslatorImpl::end_visit(zorba::ModuleImport
const, void*) (translator.cpp:3589)
==20325==by 0x55635D3:
zorba::ModuleImport::accept(zorba::parsenode_visitor) const
(parsenodes.cpp:667)
==20325==by 0x55A99BB: zorba::TranslatorImpl::begin_visit(zorba::Prolog
const) (translator.cpp:2899)
- In the following query, no dataguide seems to be pushed into the parser. Why?
import module namespace f = http://expath.org/ns/file;;
for $obj in parse-json(f:read-text(fn:resolve-uri(citylots-small.json)))
group by $s := $obj.STREET
return {
street : $s,
count : count($obj)
}
--
https://code.launchpad.net/~zorba-coders/zorba/use-dataguide/+merge/176385
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
