Re: [MarkLogic Dev General] Error in getting map keys in ML8

2015-04-28 Thread Indrajeet Verma
Thank you Bryan for your help!

Regards,
Indy

On Wed, Apr 29, 2015 at 12:38 AM, Joe Bryan  wrote:

>  Hi Indrajeect,
>
>  Your $params variable is an element (the serialization of a map), not a
> map. You can use the map constructor function to convert the XML
> serialization of a map to map:
>
>  return map:keys(map:map($params))
>
>  Thanks.
>
>  -jb
>
>   From: Indrajeet Verma 
> Reply-To: MarkLogic Developer Discussion 
> Date: Tuesday, April 28, 2015 at 3:05 PM
> To: MarkLogic Developer Discussion 
> Subject: [MarkLogic Dev General] Error in getting map keys in ML8
>
>   Hi,
>
>  Could somebody help me what I am missing in below query. I am using this
> code in ML8
>
>  let $params := http://marklogic.com/xdmp/map";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xmlns:xs="http://www.w3.org/2001/XMLSchema";>
> 
> 
> 
> 
> 
>  wk-pid="USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39">
> Users
> 
> >USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39
>
> 
> 
> 
> 
> 
>  wk-pid="483d4d2f1a54a7f0eb42f6f1dfa6518c|4/29/2015 3:41:5 AM">
>
> USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39
>
> 2015-04-28T23:41:05.172+05:30
> 
> 
> 
> 
> 
> 
> 
> return map:keys($params)
>
>  Error that I am seeing,
>
>   [1.0-ml] XDMP-ARGTYPE: (err:XPTY0004) map:keys(http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance"; xmlns:map="
> http://marklogic.com/xdmp/map";> key="USERSESSION:483d4d2f1a54a7f0eb42f6f1dfa6518c|4/2...) -- arg1
> is not of type map:map
>
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general


Re: [MarkLogic Dev General] Error in getting map keys in ML8

2015-04-28 Thread Joe Bryan
Hi Indrajeect,

Your $params variable is an element (the serialization of a map), not a map. 
You can use the map constructor function to convert the XML serialization of a 
map to map:

return map:keys(map:map($params))

Thanks.

-jb

From: Indrajeet Verma 
mailto:indrajeet.ve...@gmail.com>>
Reply-To: MarkLogic Developer Discussion 
mailto:general@developer.marklogic.com>>
Date: Tuesday, April 28, 2015 at 3:05 PM
To: MarkLogic Developer Discussion 
mailto:general@developer.marklogic.com>>
Subject: [MarkLogic Dev General] Error in getting map keys in ML8

Hi,

Could somebody help me what I am missing in below query. I am using this code 
in ML8

let $params := http://marklogic.com/xdmp/map";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:xs="http://www.w3.org/2001/XMLSchema";>






Users
USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39







USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39
2015-04-28T23:41:05.172+05:30







return map:keys($params)

Error that I am seeing,

[1.0-ml] XDMP-ARGTYPE: (err:XPTY0004) map:keys(http://www.w3.org/2001/XMLSchema"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:map="http://marklogic.com/xdmp/map";>http://developer.marklogic.com/mailman/listinfo/general


[MarkLogic Dev General] Error in getting map keys in ML8

2015-04-28 Thread Indrajeet Verma
Hi,

Could somebody help me what I am missing in below query. I am using this
code in ML8

let $params := http://marklogic.com/xdmp/map";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:xs="http://www.w3.org/2001/XMLSchema";>






Users
USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39








USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39

2015-04-28T23:41:05.172+05:30







return map:keys($params)

Error that I am seeing,

[1.0-ml] XDMP-ARGTYPE: (err:XPTY0004) map:keys(http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"; xmlns:map="
http://marklogic.com/xdmp/map";>http://developer.marklogic.com/mailman/listinfo/general


Re: [MarkLogic Dev General] JSON sending in the request body is not being converted into map:map in ML8 using roxy however working fine with ML6

2015-04-28 Thread Christopher Hamlin
On Tue, Apr 28, 2015 at 11:42 AM, Indrajeet Verma
 wrote:
> ...
>
> Could you please help me to get the link of the APIs that have modified as
> you said in your comments? OR Should we look into release notes?
>
>

Since JSON is now native, there is much change in v8.  The relevant
section of the release notes has a lot of information:

https://docs.marklogic.com/guide/relnotes/chap4#id_92312

/ch
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general


Re: [MarkLogic Dev General] JSON sending in the request body is not being converted into map:map in ML8 using roxy however working fine with ML6

2015-04-28 Thread Indrajeet Verma
Thank you David for your explanation. I also have same understanding
however not able to find exact method/module which is not converting into
map.

I identified router.xqy (line 67) and request.xqy where the conversion is
happening and setting map to the request.

Might be good to open a ticket in the github/roxy and see if I get some
resolution.

However I am attaching the router.xqy and request.xqy for your reference.
if possible please take a look and suggest some resolution.

Could you please help me to get the link of the APIs that have modified as
you said in your comments? OR Should we look into release notes?

Thank you for your time and help.

Regards,
Indrajeet

On Tue, Apr 28, 2015 at 8:16 PM, David Lee  wrote:

>  This is probably best addressed by filing an issue on the Roxy github
> issue tracker.
>
>
>
> https://github.com/marklogic/roxy
>
>
>
> Roxy is a community/developer supported product.
>
>
>
> I suspect the issue is due to the (not backwards compatible, but
> documented) change to many JSON related API's in MarkLogic.
>
>
>
> The JSON API's have been converted to work directly with the native JSON
> data types in ML8.
>
>
>
> All of the internal MarkLogic libraries and apps have been updated as well
> to adjust to this change.   Exactly where in Roxy this is breaking I don't
> know, but the errors suggest is likely due to the ML8 changes in JSON.
>
>
>
> Happy to help with any specific questions on the changes to ML APIs.
>
>
>
> -David
>
>
>
>
>
>
>
>
>
>
> -
>
> David Lee
> Lead Engineer
> *Mark**Logic* Corporation
> d...@marklogic.com
> Phone: +1 812-482-5224
>
> Cell:  +1 812-630-7622
> www.marklogic.com
>
>
>
> *From:* general-boun...@developer.marklogic.com [mailto:
> general-boun...@developer.marklogic.com] *On Behalf Of *Indrajeet Verma
> *Sent:* Tuesday, April 28, 2015 10:15 AM
> *To:* MarkLogic Developer Discussion
> *Subject:* [MarkLogic Dev General] JSON sending in the request body is
> not being converted into map:map in ML8 using roxy however working fine
> with ML6
>
>
>
> Hi,
>
>
>
> Please help us if you have encountered same kind of problem.
>
>
>
> We are upgrading ML6 to ML8 and using roxy as well. The roxy is customized
> based on the customer requirement.
>
>
> For one of the REST request "/setcache", we are sending below data to
> cache and that is being converted into map (key value pair)
>
> ML6-
>
> Request in Body-
> ---
>
> {"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14
> PM":{"user":" wk-pid=\"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39\">Users type=\"string\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39 type=\"string\">lrunne...@wk.com
> lrunne...@wk.com type=\"string\">ActiveALL type=\"string\">user type=\"string\">2013-12-25 type=\"string\"/> type=\"string\"/> type=\"string\">Allowed type=\"string\">John type=\"string\">Doe type=\"string\">WK001 type=\"string\">LoadRunner type=\"string\"/>5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00 type=\"string\">System2014-07-11T11:05:11.589-05:00
> yun.y...@wolterskluwer.com",
> "session":" 9:48:14
> PM\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}
>
> Converted into below map using ML6-
> --
>
> map:map(http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance";
>
> xmlns:map="http://marklogic.com/xdmp/map";> key="USERSESSION:97f7d8ec0b5bba928cb025bd1d4d28d4|4/28/2015 9:57:24
> PM"> wk-pid="USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa">USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaaUsers type="string">sjauh...@innodata.com type="string">shruti type="string">jauhari type="string">FORD-101Ford Motor
> Company type="string">Allowed type="string">Active type="string">2018-12-31 type="string">user type="string">212-555-1212 type="string">Northeast type="string">False type="string">default type="string">skh...@innodata.com5f4dcc3b5aa765d61d8327deb882cf99password112015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab002015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab00 key="session"> wk-pid="97f7d8ec0b5bba928cb025bd1d4d28d4|4/28/2015 9:57:24
> PM">USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa2015-04-28T17:57:24.34+05:302015-04-28T21:57:24.34+05:302015-04-28T17:57:24.34+05:30false)
>
> ML8 -
> ---
> Input and converted output is same. i.e. without map:
>
> {"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14
> PM":{"user":" wk-pid=\"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39\">Users type=\"string\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39 type=\"string\">lrunne...@wk.com
> lrunne...@wk.com type=\"string\">ActiveALL type=\"string\">user type=\"string\">2013-12-25 type=\"string\"/> type=\"string\"/> type=\"string\">Allowed type=\"string\">John type=\"string\">Doe type=\"string\">WK001 type=\"

Re: [MarkLogic Dev General] SQL Query Performance and Tableau Connection

2015-04-28 Thread Walt Rolle
Sorry for the duplicate here, I had to subscribe first...

From: Walt Rolle
Sent: Tuesday, April 28, 2015 11:24 AM
To: 'Brad Rix'; MarkLogic Developer Discussion; Cynthia Jiang
Cc: Scott Fowler; 'sara.ma...@marklogic.com'
Subject: RE: [MarkLogic Dev General] SQL Query Performance and Tableau 
Connection

Hi Brad,
Thanks for the tip. Also it looks like it's going to make sense for us just to 
store the primary address at the patient level anyway (so no joins).

I wanted to mention some conclusions we arrived at after talking to Sara.


*If we want to use SQL at all, then we can't have repeating rows in our 
document structure, or at least need to name the first one in the collection 
something different to separate it.

*Getting SQL to work in our situation is about supporting ODBC 
connectivity options such as ODBC to Tableau. We can simplify the data model by 
de-normalizing a bit, such as sticking to just primary address and active 
eligibility. After that, we supply Tableau with our own specific SQL statement 
to limit the columns and number of "tables". Group by / order by / functions 
are best left to Tableau afterwards, so our source queries are simple.

*MarkLogic lends itself better to document search, which would use 
things like XQuery or Javascript to screen the documents and return patient 
documents that match the criteria. Because we broke apart our model to support 
SQL, we take a performance hit here in having to do joins or store data in 
duplicate locations. The other option would be to keep the large hierarchical 
documents and ETL a subset off to a data mart, if the customer wants to do 
relational things against it.

Because the customer has a lot of existing code and internal capability around 
the traditional SQL Server stack, we're thinking of having our model driven 
towards supporting SQL and NoSQL techniques simultaneously - at least as the 
first step of the roadmap. In the future, it would not be much effort to 
generate new documents that are complete hierarchies of patient history. To do 
this we need to separate objects by grain.


1.  Patient (with primary address and active eligibility)

2.  Claim Header (consider storing diagnosis codes as repeating rows in one 
field, it's essentially "factless")

3.  Claim Revenue (this is line item detail with dollar amounts)

4.  Claim Procedure (another type of line item detail with dollar amounts)

5.  Care Provider (specialty may need to be a field level collection like 
diagnosis)

6.  Facility

We'll have to revisit these decisions when we get more time to spend on looking 
at the analytics use cases to make sure we always choose the right tool for the 
job. As far as deep amounts of storage for terabytes of data, MarkLogic seems 
to be a good landing place for the variety of incoming claims files, either 
before or after we standardize them or both.

Thanks,
Walt

Walter Rolle, PMP
Project Manager
ro...@rdacorp.com
o: (770) 884.4512 | m: (678) 722.1549 | f: (770) 884.4512
2475 NorthWinds Parkway, Suite 200
Alpharetta, GA 30009-4807
[in][cid:image004.png@01CDD2D0.8A8D29B0][cid:image003.jpg@01CDD2D0.8A8D29B0]
www.rdacorp.com

From: Brad Rix [mailto:brad@flatironssolutions.com]
Sent: Tuesday, April 28, 2015 10:45 AM
To: MarkLogic Developer Discussion; Cynthia Jiang
Cc: Scott Fowler; Walt Rolle
Subject: RE: [MarkLogic Dev General] SQL Query Performance and Tableau 
Connection

One comment about your specific query.You have a "left outer join" against 
the patientaddress table.  This can actually be written as an "inner join" with 
the same results since you are restricting the right side of the outer join 
with the where clause.   You can try your query as an inner join and see if 
that improves your performance.The relational database engine probably 
optimized the query by re-writing it as an inner join.If you did not have 
the "a.State='LA'", then it would have to remain an outer join.   The left 
outer join might be greatly effecting performance.Since you are also using 
group by and order by, the total amount of results might also effect the 
performance.

select c.Patientid, ClaimNumber, FirstName, LastName, sum (PharmacyTotalCost) 
as PharmacyTotalCost
  from claim c inner join patient p on c.patientid = p.patientid
  left outer join patientaddress a on p.patientid = a.patientid
  where a.[State] = 'LA'
  group by  c.PatientId, ClaimNumber, FirstName, LastName  order by 
SUM(PharmacyTotalCost) desc limit 10

Note:  You can see a reference in mysql as an example about when outer joins 
can be re-written to inner joins:  
https://dev.mysql.com/doc/refman/5.0/en/outer-join-simplification.html

Brad Rix | Technical Lead
O: +1 303 542 2172 | M: +1 303 915 2771
Skype: Brad.Rix

Re: [MarkLogic Dev General] JSON sending in the request body is not being converted into map:map in ML8 using roxy however working fine with ML6

2015-04-28 Thread David Lee
This is probably best addressed by filing an issue on the Roxy github issue 
tracker.

https://github.com/marklogic/roxy

Roxy is a community/developer supported product.

I suspect the issue is due to the (not backwards compatible, but documented) 
change to many JSON related API's in MarkLogic.

The JSON API's have been converted to work directly with the native JSON data 
types in ML8.

All of the internal MarkLogic libraries and apps have been updated as well to 
adjust to this change.   Exactly where in Roxy this is breaking I don't know, 
but the errors suggest is likely due to the ML8 changes in JSON.

Happy to help with any specific questions on the changes to ML APIs.

-David




-
David Lee
Lead Engineer
MarkLogic Corporation
d...@marklogic.com
Phone: +1 812-482-5224
Cell:  +1 812-630-7622
www.marklogic.com

From: general-boun...@developer.marklogic.com 
[mailto:general-boun...@developer.marklogic.com] On Behalf Of Indrajeet Verma
Sent: Tuesday, April 28, 2015 10:15 AM
To: MarkLogic Developer Discussion
Subject: [MarkLogic Dev General] JSON sending in the request body is not being 
converted into map:map in ML8 using roxy however working fine with ML6

Hi,

Please help us if you have encountered same kind of problem.

We are upgrading ML6 to ML8 and using roxy as well. The roxy is customized 
based on the customer requirement.


For one of the REST request "/setcache", we are sending below data to cache and 
that is being converted into map (key value pair)

ML6-

Request in Body-
---

{"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14 
PM":{"user":"UsersUSR-b740cbc5-5e27-8b1c-d97a-1329625f6b39lrunne...@wk.comlrunne...@wk.comActiveALLuser2013-12-25AllowedJohnDoeWK001LoadRunner5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00System2014-07-11T11:05:11.589-05:00yun.y...@wolterskluwer.com",
 "session":"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}

Converted into below map using ML6-
--

map:map(http://www.w3.org/2001/XMLSchema"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:map="http://marklogic.com/xdmp/map";>USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaaUserssjauh...@innodata.comshrutijauhariFORD-101Ford Motor 
CompanyAllowedActive2018-12-31user212-555-1212NortheastFalsedefaultskh...@innodata.com5f4dcc3b5aa765d61d8327deb882cf99password112015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab002015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab00USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa2015-04-28T17:57:24.34+05:302015-04-28T21:57:24.34+05:302015-04-28T17:57:24.34+05:30false)

ML8 -
---
Input and converted output is same. i.e. without map:

{"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14 
PM":{"user":"UsersUSR-b740cbc5-5e27-8b1c-d97a-1329625f6b39lrunne...@wk.comlrunne...@wk.comActiveALLuser2013-12-25AllowedJohnDoeWK001LoadRunner5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00System2014-07-11T11:05:11.589-05:00yun.y...@wolterskluwer.com",
 "session":"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}

I suspect roxy router.xqy and request.xqy are creating some issues however not 
able to identify the issue yet.

Please suggest the fix for the same.

Other errors in Errorlog.txt-
-

2015-04-28 17:11:18.302 Notice: FilingsTool: Expecting map (INVALID-PARAMS):
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/controllers/setcache.xqy, 
at 29:22,
2015-04-28 17:11:18.302 Notice: FilingsTool: in 
xdmp:function(fn:QName("http://marklogic.com/roxy/controller/setcache","main";), 
"/app/controllers/setcache.xqy")() [1.0-ml]
2015-04-28 17:11:18.302 Notice: FilingsTool: $params = 
document{text{"{"USERSESSION:3aa2eb094d15d973e355d54812cf5720|4/28/2015 8:..."}}
2015-04-28 17:11:18.302 Notice: FilingsTool: $e = http://marklogic.com/xdmp/error error.xsd" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:error="http://marklogic.com/xdmp/error";>XDMP-UNEXPECTEDerr:XPST0003...
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/lib/router.xqy, at 97:16,
2015-04-28 17:11:18.302 Notice: FilingsTool: in router:route() [1.0-ml]
2015-04-28 17:11:18.302 Notice: FilingsTool: $x = ()
2015-04-28 17:11:18.302 Notice: FilingsTool: $valid-request = fn:true()
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/restful-router.xqy, at 
20:0 [1.0-ml]

Regards,
Indrajeet
_

Re: [MarkLogic Dev General] SQL Query Performance and Tableau Connection

2015-04-28 Thread Brad Rix
One comment about your specific query.You have a "left outer join" against 
the patientaddress table.  This can actually be written as an "inner join" with 
the same results since you are restricting the right side of the outer join 
with the where clause.   You can try your query as an inner join and see if 
that improves your performance.The relational database engine probably 
optimized the query by re-writing it as an inner join.If you did not have 
the "a.State='LA'", then it would have to remain an outer join.   The left 
outer join might be greatly effecting performance.Since you are also using 
group by and order by, the total amount of results might also effect the 
performance.

select c.Patientid, ClaimNumber, FirstName, LastName, sum (PharmacyTotalCost) 
as PharmacyTotalCost
  from claim c inner join patient p on c.patientid = p.patientid
  left outer join patientaddress a on p.patientid = a.patientid
  where a.[State] = 'LA'
  group by  c.PatientId, ClaimNumber, FirstName, LastName  order by 
SUM(PharmacyTotalCost) desc limit 10

Note:  You can see a reference in mysql as an example about when outer joins 
can be re-written to inner joins:  
https://dev.mysql.com/doc/refman/5.0/en/outer-join-simplification.html

Brad Rix | Technical Lead
O: +1 303 542 2172 | M: +1 303 915 2771
Skype: Brad.Rix | Google Talk: 
bradford@gmail.com

From: 
general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Sara Mazer
Sent: Monday, April 27, 2015 6:30 PM
To: MarkLogic Developer Discussion; ji...@rdacorp.com
Cc: Scott Fowler; Walt Rolle
Subject: Re: [MarkLogic Dev General] SQL Query Performance and Tableau 
Connection

Hi Cynthia,
My name is Sara Mazer, and I am a solutions consultant in the DC area.  I've 
worked a lot with Tableau and MarkLogic and would like to set up some time to 
discuss your evaluation and give you some best practices so that you can get 
the best performance possible.  I was the person who worked with Tableau to add 
MarkLogic as a named connection and supported their certification of MarkLogic, 
and I think I can help you.

I have a too many best practices to go over by email, so I'd like to speak with 
you and walk you through setting up databases for successful analytics through 
WebEx.  Would that be possible?  I have a meeting from 9:30-11:00 but can move 
any other meeting I have tomorrow to suit your schedule, just let me know a 
good time for you.

In general, I plan on:
Recommending the best data model for MarkLogic of this type of data (hint: it 
doesn't look like what would be in an RDBMs)
Showing you a demo of patient and claims data with Tableau and review how it 
was done
Discussing the best practices of working with Tableau (minimize the data coming 
back from MarkLogic by using custom SQL for charts, using MATCH keyword, having 
Tableau do grouping/sorting instead of in SQL)
Reviewing common connection issues between MarkLogic and Tableau and how to 
avoid them (most likely you are corrupting your SQL views:a best practice is to 
have a unique schema database for each MarkLogic database)
Reviewing performance of XQuery/JavaScript vs. SQL in MarkLogic

Best regards,
Sara Mazer
sara dot mazer "at" marklogic dot com
This e-mail and any accompanying attachments are confidential. The information 
is intended solely for the use of the individual to whom it is addressed. Any 
review, disclosure, copying, distribution, or use of this e-mail communication 
by others is strictly prohibited. If you are not the intended recipient, please 
notify us immediately by returning this message to the sender and delete all 
copies. Thank you for your cooperation.



From: 
general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Cynthia Jiang
Sent: Monday, April 27, 2015 4:56 PM
To: General@developer.marklogic.com
Cc: Walt Rolle
Subject: [MarkLogic Dev General] SQL Query Performance and Tableau Connection

Hello,

We have been testing MarkLogic 8 downloaded from your website for a POC.

Our environment is very straight forward.

A single VM (windows 7 Enterprise - 64bit) with 16GM RAM and 2 Intel (E5-2690 
v2) 3GHz processors.

We have around 4 SQL tables that we modeled in Json format so each row ends up 
becoming a document, and here are some of the stats:

Patient : 8 fields / 20,924 documents
PatientAddress: 5 fields / 36,193 documents
Claim: less than 30 fields / 335,349 documents
ClaimDiagnosis: 2 fields / 8,676 documents

We created element range index for all the fields, and created 4 views that 
each represent a SQL table.

Then we were able to use SQL to query the data and get the result in the query 
console.

But the performance is not nearly as what we expected, we

Re: [MarkLogic Dev General] JSON sending in the request body is not being converted into map:map in ML8 using roxy however working fine with ML6

2015-04-28 Thread David Ennis
HI.

The issue here may be is that json is a first-class citizen in MarkLogic 8
and is happy staying as json.

If all of your code expects a map, then perhaps you can test the type of
the document and if its json, transform it to the map on the fly.

Kind Regards,
David Ennis



Kind Regards,
David Ennis


David Ennis
*Content Engineer*

[image: HintTech]  
Mastering the value of content
creative | technology | content

Delftechpark 37i
2628 XJ Delft
The Netherlands
T: +31 88 268 25 00
M: +31 63 091 72 80

[image: http://www.hinttech.com] 
  


On 28 April 2015 at 16:14, Indrajeet Verma 
wrote:

> Hi,
>
> Please help us if you have encountered same kind of problem.
>
> We are upgrading ML6 to ML8 and using roxy as well. The roxy is customized
> based on the customer requirement.
>
>
> For one of the REST request "/setcache", we are sending below data to
> cache and that is being converted into map (key value pair)
>
> ML6-
>
> Request in Body-
> ---
>
> {"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14
> PM":{"user":" wk-pid=\"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39\">Users type=\"string\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39 type=\"string\">lrunne...@wk.com
> lrunne...@wk.com type=\"string\">ActiveALL type=\"string\">user type=\"string\">2013-12-25 type=\"string\"/> type=\"string\"/> type=\"string\">Allowed type=\"string\">John type=\"string\">Doe type=\"string\">WK001 type=\"string\">LoadRunner type=\"string\"/>5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00 type=\"string\">System2014-07-11T11:05:11.589-05:00
> yun.y...@wolterskluwer.com",
> "session":" 9:48:14
> PM\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}
>
> Converted into below map using ML6-
> --
>
> map:map(http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance";
> xmlns:map="http://marklogic.com/xdmp/map";> key="USERSESSION:97f7d8ec0b5bba928cb025bd1d4d28d4|4/28/2015 9:57:24
> PM"> wk-pid="USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa">USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaaUsers type="string">sjauh...@innodata.com type="string">shruti type="string">jauhari type="string">FORD-101Ford Motor
> Company type="string">Allowed type="string">Active type="string">2018-12-31 type="string">user type="string">212-555-1212 type="string">Northeast type="string">False type="string">default type="string">skh...@innodata.com5f4dcc3b5aa765d61d8327deb882cf99password112015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab002015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab00 key="session"> wk-pid="97f7d8ec0b5bba928cb025bd1d4d28d4|4/28/2015 9:57:24
> PM">USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa2015-04-28T17:57:24.34+05:302015-04-28T21:57:24.34+05:302015-04-28T17:57:24.34+05:30false)
>
> ML8 -
> ---
> Input and converted output is same. i.e. without map:
>
> {"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14
> PM":{"user":" wk-pid=\"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39\">Users type=\"string\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39 type=\"string\">lrunne...@wk.com
> lrunne...@wk.com type=\"string\">ActiveALL type=\"string\">user type=\"string\">2013-12-25 type=\"string\"/> type=\"string\"/> type=\"string\">Allowed type=\"string\">John type=\"string\">Doe type=\"string\">WK001 type=\"string\">LoadRunner type=\"string\"/>5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00 type=\"string\">System2014-07-11T11:05:11.589-05:00
> yun.y...@wolterskluwer.com",
> "session":" 9:48:14
> PM\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}
>
> I suspect roxy router.xqy and request.xqy are creating some issues however
> not able to identify the issue yet.
>
> Please suggest the fix for the same.
>
> Other errors in Errorlog.txt-
> -
>
> 2015-04-28 17:11:18.302 Notice: FilingsTool: Expecting map
> (INVALID-PARAMS):
> 2015-04-28 17:11:18.302 Notice: FilingsTool: in
> /app/controllers/setcache.xqy, at 29:22,
> 2015-04-28 17:11:18.302 Notice: FilingsTool: in xdmp:function(fn:QName("
> http://marklogic.com/roxy/controller/setcache","main";),
> "/app/controllers/setcache.xqy")() [1.0-ml]
> 2015-04-28 17:11:18.302 Notice: FilingsTool: $params =
> document{text{"{"USERSESSION:3aa2eb094d15d973e355d54812cf5720|4/28/2015
> 8:..."}}
> 2015-04-28 17:11:18.302 Notice: FilingsTool: $e =  xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance"; xmlns:error="
> http://marklogic.com/xdmp/error
> ">XDMP-UNEXPECTEDerr:XPST0003...
> 2015-04-28 17:11:18.302 N

[MarkLogic Dev General] JSON sending in the request body is not being converted into map:map in ML8 using roxy however working fine with ML6

2015-04-28 Thread Indrajeet Verma
Hi,

Please help us if you have encountered same kind of problem.

We are upgrading ML6 to ML8 and using roxy as well. The roxy is customized
based on the customer requirement.


For one of the REST request "/setcache", we are sending below data to cache
and that is being converted into map (key value pair)

ML6-

Request in Body-
---

{"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14
PM":{"user":"UsersUSR-b740cbc5-5e27-8b1c-d97a-1329625f6b39lrunne...@wk.com
lrunne...@wk.comActiveALLuser2013-12-25AllowedJohnDoeWK001LoadRunner5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00System2014-07-11T11:05:11.589-05:00
yun.y...@wolterskluwer.com",
"session":"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}

Converted into below map using ML6-
--

map:map(http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance";
xmlns:map="http://marklogic.com/xdmp/map";>USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaaUserssjauh...@innodata.comshrutijauhariFORD-101Ford Motor
CompanyAllowedActive2018-12-31user212-555-1212NortheastFalsedefaultskh...@innodata.com5f4dcc3b5aa765d61d8327deb882cf99password112015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab002015-04-22T15:37:32.151+05:30USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab00USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa2015-04-28T17:57:24.34+05:302015-04-28T21:57:24.34+05:302015-04-28T17:57:24.34+05:30false)

ML8 -
---
Input and converted output is same. i.e. without map:

{"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14
PM":{"user":"UsersUSR-b740cbc5-5e27-8b1c-d97a-1329625f6b39lrunne...@wk.com
lrunne...@wk.comActiveALLuser2013-12-25AllowedJohnDoeWK001LoadRunner5f4dcc3b5aa765d61d8327deb882cf99password112014-07-11T11:05:11.589-05:00System2014-07-11T11:05:11.589-05:00
yun.y...@wolterskluwer.com",
"session":"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b392015-04-28T17:48:14.661+05:302015-04-28T21:48:14.661+05:302015-04-28T17:48:14.661+05:30false"}}

I suspect roxy router.xqy and request.xqy are creating some issues however
not able to identify the issue yet.

Please suggest the fix for the same.

Other errors in Errorlog.txt-
-

2015-04-28 17:11:18.302 Notice: FilingsTool: Expecting map (INVALID-PARAMS):
2015-04-28 17:11:18.302 Notice: FilingsTool: in
/app/controllers/setcache.xqy, at 29:22,
2015-04-28 17:11:18.302 Notice: FilingsTool: in xdmp:function(fn:QName("
http://marklogic.com/roxy/controller/setcache","main";),
"/app/controllers/setcache.xqy")() [1.0-ml]
2015-04-28 17:11:18.302 Notice: FilingsTool: $params =
document{text{"{"USERSESSION:3aa2eb094d15d973e355d54812cf5720|4/28/2015
8:..."}}
2015-04-28 17:11:18.302 Notice: FilingsTool: $e = http://marklogic.com/xdmp/error error.xsd" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"; xmlns:error="
http://marklogic.com/xdmp/error
">XDMP-UNEXPECTEDerr:XPST0003...
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/lib/router.xqy, at
97:16,
2015-04-28 17:11:18.302 Notice: FilingsTool: in router:route() [1.0-ml]
2015-04-28 17:11:18.302 Notice: FilingsTool: $x = ()
2015-04-28 17:11:18.302 Notice: FilingsTool: $valid-request = fn:true()
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/restful-router.xqy, at
20:0 [1.0-ml]

Regards,
Indrajeet
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general


Re: [MarkLogic Dev General] SQL Query Performance and Tableau

2015-04-28 Thread Freddy Holwerda
top running for 20 minutes after adding one dimension and one simple 
> measure the columns and rows shelves.
> 
> We are under a very tight timeline and have to report our findings back to 
> our client within 24 hours. With the current performance, we will not be able 
> to recommend any MarkLogic product to our client.
> 
> Please help us if there is anything that we could do to improve the SQL query 
> performance and data connectivity between Tableau and MarkLogic.
> 
> Also, for the query as the simple example above, is there any XQuery or 
> Javascript Query that we can use that could produce the similar results? We 
> cannot find any documented examples.
> 
> Any help will be really appreciated!
> 
> Thank you very much,
> 
> Cynthia Jiang
> RDA Corpration
> 
> 
> -- next part --
> An HTML attachment was scrubbed...
> URL: 
> http://developer.marklogic.com/pipermail/general/attachments/20150428/c22d795f/attachment-0001.html
>  
> 
> --
> 
> Message: 3
> Date: Tue, 28 Apr 2015 11:22:56 +0530
> From: Shashidhar Rao 
> Subject: [MarkLogic Dev General] Is data also store in forest
> To: general@developer.marklogic.com
> Message-ID:
>   
> Content-Type: text/plain; charset="utf-8"
> 
> Hi,
> 
> We are running out of disk space due to the indexing .
> 
> Could some answer a basic question.
> 
> I have loaded data to Marklogic server from /home/data/data1 . Now the
> loading is finished and re-indexing and merging started.
> 
> Does Marklogic server still require data to be in /home/data/data1 or can
> this data be deleted .
> 
> I have always assumed that even after loading the data from the path above
> Marklogic still requires data to be in this folder.
> 
> Basically I want the data to be deleted once the loading is complete as
> space is running out due to heavy re-indexing.
> 
> Thanks
> -- next part --
> An HTML attachment was scrubbed...
> URL: 
> http://developer.marklogic.com/pipermail/general/attachments/20150428/67f143ed/attachment.html
>  
> 
> --
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general
> 
> 
> End of General Digest, Vol 130, Issue 106
> *
  ___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general


Re: [MarkLogic Dev General] Is data also store in forest

2015-04-28 Thread Shashidhar Rao
Thanks all for the info.

On Tue, Apr 28, 2015 at 11:22 AM, Shashidhar Rao  wrote:

> Hi,
>
> We are running out of disk space due to the indexing .
>
> Could some answer a basic question.
>
> I have loaded data to Marklogic server from /home/data/data1 . Now the
> loading is finished and re-indexing and merging started.
>
> Does Marklogic server still require data to be in /home/data/data1 or can
> this data be deleted .
>
> I have always assumed that even after loading the data from the path above
> Marklogic still requires data to be in this folder.
>
> Basically I want the data to be deleted once the loading is complete as
> space is running out due to heavy re-indexing.
>
> Thanks
>
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general


Re: [MarkLogic Dev General] Is data also store in forest

2015-04-28 Thread Geert Josten
It is wise to keep the free space twice or maybe even three times as large as 
the total forest sizes as a general rule.

All data, including binary is normally inserted into the forest itself, though 
large binaries are stored outside the search scope, but within MarkLogic data 
folder. The only exception to this is external binaries, but those are loaded 
in a separate way..

Kind regards,
Geert

From: 
"rajamani.marimu...@cognizant.com" 
mailto:rajamani.marimu...@cognizant.com>>
Reply-To: MarkLogic Developer Discussion 
mailto:general@developer.marklogic.com>>
Date: Tuesday, April 28, 2015 at 8:58 AM
To: "general@developer.marklogic.com" 
mailto:general@developer.marklogic.com>>
Subject: Re: [MarkLogic Dev General] Is data also store in forest

Hi Shashi,

Yes, You can delete the data once the ML ingestion s done. And always configure 
the great disk space for the forest more than the actual content size 
recommended by ML .

By
Raja >>>

From: 
general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Shashidhar Rao
Sent: Tuesday, April 28, 2015 11:23 AM
To: general@developer.marklogic.com
Subject: [MarkLogic Dev General] Is data also store in forest

Hi,
We are running out of disk space due to the indexing .
Could some answer a basic question.
I have loaded data to Marklogic server from /home/data/data1 . Now the loading 
is finished and re-indexing and merging started.

Does Marklogic server still require data to be in /home/data/data1 or can this 
data be deleted .
I have always assumed that even after loading the data from the path above 
Marklogic still requires data to be in this folder.
Basically I want the data to be deleted once the loading is complete as space 
is running out due to heavy re-indexing.
Thanks

This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general