Keith, I'm still not sure how encoding 1-MB could take that long. Let's
try to figure out why. I'll take a naive approach to the problem, and
perhaps the difference in approaches will tell us where you are hitting
a problem.
Let's see... I have an old pdf of our developer guide on my laptop. It
is just about 2MB (2090571-B). Let's write a simple service to consume
it. This will just return the 64-bit hash as a hex string, and the
elapsed time to do so.
(: service.xqy :)
xdmp:integer-to-hex(
xdmp:hash64(
xdmp:get-request-body() ) ),
xdmp:elapsed-time()
Now let's test it with a loopback post.
(: test 2MB post :)
xdmp:http-post(
"http://localhost:8003/service.xqy";,
<options xmlns="xdmp:http">
<data>{
xs:base64Binary(
xdmp:document-get("/tmp/dev_guide.pdf"))
}</data>
</options>),
xdmp:elapsed-time()
=>
<response xmlns="xdmp:http">
<code>200</code>
<message>OK</message>
<headers>
<server>MarkLogic</server>
<content-type>text/plain; charset=UTF-8</content-type>
<content-length>28</content-length>
<connection>close</connection>
</headers>
</response>
e2b8e9636192cbe4
PT0.181185S
PT0.354548S
So that's about 2-MB in less that 360-ms, and roughly half of that
appears to be request time. Going over a real network rather than
loopback will increase the time a little bit, but it shouldn't be 9-sec.
-- Mike
On 2010-04-01 08:06, Keith L. Breinholt wrote:
I need to correct myself. It was 9 seconds to convert a 1MB image from binary
to hex string...not minutes.
My apologies to MarkLogic and anyone else who were wondering about the extreme
amount of time.
We've been spoiled by MarkLogic's response times for everything else we do. 9
seconds is more time than we want to spend on any request.
Keith L. Breinholt
breinhol...@ldschurch.org
-----Original Message-----
From: general-boun...@developer.marklogic.com
[mailto:general-boun...@developer.marklogic.com] On Behalf Of Geert Josten
Sent: Thursday, April 01, 2010 8:50 AM
To: General Mark Logic Developer Discussion
Subject: RE: [MarkLogic Dev General] Image cropping and binary manipulation
Hi Keith,
Do you now that you don't really need to do any calculation to encode binary?
You can simply apply string() to it to get a hex encoded string. But I agree
that sending binary as binary makes more sense...
Kind regards,
Geert
-----Original Message-----
From: general-boun...@developer.marklogic.com
[mailto:general-boun...@developer.marklogic.com] On Behalf Of
Keith L. Breinholt
Sent: donderdag 1 april 2010 16:39
To: General Mark Logic Developer Discussion
Subject: RE: [MarkLogic Dev General] Image cropping and
binary manipulation
It's not the sending so much as the fact that
xdmp:http-post() and xdmp:http-put() won't allow you to send
binary objects. So you have to hex encode the binary first
and then decode it at the other end.
We did several shots at this and on a 32-bit machine it was
taking close to 9 minutes to encode a 1MB file. Now I assume
that performance will be better on a 64-bit machine but even
if we cut the time in a quarter that's in the neighborhood of
2 minutes for something that should only take several seconds
at most to send and receive.
I've put a request for enhancement to MarkLogic because there
really isn't a good reason not to support sending binary objects.
In the meantime we're writing the file to an NFS mount and
having an external process do the manipulations and write it
back to a drop directory. Takes less than a minute in worst
case scenario and it doesn't tie up a MarkLogic server or its
resources.
Keith L. Breinholt
breinhol...@ldschurch.org
-----Original Message-----
From: general-boun...@developer.marklogic.com
[mailto:general-boun...@developer.marklogic.com] On Behalf Of
Geert Josten
Sent: Wednesday, March 31, 2010 11:45 PM
To: General Mark Logic Developer Discussion
Subject: RE: [MarkLogic Dev General] Image cropping and
binary manipulation
Hi Keith,
Have you actually measured the performance of sending
binaries back and forth to MLJAM? And did you compare with
plain HTTP attachment submission (or some other method)? I'd
be interested in knowing timing differences..
Kind regards,
Geert
drs. G.P.H. (Geert) Josten
Consultant
Daidalos BV
Hoekeindsehof 1-4
2665 JZ Bleiswijk
T +31 (0)10 850 1200
F +31 (0)10 850 1199
mailto:geert.jos...@daidalos.nl
http://www.daidalos.nl/
KvK 27164984
P Please consider the environment before printing this mail.
De informatie - verzonden in of met dit e-mailbericht - is
afkomstig van Daidalos BV en is uitsluitend bestemd voor de
geadresseerde. Indien u dit bericht onbedoeld hebt ontvangen,
verzoeken wij u het te verwijderen. Aan dit bericht kunnen
geen rechten worden ontleend.
Good suggestion Gert. Unfortunately the performance for the
encode/decode of the binary makes MLJAM a nonstarter.
I appreciate the suggestion though.
Keith L. Breinholt
breinhol...@ldschurch.org
_______________________________________________
General mailing list
General@developer.marklogic.com
http://xqzone.com/mailman/listinfo/general
NOTICE: This email message is for the sole use of the
intended recipient(s) and may contain confidential and
privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email
and destroy all copies of the original message.
_______________________________________________
General mailing list
General@developer.marklogic.com
http://xqzone.com/mailman/listinfo/general
_______________________________________________
General mailing list
General@developer.marklogic.com
http://xqzone.com/mailman/listinfo/general
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy all
copies of the original message.
_______________________________________________
General mailing list
General@developer.marklogic.com
http://xqzone.com/mailman/listinfo/general
_______________________________________________
General mailing list
General@developer.marklogic.com
http://xqzone.com/mailman/listinfo/general
