[basex-talk] html document retrieval runs out of main memory

[Graydon Saunders]

Hello --

I'm using the basexgui to run (minus some identifying actual values defined
previously in the query)

(: for each path, retrieve the document :)
for $remote in $paths
  let $name as xs:string := file:name($remote)
  let $target as xs:string := file:resolve-path($name,$targetBase)
  let $fetched :=
    http:send-request(<http:request method='get'
override-media-type='application/octet-stream' username='{$id}'
password='{$pass}' />,
     $remote)[2]
  let $use as item() := try {
    html:parse($fetched)
  } catch * {
    $fetched
  }
  return if ($use instance of document-node())
     then file:write($target,$use)
     else file:write-binary($target,$use)

It works, in that I get exactly 100 documents retrieved.  (There are
unfortunately 140+ documents in the list.)

However, the query fails with an "out of main memory" error when using a
recent 10.0 beta or 9.7 with Xmx set to 2g.  Setting Xmx to 16g with 9.7
produces the same "out of memory" error in the same length of time (about 5
minutes).

java -version says
20:27 test % java -version
openjdk version "11.0.14.1" 2022-02-08
OpenJDK Runtime Environment 18.9 (build 11.0.14.1+1)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.14.1+1, mixed mode, sharing)

It's entirely possible I'm going about fetching files off a web server the
wrong way; it's possible there's something there that's rather large, but I
doubt it's that large.

What should I be doing instead?

Thanks!
Graydon