On Wednesday, November 1, 2017 at 6:36:25 PM UTC+2, Felipe da Veiga
Leprevost wrote:
>
> Hey;
>
> I'm developing a pipeline that takes as input XML files with a reasonable
> complex structure and size, the nested structure might have several layers
> and these files can go from 900 MB to 3 GB in size. I can't change that
> because it's not on my control how they are formed, I just need to read
> them.
>
> Right now I have the pipeline working by mapping a nested group of structs
> to the XML file, something like this:
>
> (code already works, this is just to server as an example on how I'm doing)
> ```
> xmlFile, e := os.Open(f)
> if e != nil {
> return e
> }
>
> defer xmlFile.Close()
>
> b, _ := ioutil.ReadAll(xmlFile)
>
> var xmlHead XmlHeadStruct
>
> reader := bytes.NewReader(b)
> decoder := xml.NewDecoder(reader)
> decoder.CharsetReader = charset.NewReader
>
> if e = decoder.Decode(&xmlHead); e != nil {
> return e
> }
> ```
>
> Now, everything works perfectly but with one exception, the memory usage!
> A file with approximately 900 MB is consuming 4 GB RAM and one with 3.3 GB
> can consume almost 14 GB RAM. Not cool!
>
> Do you guys have any suggestions on how to circumvent this ?
>
One option is to use "streaming mode", see ie
http://blog.davidsingleton.org/parsing-huge-xml-files-with-go/
HTH
ain
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
