Re: [Go] High memory usage on CSV read into table

://docs.google.com/spreadsheets/d/1SE4S-wcKQ5cwlHoN7rQm7XOZLjI0HSyMje6q-zLvUHM/edit?usp=sharing Looks to me like the change has definitely helped, with memory usage dropping to around 300mb, although the usage doesn't really change that much once chunk size is > 1000 Daniel Harper h

Re: [Go] High memory usage on CSV read into table

Uncompressed $ ls -la concurrent_streams.csv -rw-r--r-- 1 danielharper 112M Nov 16 19:21 concurrent_streams.csv $ wc -l concurrent_streams.csv 1007481 concurrent_streams.csv Daniel Harper http://djhworld.github.io On Mon, 19 Nov 2018 at 21:55, Wes McKinney wrote: > I'm curious

Re: [Go] High memory usage on CSV read into table

on the other Arrow implementations (e.g. Java) to see if they allocate a similar amount. Daniel Harper http://djhworld.github.io On Mon, 19 Nov 2018 at 10:17, Sebastien Binet wrote: > hi Daniel, > On Sun, Nov 18, 2018 at 10:17 PM Daniel Harper > wrote: > > > Sorry just realise

Re: [Go] High memory usage on CSV read into table

Sorry just realised SVG doesn't work. PNG of the pprof can be found here: https://i.imgur.com/BVXv1Jm.png Daniel Harper http://djhworld.github.io On Sun, 18 Nov 2018 at 21:07, Daniel Harper wrote: > Wasn't sure where the best place to discuss this, but I've noticed that > when r