I don't know if the CGo transitions for 16 million images will completely
swamp the speed gains, or how much post-processing you need to do to these
images, but you might try https://github.com/h2non/bimg and see if it gives
you any wins. It claims 'typically 4x faster' then the Go Image package
Just parallelize in the cloud. At a minimum parallelize locally with multiple
Go routines.
> On Jun 26, 2020, at 7:51 AM, howardcs...@gmail.com wrote:
>
>
> I don't know if the CGo transitions for 16 million images will completely
> swamp the speed gains, or how much post-processing you need
Thanks, I am already maxing out some servers but wondered if it could be
sped up.
I will give the bimg package a go!
On Friday, June 26, 2020 at 1:55:40 PM UTC+1 ren...@ix.netcom.com wrote:
> Just parallelize in the cloud. At a minimum parallelize locally with
> multiple Go routines.
>
> On J
Without information about the particular images - the data set you're
working with - it's hard to provide any accurate advice. If the images are
small, maybe it's all overhead. If the images are of a particular form of
PNG, maybe a fast path is missing. And so on.
To get meaningful help for proble
Thanks Rob,
Yes you are right these images are small. <100kb each with an example
below. And as I mentioned I have millions of these.
I am able to max the CPUs out with a worker pool but that is when I noticed
a lot of the work was actually decoding the PNG images which made we wonder
if if was a
Just because the bulk of the time is in the decode doesn’t mean the decode is
inefficient or can be improved upon. It might be the most expensive stage in
the process regardless of the implementation.
> On Jun 27, 2020, at 12:15 AM, Lee Armstrong wrote:
>
>
> Thanks Rob,
>
> Yes you are ri
An easy "comfort" exercise:
save 1000 images to a directory in png form.
decode with go and with several tools such as imagemagick
compare times
for format in jpeg, tiff, ...
convert all images to that format; do the test above anew
the result will be a comparison chart of Go existing image deco
On Jun 27, 2020, at 8:30 AM, Robert Engels wrote:
>
> Just because the bulk of the time is in the decode doesn’t mean the decode is
> inefficient or can be improved upon. It might be the most expensive stage in
> the process regardless of the implementation.
This is I think the most important
