[
https://issues.apache.org/jira/browse/MADLIB-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Domino Valdano updated MADLIB-1340:
-----------------------------------
Description:
The minibatcher's internal logic for picking a default batch size isn't strict
enough. It can crash for arrays of datatypes which are less than 32-bits. I
tried to come up with a simple repro, but it still needs some work. Here's
what I have now, for 16-bit type REAL[], haven't had a chance to test it yet:
madlib=# CREATE TABLE foo AS SELECT ARRAY[i,i,i,i,i] AS x, 1 as y FROM (SELECT
ARRAY[i,i,i,i,i] AS i FROM (SELECT GENERATE_SERIES(1,6*1024*1024) AS i) a1 ) a;
madlib=# \d foo;
Table "public.foo"
Column | Type | Modifiers
--------+---------+-----------
x | integer[] |
y | integer |
Distributed randomly
madlib=# SELECT madlib.minibatch_preprocessor_dl('foo','foo_batched', 'y',
'x');
TODO: above example doesn't actually work, because it only has 6-million rows.
Generate an example with at least 150-million rows, and it should work (ie,
crash).
was:
The minibatcher's internal logic for picking a default batch size isn't strict
enough. It can crash for arrays of datatypes which are less than 32-bits. I
tried to come up with a simple repro, but it still needs some work. Here's
what I have now, for 16-bit type REAL[], haven't had a chance to test it yet:
madlib=# CREATE TABLE foo AS SELECT ARRAY[i,i,i,i,i] AS x, 1 as y FROM (SELECT
ARRAY[i,i,i,i,i] AS i FROM (SELECT GENERATE_SERIES(1,6*1024*1024) AS i) a1 ) a;
madlib=# \d foo;
Table "public.foo"
Column | Type | Modifiers
--------+---------+-----------
x | integer[] |
y | integer |
Distributed randomly
madlib=# SELECT madlib.minibatch_preprocessor_dl('foo','foo_batched', 'y',
'x');
> minibatch_preprocessor_dl crashes with default batch size
> ---------------------------------------------------------
>
> Key: MADLIB-1340
> URL: https://issues.apache.org/jira/browse/MADLIB-1340
> Project: Apache MADlib
> Issue Type: Bug
> Components: Deep Learning
> Affects Versions: v1.16
> Reporter: Domino Valdano
> Priority: Minor
> Fix For: v1.16
>
>
> The minibatcher's internal logic for picking a default batch size isn't
> strict enough. It can crash for arrays of datatypes which are less than
> 32-bits. I tried to come up with a simple repro, but it still needs some
> work. Here's what I have now, for 16-bit type REAL[], haven't had a chance
> to test it yet:
> madlib=# CREATE TABLE foo AS SELECT ARRAY[i,i,i,i,i] AS x, 1 as y FROM
> (SELECT ARRAY[i,i,i,i,i] AS i FROM (SELECT GENERATE_SERIES(1,6*1024*1024) AS
> i) a1 ) a;
> madlib=# \d foo;
> Table "public.foo"
> Column | Type | Modifiers
> --------+---------+-----------
> x | integer[] |
> y | integer |
> Distributed randomly
> madlib=# SELECT madlib.minibatch_preprocessor_dl('foo','foo_batched', 'y',
> 'x');
> TODO: above example doesn't actually work, because it only has 6-million
> rows. Generate an example with at least 150-million rows, and it should work
> (ie, crash).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
