Re: Out of memory for neural network

[LUYAO CHEN]

Hi NJ,


I use the same date set for the issue 
https://issues.apache.org/jira/browse/MADLIB-1257.


It has 23K instances / 300 features in 263 groups.

SELECT madlib.mlp_classification(
    'train_data',      -- Source table
    'mlp_model',      -- Destination table
    'features',     -- Input features
    'positive',     -- Label
    ARRAY[5],         -- Number of units per layer
    'learning_rate_init=0.003,
    n_iterations=500,
    tolerance=0',     -- Optimizer params
    'tanh',           -- Activation function
    NULL,             -- Default weight (1)
    FALSE,            -- No warm start
    true,             -- verbose
    'caseid'         -- Grouping
);
PostgreSQL memory setup.

 checkpoint_completion_target = '0.9';
 default_statistics_target = '500';
 effective_cache_size = '12GB';
 effective_io_concurrency = '200';
 maintenance_work_mem = '2GB';
 max_connections = '20';
 max_parallel_workers = '4';
 max_parallel_workers_per_gather = '2';
 max_wal_size = '8GB';
 max_worker_processes = '4';
 min_wal_size = '4GB';
 random_page_cost = '1.1';
 shared_buffers = '4GB';
 wal_buffers = '16MB';
 work_mem = '52428kB';

also i setup sysctl -w vm.overcommit_memory=2 to avoide the crash of postmaster

Regards,
Luyao


________________________________
From: Nandish Jayaram <njaya...@pivotal.io>
Sent: Friday, July 27, 2018 10:05 AM
To: user@madlib.apache.org
Subject: Re: Out of memory for neural network

Hi Luyao,

Can you kindly include details about the database settings (btw, are you using 
Postgres or Greenplum?) and the training dataset size used?
If your dataset is publicly available, please do share the details of where we 
can access it too.

NJ

Sent from my iPhone

On Jul 27, 2018, at 7:41 AM, LUYAO CHEN 
<luyao_c...@hotmail.com<mailto:luyao_c...@hotmail.com>> wrote:


Dear user community,


I would report a problem regarding in neural network. I am using a 16G RAM 
machine.


After some iterations (~100 ). I got the below error


ERROR:  spiexceptions.OutOfMemory: out of memory
DETAIL:  Failed on request of size 32800.
CONTEXT:  Traceback (most recent call last):
  PL/Python function "mlp_classification", line 36, in <module>
    grouping_col
  PL/Python function "mlp_classification", line 45, in wrapper
  PL/Python function "mlp_classification", line 325, in mlp
  PL/Python function "mlp_classification", line 580, in update
PL/Python function "mlp_classification"

Below is the command ,

SELECT madlib.mlp_classification(
    'train_data_sub',      -- Source table
    'mlp_model',      -- Destination table
    'features',     -- Input features
    'positive',     -- Label
    ARRAY[5],         -- Number of units per layer
    'learning_rate_init=0.003,
    n_iterations=500,
    tolerance=0',     -- Optimizer params
    'tanh',           -- Activation function
    NULL,             -- Default weight (1)
    FALSE,            -- No warm start
    true,             -- verbose
    'case_icd'         -- Grouping
);

Is that a problem or just caused by the data size?

Regards,
Luyao Chen