Re: [Scilab-users] Saving and Loading big data

2014-10-24 Thread Antoine Monmayrant

On 10/23/2014 05:23 PM, Matiasb wrote:

Thanks Antoine! yes, that is exactly the problem I'm experiencing.

For the time being, I'm trying to do some ugly stuff with 'listvarinfile'
and 'eval' to load variables one by one. Same for the save, saving variables
into different files one by one helps (not if the problematic variable is
only one of course).

Other possibility, split your big data in smaller chunks:

a_1=a(1:$/2);
a_2=a($/2+1:$);
save('a_1.sod','a_1');
save('a_2.sod','a_2');


Hope this gets solved soon or I will have to move to another platform (maybe
ROOT) to handle big data.


Arggh, no, don't.
And if you do, have a look at julia (julialang.org).

Antoine



Cheers!





--
View this message in context: 
http://mailinglists.scilab.org/Saving-and-Loading-big-data-tp4031419p4031433.html
Sent from the Scilab users - Mailing Lists Archives mailing list archive at 
Nabble.com.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users




--
+++

 Antoine Monmayrant LAAS - CNRS
 7 avenue du Colonel Roche
 BP 54200
 31031 TOULOUSE Cedex 4
 FRANCE

 Tel:+33 5 61 33 64 59
 
 email : antoine.monmayr...@laas.fr

 permanent email : antoine.monmayr...@polytechnique.org

+++

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users


Re: [Scilab-users] Saving and Loading big data

2014-10-23 Thread Antoine Monmayrant

On 10/22/2014 04:02 PM, Matiasb wrote:

Hi,

Just yesterday I solved some issues I was experiencing with the save
function by upgrading to scilab 5.5.1 as suggested by Samuel in  this thread
http://mailinglists.scilab.org/Problems-using-save-td4031410.html  .

Now, being able to save all the environment variables, I'm experiencing the
next problem when there is a lot of data to be saved/loaded.

The first one to have problems is the load function. My script created
several (aprox 200) big matrices and when they are saved they produce a
850MB file. The save function works well in this case, but when I try to
load the variables back again scilab runs out of memory as shown below. If
the scripts creates even bigger matrices, scilab can handle them, but when I
try calling save it does not work (also the problematic code below).

Scilab can handle these big matrices, but is there anything I can do to make
a reliable backup of them?
For example increasing scilab's stacksize even more (which it states its
maximum is 268435455, but my PC has more RAM can be used). Or using
save/load in a more efficient way so that they don't require so much extra
memory in the stack.

Any help would be appreciated.
Thanks you very much!

Code samples


Hi,

I experienced similar issues when trying to save/load big data.

1) save/load do not seem to be really efficient and it looks like they 
create copies of variables to save/load.
As a result, if a matrix uses more than 50% of the available stack, load 
tend to fail:


%%%
--n=2250,a=zeros(n,n);s=stacksize();s(2)/s(1)
 n  =

2250.
 ans  =

0.5071616  % a is using a bit more than 50% of the stack

--save('/tmp/a.sod','a')
%val=[a;
!--error 17
stack size exceeded!
Use stacksize function to increase it.
Memory used for variables: 5094842
Intermediate memory needed: 5062522
Total memory available: 1000
in  execstr instructioncalled by :
at line  35 of function evstr called by :
at line1003 of function %_save called by :
save('/tmp/a.sod','a')
%%%

if we increase the stack, it works:
--stacksize('max')
--save('/tmp/a.sod','a')

What's worse, load seem to try to allocate memory without checking 
allocation was successful:


%%% scilab restarted with the default stacksize
--load('/tmp/a.sod')
 !--error 42
A fatal error has been detected by Scilab.
Your instance will probably quit unexpectedly soon.

%%%


2) when trying to load huge sod file, you can try to load saved 
variables one by one instead of all of them at once.


Hope it helps,

Antoine


*Load*
I make the backup as follows:
-- stacksize('max') // increase scilab memory
-- save('bigFile.sod')

And when I  load the backup again I get the following error:
-- stacksize('max') // increase scilab memory
-- load('bigFile.sod)
* !--error 17
stack size exceeded!
Use stacksize function to increase it.
Memory used for variables: 90307998
Intermediate memory needed: 180528916
Total memory available: 268435455
at line 971 of function %_sodload called by :
load('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod')*

Just to get an idea of the amout of variables and sizes:
-- [names, typs, dims, vols] =
listvarinfile('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod');
-- size(names)   -- 237
-- sum(vols)   -- 8.920D+08

Some of the matrices are big:
Name Type   SizeBytes

CoreSwitchQueueR_discard constant   9 by 113654681831328
CoreSwitchQueueR_queueLe constant   9 by 113654681831328
CoreSwitchQueueR_t   constant   9 by 113654681831328
CoreSwitchQueueR_waittim constant   9 by 113654681831328


*save*
When make the backup I get the following error :
-- stacksize('max') // increase scilab memory
-- save('bigFile.sod')
*stack size exceeded!
Use stacksize function to increase it.
Memory used for variables: 235237999
Intermediate memory needed: 44657871
Total memory available: 268435455
at line  71 of function evstr called by :
at line1003 of function %_save called by :
save(backupDirectory + logFileName + '.sod'); // saving all variables
  
  !--error 4

Undefined variable: %val*

The output from 'whos' shows that there are huge matrices, but I can
manipulate them without problem as long as I dont call save.
-- whos
Name Type   Size   Bytes

.
RejectedEv_arrived   constant   15 by 299353592216
RejectedEv_evBuiltLatenc constant   15 by 299353592216
RejectedEv_L2Latency constant   15 by 299353592216
RejectedEv_packetLatency constant   15 by 299353592216
RejectedEv_roundTripLate constant   15 by 299353592216
RejectedEv_t constant   15 by 299353592216
HLTSV_idlePUsconstant   15 by 59872   

Re: [Scilab-users] Saving and Loading big data

2014-10-23 Thread Antoine Monmayrant
For information, I filled a bug about 'load' crashing scilab when trying 
to load a big matrix from a sod file:

http://bugzilla.scilab.org/show_bug.cgi?id=13641

Cheers,

Antoine

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users


Re: [Scilab-users] Saving and Loading big data

2014-10-23 Thread Matiasb
Thanks Antoine! yes, that is exactly the problem I'm experiencing.

For the time being, I'm trying to do some ugly stuff with 'listvarinfile'
and 'eval' to load variables one by one. Same for the save, saving variables
into different files one by one helps (not if the problematic variable is
only one of course).

Hope this gets solved soon or I will have to move to another platform (maybe
ROOT) to handle big data.

Cheers!





--
View this message in context: 
http://mailinglists.scilab.org/Saving-and-Loading-big-data-tp4031419p4031433.html
Sent from the Scilab users - Mailing Lists Archives mailing list archive at 
Nabble.com.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users


[Scilab-users] Saving and Loading big data

2014-10-22 Thread Matiasb
Hi,

Just yesterday I solved some issues I was experiencing with the save
function by upgrading to scilab 5.5.1 as suggested by Samuel in  this thread
http://mailinglists.scilab.org/Problems-using-save-td4031410.html  .

Now, being able to save all the environment variables, I'm experiencing the
next problem when there is a lot of data to be saved/loaded.

The first one to have problems is the load function. My script created
several (aprox 200) big matrices and when they are saved they produce a
850MB file. The save function works well in this case, but when I try to
load the variables back again scilab runs out of memory as shown below. If
the scripts creates even bigger matrices, scilab can handle them, but when I
try calling save it does not work (also the problematic code below).

Scilab can handle these big matrices, but is there anything I can do to make
a reliable backup of them?
For example increasing scilab's stacksize even more (which it states its
maximum is 268435455, but my PC has more RAM can be used). Or using
save/load in a more efficient way so that they don't require so much extra
memory in the stack. 

Any help would be appreciated.
Thanks you very much!

Code samples


*Load*
I make the backup as follows:
-- stacksize('max') // increase scilab memory
-- save('bigFile.sod')

And when I  load the backup again I get the following error:
-- stacksize('max') // increase scilab memory
-- load('bigFile.sod)
* !--error 17 
stack size exceeded!
Use stacksize function to increase it.
Memory used for variables: 90307998
Intermediate memory needed: 180528916
Total memory available: 268435455
at line 971 of function %_sodload called by :  
load('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod')*

Just to get an idea of the amout of variables and sizes:
-- [names, typs, dims, vols] =
listvarinfile('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod');
-- size(names)   -- 237
-- sum(vols)   -- 8.920D+08 

Some of the matrices are big: 
Name Type   SizeBytes

CoreSwitchQueueR_discard constant   9 by 113654681831328  
CoreSwitchQueueR_queueLe constant   9 by 113654681831328  
CoreSwitchQueueR_t   constant   9 by 113654681831328  
CoreSwitchQueueR_waittim constant   9 by 113654681831328 


*save*
When make the backup I get the following error :
-- stacksize('max') // increase scilab memory
-- save('bigFile.sod')
*stack size exceeded!
Use stacksize function to increase it.
Memory used for variables: 235237999
Intermediate memory needed: 44657871
Total memory available: 268435455
at line  71 of function evstr called by :  
at line1003 of function %_save called by :  
save(backupDirectory + logFileName + '.sod'); // saving all variables
 
 !--error 4 
Undefined variable: %val*

The output from 'whos' shows that there are huge matrices, but I can
manipulate them without problem as long as I dont call save.
-- whos
Name Type   Size   Bytes

.
RejectedEv_arrived   constant   15 by 299353592216  
RejectedEv_evBuiltLatenc constant   15 by 299353592216  
RejectedEv_L2Latency constant   15 by 299353592216  
RejectedEv_packetLatency constant   15 by 299353592216  
RejectedEv_roundTripLate constant   15 by 299353592216  
RejectedEv_t constant   15 by 299353592216   
HLTSV_idlePUsconstant   15 by 598727184656  
HLTSV_queuedEventsToSend constant   15 by 598727184656  
HLTSV_sent   constant   15 by 598727184656  
HLTSV_t  constant   15 by 598727184656   
coreSwitchQueueCapacity  constant   1 by 1 24   
CoreSwitchQueueR_discard constant   8 by 5582233   3.573D+08
CoreSwitchQueueR_queueLe constant   8 by 5582233   3.573D+08
CoreSwitchQueueR_t   constant   9 by 5582233   4.019D+08
CoreSwitchQueueR_waittim constant   8 by 5582233   3.573D+08  
.



--
View this message in context: 
http://mailinglists.scilab.org/Saving-and-Loading-big-data-tp4031419.html
Sent from the Scilab users - Mailing Lists Archives mailing list archive at 
Nabble.com.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users