Re: [Scilab-users] Saving and Loading big data
On 10/23/2014 05:23 PM, Matiasb wrote: Thanks Antoine! yes, that is exactly the problem I'm experiencing. For the time being, I'm trying to do some ugly stuff with 'listvarinfile' and 'eval' to load variables one by one. Same for the save, saving variables into different files one by one helps (not if the problematic variable is only one of course). Other possibility, split your big data in smaller chunks: a_1=a(1:$/2); a_2=a($/2+1:$); save('a_1.sod','a_1'); save('a_2.sod','a_2'); Hope this gets solved soon or I will have to move to another platform (maybe ROOT) to handle big data. Arggh, no, don't. And if you do, have a look at julia (julialang.org). Antoine Cheers! -- View this message in context: http://mailinglists.scilab.org/Saving-and-Loading-big-data-tp4031419p4031433.html Sent from the Scilab users - Mailing Lists Archives mailing list archive at Nabble.com. ___ users mailing list users@lists.scilab.org http://lists.scilab.org/mailman/listinfo/users -- +++ Antoine Monmayrant LAAS - CNRS 7 avenue du Colonel Roche BP 54200 31031 TOULOUSE Cedex 4 FRANCE Tel:+33 5 61 33 64 59 email : antoine.monmayr...@laas.fr permanent email : antoine.monmayr...@polytechnique.org +++ ___ users mailing list users@lists.scilab.org http://lists.scilab.org/mailman/listinfo/users
Re: [Scilab-users] Saving and Loading big data
On 10/22/2014 04:02 PM, Matiasb wrote: Hi, Just yesterday I solved some issues I was experiencing with the save function by upgrading to scilab 5.5.1 as suggested by Samuel in this thread http://mailinglists.scilab.org/Problems-using-save-td4031410.html . Now, being able to save all the environment variables, I'm experiencing the next problem when there is a lot of data to be saved/loaded. The first one to have problems is the load function. My script created several (aprox 200) big matrices and when they are saved they produce a 850MB file. The save function works well in this case, but when I try to load the variables back again scilab runs out of memory as shown below. If the scripts creates even bigger matrices, scilab can handle them, but when I try calling save it does not work (also the problematic code below). Scilab can handle these big matrices, but is there anything I can do to make a reliable backup of them? For example increasing scilab's stacksize even more (which it states its maximum is 268435455, but my PC has more RAM can be used). Or using save/load in a more efficient way so that they don't require so much extra memory in the stack. Any help would be appreciated. Thanks you very much! Code samples Hi, I experienced similar issues when trying to save/load big data. 1) save/load do not seem to be really efficient and it looks like they create copies of variables to save/load. As a result, if a matrix uses more than 50% of the available stack, load tend to fail: %%% --n=2250,a=zeros(n,n);s=stacksize();s(2)/s(1) n = 2250. ans = 0.5071616 % a is using a bit more than 50% of the stack --save('/tmp/a.sod','a') %val=[a; !--error 17 stack size exceeded! Use stacksize function to increase it. Memory used for variables: 5094842 Intermediate memory needed: 5062522 Total memory available: 1000 in execstr instructioncalled by : at line 35 of function evstr called by : at line1003 of function %_save called by : save('/tmp/a.sod','a') %%% if we increase the stack, it works: --stacksize('max') --save('/tmp/a.sod','a') What's worse, load seem to try to allocate memory without checking allocation was successful: %%% scilab restarted with the default stacksize --load('/tmp/a.sod') !--error 42 A fatal error has been detected by Scilab. Your instance will probably quit unexpectedly soon. %%% 2) when trying to load huge sod file, you can try to load saved variables one by one instead of all of them at once. Hope it helps, Antoine *Load* I make the backup as follows: -- stacksize('max') // increase scilab memory -- save('bigFile.sod') And when I load the backup again I get the following error: -- stacksize('max') // increase scilab memory -- load('bigFile.sod) * !--error 17 stack size exceeded! Use stacksize function to increase it. Memory used for variables: 90307998 Intermediate memory needed: 180528916 Total memory available: 268435455 at line 971 of function %_sodload called by : load('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod')* Just to get an idea of the amout of variables and sizes: -- [names, typs, dims, vols] = listvarinfile('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod'); -- size(names) -- 237 -- sum(vols) -- 8.920D+08 Some of the matrices are big: Name Type SizeBytes CoreSwitchQueueR_discard constant 9 by 113654681831328 CoreSwitchQueueR_queueLe constant 9 by 113654681831328 CoreSwitchQueueR_t constant 9 by 113654681831328 CoreSwitchQueueR_waittim constant 9 by 113654681831328 *save* When make the backup I get the following error : -- stacksize('max') // increase scilab memory -- save('bigFile.sod') *stack size exceeded! Use stacksize function to increase it. Memory used for variables: 235237999 Intermediate memory needed: 44657871 Total memory available: 268435455 at line 71 of function evstr called by : at line1003 of function %_save called by : save(backupDirectory + logFileName + '.sod'); // saving all variables !--error 4 Undefined variable: %val* The output from 'whos' shows that there are huge matrices, but I can manipulate them without problem as long as I dont call save. -- whos Name Type Size Bytes . RejectedEv_arrived constant 15 by 299353592216 RejectedEv_evBuiltLatenc constant 15 by 299353592216 RejectedEv_L2Latency constant 15 by 299353592216 RejectedEv_packetLatency constant 15 by 299353592216 RejectedEv_roundTripLate constant 15 by 299353592216 RejectedEv_t constant 15 by 299353592216 HLTSV_idlePUsconstant 15 by 59872
Re: [Scilab-users] Saving and Loading big data
For information, I filled a bug about 'load' crashing scilab when trying to load a big matrix from a sod file: http://bugzilla.scilab.org/show_bug.cgi?id=13641 Cheers, Antoine ___ users mailing list users@lists.scilab.org http://lists.scilab.org/mailman/listinfo/users
Re: [Scilab-users] Saving and Loading big data
Thanks Antoine! yes, that is exactly the problem I'm experiencing. For the time being, I'm trying to do some ugly stuff with 'listvarinfile' and 'eval' to load variables one by one. Same for the save, saving variables into different files one by one helps (not if the problematic variable is only one of course). Hope this gets solved soon or I will have to move to another platform (maybe ROOT) to handle big data. Cheers! -- View this message in context: http://mailinglists.scilab.org/Saving-and-Loading-big-data-tp4031419p4031433.html Sent from the Scilab users - Mailing Lists Archives mailing list archive at Nabble.com. ___ users mailing list users@lists.scilab.org http://lists.scilab.org/mailman/listinfo/users
[Scilab-users] Saving and Loading big data
Hi, Just yesterday I solved some issues I was experiencing with the save function by upgrading to scilab 5.5.1 as suggested by Samuel in this thread http://mailinglists.scilab.org/Problems-using-save-td4031410.html . Now, being able to save all the environment variables, I'm experiencing the next problem when there is a lot of data to be saved/loaded. The first one to have problems is the load function. My script created several (aprox 200) big matrices and when they are saved they produce a 850MB file. The save function works well in this case, but when I try to load the variables back again scilab runs out of memory as shown below. If the scripts creates even bigger matrices, scilab can handle them, but when I try calling save it does not work (also the problematic code below). Scilab can handle these big matrices, but is there anything I can do to make a reliable backup of them? For example increasing scilab's stacksize even more (which it states its maximum is 268435455, but my PC has more RAM can be used). Or using save/load in a more efficient way so that they don't require so much extra memory in the stack. Any help would be appreciated. Thanks you very much! Code samples *Load* I make the backup as follows: -- stacksize('max') // increase scilab memory -- save('bigFile.sod') And when I load the backup again I get the following error: -- stacksize('max') // increase scilab memory -- load('bigFile.sod) * !--error 17 stack size exceeded! Use stacksize function to increase it. Memory used for variables: 90307998 Intermediate memory needed: 180528916 Total memory available: 268435455 at line 971 of function %_sodload called by : load('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod')* Just to get an idea of the amout of variables and sizes: -- [names, typs, dims, vols] = listvarinfile('ScilabLogs/local_39DCM_100HZ_20141022_142845.sod'); -- size(names) -- 237 -- sum(vols) -- 8.920D+08 Some of the matrices are big: Name Type SizeBytes CoreSwitchQueueR_discard constant 9 by 113654681831328 CoreSwitchQueueR_queueLe constant 9 by 113654681831328 CoreSwitchQueueR_t constant 9 by 113654681831328 CoreSwitchQueueR_waittim constant 9 by 113654681831328 *save* When make the backup I get the following error : -- stacksize('max') // increase scilab memory -- save('bigFile.sod') *stack size exceeded! Use stacksize function to increase it. Memory used for variables: 235237999 Intermediate memory needed: 44657871 Total memory available: 268435455 at line 71 of function evstr called by : at line1003 of function %_save called by : save(backupDirectory + logFileName + '.sod'); // saving all variables !--error 4 Undefined variable: %val* The output from 'whos' shows that there are huge matrices, but I can manipulate them without problem as long as I dont call save. -- whos Name Type Size Bytes . RejectedEv_arrived constant 15 by 299353592216 RejectedEv_evBuiltLatenc constant 15 by 299353592216 RejectedEv_L2Latency constant 15 by 299353592216 RejectedEv_packetLatency constant 15 by 299353592216 RejectedEv_roundTripLate constant 15 by 299353592216 RejectedEv_t constant 15 by 299353592216 HLTSV_idlePUsconstant 15 by 598727184656 HLTSV_queuedEventsToSend constant 15 by 598727184656 HLTSV_sent constant 15 by 598727184656 HLTSV_t constant 15 by 598727184656 coreSwitchQueueCapacity constant 1 by 1 24 CoreSwitchQueueR_discard constant 8 by 5582233 3.573D+08 CoreSwitchQueueR_queueLe constant 8 by 5582233 3.573D+08 CoreSwitchQueueR_t constant 9 by 5582233 4.019D+08 CoreSwitchQueueR_waittim constant 8 by 5582233 3.573D+08 . -- View this message in context: http://mailinglists.scilab.org/Saving-and-Loading-big-data-tp4031419.html Sent from the Scilab users - Mailing Lists Archives mailing list archive at Nabble.com. ___ users mailing list users@lists.scilab.org http://lists.scilab.org/mailman/listinfo/users