From help-octave-request at bevo dot che dot wisc dot edu Wed Feb 2 06:09:36 2000 Subject: Re: memory problem From: Daniel Heiserer To: jwe at bevo dot che dot wisc dot edu Cc: help-octave at bevo dot che dot wisc dot edu Date: Wed, 02 Feb 2000 13:08:55 +0100 jwe at bevo dot che dot wisc dot edu wrote: > > On 1-Feb-2000, Daniel Heiserer wrote: > > | The machine has sufficient memory to process my request. > > Apparently not. :-( > > How much virtual memory do you have? > > | octave:2> tic;eval(sprintf("L_%08d=eye(1000);",[1:1000]));toc > | error: memory exhausted -- trying to return to prompt > | > | ......... > | wd matrix 1000 1000 L_00000232 > | wd matrix 1000 1000 L_00000233 > | > | ==> octave could create only 233 matrizes. > > That's about 233 * 1000 * 1000 * 8 ==> 1.8GB of data (unfortunately, > even though eye produces a diagonal matrix, Octave currently uses a > full matrix to store it). I know. I have 16Gb of memory. :-) > > | In matlab it works fine. See below. > | > | I had an experience with fortran before, > | where I had a memory limitation inside > | a subroutine, but defining everything as > | a common block it worked. Maybe that is > | a hint .......... > | > | matlab: > | >> tic;eval(sprintf('L_%08d=eye(1000);',[1:1000]));toc > | > | elapsed_time = > | > | 72.8044 > | ................... > | > | L_00000999 1000x1000 8000000 double array > | L_00001000 1000x1000 8000000 double array > | > | Grand total is 1000000000 elements using 8000000000 bytes > > Thats about 8GB of data. Do you really have that much virtual memory I know that these are 8Gb. I have 16GB. > available? If not, I'd guess that Matlab is actually using a smarter > storage scheme (either storing only the diagonal, or even just > recognizing an identity matrix as special, or recognizing that it > really only needs data for one 1000x1000 identity matrix that can be > shared among all your identifiers). > No. matlab stores all matrices as full unless i tell matlab that the matrix is sparse. And even then he stores null elements if they haven't been excluded. Sparse stuff in matlab is not perfect. The limits are: cputime unlimited filesize unlimited datasize unlimited stacksize 524288 kbytes coredumpsize unlimited memoryuse 524288 kbytes vmemoryuse unlimited descriptors 200 Main memory size: 16384 Mbytes > What happens if you replace eye() with rand()? Do you still have > enough memory to do this operation? ---------------------------------------------------------------------- matlab: >> tic;eval(sprintf('L_%08d=rand(1000);',[1:1000]));toc elapsed_time = 229.8259 >> whos Name Size Bytes Class L_00000001 1000x1000 8000000 double array L_00000002 1000x1000 8000000 double array L_00000003 1000x1000 8000000 double array ................... L_00000999 1000x1000 8000000 double array L_00001000 1000x1000 8000000 double array Grand total is 1000000000 elements using 8000000000 bytes >> tic;eval(sprintf('clear L_%08d;',[100:1000]));toc elapsed_time = 0.5240 ---------------------------------------------------------------------- GNU Octave, version 2.0.14 (mips-sgi-irix6.4). Copyright (C) 1996, 1997, 1998, 1999 John W. Eaton. This is free software with ABSOLUTELY NO WARRANTY. For details, type `warranty'. octave:3> tic;eval(sprintf('L_%08d=eye(1000);',[1:1000]));toc error: memory exhausted -- trying to return to prompt octave:3> whos *** currently compiled functions: prot type rows cols name ==== ==== ==== ==== ==== wd user function - - clock wd user function - - tic *** local user variables: prot type rows cols name ==== ==== ==== ==== ==== wd matrix 1000 1000 L_00000001 wd matrix 1000 1000 L_00000002 .......................... wd matrix 1000 1000 L_00000229 wd matrix 1000 1000 L_00000230 wd matrix 1000 1000 L_00000231 octave:4> ---------------------------------------------------------------------- Trying the rand function is HOPELESS in octave. Some benchmarks before showed me, that rand is one of the operations where matlab is factors faster. I stopped the "tic;eval(sprintf('L_%08d=rand(1000);',[1:1000]));toc" after a few minutes and he had only done 38 matrices! Regardless of the inefficieny of "rand" i just have no idea why octave cannot allocate the 8GB. Daniel ----------------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web: http://www.che.wisc.edu/octave/octave.html How to fund new projects: http://www.che.wisc.edu/octave/funding.html Subscription information: http://www.che.wisc.edu/octave/archive.html -----------------------------------------------------------------------