From bug-octave-request at bevo dot che dot wisc dot edu Sun Dec 19 10:34:09 1999 Subject: LateFeedback(Gilsing,19.12.1999) From: Hagen Gilsing To: bug-octave at bevo dot che dot wisc dot edu Date: Sun, 19 Dec 1999 10:34:12 -0600 (CST) 1. Bugreport, 11.03.1999,15:24:59, split.m ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >From your answer: > When making patches, please don't reformat the existing code > unnecessarily. It makes the patch larger and it makes it harder to > find the actual changes. Whitespace is not evil. :-) I changed more than was necessary to patch the error, because avoiding unnecessary calls of other scripts makes split1 faster than split (split - your patch, split1 - my patch, time comparison: Line 207). In general, I prefere compact writing. As the essential part consists of some 10 lines, I didn't expext this to cause problems. But you are right, I should have written it with more white spaces to make it easier for you. I'm sorry. 2. Bugreport, 20.02.1999,19:11:00, various problems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I described memory problems with large matrices when using Octave. The reason was an error in glibc. To remember: - Bugreport sent on 20.02.1999, 19:11:00 (Line 31) - Problems with glibc (Line 87) Bugreport sent on 20.02.1999, 19:11:00 -------------------------------------- ... a) octave-2.1.12:1> clear;OCTAVE_VERSION OCTAVE_VERSION = 2.1.12 ans = 2.1.12 octave-2.1.12:2> x=zeros(1,10^8); error: memory exhausted -- trying to return to prompt octave-2.1.12:2> x=zeros(1,10^8); error: memory exhausted -- trying to return to prompt ... b) octave-2.1.12:1> clear;OCTAVE_VERSION OCTAVE_VERSION = 2.1.12 ans = 2.1.12 octave-2.1.12:2> x=zeros(1,10^9); error: Segmentation fault -- stopping myself... attempting to save variables to `octave-core'... save to `octave-core' complete Segmentation fault ../Octave990219 03:45:53 503 # ... c) octave-2.1.12:1> clear;OCTAVE_VERSION OCTAVE_VERSION = 2.1.12 ans = 2.1.12 octave-2.1.12:2> x=zeros(10^4); error: memory exhausted -- trying to return to prompt octave-2.1.12:2> x=zeros(10^4); error: memory exhausted -- trying to return to prompt ... d) octave-2.1.12:1> clear;OCTAVE_VERSION OCTAVE_VERSION = 2.1.12 ans = 2.1.12 octave-2.1.12:2> x=zeros(10^5); error: Segmentation fault -- stopping myself... attempting to save variables to `octave-core'... save to `octave-core' complete Segmentation fault ../Octave990219 02:42:49 503 # ... The examples b), d), e), f) demonstrate, that Octave is insecure in cases of large memory requirements. Such re- quirements occure for instance, if one would do simulations like simulations of Stochastic Differential Equations, where computed path should be kept, accompanying path statistics is required, the problems are parametrized with multitud of parameters, etc. . So there's no problem at all in reaching the memory limits. ... Problems with glibc ------------------- >Submitter-Id: net >Originator: gilsing at informatik dot hu-berlin dot de >Organization: no >Confidential: no >Synopsis: Malloc fails (large sizes) >Severity: serious >Priority: medium >Category: libc >Class: sw-bug >Release: glibc-2.0.7pre6 >Environment: i686,Linux 2.0.36,i686,glibc-2.0.7pre6 Host type: i686-unknown-linux-gnu System: Linux Merlin 2.0.36 #40 Tue Feb 2 10:26:45 CET 1999 i686 unknown Architecture: i686 Addons: crypt linuxthreads localedata Build CFLAGS: -g -O2 Build CC: gcc (egcs-2.91.60) Build shared: yes Build profile: no Build omitfp: no Stdio: libio >Description: When trying to allocate large chunks of memory, it can occur, that malloc returns a non-zero pointer although the requested memory is not available. The is caused by request2size. #define request2size(req) \ (((long)((req) + (SIZE_SZ + MALLOC_ALIGN_MASK)) < \ (long)(MINSIZE + MALLOC_ALIGN_MASK)) ? MINSIZE : \ (((req) + (SIZE_SZ + MALLOC_ALIGN_MASK)) & ~(MALLOC_ALIGN_MASK))) For req of type size_t holds, denoting maxst=(size_t)(-1), ((long)((req) + (SIZE_SZ + MALLOC_ALIGN_MASK)) < \ (long)(MINSIZE + MALLOC_ALIGN_MASK)) <=> ((maxst/2)+1 - (SIZE_SZ+MALLOC_ALIGN_MASK)) < req < maxst So for req > ((maxst/2)+1-(SIZE_SZ+MALLOC_ALIGN_MASK)), request2size always returns MINSIZE. As request2size in malloc returns the adjusted number of bytes to allocate, this is a potential source for subsequent segmentation faults. To a large extend, the problem could be avoided by replaceing long by INTERNAL_SIZE_T in request2size. But for very large values adding (SIZE_SZ+MALLOC_ALIGN_MASK) to the requested number of bytes could result in an "size_t-overflow" and cause the same problem. Further potential sources of "size_t-overflows" are the assignments size = (size + SIZE_SZ + page_mask) & ~page_mask (in mmap_chunk) INTERNAL_SIZE_T sbrk_size = nb + top_pad + MINSIZE; (in memory_extend_top) which add different numbers to the user request, f.i. page_mask=4K-1 which is greater than (SIZE_SZ+MALLOC_ALIGN_MASK). To meet the specification, I would propose to change long into INTERNAL_SIZE_T in macro request2size. In order to prevent "size_t-overflows", I would propose per function checks like f.i. if(bytes+SIZE_SZ+MALLOC_ALIGN_MASKFrom aj at arthur dot rhein-neckar dot de Wed Jul 7 08:04:57 1999 Received: from mail.informatik.hu-berlin.de (mail [141.20.20.50]) by sigma.informatik.hu-berlin.de (8.9.1/8.9.1/INF-2.0-MA-UNIX-4.0D) with ESMTP id IAA15374 for ; Wed, 7 Jul 1999 08:04:51 +0200 (MET DST) Received: from news-ma.rhein-neckar.de (news-ma.rhein-neckar.de [193.197.90.3]) by mail.informatik.hu-berlin.de (8.9.1/8.9.1/INF-2.0-MA-SOLARIS-2.7) with ESMTP id IAA02166 for ; Wed, 7 Jul 1999 08:04:50 +0200 (MET DST) Received: from arthur.rhein-neckar.de (uucp at localhost) by news-ma.rhein-neckar.de (8.8.8/8.8.8) with bsmtp id IAA17590; Wed, 7 Jul 1999 08:04:47 +0200 (CEST) (envelope-from aj at arthur dot rhein-neckar dot de) Received: from aj by arthur.rhein-neckar.de with local (Exim 3.02 #1) id 111kPO-0002MX-00; Wed, 07 Jul 1999 07:38:14 +0200 Mail-Copies-To: never To: Hagen Gilsing Cc: bugs at gnu dot org Subject: Re: libc/1045: Malloc fails on large sizes References: <199903250433 dot FAA16519 at data dot informatik dot hu-berlin dot de> From: Andreas Jaeger Date: 07 Jul 1999 07:38:14 +0200 In-Reply-To: Hagen Gilsing's message of "Thu, 25 Mar 1999 05:33:58 +0100 (MET)" Message-ID: Lines: 153 User-Agent: Gnus/5.070092 (Pterodactyl Gnus v0.92) XEmacs/21.1 (Acadia) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Status: RO >>>>> Hagen Gilsing writes: >> Number: 1045 >> Category: libc >> Synopsis: Malloc fails (large sizes) Hi Hagen, sorry for the late answer. Wolfram has made the appended patch which should fix your problem. The patch will be in glibc 2.1.2. Andreas 1999-07-04 Wolfram Gloger * malloc/malloc.c (request2size): Check for overflow and return NULL whenever it is encountered. --- ptmalloc.c 1999/07/04 20:34:03 1.1.1.10 +++ ptmalloc.c 1999/07/04 21:08:24 at @ -1256,12 +1256,12 @@ ... Split-timing ------------ octave-2.1.12:1> clear;fprintf(2,'%s\n',OCTAVE_VERSION);fflush(2); 2.1.12 octave-2.1.12:2> itest=15; octave-2.1.12:3> s='abc\"def\"ghi\"jkl';t='"'; octave-2.1.12:4> for i=1:itest > tm2=clock;split(s,t);tm1=clock; > fprintf(2,"split: %7.3f sec",etime(tm1,tm2));fflush(2); > tm2=clock;split1(s,t);tm1=clock; > fprintf(2,", split1: %7.3f sec\n",etime(tm1,tm2));fflush(2); > endfor; split: 0.073 sec, split1: 0.025 sec split: 0.014 sec, split1: 0.007 sec split: 0.013 sec, split1: 0.008 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.013 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.008 sec split: 0.012 sec, split1: 0.007 sec split: 0.012 sec, split1: 0.007 sec octave-2.1.12:5> a=31.5+10^-6;b=95-2*10^-6;ls=10000;lt=1; octave-2.1.12:6> s=setstr(round(a+b*rand(1,ls)));t=setstr(round(a+b*rand(1,lt))); octave-2.1.12:7> for i=1:itest > tm2=clock;split(s,t);tm1=clock; > fprintf(2,"split: %7.3f sec",etime(tm1,tm2));fflush(2); > tm2=clock;split1(s,t);tm1=clock; > fprintf(2,", split1: %7.3f sec\n",etime(tm1,tm2));fflush(2); > endfor; split: 1.752 sec, split1: 1.424 sec split: 1.632 sec, split1: 1.473 sec split: 1.682 sec, split1: 1.459 sec split: 1.665 sec, split1: 1.453 sec split: 1.665 sec, split1: 1.444 sec split: 1.674 sec, split1: 1.454 sec split: 1.686 sec, split1: 1.453 sec split: 1.678 sec, split1: 1.439 sec split: 1.659 sec, split1: 1.459 sec split: 1.665 sec, split1: 1.461 sec split: 1.665 sec, split1: 1.460 sec split: 1.685 sec, split1: 1.445 sec split: 1.678 sec, split1: 1.449 sec split: 1.680 sec, split1: 1.445 sec split: 1.656 sec, split1: 1.453 sec octave-2.1.12:8> exit ../reports 02:25:49 534 # Configuration (please do not edit this section): ----------------------------------------------- uname output: Linux Merlin 2.0.36 #40 Tue Feb 2 10:26:45 CET 1999 i686 unknown configure opts: --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --datadir=/usr/share --libexecdir=/usr/libexec --libdir=/usr/lib --includedir=/usr/include --infodir=/usr/info/octave-2.1.12 --mandir=/usr/man --srcdir=/root/MathsPlotting/Octave/octave-2.1.12.src --enable-dl --enable-static --enable-shared --target=i686-pc-linux-gnu Fortran compiler: g77 FFLAGS: -O F2C: F2CFLAGS: FLIBS: -lg2c -lm -L/usr/lib/gcc-lib/i486-linux/egcs-2.91.60 -L/usr/i486-linux/lib -lm CPPFLAGS: INCFLAGS: -I/usr/include/octave-2.1.12/octave -I/usr/include C compiler: gcc, version 2.91.60 19981201 (egcs-1.1.1 release) CFLAGS: -DHAVE_CONFIG_H -mieee-fp -g -O2 -Wall CPICFLAG: -fPIC C++ compiler: c++, version 2.91.60 19981201 (egcs-1.1.1 release) CXXFLAGS: -DHAVE_CONFIG_H -mieee-fp -fno-rtti -fno-exceptions -fno-implicit-templates -g -O2 -Wall CXXPICFLAG: -fPIC LDFLAGS: -g LIBFLAGS: -L/usr/lib/octave-2.1.12 RLD_FLAG: -Xlinker -rpath -Xlinker /usr/lib/octave-2.1.12 TERMLIBS: -lncurses LIBS: LEXLIB: LIBPLPLOT: LIBDLFCN: LIBGLOB: ./glob/libglob.a DEFS: -DOCTAVE_SOURCE=1 -DSEPCHAR=':' -DSEPCHAR_STR=":" -DUSE_READLINE=1 -D__NO_MATH_INLINES=1 -DCXX_NEW_FRIEND_TEMPLATE_DECL=1 -DHAVE_LIBM=1 -DF77_APPEND_UNDERSCORE=1 -DSIZEOF_SHORT=2 -DSIZEOF_INT=4 -DSIZEOF_LONG=4 -DHAVE_ALLOCA_H=1 -DHAVE_ALLOCA=1 -DNPOS=string::npos -DSTDC_HEADERS=1 -DHAVE_DIRENT_H=1 -DTIME_WITH_SYS_TIME=1 -DHAVE_SYS_WAIT_H=1 -DHAVE_ASSERT_H=1 -DHAVE_CURSES_H=1 -DHAVE_DLFCN_H=1 -DHAVE_FCNTL_H=1 -DHAVE_FLOAT_H=1 -DHAVE_FNMATCH_H=1 -DHAVE_GLOB_H=1 -DHAVE_GRP_H=1 -DHAVE_LIMITS_H=1 -DHAVE_MEMORY_H=1 -DHAVE_NAN_H=1 -DHAVE_NCURSES_H=1 -DHAVE_PWD_H=1 -DHAVE_SGTTY_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_PARAM_H=1 -DHAVE_SYS_RESOURCE_H=1 -DHAVE_SYS_SELECT_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_SYS_TIMES_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_UTSNAME_H=1 -DHAVE_TERMCAP_H=1 -DHAVE_TERMIO_H=1 -DHAVE_TERMIOS_H=1 -DHAVE_UNISTD_H=1 -DHAVE_VARARGS_H=1 -DHAVE_ATEXIT=1 -DHAVE_BCOPY=1 -DHAVE_BZERO=1 -DHAVE_DUP2=1 -DHAVE_ENDGRENT=1 -DHAVE_ENDPWENT=1 -DHAVE_EXECVP=1 -DHAVE_FCNTL=1 -DHAVE_FORK=1 -DHAVE_GETCWD=1 -DHAVE_GETEGID=1 -DHAVE_GETEUID=1 -DHAVE_GETGID=1 -DHAVE_GETGRENT=1 -DHAVE_GETGRGID=1 -DHAVE_GETGRNAM=1 -DHAVE_GETHOSTNAME=1 -DHAVE_GETPGRP=1 -DHAVE_GETPID=1 -DHAVE_GETPPID=1 -DHAVE_GETPWENT=1 -DHAVE_GETPWNAM=1 -DHAVE_GETPWUID=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_GETUID=1 -DHAVE_GETWD=1 -DHAVE_LSTAT=1 -DHAVE_MEMMOVE=1 -DHAVE_MKDIR=1 -DHAVE_MKFIFO=1 -DHAVE_ON_EXIT=1 -DHAVE_PIPE=1 -DHAVE_PUTENV=1 -DHAVE_RENAME=1 -DHAVE_RINDEX=1 -DHAVE_RMDIR=1 -DHAVE_SETGRENT=1 -DHAVE_SETPWENT=1 -DHAVE_SETVBUF=1 -DHAVE_SIGACTION=1 -DHAVE_SIGPENDING=1 -DHAVE_SIGPROCMASK=1 -DHAVE_SIGSUSPEND=1 -DHAVE_STAT=1 -DHAVE_STRCASECMP=1 -DHAVE_STRDUP=1 -DHAVE_STRERROR=1 -DHAVE_STRFTIME=1 -DHAVE_STRNCASECMP=1 -DHAVE_TEMPNAM=1 -DHAVE_UMASK=1 -DHAVE_UNLINK=1 -DHAVE_USLEEP=1 -DHAVE_VFPRINTF=1 -DHAVE_VSPRINTF=1 -DHAVE_WAITPID=1 -DSMART_PUTENV=1 -DHAVE_LIBDL=1 -DHAVE_DLOPEN=1 -DHAVE_DLSYM=1 -DHAVE_DLERROR=1 -DHAVE_DLCLOSE=1 -DWITH_DL=1 -DWITH_DYNAMIC_LINKING=1 -DHAVE_TIMEVAL=1 -DHAVE_FINITE=1 -DHAVE_ISNAN=1 -DHAVE_ISINF=1 -DHAVE_ACOSH=1 -DHAVE_ASINH=1 -DHAVE_ATANH=1 -DHAVE_ERF=1 -DHAVE_ERFC=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_ST_BLOCKS=1 -DHAVE_ST_RDEV=1 -DHAVE_TM_ZONE=1 -DHAVE_GR_PASSWD=1 -DEXCEPTION_IN_MATH=1 -DRETSIGTYPE=void -DSYS_SIGLIST_DECLARED=1 -DHAVE_SYS_SIGLIST=1 -DHAVE_POSIX_SIGNALS=1 -DHAVE_GETRUSAGE=1 -DHAVE_TIMES=1 -DGNUPLOT_HAS_MULTIPLOT=1 -DGNUPLOT_HAS_FRAMES=1 User-preferences (please do not edit this section): -------------------------------------------------- EDITOR = "vim" EXEC_PATH = ":.:/sbin:/usr/sbin:/root/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/openwin/bin:/usr/lib/java/bin:/usr/games/bin:/usr/games:/opt/kde/bin:/usr/lib/pgsql/bin:/usr/bin/tex" IMAGEPATH = ".:/usr/share/octave/2.1.12/imagelib//" INFO_FILE = "/usr/lib/octave/info/octave-2.1.12/octave.info" INFO_PROGRAM = "info" LOADPATH = ".:/root/.octave/.octavecommon//:/usr/lib/octave/libexec/octave/site/oct/i686-pc-linux-gnu//:/usr/lib/octave/share/octave/site/m//:/usr/lib/octave/libexec/octave/2.1.12/oct/i686-pc-linux-gnu//:/usr/lib/octave/share/octave/2.1.12/m//" PAGER = "less" PS1 = "\\s:\\#> " PS2 = "> " PS4 = "+ " automatic_replot = 0 beep_on_error = 0 completion_append_char = " " default_eval_print_flag = 1 # default_global_variable_value = default_return_value = [] default_save_format = "ascii" define_all_return_values = 0 do_fortran_indexing = 0 echo_executing_commands = 0 empty_list_elements_ok = "warn" fixed_point_format = 0 gnuplot_binary = "gnuplot" gnuplot_command_end = "\n" gnuplot_command_plot = "pl" gnuplot_command_replot = "rep" gnuplot_command_splot = "sp" gnuplot_command_title = "t" gnuplot_command_using = "u" gnuplot_command_with = "w" gnuplot_has_frames = 1 gnuplot_has_multiplot = 1 history_file = "/root/.octave_hist" history_size = 1024 ignore_function_time_stamp = "system" implicit_num_to_str_ok = 0 implicit_str_to_num_ok = 0 initialize_global_variables = 0 max_recursion_depth = 256 ok_to_lose_imaginary_part = "warn" output_max_field_width = 10 output_precision = 5 page_output_immediately = 0 page_screen_output = 1 prefer_column_vectors = 1 print_answer_id_name = 1 print_empty_dimensions = 1 print_rhs_assign_val = 0 propagate_empty_matrices = 1 resize_on_range_error = 1 return_last_computed_value = 0 save_precision = 15 saving_history = 1 silent_functions = 0 split_long_rows = 1 string_fill_char = " " struct_levels_to_print = 2 suppress_verbose_help_message = 0 treat_neg_dim_as_zero = 0 warn_assign_as_truth_value = 1 warn_divide_by_zero = 1 warn_function_name_clash = 1 warn_future_time_stamp = 1 warn_missing_semicolon = 0 warn_variable_switch_label = 0 whitespace_in_literal_matrix = ----------------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web: http://www.che.wisc.edu/octave/octave.html How to fund new projects: http://www.che.wisc.edu/octave/funding.html Subscription information: http://www.che.wisc.edu/octave/archive.html -----------------------------------------------------------------------