[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Memory Management



Hi,

This is rather technical, so if you are easily intimidated/offended by
kernel blabbering, skip this message.

I have two MM questions:

1. When mallocing a large buffer, you only get a ptr to the 1st page,
   and later the kernel will allocate you more pages upon mem writes,
   and chain them to the buffer in your virtual address space.
   By default, kernels allow you to allocate more than it can afford,
   so running two procs that allocate a big buffer ( > vmem/2 ), sleep 5
   secs to allow the other one to malloc, than sequentially fill
   those buffers results in a sigbus (Bus Error) for both procs.
   Looks like kernel failed to allocate a new page so it just chained
   real-address 0 to the virtual address space.  This means that even if
   malloc returned != NULL, you can't safely assume that you have the
   requested buffer.   This might sound hypothetic, but it might happen in
   real life when mallocing a large buffer and forking before starting to
   use it.  in such case one could easily run into the above situation and
   crash in a random location.
   This might happen in most unices but in most of them you can set a
   kernel param (boot time or compilation) that forbids promising more
   memory than you have.  I failed to find such parameter in linux.
   Is there a way to prevent kernel from being too generous for its own
   good ?
   If not, I plan to add something that will at least allow me to allocate
   my own critical buffers safely.  I was thinking of adding a syscall
   that really binds all the pages you requested, given a ptr to the
   allocated buffer, return the ptr to this buffer, or NULL if failed.
   Something like this:

   if (!(x=real_alloc(y=malloc(BIGBUF))) {
	free(y);
	fprintf(stderr,"whatever");
	/* handle this case just like a regular malloc failure.
   	   if it ain't real, we don't want it! */
   }
   /* Here we can safely assume x is pointing to a buffer safe enough for
      our mission critical function */

   Anyone can think of a better way to handle this ?
   Maybe something that will put the whole system in a stable mode ?
   This would be better since protecting my own buffers while important
   daemons who malloced before are crashing might result in an unstable
   system which cannot hold my protected app.
   The problem with this concept is that the system might already be in
   memory overdraft due to allocated-but-unused big buffers, and waiting
   for them to be freed might be an endless deadlock.


2. Another thing I couldnt find in linux MM is plock/mlock/memlock
   equivalent.  (something that locks a buffer in RAM, and if succeeded,
   garantees that your buffer will never be swapped out.)
   Anyone knows how to do it in linux ?  (Its possible inside the kernel,
   but creating and locking a buffer in the kernel before UIOing to user
   space seems like a real burden for high-level app developers.)


Anyone still with me here ?  ;-)

Let me know if you have any ideas...

	Yoav Weiss

p.s. Anyone knows whats the exact use of those huge AVL trees mm uses ?