Comments on: Malloc free space not reclaimed

By: Adam Strzelecki

Adam Strzelecki — Tue, 20 May 2008 13:40:15 +0000

There’s actually one allocator ptmalloc http://www.malloc.de/en/index.html that reclaims the memory for small blocks. I’ve tested also Hoard but it seems it only grows the memory. But with ptmalloc simple program below that never gave back 160MB when running default OSX allocator, now dropped to 512KB after Freeing… loop! BTW. ptmalloc is used by GNU libc.

#include “stdio.h” #include “stdlib.h” #include “string.h”

int main(int argc, char const *argv[])
{
char b[4*1024], c = 0;
void *p[4*10*1024];
size_t i, len = sizeof(b)/sizeof(b⁰);

printf(“Preparing buffer…\n”);
for(i = 0; i < sizeof(b)/sizeof(b^{0); ++i)

{

b[i] = c + ‘a’;

c = (c + 1) % 30;

}}

printf(“Allocating…\n”);
for(i = 0; i < sizeof(p)/sizeof(p^{0); ++i)

{

p[i] = malloc(len);

memcpy(p[i], b, len);

}

printf(“Press any key…\n”);

getchar();}

printf(“Freing…\n”);
for(i = 0; i < sizeof(p)/sizeof(p^{0); ++i)

free(p[i]);

printf(“Press any key…\n”);

getchar();}

return 0;
}

By: Ankur

Ankur — Thu, 10 Apr 2008 06:26:34 +0000

Right now Quicksilver is using roughly 235MB of Real Memory.

Wow. Turning off unused plugins and catalog entries can help there. My (admittedly heavily modded) QS sits between 5 – 12 MB of real memory usage.

By: Kevin Lyda

Kevin Lyda — Wed, 02 Jan 2008 05:18:35 +0000

I’m not exactly sure why on OS X, but generally on unix free()s don’t reduce the process size. It just marks the memory as unused for future malloc calls to reclaim.

By: http://toba.ath.cx:724/blog/

http://toba.ath.cx:724/blog/ — Wed, 14 Nov 2007 17:12:52 +0000

Used memory fragmentation – the system can’t reclaim the memory unless the entire page is unused, IIRC.

By: Adonis

Adonis — Mon, 01 Oct 2007 17:52:48 +0000

@Kevin:
Thanks for your blog entry! I finally get my answer to one of my long-standing questions.

@Ryan:
Thanks for the in-depth explanation! I’ve been having this hunch about OS X for at least a year now and never delved deep enough into it to find the answers!

@Paul.G:
If you see this post, could you explain the second part of your reply? (I understand that relying on the kernel to do the work would be slower, but not the grinding to a halt… but what do you mean by “issues with different shared libraries”? what issues are you referring to?)

By: Paul G

Paul G — Fri, 07 Sep 2007 20:04:52 +0000

Well, if malloc didn’t do this, it might be a lot slower and you’d be surprised how that might really cause many applications to grind to a halt. Also, by going against one system malloc, there are fewer issues with different shared libraries allocating and disposing of memory (note: this can lead to bad programming practices ;-). I suppose that background garbage collection might be a possibility?

By: Paul G

Paul G — Fri, 07 Sep 2007 20:04:09 +0000

By: Paul G.

Paul G. — Fri, 07 Sep 2007 20:02:29 +0000

By: itistoday

itistoday — Tue, 17 Jul 2007 09:32:21 +0000

Because apple stupidly implemented their malloc system. For tiny allocations the freed blocks are put on a linked list so that when the program asks for a small block again it can quickly pull them off the list instead of asking the kernel to allocate a new block. This speeds up performance but because Apple apparently didn’t implement a way for this list of freed blocks to be freed, it will just grow in size as the program runs.

By: Ryan Brown

Ryan Brown — Fri, 06 Jul 2007 22:04:04 +0000

In general, most Unix systems do not return malloced memory to the system when freed (although there are exceptions for large allocations). On OS X allocations take place in one of four categories: tiny (1 – 496 byes), small (497 – 15,359), large (15,360 – 16,773,120) and huge (16,773,121+) (note that those ranges are for 32-bit systems). For tiny and small allocations malloc maintains its own internal memory pool. So if you malloc 16 bytes 65,536 times the tiny pool will grow to 1mb. This memory will never be reclaimed by the system. However, if you free that memory it will be reused by the same process for future tiny allocations. Large and huge allocations are reclaimed by the system when freed.

This means that if a program makes a large number of small allocations one time, that one time will grow the process’ resident memory to it’s largest level, which may be significantly larger than the programs regular memory usage. The system will eventually move the unused pages to disk if necessary.

Really I think this behavior may not be optimal for OS X. In a GUI app it is possible for user to do something which will make a large number small allocations one time and then free that memory all at once, bring the app’s resident memory to a high level. If enough programs do this then RAM will become filled with freed memory, and the system will start swapping, which may make things sluggish. If freed memory was actually reclaimed by the system you wouldn’t see this. OpenBSD’s implementation of free() actually does this.