randcall - fork: out of memory

I passed every test except the randcall test, if I run the test as a standalone test test161 run stability/randcall.t I pass the test every time. When I run it as a whole it fails with unexpected shutdown.
I then ran it from the shell as /testbin/randcall -f 2, this passes with no problem at all. However when I run it without suppressing fork it fails with fork: out of memory.

Using gdb I tracked the issue to thread_create that is called by thread_fork in my sys_fork function. thread_create fails when attempting to allocate space for the thread stack.
I noticed the comments for thread_exit say that threads leak memory and will cause some test161 checks to fail but i’m not convinced this is why fork is running out of memory.
I have written unit tests for the table structure/api I use for my file handle table and process table and I am sure they are not leaking any memory. I’ve gone through all the code I wrote and made sure all kmallocs have equivalent kfrees somewhere.
Yet I know for sure that I am leaking memory because everytime I run a test from the kernel menu that uses fork, khu increases by 176. There is an orphaned process, the first process that runs in userspace that remains a zombie process until shutdown but only the bare bones of that proc are left sizeof(struct proc) = 56. So that leaves me with 120bytes leaking from somewhere.
But even in the case where everytime fork is called with a matching waitpid, leaking 120bytes; this shouldn’t cause fork to run out of memory.

Also when fork runs out of memory in the shell, if I quit the shell (freeing the 4096bytes of space allocated for the shell’s process’ stack, when I try to run the shell again from the kernel cmdline, it fails with fork out of memory again.

Any ideas for solving this would be much appreciated

Remember that when you are using dumbvm all allocations over a certain size will leak, as the underlying “dumb VM” does not track page allocations. (You get to fix this for ASST3.1.) khu accounts for this properly—find and read Scott’s post on that subject.

So there are two things that could be going on here:

  1. sys_fork is leaking memory. This definitely seems to be happening based on your post. However, a small leak like this shouldn’t cause your system to run out of memory. ASST2 tests are run with enough memory to account for dumbvm leakage, since (for example) things like kernel stacks and process pages are always leaked. Of course, if you’re really close to hitting the limit, then some small amount of extra leaking could push you over. But if you’re that close, something else is going wrong. Which could be…
  2. Some of your other data structures are too large and are being leaked by dumbvm. For example, if your per-process file table is 32 entries, then it is 128 bytes in size and will not be leaked during proc_destroy. But if it is 1024 entries, then it is 4096 bytes and will be leaked during proc_destroy. Correct but poorly-written execv implementations can also leak a lot of memory. For example, if for some reason you’re using kmaloc to allocate really large string buffers for every argument during the copy in (which is unnecessary), then all of those allocations would be leaked. (Although that would probably cause things to fail much earlier.)

Given that it’s sys_fork that seems to be the culprit, I’d start looking there. First I’d hunt down the leak that khu detects and see if that helps. Then I’d look at my process data structures and see if they include large allocations that I could reduce in size.



  1. When calls were failing I wasn’t freeing a few things
  2. I was allocating 1024 buffers for every path that was copied in
  3. for the large arguments passed when calling execv, I was just increasing the size of the kernel buffer by a factor of 2 until the argument fit or the arg max limit was reached


  1. khu remains constant
  2. I now allocate 32 bytes for a path buffer and increase up to path max if needed
  3. If the argument is larger than 1024 bytes I use memmove to place that argument incrementally on the new address space stack along with copyinstr/copyin

Data structures:
I use a table structure and api, the table struct contains amongst other things a resizable array that I use to keep track of sections. Each section allocates 256 void pointers and these sections are allocated based on where in the table you add to. So unless elements are added to indexes in multiples of 256 there shouldn’t be any problem.

I’ve run unit tests that recursively add then remove 4000 processes to the proc table until max pid is hit and that works fine, as well as a test that adds 60,000 file handles to the file handle table then removes them. Neither of these tests leak memory.
I ran a combination of both tests adding then removing 4000 processes with 3 file handles per process recursively until max pid is reached; this test actually fails when the pid reaches somewhere around 24,000 but still I don’ think test161 comes close to this, especially just the randcall test.

For test161 run asst2, randcall now fails on and off so I do sometimes get full marks but not on every run. Run through the shell randcall still fails with fork out of memory, I’ve noticed that this pretty much happens for any test that calls fork as long as it is run it enough times. Yet when I exit the shell khu is now constant so no memory leaking and I don’t allocate anything greater than 1024bytes.
I should probably give up at this point and call it quits but i’d rather get this right and not have problems down the line.

I’m very impressed with the amount of testing you’re doing.

At this point, are you sure that the problem is still in fork rather than exec?

What call is using this approach to copying in the path? That seems needlessly complicated. You should be easily able to get away with just creating one buffer of PATH_MAX and then using it and discarding it when done. There’s no need to do this incremental approach, which is both slow and creates a lot of needless heap churn. But open and the filename part of exec are really the only things that use this.

Some of your description seems to involve exec. People get really confused about this for some reason. The limit on exec is for all of the arguments combined. That makes things a lot easier from a memory allocation perspective:

  1. Allocate a buffer of size ARGS_MAX.
  2. Copy the first string argument into it with limit ARGS_MAX.
  3. Adjust the limit to ARGS_MAX - length of the string that was copied.
  4. Repeat.

You’ll notice that there is only one call to kmalloc here—not one per argument. If you are using one per argument you will run into problems.

Sycall execv:
Ok I assumed allocating such huge buffers was the problem so what I do is I alloc a buffer of 1024 bytes and copy in the first argument then switch to the new addrspace and copy out that argument (I put the arg pointer in a linked list to add to the new addrspace at the end), I decrement a variable initialized to ARG_MAX by the size of the arg, switch back to old addrspace and repeat.
If copyinstr returns ENAMETOOLONG then I call a helper function that incrementally brings in the long argument by copying in 1024 bytes to a buffer, copying out to the new addrspace, then increments that argument pointer by 1024 copies in (first using copyinstr to check that the remainder of the arg isn’t less than 1024, if it is then just copy that amount into the buffer), switch to new addrspace and use memmove to move the previous section of the arg already on the stack up making enough space for the next section of the arg to be copied below, and do this recursively until that entire argument has been copied over.
I thought allocating huge buffers of ARG_MAX every time execv was called was what was causing me to run out of memory.
In the shell /testbin/bigexec currently takes 4.8 seconds to complete or 2.9s real time (55.9mhz), what is the benchmark for this?

Again: this isn’t necessary. Follow my instructions above. exec should only have to allocate a single 32K buffer. You can also avoid this allocation entirely by using a single static buffer and locking across it, although this has the effect of serializing multiple parallel calls to exec. (The second option simplifies the page allocator that you have to implement for ASST 3.1 somewhat.)

So I now simply use a static buffer but regardless that wasn’t my problem because randcall still fails in the shell and still passes intermittently running test161 run asst2. I just can’t figure out how so much memory could be leaked per call to fork. I have a ridiculous amount of leeway (even leaking 20k per fork shouldn’t cause me to run out of memory) and if this is failing in less than 100 calls to fork I must be leaking (and therefore allocating) huge amounts of memory somewhere. :frowning:

Could someone run say /testbin/forktest on the shell repeatedly to see if at any point they run out of memory? I get 11 runs before it fails

You could add some logging to kmalloc to track large (i.e., page size) allocations and see if that helps.

You should also be able to do some back-of-the-envelope calculations about how much memory fork would need. Figure out how many pages forktest needs for its address space, add one for the kernel stack, and then divide into the number of kernel pages left after you boot.

Another question: do you have any huge static kernel data structures? Remember that kernel itself takes up memory. So if you have (say) a huge process table, that leaves less memory for kmalloc and for everything else.

The majority of static data structures are part of tests and I haven’t included them in the build. The only static data structure larger than a pointer that I’ve declared is the buffer for execv of size ARG_MAX.

I tried to enable LABELS so I could use khdump but I got an address error on load occuring on line 660 of kern/vm/kmalloc.c:
Remaining allocations from generation 3: panic: Fatal exception 4 (Address error on load) in kernel mode panic: EPC 0x800223c0, exception vaddr 0x80055fff
Is this not supposed to work or is this a sign that something else is going wrong?

Convert the EPC and find out.