[olug] understanding Linux memory usage

[Christopher Cashell]

I'm going to simplify some aspects of Linux memory management below to
make for shorter explanations.  If anyone wishes to expand on any of
the oversimplified places, feel free.  Memory management, and trying
to sort out exact memory usage, can be complicated and convoluted.

On Wed, Feb 20, 2008 at 2:58 PM, Ryan Stille <ryan at cfwebtools.com> wrote:
> I have a process (a java virtual machine) that according to ps and top,
>  is taking up 3GB of "virtual memory" size.  I would think "virtual
>  memory" would be swap (not real memory, but virtual, like swap, right?)
>  but the man page says it is swap + resident (physical) combined.

The Linux Virtual Memory subsystem takes all available memory,
physical and swap, and manages it internally, while presenting a
single memory "view" to an application.  For this reason, many
references to virtual memory, such as the one you point out, are
meaning memory allocated by the virtual memory system, as opposed to
disk-based "virtual" memory (swap).

>  Here is the ps output:
>  USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
>  nobody    7147 21.1  6.6 3013876 271796 pts/0 Sl  14:44   1:04
>  /opt/coldfusion8/runtime/bin/....
>
>  Its corresponding line from top agrees:
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  7147 nobody    25   0 2943m 265m  12m S    0  6.7   1:06.01 coldfusion8

The first thing to note, is that this information can be a bit
misleading, as the information is kind of an idealized report of how
much memory the given application would use if it were the only
application running on the system.  The 265M resident memory
represents memory being used by this program alone, along with memory
being used by shared libraries that are shared among multiple
programs.

For an example of how misleading ps/top can seem to be, I have a web
server that is running apache.  It shows that each instance of apache
is using 22M of resident (RAM) memory.  And there are 50 apache
processes running on that web server.  Which should mean that apache
is using 1100MB of RAM, right?  Except this box only has 512MB of RAM,
and of that, almost 200MB is currently free.  The reason is because
each of those apache processes is sharing most of the memory with all
other apache processes, and only about 1M each is probably unique.

>  So why does the summary info at the top of the top output say that there
>  is only 687400k being used on the whole box, and that 0k swap is being used?
>
>  Here are those lines from top:
>  Mem:   4090516k total,   687400k used,  3403116k free,    42012k buffers
>  Swap:   787176k total,        0k used,   787176k free,   206808k cached
>
>  I know that the 265m of resident memory is contributing to the 687400k
>  total used... but its still not all making sense.  I don't understand
>  how top and "free -m" can say there is over 3GB of free memory, when the
>  java process is supposed to be taking up 3GB or so (there is only 4GB on
>  the box).

This has to do with quirks of Java and Linux memory.  An application
can do a number of things to reserve memory, even though they aren't
using them.  Common examples are linked libraries, shared memory,
mmap()ed files, special memory mapped to a process (i.e. video card's
RAM mapped to X), COW (Copy On Write) semantics, and Linux memory
overcommit.

When the JVM starts up, it is reserving enough memory for all of the
libraries it thinks it will need.  It also reserves memory for it's
heap, as specified.  However, when it does that, the kernel is giving
it a pointer to this memory address space, but internally, it will
allocate that memory as needed.  Because the memory isn't actually in
use, just reserved, it doesn't show up as used memory.

Once the application actually makes use of the memory, it will start
to show up in top and ps as being taken by the application.

>  The end goal here is to allow this java process to allocate as much
>  memory as possible for itself.  If free -m is showing 3GB free, couldn't
>  other processes end up using that memory?  My java memory switches are
>  -Xmx2500m -Xms2500m, BTW.

Setting the max memory isn't a bad idea, to prevent the application
from going crazy.  Setting the minimum is rarely a good idea, in my
experience.  It limits the amount of memory anything else on the box
can potentially allocate, even though Java may not be using it.  Also,
if you do end up with other programs on the box requesting memory,
it's possible for Linux to start killing processes as OOM, even though
it isn't actually out of memory, it's just that all of the memory is
reserved (even though it may not be used).  Your java program could
even end up getting killed that way.

Also don't forget that Linux will use all available (unused) memory
for disk cache, and that can have a significant impact on the
performance of a machine, especially if it's dealing with any
significant amount of disk IO.  I always make it a point to keep a
certain amount of memory free on my Linux boxes specifically for disk
cache use.

Hopefully this made a little bit of sense.  If you want to know a
little more about your specific application, check out the 'pmap'
command.  It will give you a very detailed breakdown of how memory is
mapped for it.  If you really want to get down and dirty with Linux
memory, google for "linux memory management".  There's a number of
really good references that'll come up from that.

>  -Ryan

-- 
Christopher