Could someone give more details on the real function and usage of swap on Linux. I know its something that creates virtual memory on the disk but not sure. What are the recommended limits for swap?

Swap is like virtual memory in Windows. It is used only when the physical memory is insufficient to totally handle the running of the applications. Of course, working in RAM is faster than working in harddisk but having set Swap will prevent your application from failing to run when you have insufficient memory.

I believe the benchmark is to set swap twice the size of RAM. But of course, as for me, I rather not have the swap from being used as it means a slowdown of activities. If I see about 10% of swap being used at most of the times, I will add more RAM but that is just me.

Great. My box was using 25% swap or so. That means more RAM right?

I dont know how much is the swap set, since TP configured the machine, it came with 1 GB default.

The weard thing is that the Watch monitor module says i only use about half the ram on the box.

I believe you might not be able to change the swap that easily since it is a partition. If you do not experience any slowdown then 25% is fine.

Linux memory is not something that easy to comprehend. I know everything seems cryptic but I am sure an expert would chime in with a better and probably more accurate answer.

Keep in mind, the kernel puts data into swap based on calculations and expectations, not necissarily when it is out of RAM to hold data. By default, the swappiness setting favores to use swap by 60% (the exact calculations/methods I am not familiar with). But I know that the RAM was never 80+% before it starts swapping out data on my box. To compensate to keep data in RAM as much as possible, I changed this setting to make the decision to swap data out with a 90% favor to stick to RAM.

To do this, add the following to /etc/sysctl.conf:
vm.swappiness = 10

Save that file and then run sysctl -p for this to take effect immediately.

But keep in mind, swap is still a good thing. Daemons and applications running using up RAM but not actually being accessed are the programs being swapped out to make more room in RAM for active applications. If a daemon like httpd or mysqld uses up a lot of RAM and swap (which it does over time due to memory leaks and caches), then restart the service dumping that data in RAM and swap and start fresh.