QUOTE (JJ)
Any harm?
Your system is swapping because it's running out of ram. If you run out of ram with swap turned off, bad things will happen. Things like processes dieing, or the system crashing.
There's a difference between, "there is some data in swap" and "actively swapping." If a process is idle and has not been used for quite some time, most operating systems will page them out to free up memory for other uses like disk block caching and other, active, processes. Use top(1) or something to monitor how much used swap changes. If you can see the size of swap change, then you're actively swapping. If you can't see the size of swap change, or it doesn't change very often then there's probably no real problem.
If you are actively swapping (it seems that you are, as you've stated that the system gets slow), there is a reason for it. Since you've disclosed that this is a "LAMP" system, check for very large php or apache processes. I've found that php will grow to be very large and not release it's memory back to the operating system under certain conditions. If you have very large apache processes, try setting the httpd.conf 'MaxRequestsPerChild' setting to something sane (like 100 requests or so). This will force apache to kill it's child processes before they grow to be to large.