Memory Behavior testing in craftbukkit

We've seen some interesting behavior from craftbukkit on a server that normally has between 3-10 players on at a given time. These tests began this past weekend, once minecraft.net was back to fully stable.

Tested builds include 118, 132, and 66 running on a quad-core 3 Ghz Athlon system, Windows 7, 6 GB RAM, 64-bit Java with 2 GB RAM allocated. Plugins are BigBrother, Minecart Mania, MyHome, MyWarps, Permissions, General, General Essentials (the one that only adds /kit), TimeShift, and BorderGuard. The same plugins were used on all versions with the exception of MyHome and MyWarps, which were downgraded to older versions - 1.4 and 1.6 - to run on 66.

Memory allocated as measured by Windows task manager tends to increase to maximum over time, but did so much more rapidly on 118 and 132 (4-8 hours). Once reaching maximum, users are able to continue playing normally for a period of time, but eventually block-redraw lag sets in to the point where the server becomes unusable. On 118 and 132 the time from startup to requiring a reboot was anywhere from 8-16 hours; towards the end of that time, already-logged in users could continue playing but new login attempts would time out. Test runs of 118 and 132 with only the timeshift plugin showed similar memory growth over time. Test runs with no plugins are an option, but are likely to have much lower player activity - we don't normally build without continuous daylight to make it more workable.

On 66, uptime before beginning to see block-update lag has been closer to 2 days. New users were still able to log in without issues even at the end of that time. We've been running on this version for normal use, since it's stable enough and supports the really key features for our server - minecart mania, timeshift, and warps.

The performance issues would set in regardless of number of users online at the time, although more users online early would accelerate the problem on 118 and 132. The issue seems to be solely related to memory load and time since startup. This seems to me to show a memory leak in craftbukkit as a whole, and possibly multiple memory leaks in the newer versions. I will be adding the -Xincgc option to my java command line as mentioned in another recent thread, but wanted to post these results. If anyone on the development side would like me to run additional debugging steps, I'm willing to do so but would need instructions.