Bukkit Server hangs on boot, thrashes CPU, no errors, does not answer console.

After running for some time (days), something happens, and then the server stops responding to requests. From that point forward, any server restart has the same condition -- the server log says its up, but it does not answer (even to a console "help" command.) There are no errors.

I removed all plugins, and tried just starting up bukkit, but that did not resolve the issue. I previously fixed the issue by restoring the world directory from backup, presuming it was some random corruption, but it happened again a few days later.

[root@dl380 bukkit1]# uname -a
Linux dl380.knodel.net 2.6.35.6-45.fc14.i686 #1 SMP Mon Oct 18 23:56:17 UTC 2010 i686 i686 i386 GNU/Linux

[root@dl380 bukkit1]# java -version
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) Client VM (build 20.0-b11, mixed mode, sharing)

Plugins:
Appleseed.jar DwarfForge.jar PermissionsBukkit-1.2.jar ScubaKit.jar WoolTrees.jar
ChopTree.jar ExtendedReach.jar PortalStick.jar skypirates.jar
ClayGen.jar GiveTo-0.8.jar PorteCoulissante.jar TorchArrow.jar
CommandBin.jar Lift-3.jar RetractableBridge.jar Vault.jar
DirtPaths.jar MobRider.jar runecraft2.9.1.jar VirtualChest.jar

Startup Script
[root@dl380 bukkit1]# more serverstart.sh
#!/bin/sh
java -Xincgc -Xmx2560M -jar craftbukkit.jar

Server log after plugin removal:

[root@dl380 bukkit1]# ./serverstart.sh
174 recipes
27 achievements
00:58:40 [INFO] Starting minecraft server version 1.0.1
00:58:40 [INFO] Loading properties
00:58:40 [INFO] Starting Minecraft server on *:25565
00:58:40 [INFO] This server is running Craftbukkit version git-Bukkit-1.0.1-R1-1-g1e6a083-b1602jnks (MC: 1.0.1) (Implementing API version 1.0.1-R2-SNAPSHOT)
00:58:40 [INFO] Preparing level "world"
00:58:40 [INFO] Default game type: 0
00:58:41 [INFO] Preparing start region for level 0 (Seed: -5103884437711304285)
00:58:42 [INFO] Preparing spawn area: 12%
00:58:43 [INFO] Preparing spawn area: 48%
00:58:44 [INFO] Preparing start region for level 1 (Seed: -3034072336482123482)
00:58:44 [INFO] Preparing spawn area: 0%
00:58:45 [INFO] Preparing spawn area: 44%
00:58:46 [INFO] Preparing spawn area: 97%
00:58:46 [INFO] Preparing start region for level 2 (Seed: -3034072336482123482)
00:58:47 [INFO] Preparing spawn area: 40%
00:58:48 [INFO] Preparing spawn area: 85%
00:58:48 [INFO] Server permissions file permissions.yml is empty, ignoring it
00:58:48 [INFO] Done (0.799s)! For help, type "help" or "?"
>help
>?
>^C^C

Server log with plugins:
[root@dl380 bukkit1]# ./serverstart.sh
174 recipes
27 achievements
00:57:51 [INFO] Starting minecraft server version 1.0.1
00:57:51 [INFO] Loading properties
00:57:51 [INFO] Starting Minecraft server on *:25565
00:57:52 [INFO] This server is running Craftbukkit version git-Bukkit-1.0.1-R1-1-g1e6a083-b1602jnks (MC: 1.0.1) (Implementing API version 1.0.1-R2-SNAPSHOT)
00:57:53 [INFO] ClayGen: Setting activation block to: 45
00:57:53 [INFO] Preparing level "world"
00:57:53 [INFO] Default game type: 0
00:57:54 [INFO] Preparing start region for level 0 (Seed: -5103884437711304285)
00:57:55 [INFO] Preparing spawn area: 28%
00:57:56 [INFO] Preparing spawn area: 73%
00:57:56 [INFO] Preparing start region for level 1 (Seed: -3034072336482123482)
00:57:57 [INFO] Preparing spawn area: 20%
00:57:58 [INFO] Preparing spawn area: 73%
00:57:58 [INFO] Preparing start region for level 2 (Seed: -3034072336482123482)
00:57:59 [INFO] Preparing spawn area: 16%
00:58:00 [INFO] Preparing spawn area: 73%
00:58:01 [INFO] [Vault][Permission] PermissionsBukkit hooked.
00:58:01 [INFO] [Vault][Permission] PermissionsBukkit found: Waiting
00:58:01 [INFO] [Vault][Permission] SuperPermissions loaded as backup permission system.
00:58:01 [INFO] [Vault] Enabled Version 1.1.9-b115
00:58:02 [INFO] [VirtualChest] (version 5.4)
00:58:02 [INFO] [VirtualChest] Chests loaded !
00:58:03 [INFO] [PortalStick] 1 region(s) loaded
00:58:03 [INFO] [PortalStick] 0 grill(s) loaded
00:58:03 [INFO] [PortalStick] No permission handler detected, only ops can use commands
00:58:03 [INFO] [PortalStick] PortalStick enabled
00:58:03 [INFO] Bukkit permissions detected, DirtPaths permissions enabled...
00:58:03 [INFO] DirtPaths version 1.0 is enabled!
00:58:03 [INFO] Lift v3 is now enabled!
00:58:03 [INFO] PermissionsBukkit v1.2 is now enabled
00:58:03 [INFO] [Vault][Permission] PermissionsBukkit hooked.
00:58:03 [INFO] [CommandBin] I could not detect any Permissions on your server. Defaulting to OP!
00:58:03 [INFO] [CommandBin] Enabled successfully.
00:58:03 [INFO] [CommandBin] You are running version: v1.34
00:58:03 [INFO] [SkyPirates]: version [1.4.0] loaded
00:58:03 [INFO] [MobRider] 0.43 enabled.
00:58:03 [INFO] [PorteCoulissante] plugin version 1.2.6 by Captain_Chaos enabled
00:58:03 [INFO] [ChopTree] Permissions not detected, defaulting to ops.
00:58:03 [INFO] ChopTree 1.2 enabled!
00:58:03 [INFO] [WoolTrees] v1.1.1 Started
00:58:03 [INFO] [WoolTrees] Using Permission Provider PermissionsBukkit
00:58:03 [INFO] [WoolTrees] No Economy Provider loaded.
00:58:03 [INFO] [DwarfForge] Configuration v.2.0 loaded.
00:58:03 [INFO] [DwarfForge] Config is up to date
00:58:05 [INFO] [DwarfForge] is up to date at version 2.2.
00:58:05 [INFO] [DwarfForge] Restored 1 active Forges.
00:58:05 [INFO] [DwarfForge] v2.2 enabled
00:58:05 [INFO] [RetractableBridge] plugin version 1.3.12 by Captain_Chaos enabled
00:58:05 [INFO] [Runecraft] Rune blacklist with 0 runes loaded.
00:58:05 [INFO] [Runecraft] Loaded:
00:58:05 [INFO] [Runecraft] - 2 waypoints, 1 teleporters
00:58:05 [INFO] [Runecraft] - 0 position runes
00:58:05 [INFO] [Runecraft] - 0 rune pads
00:58:05 [INFO] [Runecraft] - 2 block runes
00:58:05 [INFO] [Runecraft] World 'world_nether': id 1.
00:58:05 [INFO] [Runecraft] World 'world': id 0.
00:58:05 [INFO] [Runecraft] World 'world_the_end': id 2.
00:58:05 [INFO] [Runecraft] Runecraft 2.9.1 initialized.
00:58:05 [INFO] Runecraft plugin version 0.5 is enabled!
00:58:05 [INFO] TorchArrow v0.8 - Plugin Enabled.
00:58:05 [INFO] Appleseed: ShowErrorsInClient=true
00:58:05 [INFO] Appleseed: DropInterval=60 seconds
00:58:05 [INFO] Appleseed: MaxUncollectedItems=6
00:58:05 [INFO] Appleseed: WandItem=wood_hoe
00:58:05 [INFO] Appleseed: FertilizerItem=bone_meal
00:58:05 [INFO] Appleseed: MinimumTreeDistance=disabled
00:58:05 [INFO] Appleseed: SignTag=Appleseed
00:58:05 [INFO] Appleseed: TreeTypes=(cocoa_beans,apple,golden_apple,cookie)
00:58:05 [INFO] Appleseed: 11 trees loaded in world world.
00:58:05 [INFO] Appleseed: 0 trees loaded in world world_nether.
00:58:06 [INFO] Appleseed: 0 trees loaded in world world_the_end.
00:58:06 [INFO] Appleseed version 0.6.5 is enabled!
00:58:06 [INFO] ClayGen version 1.5.2 is enabled!
00:58:06 [INFO] [ExtendedReach] has been enabled
00:58:06 [INFO] [ScubaKit][3.0.0.D IN 3D!] Enabled started
00:58:06 [INFO] [ScubaKit][3.0.0.D IN 3D!] Remember to update any permissions nodes of ScubaKit.ScubaGear to ScubaKit.ScubaGear.*
00:58:06 [WARNING] [ScubaKit][3.0.0.D IN 3D!] this message's priority was not properly set
00:58:06 [WARNING] [ScubaKit][3.0.0.D IN 3D!] Nature Trail to Hell! (In 3D!) Nature Trail to Hell! (In 3D!) Nature Trail to Hell! (In 3D!) IN THREEE DEEEEE!
00:58:06 [INFO] [ScubaKit][3.0.0.D IN 3D!] Permission system not detected, ignorePermissions has be set to true
00:58:06 [INFO] [GiveTo] v0.8 enabled (loaded items: 266)
00:58:06 [INFO] Server permissions file permissions.yml is empty, ignoring it
00:58:06 [INFO] Done (1.301s)! For help, type "help" or "?"
>help
>?
>^C

And, for those of you who are truly geeky, here is the strace:
[root@dl380 bukkit1]# more strace.out
execve("./serverstart.sh", ["./serverstart.sh"], [/* 19 vars */]) = 0
brk(0) = 0x9459000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb775d000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=58941, ...}) = 0
mmap2(NULL, 58941, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb774e000
close(3) = 0
open("/lib/libtinfo.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0}\220\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=125880, ...}) = 0
mmap2(0x902000, 127796, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x902000
mmap2(0x91f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c) = 0x91f000
close(3) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`:\311\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=19780, ...}) = 0
mmap2(0xc93000, 16500, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xc93000
mmap2(0xc96000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xc96000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0po\257\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1889628, ...}) = 0
mmap2(0xae0000, 1649160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x110000
mmap2(0x29d000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18c) = 0x29d000
mmap2(0x2a0000, 10760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2a0000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb774d000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb774d8d0, limit:1048575, seg_32bit:1, contents:0, read
_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x29d000, 8192, PROT_READ) = 0
mprotect(0xc96000, 4096, PROT_READ) = 0
mprotect(0xadc000, 4096, PROT_READ) = 0
munmap(0xb774e000, 58941) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/dev/tty", O_RDWR|O_NONBLOCK|O_LARGEFILE) = 3
close(3) = 0
brk(0) = 0x9459000
brk(0x947a000) = 0x947a000
brk(0) = 0x947a000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=99158720, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb754d000
close(3) = 0
brk(0) = 0x947a000
getuid32() = 0
getgid32() = 0
geteuid32() = 0
getegid32() = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
time(NULL) = 1324621472
open("/proc/meminfo", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb775c000
read(3, "MemTotal: 3872860 kB\nMemF"..., 1024) = 1024
close(3) = 0
munmap(0xb775c000, 4096) = 0
rt_sigaction(SIGCHLD, {SIG_DFL, [], 0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGCHLD, {SIG_DFL, [], 0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], 0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], 0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], 0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], 0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], 0}, {SIG_DFL, [], 0}, 8) = 0
uname({sys="Linux", node="dl380.knodel.net", ...}) = 0
stat64("/export/bukkit1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getpid() = 22807
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=26048, ...}) = 0
mmap2(NULL, 26048, PROT_READ, MAP_SHARED, 3, 0) = 0xb7756000
close(3) = 0
getppid() = 22806
gettimeofday({1324621472, 81535}, NULL) = 0
getpgrp() = 22806
rt_sigaction(SIGCHLD, {0x80835f0, [], 0}, {SIG_DFL, [], 0}, 8) = 0
getrlimit(RLIMIT_NPROC, {rlim_cur=1024, rlim_max=30156}) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("./serverstart.sh", O_RDONLY|O_LARGEFILE) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbfa4eb38) = -1 ENOTTY (Inappro
priate ioctl for device)
_llseek(3, 0, [0], SEEK_CUR) = 0
read(3, "#!/bin/sh \njava -Xincgc -Xmx2560"..., 80) = 55
_llseek(3, 0, [0], SEEK_SET) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
fcntl64(255, F_GETFD) = -1 EBADF (Bad file descriptor)
dup2(3, 255) = 255
close(3) = 0
fcntl64(255, F_SETFD, FD_CLOEXEC) = 0
fcntl64(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat64(255, {st_mode=S_IFREG|0755, st_size=55, ...}) = 0
_llseek(255, 0, [0], SEEK_CUR) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "#!/bin/sh \njava -Xincgc -Xmx2560"..., 55) = 55
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
stat64(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/usr/local/sbin/java", 0xbfa4e9b0) = -1 ENOENT (No such file or directory)
stat64("/usr/local/bin/java", 0xbfa4e9b0) = -1 ENOENT (No such file or directory)
stat64("/sbin/java", 0xbfa4e9b0) = -1 ENOENT (No such file or directory)
stat64("/bin/java", 0xbfa4e9b0) = -1 ENOENT (No such file or directory)
stat64("/usr/sbin/java", 0xbfa4e9b0) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/java", {st_mode=S_IFREG|0755, st_size=47308, ...}) = 0
stat64("/usr/bin/java", {st_mode=S_IFREG|0755, st_size=47308, ...}) = 0
geteuid32() = 0
getegid32() = 0
getuid32() = 0
getgid32() = 0
access("/usr/bin/java", X_OK) = 0
stat64("/usr/bin/java", {st_mode=S_IFREG|0755, st_size=47308, ...}) = 0
geteuid32() = 0
getegid32() = 0
getuid32() = 0
getgid32() = 0
access("/usr/bin/java", R_OK) = 0
stat64("/usr/bin/java", {st_mode=S_IFREG|0755, st_size=47308, ...}) = 0
stat64("/usr/bin/java", {st_mode=S_IFREG|0755, st_size=47308, ...}) = 0
geteuid32() = 0
getegid32() = 0
getuid32() = 0
getgid32() = 0
access("/usr/bin/java", X_OK) = 0
stat64("/usr/bin/java", {st_mode=S_IFREG|0755, st_size=47308, ...}) = 0
geteuid32() = 0
getegid32() = 0
getuid32() = 0
getgid32() = 0
access("/usr/bin/java", R_OK) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb774d938) = 22
808
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x8081350, [], 0}, {SIG_DFL, [], 0}, 8) = 0
waitpid(-1,


And there it hangs... strace does not see it doing anything, but it pegs the CPU at 100% and stays there.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.