Memory Leak

Discussion in 'Bukkit Help' started by Retricide, Apr 29, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    TheBeast808

    Look into jmap or a CentOS alternative. On Ubuntu, you can use jmap to write everything in your RAM for a specific PID to a file (ex: dump.hprof). Then, download it onto a computer and use Eclipse's Memory Analyzer Tool (MAT) to generate a leak suspect report.

    There are lots of guides to doing this online, and it's the easiest way I know of to pin point where I'm losing memory.
     
  2. Offline

    Retricide

    maniacmusic Bakuhatsu
    What OS are you guys running? Also, are you running any other processes such as MySQL or apache?

    I think I found the problem; my server might not be caching memory correctly. I started another thread on this topic, in case I'm misunderstanding something.

    Also, please post a print-out of the "free" command; I'd like to see if your servers are running out of memory because it's being cached incorrectly.

    TheBeast808
    Thanks for that suggestion - I will definitely look into that!
    Regarding what I said just above this, should cached memory be released after the processes caching the memory is killed? My cache will grow indefinitely and never seems to be used or decrease in size. Additionally, even after killing every single active processes, the memory still remains cached.
    Thanks for the help, Beast.
     
  3. Offline

    Bakuhatsu

    Retricide
    Using MineOS crux (linux crux mod), mysql and hiawatha running. (hiawatha is similar to apache)

    I have NOT tested to run the server under another OS yet to see if the issue is bukkit or specific hardware etc or just the OS.

    mc:~/ $ free
    total used free shared buffers cached
    Mem: 15900940 12919276 2981664 0 33204 10015512
    -/+ buffers/cache: 2870560 13030380
    Swap: 48933984 0 48933984

    And that is while mc is running though
     
  4. Offline

    04blatca

    Can you run the 'free -m' command for me? I know that the values in top -c include cached memory that is used for storing commonly used files. Read the following site: http://www.linuxatemyram.com/
     
  5. Offline

    Retricide

    I saw that site a while back - that's what I originally read to help me understand cached memory.

    "free -m" does basically the same thing as "free," it just displays it in terms of MB instead of KB.
    The "buffers/cache" section of "free" displays memory that is supposedly "free;" that memory should be accessible.

    Using Bakuhatsu 's "free" printout, we can see that he has about 13GB of free-usable memory under "buffers/cache." That memory should be usable - but, my point is, something is happening that is rendering this memory useless. He is only using 2.8GB of memory, the other 10GB is cached.

    Bakuhatsu , see if you can get another print-out of the "free" command after your server crashes or when it starts lagging.
     
  6. Offline

    Bakuhatsu

    04blatca

    mc:~/ $ free
    total used free shared buffers cached
    Mem: 15900940 14576260 1324680 0 38204 11500264
    -/+ buffers/cache: 3037792 12863148
    Swap: 48933984 152 48933832

    mc:~/ $ free -m
    total used free shared buffers cached
    Mem: 15528 14234 1293 0 37 11230
    -/+ buffers/cache: 2966 12561
    Swap: 47787 0 47786

    This is right after a crash but restarted the mc server.
    I have 5gb allocated to minecraft while 5.5gb towards a ramdisk and a total of 16gb in the system itself.
     
  7. Offline

    04blatca

    As you can see from the following:

    total used free shared buffers cached
    Mem: 15528 14234 1293 0 37 11230
    -/+ buffers/cache: 2966 12561

    You are currently using 14GB of RAM with cache. However, you are only using 3GB of RAM without cache.
     
  8. Offline

    Retricide

    Right. I completely agree - the diagnosis was similar in my above post.
    And I do understand that cached RAM is usable, but my question is should his server be crashing when it has 12GB of cached RAM? Shouldn't it be using that RAM, not keeping it cached? And, to that point, does cached RAM ever un-cache, to your knowledge? Wouldn't it make sense for it to un-cache itself after the processes using the RAM is killed?

    I'm just trying to wrap my head around what's going on because it doesn't make sense for his server to be crashing when it's only using 3GB of RAM.

    Edit: Not trying to sound hostile here, I'm just pretty confused. I do appreciate your help, 04blatca
     
  9. Offline

    Bakuhatsu

    No idea if interesting.. but:
    (phpsysinfo)
    Physical Memory 94%
    free 1.03 GiB
    used 14.13 GiB
    size 15.16 GiB

    Kernel + applications 21%
    used 3.11 GiB

    Buffers 1%
    used 39.57 MiB

    Cached 73%
    used 10.98 GiB
     
  10. Offline

    04blatca

    What is the error that you get when the server crashes? Whenever memory is needed for a process, it is taken away from the memory that is used for cache. From what I have read, there is no problem with the amount of memory that you are using.
     
  11. Offline

    Bakuhatsu

    Sometimes nothing happen, it begins with etc 5gb ram usable but it keeps slowly dropping each time the GC run (tried multiple GC's but just the time vary)... and eventually it reaches zero and the server usually do not crash but freeze. It's still running but refuses any connection and at some times saying "Read timed out".

    In rare cases the server actually _crash_ with a hs error file but it usually just contain whatever the server was doing as server reached zero ram, but some other cases it just tell the fault is actually the GC collector.
     
  12. Offline

    04blatca

    Hmmm, it sounds to me like a plugin is misbehaving. How long does it normally take before the server crashes?
     
  13. Offline

    Retricide

    Bakuhatsu


    Cached memory shows up as "used;" however, it should not be considered "used" memory. It is memory that is constantly used and is optimized for a certain process, so it's actually usable - it just doesn't appear that way.

    For some reason, however, cached memory is never being re-used; once it's cached, it doesn't seem to be re-used. At least, that's the case on my server. My cached memory grows indefinitely, never decreasing so it's seemingly never being used. My point is something is going wrong when the server "caches" memory and I'm not sure what that problem is.

    I've replicated the problem on a vanilla-craftbukkit server.
    Perhaps you, too, could try and replicate it on a vanilla-craftbukkit server, Bakuhatsu .

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 25, 2016
  14. Offline

    Bakuhatsu

    It's not plugins, I've already tested that. It's java garbage collector related together with bukkit.
    I am however about to try all this with ubuntu server or something different because this OS is annoying me (no apt-get)

    Also:

    mc:~/ $ java -version
    java version "1.7.0"
    Java(TM) SE Runtime Environment (build 1.7.0-b147)
    Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
     
  15. Offline

    Retricide

    I think I've gotten the same problem just using the basic -Xms and -Xmx args, with no garbage collector.
     
  16. Offline

    Bakuhatsu

    I might try another OS tomorrow so... if that solves the issue it's either OS related or java version related.
     
  17. Offline

    horrorduck123

    ***sigh*** i've been have this problem for about 2 days now my server crashing cause of ram running out on server i have 4gb dedicated to my bukkit server and with only 10 people on and 26 plugins i only have about 400 mb ram left that isint normal; :( how can i fix this .. i am useing a dedicated centos server btw. PLUS am even useing ptweaks to see if it would reduce ram usage and it would still be leaking :( little by little
     
  18. Offline

    Bakuhatsu

    This is the main issue:
    Stack: [0x00007f9fd91be000,0x00007f9fd92bf000], sp=0x00007f9fd92bd5f0, free space=1021k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V [libjvm.so+0x667bce] JVM_FindSignal+0xf3b9e
    V [libjvm.so+0x745112] JVM_handle_linux_signal+0x4a7a2
    V [libjvm.so+0x745da3] JVM_handle_linux_signal+0x4b433
    V [libjvm.so+0x7467a2] JVM_handle_linux_signal+0x4be32
    V [libjvm.so+0x84cc9a] JVM_handle_linux_signal+0x15232a
    V [libjvm.so+0x85285c] JVM_handle_linux_signal+0x157eec
    V [libjvm.so+0x851340] JVM_handle_linux_signal+0x1569d0
    V [libjvm.so+0x851881] JVM_handle_linux_signal+0x156f11
    V [libjvm.so+0x851f00] JVM_handle_linux_signal+0x157590
    V [libjvm.so+0x6f84b0] fork1+0x3fd0

    VM_Operation (0x00007f9fd07ec650): ParallelGCSystemGC, mode: safepoint, requested by thread 0x0000000005d37800

    Also... my temporary solution, a quite simple bashscript that shuts down and then starts the server again if it's frozen using cronjobs with 5min interval.

    Code:
    #!/bin/bash
     
    line=`tail -1 /home/mc/servers/F-craft/server.log`
    line2=`tail -2 /home/mc/servers/F-craft/server.log`
     
    fisk=${line:27:14}
    fisk2=${line2:27:14}
     
    if [ "$fisk" == "Read timed out" ] && [ "$fisk2" == "Read timed out" ]; then
     
    cd /usr/games/minecraft; ./mineos_console.py command F-craft save-all
    cd /usr/games/minecraft; ./mineos_console.py stop F-craft
     
    killall java
     
    cd /usr/games/minecraft; ./mineos_console.py start F-craft
     
    fi
    The "mineos_console.py" is just a python thingy that mineos contain, it's possible to do this in other ways... "screen" etc and using it to send commands to the console. And starting the server is just to call that .sh file :3

    It's not pretty but it checks for two repeated "Read timed out" in the console which often spams when it freezes, and if match it restarts the server... good if it must be slightly up, and especially when you're going to sleep like I will now.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 25, 2016
  19. Offline

    maniacmusic

    I use Ubuntu but I figured my problem was Orebfuscator.
    Now i have to deal with Xrayer - not much better. -.-
     
  20. Offline

    Bakuhatsu

    I've reinstalled the server with latest ubuntu server version, will update this post if the memory leak remain. So far it looks fairly bad but can't know until it keeps going a while longer.

    Edit: It did crash. Currently running a second test with a bit different args and also running a vanilla bukkit alongside.

    I am curious though.. what kinda cpu/motherboard/ram brand does your servers have?

    Edit2: Both servers crashed, doing a new test of the same deal with java 1.6.

    All tests I did pretty much failed but I did something that might affect all.. I ran a memtest and saw that there was a shitload of errors. Swapped with ram that I know works flawlessly and did a memtest with as well, no errors.

    Currently running server with these ram sticks to see if it works, will report after my sleep.

    I suggest to try running a memtest, if your ram is also bad... then that might be the reason.

    Edit: It seems the ram leaks is gone but other issues has popped up instead that is unrelated.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 25, 2016
  21. Offline

    maniacmusic

    I thought it's fixed, but it's not :mad:
    [​IMG]

    This is just crazy.
     
  22. Offline

    Bakuhatsu

    maniacmusic
    Did you try to run memtest86+?

    You can install it onto etc a cd or usb stick and choose to boot up from it, if any errors... one or more ramsticks may be bad. One of mine was in bad condition so swapped that one out, seems to have solved it.
    (if linux os you may have it as an option in grub)
     
  23. Offline

    maniacmusic

    This is a dedicated server, if the ram is bad the host will change it for me or move me to a new node.
    But I can't just put it on a CD or USB stick and boot the computer with it. :p

    It's also worth mentioning that I have 20 worlds.
    But even all together they are not 8GB - or even 1GB.
    I'm not sure if that's the problem or if a plugin is causing this...
    Code:
    Plugins (38): WorldBorder, CaptureThePoints, Minequery, JustWarp, BukkitCompat, NoPl, GravitySucks, Spout, Vault, Multiverse-Core, PermissionsEx, RankMarket, TopPVP, SignColours, HardCoreWorlds, iConomy, SurvivalGames, Like, World-Of-Spoot, Lockette, ScheduledAnnouncer2, ColorMe, Votifier, Herochat, MobArena, mcMMO, SkyGrid, CommandSigns, ChunkOwn, WorldEdit, DynamicEconomy, CreativeGates, Multiverse-Inventories, Multiverse-NetherPortals, CommandBook, WorldGuard, SimpleSpleef, ChessCraft
     
  24. Offline

    TnT Retired Staff

    For those of you that have replicated this without plugins (very important that it be replicated without plugins), please list your OS, its architecture (x64 or x86), and its version. Please also provide the output of your java -version command. A full server.log from the start until the point you had to stop/kill the server is great. A visualvm snapshot for that server run would be cool as well.

    On top of that, please provide a list of what commands you have used to start the server without plugins.
    (Aka, java -Xincgc -Xmx1G -jar craftbukkit.jar). If you have tried with multiple, please list them all.

    If you can replicate this by doing anything other than waiting for it to crash (or become unresponsive due to a lack of RAM), please include those steps.

    If there is a leaky ticket already for this issue, please also include the ticket number.
     
  25. Offline

    Bakuhatsu

    I've been running my server for about 15 hours now and not a single crash, my issue was one bad ramstick. However, I did a few other things at the same time but I do not believe those fixed it.

    Current setup:
    Show Spoiler
    Ubuntu Server 10.04
    Craftbukkit dev build 2182

    java version "1.6.0_20"
    OpenJDK Runtime Environment (IcedTea6 1.9.13) (6b20-1.9.13-0ubuntu1~10.04.1)
    OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

    Startup script (start.sh)
    Code:
    screen -dmS minecraft java -server -Xmx6G -Xms6G -Djline.terminal=jline.UnsupportedTerminal -jar craftbukkit.jar
    
    6gb dedicated towards the minecraft server, 6gb dedicated for a ramdisk where all server files and worlds are placed and the remaining ram for w/e.

    Plugins (16)
    Code:
    ColorMe, CommandBook, FilterChat, HomeSpawnPlus, Lockette, Logblock, Multiverse-Core-2.4, NoCheatPlus, PermissionsBukkit-1.6, RealTeleporter, SpawnerAdjuster, VanishNoPacket, WorldBorder, WorldEdit, WorldGuard
    Specs:
    Msi 890gx
    Amd Phenom II x6 1090T
    WD Caviar blue 500gb
    (4x4gb) Corsair Value Ram
    So... for those who can, run a Memtest86+. (Alternative, ask your host to run one)
     
  26. Offline

    TnT Retired Staff

  27. Offline

    Retricide

    Edit:
    Just had my host run a memTest86+ and they found 3 faulty sticks of RAM.
    Go figure, right?

    Edit x2: The RAM slots on my motherboard were actually fried. I got moved to a new machine and now everything is perfect.

    Thank you, Bakuhatsu , for the suggestion. Hopefully that was my only problem.
     
  28. Offline

    Mordy2011

    I run a server as well. Its is an E3-1270 (quad xeon) with 16GB RAM. We also get massive memory leaks and must restart the server about 6X per day in order to continue hosting Minecraft. If we do not the service leaks until the system hangs and then it remains frozen until we terminate the process or stop the server.

    I do need some updates, but wanted to also sign up to this forum so I could also confirm that this appears to happen regardless of your platform or plugins. I believe the issue is Minecraft or Bukkit. I am hoping the new update can offer a little more benefit to server operators. Minecraft is by far one of the most difficult and tedious applications to host because the code is so poorly optimized.


    BTW I am not complaining. I am after all using software made by dozens of volunteers and a rookie game developer. So I no longer get upset, I just do my best to cope. I love the game and the people behind it are impressive. So we have no choice but to wait patiently until Mojang/Bukkit does some server side work to address these things.

    I too wish there was an option today. I've tried plugins and other stuff to help, but nothing really works well, so I just restart, or set mcmyadmin restart on its own every hour when ram goes over 4GB.
     
  29. Offline

    Jefle

    Ive got the same with 32Gb ram i7 and a CENT OS operating system
     
Thread Status:
Not open for further replies.

Share This Page