Memory Leak

Discussion in 'Bukkit Help' started by Retricide, Apr 29, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    Retricide

    Hello,

    Recently, I've noticed my server has been suffering from a huge memory leak.
    The leak does not seem plugin related, as it happens when just running craftbukkit by itself.

    The server will start up fine, using a reasonable amount of memory, but after a few minutes it will start consuming memory at an uncontrollable rate. This memory is not recoverable as the amount of "free" memory does not increase significantly once the server is stopped. Only about 1.5GB of memory is recovered.

    Until minecraft 1.2, I hadn't noticed this issue - but now it's extremely overt.

    I can provide screenshots myself using the "free" command, if anyone would like to see hard-proof.

    I thought it could be my java arguments, so I've tried multiple optimization guides, such as TnT 's suggestions and the ones detailed in these threads:
    However, the problem still persists.
    As a side note, it seems that whenever I use the -server argument, the problem is exacerbated and the server consumes memory quicker then usual.
    Here are my server's specs:
    Code:
    CentOS 6 - RAID 1 - x64 bit
    2x2000GB SATA HD
    AMD Opteron 3280 Octacore
    32GB RAM
    JRE (build 1.7.0_03-b04)
    
    I'm not that proficient with Linux, so perhaps I have something misconfigured? The server does not give any error message no matter how much memory I allocate. However, I have noticed that when I set the -Xms flag to, say, 5GB, the server will not allocate 5GB of memory on startup. It will eventually surpass that as it leaks memory indefinitely, but for some reason it does not allocate that much on startup.
    Has anyone experienced similar issues? Or, could perhaps suggest some startup arguments for me to try?
    Thanks,
    Eric

    EDIT: It seems that using JRE build 1.6.0_31 manages memory better (leaks memory slower) than JRE 1.7_X.
     
  2. Offline

    ashley

    try using these perameters
    i dont know if these work on CentOs
    these perameters use 4Gb minimum 8G a startup and 16Gb at max
    change to you're likings

    Echo OFF
    SET BINDIR=%~dp0
    CD /D "%BINDIR%"
    java -server -Xmn4G -Xms8G -Xmx16G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:parallelGCThreads=5 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -jar craftbukkit.jar
    PAUSE
     
  3. Offline

    Retricide

    I already tried those arguments, as they're right out of the thread I cited, but had no success with them.
    Also, aside from the java arguments you posted, the rest of your startup file is for Windows, I believe. I only need the java arguments, but that's just a heads up.
     
  4. Offline

    TnT Retired Staff

    What line are you using to start your server?

    What do you see when you run htop or top?
     
  5. Offline

    ShootToMaim

    1. Stop using /reload!
    2. Update your JRE.
     
  6. Offline

    Retricide

    I've tried a variety of startup lines from this thread:
    • Code:
      java -server -Xmn4G -Xms8G -Xmx16G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=5 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -jar craftbukkit.jar
    • Code:
      java -server -Xmx12G -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseNUMA -XX:+CMSParallelRemarkEnabled -XX:MaxGCPauseMillis=50 -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:UseSSE=3 -XX:+UseLargePages -XX:+UseFastAccessorMethods -XX:+UseStringCache -XX:+UseCompressedStrings -XX:+UseCompressedOops -XX:+OptimizeStringConcat -XX:+AggressiveOpts -jar craftbukkit.jar nogui
    Right now I've upgraded back to Java 1.7_03 and I'm running these, basic, arguments through RTK:
    Code:
    java -Xincgc -Xmx15024M -jar craftbukkit.jar nogui
    Memory seems to leak the slowest using these parameters - but it' still pretty rapid. Also, I've definitely noticed that using the -server flag has a drastic, negative, effect on the memory leak.

    Top:
    Code:
    top - 17:29:43 up 35 min,  1 user,  load average: 0.21, 0.62, 0.43
    Tasks: 198 total,   1 running, 197 sleeping,   0 stopped,   0 zombie
    Cpu(s):  1.8%us,  0.0%sy,  0.0%ni, 96.7%id,  1.4%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:  32814452k total,  8919764k used, 23894688k free,    19788k buffers
    Swap:  2097136k total,        0k used,  2097136k free,  7245688k cached
     
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     4260 root      20   0 18.7g 803m  11m S 17.3  2.5   5:15.22 java
       42 root      20   0     0    0    0 S  0.3  0.0   0:01.04 events/7
     5624 root      20   0 15012 1368  988 R  0.3  0.0   0:00.05 top
        1 root      20   0 19204 1508 1220 S  0.0  0.0   0:00.94 init
    
    Thanks for the help, TnT.

    I've already updated to the most recent jre, 1.7.0_03-b04.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 25, 2016
  7. Offline

    ShootToMaim

    Have you tried re-installing your craftbukkit.jar?
     
  8. Offline

    Retricide

    Re-installing? Updating to the newest version? Yes.
    I was on the RB for 1.2.5 for a while, but decided to update to the newest beta for 1.2.5 a few days ago, in hopes of seeing some improvements. I haven't seen any.

    Maybe I've got a configuration issue with Java or my CentOS/ Linux configuration?
    Maybe it's a CentOS related problem - I know Cent is drastically divergent from most other Linux Distros.

    I, honestly, have no idea why I'm leaking memory like crazy. It isn't a bukkit issue - or is it?

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 25, 2016
  9. Offline

    ShootToMaim

    It could be a bukkit issue. Users have been reporting memory leaks...
    As I said before, are any of your users/you using /reload? /reload has been known to cause memory leaks.
    CentOS shouldn't be the problem, unless you misconfigured it.
    Also, try running a virus scan. It could be some virus.
     
  10. Offline

    Retricide

    Already did a scan and I never use /reload - partially, for that reason.
     
  11. Offline

    ShootToMaim

    I've got no solution then :/. Wait for TnT, or tag some more moderators xD.
     
  12. Offline

    TnT Retired Staff

    Why do you think you have a memory leak? What numbers are you looking at?
     
  13. Offline

    KakusCraft

    ^stop posting like that, lots of people got memory leaks, i got memory leaks too...

    i have to restart my server every 12 hours or it will blew up and start consuming 200% CPU...

    bukkit needs to optimize their server ASAP
     
    Thasan likes this.
  14. Offline

    Retricide

    Was that in response to my thread or TnT 's post? Either way, use the "reply" button in the bottom right of posts to make yourself more clear, next time.


    Despite this being bad news, I am actually glad to hear this as it reassures me the issue is reproducible - it's not just my server. Thanks for the confirmation, Kakus.



    Here are some screenshots of me using "top" along with relevant information at the time of each "top."
    Note: When I took these screenshots, my server was running a heavy load of plugins (and it still is). However, I can confirm that the memory leak still persists on vanilla craftbukkit and at the same rate as with a variety of plugins. If you are doubtful, though, I am willing to documentmy server's memory usage using just vanilla craftbukkit.jar, as well.


    Prior to starting Minecraft/Java (MySQL and httpd services are active throughout testing).
    beforemc.png


    Initiation of Minecraft/Java (taken a few minutes after "Done" was displayed in console; prior to any players joining
    startmc.png


    On this particular day, the memory leak seemed to be less noticable. It usually occurs at a faster rate, but it took longer today. It is important to note, however, that the lead did occur steadily and at a constant rate. It seemed to level out at ~5GB of RAM for a few hours (player activity varied from 0 to 20, as usual), but resumed its regular leaky, activity after. Also, the number of players online did not seem to affect the rate of memory leak; the memory leak was constant whether 0, 5, 10, or 20 players were online.

    Minecraft/Java Running for 12 hours (taken with 3 players online when taken)
    duringmc.png

    In the above screenshot, note how Minecraft/Java is only using 3.1% of the availabe RAM, providing no explanation for why 7GB of memory is being used. This, alone, is evidence enough of a leak.

    Minecraft/Java running for 24 hours | Taken 1 minute before the server was stopped (10 players online when taken)
    duringmc2.png


    After terminating Minecraft/java process
    aftermc.png


    As you can see, the server starts out with 82MB of used memory (~32GB free), but after minecraft is gracefully stopped and all java processes (only minecraft) cease to exist, the server does not recover RAM as it still uses 8.8GB of memory (only 24GBs are free as opposed to the original 32GB free). However, no processes seems to actually using that memory.

    I do believe that this qualifies as a memory leak.


    I am now aware that bukkit has released a new beta build, which supposedly fixed performance issues.
    I'm not sure this potential issue was addressed, but I'll post my findings after updating, later.
     

    Attached Files:

  15. Offline

    TnT Retired Staff

    Alright. Compile your findings, all of them (thanks for including them) to leaky.bukkit.org as a bug. You can link to this thread if you wish. Please include a full server.log along with this at the time of these events.

    It would be great to see the documentation of this without any plugins.

    Can you also post your htop results?
     
  16. Offline

    Retricide

    I don't believe there is an "htop" command in CentOS; or, if there is, it's under some other alias.

    I'll go ahead and post a bug report tomorrow. Right now it seems as if the memory leak has been fixed or mitigated, but more testing will reveal what actually is the case. Thanks for the assistance, TnT .

    After some preliminary testing, memory consumption and leakage seems to be little to none.
    Whatever the Bukkit-Team did in the latest Beta worked, I suppose!

    However, there seems to be one caveat:
    Ironically, my CPU usage on the latest Beta is through the roof.
    Before, when I had the memory leak, my CPU usage hovered between 10% to 75%, depending on how many players were on and other factors.

    Now, however, my RAM does not go above 3.5GB but my CPU stays between 40% to 90%, usually hovering around 60%. Perhaps the spike in CPU could be due to my server compensating for using little memory, as my java arguments aren't that bold nor efficient.

    I'll do some more tweaking and testing and post what I find.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 25, 2016
  17. Offline

    TnT Retired Staff

    What java arguments are you using for this testing?
     
  18. Offline

    Lelo

    I had a big memory leak for an entire week. I know how to deal with leaks now. I might be able to help you.
    Do you mind listing all your plugins first of all? A memory leak can be caused by plugins or a world.
     
  19. Offline

    Retricide

    Probably could've made posted this again, to clarify:
    I'll try one of the more advanced arguments that I cited and see what effect that has on overall performance.

    I was experiencing a memory leak running just vanilla bukkit, so it probably wasn't plugin related. Thank you for the offer, though.
     
  20. Offline

    Lelo

    Remove your current worlds temporary and have new worlds generate and see if the problem still occurs.
     
  21. Offline

    TnT Retired Staff

    Retricide I asked what arguments you ran, cause your post was quite detailed, but only had the results from what I assume was a single run of the server. Changing the GC methods like you are doing would change how much RAM is consumed and if changed drastically, can cause you more problems than they solve. I would expect to see results like that three times over - watching RAM usage with each modification of the arguments. I am looking for as much information as possible. One can never have too much.

    Its always a good idea to test simply with only the Xms and Xmx arguments. They are the default arguments. Also, I wouldn't recommend -Xincgc anymore for a heap size that large.
     
  22. Offline

    Retricide

    I'll give it a try. I'm going to just test craftbukkit by-itself, without any plugins, so I'll use a new map for that.

    I didn't mean to sound snarky - I apologize if I came off that way (I was in a rush, at that time). I was just trying to save time by quoting myself instead of going through my server files to check :p

    Any specific arguments you would like me to test, when I test CB by itself? Or just something along the lines of -Xms6G -Xmx20G ?
     
  23. Offline

    KakusCraft

    try this command, i got memory leaks every 24 hours though, but only every 24 hours...

    java -Xms<InsertRAMhere> -Xmx<InsertRAMhere> -jar craftbukkit.jar nogui
     
  24. Offline

    TnT Retired Staff

    No worries. I know this is almost like writing an essay at this point but it really helps to have as much data as possible.

    That would be a good baseline to start with.
     
  25. Offline

    bob7

    I'm having the same problem, i have 2x 120 SSD, Xeon E3-120, and 32gb ram... I'm starting to beleive it has something to do with the world or maybe some plugin. Mine uses 32/32gb though :(
     
  26. Offline

    Nathan C

    I can bet it is a plugin.

    Don't use every plugin that you see.........only use ones that you need and know are coded properly, to prevent memory leaks.
     
  27. Offline

    Retricide

    It's not plugin related; I've tested with just vanilla bukkit and produced the same results.

    TnT In other news, I tested out my bukkit server on a new OS (Debian, freshly installed) up until today.
    I tested with plugins (same exact minecraft folder I use on the server with a leak) and without plugins and, for some reason, the memory leak disappeared on the fresh install.

    After doing some more digging, I realized I wasn't running a webserver (though MySQL was running) on the new OS. I turned on Apache2 and began testing, again; oddly enough, the memory leak reappeared. For some reason though, the leak doesn't occur unless another process is consuming memory/CPU (a bukkit server would seem to fulfill that criteria).

    I'm going to continue to test some more and see what definitive answer I come to, but, for now, it seems like the problem isn't bukkit related; bukkit's heavy CPU and RAM consumption induces the leak, but it doesn't create the leak. If anyone else is experiencing a leak ( bob7 ) and is running a web-server (Apache, Apache2), I'd be interested in seeing is the leak desists after disabling you're web-server process.
     
  28. Offline

    Retricide

    Nope, guess it's not the webserver. The leak still persists, it's just extremely gradual - but, nonetheless, existent.

    TnT I'll post a bug report to leaky sometime next week, when I have time to properly document it. Do you know if anyone else is experiencing this issue, or is it only me?
     
  29. Offline

    maniacmusic

    Hey I have exactly the same server with 24GB RAM (I assume your's is also from Server4you) and I have similar issues.
    With only 8 players online it uses up 9GB!!! RAM after just an hour. It's crazy. It also freezes at some point. I'm honestly desperate.
     
  30. Offline

    Bakuhatsu

    Yep, same issue. Sometimes some args function to keep it stable but once the server would be rebooted those args refuse to properly work any longer and it keeps eating up more ram than it frees, and eventually hanging itself up with tons of "read time out" because ram dropped to 0. (5gb allocated)
     
Thread Status:
Not open for further replies.

Share This Page