How can I optimize Java on my server?

Discussion in 'Bukkit Help' started by odielag, Feb 10, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    TnT

    Even doing a dynmap fullrender with the -Xincgc flag and the -verbose:gc, I only saw a couple instances of the full GC taking up to a second.
     
  2. Offline

    Phaedrus

    If you're using Java 7 you can try the new experimental garbage collector that will eventually be replacing the concurrent mark sweep collector.

    Code:
    -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+G1ParallelRSetUpdatingEnabled -XX:+G1ParallelRSetScanningEnabled
    
    It works in Java 6 as well, but performance is described as being as good or worse than concmarksweep.
    --- merged: Feb 13, 2011 11:50 PM ---
    I found a way to redirect the -verbose:GC messages to a log file. Add -Xloggc:gc.log t your line. it will output the message to a log file in the craftbukkit directory called gc.log
     
    Rokrox likes this.
  3. Offline

    TnT

    Sweet, that works. Thanks.
     
  4. Offline

    Binaryclock

    Disable Turbo Boost on your machine in BIOS. I found through my testing Turbo Boost was improperly setting the multipliers of the cores and also putting too many processes in states greater than C1 (which are the power saving states.)

    Also, disable hyperthreading.
     
  5. Offline

    Toasty

    Alright, I decided to do some testing myself.

    Original:
    Code:
    #!/bin/sh
    java -Xincgc -Xmx2G -jar craftbukkit.jar
    Tuned:
    Code:
    #!/bin/sh
    java -Xmn256M -Xms512M -Xmx2048M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=1 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -jar craftbukkit.jar
    And the respective pastebin files:

    Original

    Tuned

    Bukkit is running in a Xen VM on a dual-socket quadcore Xeon server (Core 2 Duo era) running CentOS 5.5 64-bit with 8GB of DDR2 FBDIMMs. The VM has 3GB RAM and 2 cores allocated to it, though I used to give it 3 (recently needed to make a new VM). It's running in CentOS 5.5 64-bit, fully virtualized.

    As you can see, the latter has longer GC collections on average, but does so far less frequently. Even though the GC times were longer, there were only two instances of GC lasting longer than 1/2 a second, and only one of those was a full second. I'll also note that I don't recall a single bit of lag in either session, though in the original session there were numerous overload warnings in the console (I was over a massive crater in both sessions, though I didn't notice many overload warnings in the console with the original commands until I went to the crater).

    Java is also using less memory. 397M with two people on versus 470M with one person.

    I'm not an expert on GC, but I did do a fair amount of research on it way back before I moved my server from vanilla to hmod. I remember my startup script looking similar to the "tuned" one posted above, though I came up with it myself, so it was a bit different. I was just too lazy to attempt optimization with bukkit until recently.

    [EDIT] The JVM is now up to 465M with 2 people on, though it took quite a bit longer to reach that amount than it did previously.
     
  6. Offline

    Phaedrus

    I'd be interested to see the results from a period of higher load. Say 10 users or more.

    Also, I wonder if you could try again using the -server argument. I wonder if java will detect a dual core system with 3 gigs of available ram as server class. I don't know how "aware" it is of physical v virtual hardware resources.
     
  7. Offline

    Toasty

    iirc, it classifies anything with more than 2GB and 2+ cores as a server. I could be wrong though.

    I suppose that's something I could test tomorrow. As for higher load, the most I have on my server at any given moment is about 5 people. It's still a pretty small server.
     
  8. Offline

    Phaedrus

    Same here. It's making doing stress testing pretty difficult.
     
  9. Offline

    Toasty

    Just set off some TNT. :D
     
  10. Offline

    Nathan C

    World edit and lots of water. That does a number on the server.
     
  11. Offline

    Myx0x3

    @jasonznack Can you optimize for 6GB, and 4 physical CPU cores? :)
     
  12. Offline

    M1sT3rM4n

    Set off 60k TNTs. Mine barely survived that.
     
  13. Offline

    WolwX

    Fine post :D
    I really like yours post @jasonznack !
    So I tried thoses launch options :

    Code:
    java -server -Xmn2048M -Xms4096M -Xmx5200M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=5 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -jar craftbukkit-0.0.1-SNAPSHOT.jar -hminecraft.n1-servers.fr -p25565 nogui
    And still with 76% of CPU load for only 10 players online, and so quickly over 100% when some 20-30 players :s

    10 players = 76 %
    15 players = 87.3 %

    My hardware is Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 8 cores, 12 gigas DDR3
    On Debian Linux 5.0.8, using jre 1.6

    Someone have some tips for me ?

    And a way to update my java to a 1.7 version ?
     
  14. Offline

    Phaedrus

    @WolwX I've made a few adjustments since my last post after more testing. I'll post an update tomorrow night. (I'm at work right now)

    CPU usage is going to depend more on what plugins you are using than on garbage collection. Tuning the collector can help eliminate world pauses due to the garbage collector, but it can't fix a poorly written or resource intensive plugin.
     
  15. Offline

    tekac

    It would appear the OP is not going to test out these new Java options. I have been following @TnT 's performance thread and have done a few trials myself. I just yesterday updated my server's java to 1.7-b-136. I have noticed an improvement.

    I currently run a x3460 @ 2.8Ghz, that is 4 cores, 8 threads. HT is enabled. I'm unsure as to if I should disable it or not. I have 12GB on the server. I have zero block lag, insta chat. However time and mobs are at a crawl. My CPU usage is off the charts. I was at 90%-100% prior to 1.4, but with 1.4 I bounce up into the 150%'s.. I could be mis-reading this. Here is a quick video showing hTop. http://www.youtube.com/watch?v=mCWgDgTeXfA

    EDIT: More detailed on the specs.
    Code:
    Intel Xeon Lynnfield X3460 - 4 x 2.8GHz
    12GB DDR3 1333 RDIMM
    146GB SAS 15,000RPM  x2 = Raid 0
    Cent OS 5.5 x64
    
    My current world is NOT in a RamDisk. I'm going to attempt it and see how it works out.

    The current settings I use are from TnT's thread.

    Code:
    java -d64 -Xincgc -Xmx10752M -jar craftbukkit-0.0.1-SNAPSHOT.jar nogui
    I'm am definately eager to trial your tunings @jasonznack I will let you know what I come up with !

    I average 40-50 players online, peak at 80. Since server cap is 80 :)
     
  16. Offline

    TnT

    @tekac
    At that size of RAM for your Java heap, you may want to use some more advanced flags as @jasonznack proposes. For smaller amounts of RAM for your Java heap, the flags I have are sufficient.

    I'd be eager to work with @jasonznack to update my tutorial for the more advanced flags, but I haven't approached him at all yet, so no surprise I haven't gotten it added.
     
  17. Offline

    tekac

    Well here is a pastbin of my verbosGC .. It ran for an hour period, hit 40 players, and varied between 30-40.
    http://pastebin.com/YpAEXGe8

    Used this commandline.
    Code:
    screen -S minecraft java -d64 -Xincgc -verbose:GC -Xloggc:gc.log -Xmx12G -jar craftbukkit-0.0.1-SNAPSHOT.jar nogui
    EDIT: I am now using the following commandline.
    Code:
    screen -S minecraft java -server -Xmn2G -Xms6G -Xmx12G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=5 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -jar craftbukkit-0.0.1-SNAPSHOT.jar
    
    I'll let you know how things go.
     
  18. Offline

    Phaedrus

    Try removing this entirely -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90
    The JVM seems to do a good job of sizing it on it's own, so it might be better to let it do its thing. I noticed a few runaway memory usage scenarios while using those. Not sure if its related, or just due to a bunk plugin.

    Change this value to 5 instead of 500 -XX:MaxGCPauseMillis=5
    500 milliseconds turns out to be a really long time with modern CPUs, so having it at 500 will probably never have an effect. 5 however should be a good target for modern fast hardware. You can go as low 1, but that might be a bit too low for it to achieve. No worries though, It will do it's best to meet your request, but it won't break itself to meet your demanding goals. ;)

    I also don't bother setting -XMN any more. I just let the JVM do its thing.

    -Xms6G looks good for a starting point, but -Xmx12G is too high since your max ram is 12g. this could lead to an out of memory error or at the very least force your OS to start paging like mad. Either way, not good. figure out how much your OS is using under load and subtract that from the 12gigs. 10G might be better.

    I'll post the more advanced settings I'm using in a few hours.

    Please keep in mind that this is targeted at Java 6. However, most or all arguments will work in Java 7 as well, however your millage may vary.

    Server is running Windows Server 2008 R2 SP1 64-bit, Java 1.6_25 64-bit
    Core i7 2600k OCed to 4.8Ghz, 16 gigs of RAM, 1gb RAMDisk

    Here is my my complete StartServer.cmd
    Code:
    @Echo OFF
    TITLE MINECRAFT-CRAFTBUKKIT
    START "MINECRAFT-CRAFTBUKKIT" /ABOVENORMAL /B java -server -Xmx12G -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseNUMA -XX:+CMSParallelRemarkEnabled -XX:MaxGCPauseMillis=50 -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:UseSSE=3 -XX:+UseLargePages -XX:+UseFastAccessorMethods -XX:+UseStringCache -XX:+UseCompressedStrings -XX:+UseCompressedOops -XX:+OptimizeStringConcat -XX:+AggressiveOpts -jar craftbukkit.jar nogui
    
    And to get complete Garbage Collection logs add the following:
    Code:
    -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -Xloggc:memory.log
    
    Warning! It can generate a fairly large log file after a while because it captures a lot of data. This may also have a performance impact on the server in general. Use only during testing to see how your garbage collection times are doing. This log file can be parsed by the various garbage collection analysis tools available which will give you a nice graph of your collection times.

    You can get a graphing analysis tool here: http://java.sun.com/developer/technicalArticles/Programming/GCPortal/index.html

    I'll break it down.

    Code:
    (AT) Echo OFF
    TITLE MINECRAFT-CRAFTBUKKIT
    START "MINECRAFT-CRAFTBUKKIT" /ABOVENORMAL /B java -server
    
    This starts the server with above normal priority and specifies the server JVM.

    Code:
    -Xmx12G -XX:PermSize=128m -XX:MaxPermSize=256m
    
    Specifies a 12gig maximum heap, which is just under my maximum ram. I do not specify an initial heapsize, instead allowing the JVM to take what it needs as it needs it. My server has 16gigs of ram, so I leave 2 gigs for the OS. Max physical RAM minus 2 gigs seems like a good rule of thumb. On a dedicated linux server you can get away with much less for the OS.
    The PermSize parameters set the starting and max size for the permanent portion of the heap. Default is 32mb and 64mb max I believe. From the garbage collection logs I have seen it grow up to 64mb pretty often, so I set it to something higher to give it room. 128megs isnt that much to spare.

    Code:
    -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseNUMA -XX:+CMSParallelRemarkEnabled
    
    Disables full garbage collection passes to prevent long pauses.
    Specifies the concurent mark sweep collector for the old generation heap and the parallel collector for the young generation. It also enables parallel operation for the remark phase of the collection. And finally, enables enhancements that take advantage of modern Non Uniform Memory Access architectures (NUMA) That can increase performance by optimizing the placement of objects in memory.

    Code:
    -XX:MaxGCPauseMillis=50
    
    It tells the collector to target a maximum of 50 milliseconds for collection pauses. Average GC times for modern CPUs are typically in the sub 40ms range anyway, so setting this at 50ms will help prevent spikes. Keeping the pauses lower than the server tick rate should help keep things running smooth.
    I let the JVM decide how many threads to use for garbage collection, etc. It knows best. Setting it too high or too low will cause more harm than good.

    Code:
    -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -XX:+UseBiasedLocking
    
    Allows the collector to move the boundary between young and tenured generation.
    The minus in front of GCOverHeadLimit means that it is turning that function off. This increases the amount of time it will let garbage collection go before throwing an out of memory error. Not generally something you want your server to do. I'd rather let it thrash for a bit to try and free up some memory.
    BiasedLocking should aid efficiency in multi CPU setups by biasing an objects to use the thread which first created it.

    Code:
    -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15
    
    Allows more time for objects to die before being promoted to the old generation. Try with and without these 3 items and compare. Test for long durations only for best results. 24 hours or more up time.

    Code:
     -XX:UseSSE=3 -XX:+UseLargePages
    
    Allows the usage of large memory paging to improve efficiency. Your OS will need to support it though. Windows 7 does, and most linux distros do, but you may need to do some tweaking to make it work on linux. Nothing google can't help you with. You will also need to run as administrator. If you receive an error you may need to grant some extra privileges to your user account.


    Code:
    -XX:+UseStringCache -XX:+UseCompressedStrings -XX:+Use CompressedOops -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+AggressiveOpts
    
    Enables optimizations in java 6 that are disabled by default, but may already be present in java 7.

    So that's what I use and I have pretty good results. I only have ~15 users max atm, so I'd like to hear what a larger server with 30+ experiences using these launch params. Though so far people have reported good results.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 8, 2016
  19. Offline

    Kane

    I would really love to see some recommended flags I should be running on my machine.

    Java 1.7 umm March 31st build
    CentOS 64bit
    16 gigs of ram
    6 ramdisk (16,000x16,000) world lol
    8 gigs to world (hell even 4 it only uses like 3gigs with 40 online not even...)
    2gigs for os...

    The biggest flag I used these days is -d64 and -Xincgc

    I noticed main thread seams to hit about 60-100% cpu when there is 30-40 players online and it spikes so high on /warp /home and /logouts. It seams anything to do with chunk unloading the main thread lags like a bitch.
     
  20. Offline

    WolwX

    Someone can tell where found any tutorial for use Java 7 into my Debian dedicated ? please ^^
     
  21. Offline

    tony.t

    Hi!
    can you help me with my new config please ?

    2x Intel® Xeon® E5620
    24Gigs of RAM
    Windows Server 2008 R2 x64
    Java 1.7 last build.

    thanks!
     
  22. Offline

    ioScream

    I used the same setup as you on a 3ghz dualcore and saw a HUGE decrease in CPU usage. I couldn't use:
    Code:
    XX:+UseCompressedStrings
    And I believe UseLargePages is causing me to get this on startup:
    Code:
    OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory (errno = 22).
    What do you think?
     
  23. Offline

    tekac

    @TnT I just decided to convert from Windows to Linux Mint on my personal PC. (nothing to do with server)

    I was wondering, since I run Minecraft.jar off of commandline.. would -xingc be of any benefit?

    Code:
    java -Xincgc -Xmx2G -jar minecraft.jar 
    I don't even think its necessary :confused: but now I'm all into this java tuning stuff. :p
     
  24. Offline

    Phaedrus

    Glad to hear it dropped your cpu usage.

    I assume you're using linux. As I said in my post, you may need to do some googling on how to enable large memory pages in linux. some distros support it out of the box, others do not.

    Has any one else tried my settings from a couple posts up?
     
  25. Offline

    TnT

    @tekac
    Yes, I would say it would improve your RAM usage. The command you have there looks good.
     
  26. Offline

    ioScream

    yeah, im using ubuntu.. looks like i need to edit something in my /etc/sysctl.conf.. I'll edit this post with information that might help others.

    EDIT: Looking more into, it appears beyond my scope of knowledge, and also probably not needed for me.. so I think I will leave well-enough alone.. fow now :)
     
  27. Offline

    Phaedrus

    Perhaps @TnT can be of more help with Large Page support in linux. I'm a windows server admin only.
     
    boduzapho likes this.
  28. Offline

    phondeux

    Could someone explain what each column means in garbage collection? Actually, just the first column as it keeps growing ...
    Code:
    5145.179: [GC 327851K->138772K(498112K), 0.0398640 secs]

    First Column 5145.179 - ?
    GC 327851K - GC means a minor collection, the number is The combined size of live objects before garbage collection.
    138772K - The combined size of live objects after garbage collection.
    (498112K) - The total available space, not counting the space in the permanent generation, which is the total heap minus one of the survivor spaces.
    0.0398640 secs - The time it took for garbage collection to occur.
     
  29. Offline

    anon

    Im running jre 7, should I use like that?


    Code:
    java -Xincgc -Xmx3072M -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+G1ParallelRSetUpdatingEnabled -XX:+G1ParallelRSetScanningEnable -jar craftbukkit.jar nogui
    EDIT:
    The above gives me this error:
    Code:
    user1@cl-443:~/Bukkit$ sh server_nogui.sh
    Unrecognized VM option '+G1ParallelRSetUpdatingEnabled'
    Error: Could not create the Java Virtual Machine.
    Error: A fatal exception has occurred. Program will exit.
     
  30. Offline

    phondeux

    Just cut out that parameter; i had the same problem and did that to resolve it.
     
Thread Status:
Not open for further replies.

Share This Page