What happened to CI / Jenkins?

Discussion in 'Bukkit News' started by EvilSeph, Jan 21, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    EvilSeph Retired Staff

    [​IMG]

    Three days ago we were forced to lock down our build server, ci.bukkit.org, due to high load from all of you eagerly smashing F5 to see if a new Recommended Build had come out. While it is awesome that we have such a following that this is even possible, unfortunately CI runs on the same box as the rest of our sites and took down our site and forums whenever CI was under heavy load.

    As a result, I had to make the difficult decision of stopping all public access to our build service in order to restore stability and reliability to our website and forums. Once we had done this and the site was mostly reachable and usable, we immediately went on a hunt to determine what the cause of the poor performance was on our servers. Although the site is relatively stable, we're still seeing problems crop up now and again and are doing whatever we can to get down to the bottom of things. As part of this effort, we'll be experiencing minimal downtime on Monday as we'll be updating various aspects of our main server and it will require a reboot. After doing so, we'll continue to monitor the situation and see what we can do to continue to improve things.

    So what are we doing to solve these issues? We're actively working on developing a better way for the community to find and acquire our builds and releases and hope to have a suitable solution ready within the next few days.

    While we fully intend to continue to utilise Jenkins to provide us with continuous integration and reactive build compiling, we feel that Jenkins isn't a suitable solution for providing a user-friendly download experience for our community. As a result, we've been working on an easy-to use solution better designed to meet the needs of both our developers and our server admins. By combining the invaluable information usually provided by Jenkins (build numbers, changes and associated commits between builds, and so on) with a better user experience and intuitive design, we'll be able to provide an optimised, well cached site that can withstand the level of traffic we see day to day while still providing the community with up-to-date information about the latest builds and more.

    Thanks for your patience and understanding while we investigate the cause of the issues we've been having and make changes to address them, your continued unwavering support means the world to us.
     
  2. Offline

    gianfun

    Hm, this might be a kinda lame alternative, but maybe at these times when everyone gets frantic refreshing until a new build is released, the main page of the site could be changed for a page with a small page with some info like "New build (#buildnumber) is not out yet!" or "Here is the new build!", and then, underneath that, a link to the original main page so that users that want to see other things can navegate the site normally!

    (and btw, I do like jenkins)
     
  3. Offline

    Pokkits

    And this is why Bukkit team is awesome.

    Sorry all the love took down the servers. :) Thank you guys for all the extra work you are putting in on the tech side and developing a new solution for the user experience.

    Keep up the good work!
     
  4. Offline

    Darkhand81

  5. Offline

    alfonsojon

    A nice plan would be if you were to set up a local FTP server to push new builds from the ci.bukkit.org computer over to another server in which users had permission to download from, thus preventing future bukkit meltdowns.
     
  6. Offline

    Glaive

    May I suggest torrenting. It would Take the load off the server other than the download page. It also is not illegal since you own the intellectual rights to it and MineCraft is open dev. Yes the users would have to work together but I'm sure in order to keep your site up and working people wouldn't mind sharing their bandwidth with others.

    Many companies that have such problems are doing this. The best part is with a community this big your almost guaranteed that the download is available
     
  7. Offline

    EvilSeph Retired Staff


    We have a mailing list/Google Group, see the following: http://forums.bukkit.org/threads/new-bukkit-recommended-build-announcements-system.51003/

    We're still deciding what kind of announcements (on top of RB announcements) we should be using this mailing list for.

    This is something I'm working on addressing as much as we can. Leaky / JIRA has a roadmap feature..but it isn't easy to link to.

    Getting someone to do this would mean it being more than a full time job and we don't have the resources to hire anyone - it is that much work. I wouldn't ever consider dumping this much work onto anyones lap unless they were getting paid.

    Torrenting is not a feasible solution. We have far too many builds that we need to make available every day and a torrent isn't easily controlled if we need to pull a build for whatever reason. The issue is mostly that Jenkins is such a responsive and rich system that every hit it gets is significantly larger than if people were refreshing a more static site/page.
     
    peddermaster2 likes this.
  8. Offline

    iKeirNez

    Thanks for replying, I have subscribed although would it be possible to have a seperate group for dev builds for CraftBukkit and the Bukkit API? That would be awesome.

    Keir
     
  9. Offline

    HappyPikachu

    You heard the man.

    START DONATING.
     
    Stone_Tigris likes this.
  10. Offline

    a4blue

    What about: Outsource the CI System on another system/server or
    Use a Fan-based mirror for the RB's ??
    I maybe could mirror the RB's on my website, so the traffic on the ci system could slow down
     
  11. Offline

    diffedge

    One of the easiest solutions in my mind is to COMMUNICATE and on the main page as far as progress on a new build! You should have a post every 3 days minimum. I mean it does not need to be more than a few lines!

    I would not be hitting the website half as much if you guys did that.
     
    Feartide likes this.
  12. Offline

    Deathwish001

    People need to do is stop smashing F5 and just have a life. I comeback once every 1 to 2 days to see if there is a build. if not I play single player custom maps or try out mods.

    Uh wait.... we are dealing with kids....
     
  13. Offline

    lorenzo_p

    maybe the answer is as simple as server side caching?
     
    peddermaster2 likes this.
  14. Offline

    Gravity Retired Staff

    The progress is on the right side of home page usually, but just wait, if there is a recommended build you will KNOW, you can't demand that kind of update that's totally unreasonable, there are much, much better things to be doing rather than trying to appease people spamming f5 for a new build
     
  15. Offline

    ThornRage

    As a server admin, I must say that I truly miss Jenkins. I checked once a day for a new update and installed the latest one. The people on my server enjoyed testing the latest build even if it was not the RB. However, I eagerly await the new system in the hopes that we can continue to test each new build.
     
  16. Offline

    dark_hunter

    EvilSeph - Is the downtime of CI delaying new builds of Craftbukkit?
     
    Feartide likes this.
  17. Offline

    Kozzy68

    Duno if forum is also related but if people have a server that has like 30 50 plugins and there is no proper way to get new versions notification or getting updates yes people have to click like crazy. Proper update mechanism like one you can see that even on open transport tycoon deluxe would cut 75% to 90% of the trafic. What is worse there is no even notification on new bukit dev when you get reply. So again you have to check it manualy.
     
  18. Offline

    Donny

    Having the changelog filled out would help, so people know whats been done and what not. Seems to be lacking since minecraft 1.0

    Since most users are grabbing the latest build, have a complete list of all changes ,and the latest devbuild pushed to another server/page. Where all users could download from that page instead of ci and users will flock there instead.

    Leave ci open though, please.
     
    peddermaster2 likes this.
  19. Offline

    Cory_

    Simplest solution if you're able too is considering of ordering a separate server for just the Jenkins CI, I've never used it personally, depending on the load either "Cloud VPS" or a "Regular VPS" possibly substain fine, then again depending on the rates of those you could easily order a dedicated server which would be highly recommended and since the DNS is on
    DNSMADEEASY.COM it would be simple to just point ci.bukkit.org to the new server and keep everything intact, unless your plans are to upgrade altogether.
    Now, Someone did mention "torrents" which isn't illegal and there are free tracker scripts out there and definitively would take high bandwidth load off your server as it wouldn't be using much. I would just say stick with what you know.
    Otherwise, possibly allowing us users become a mirror for you, but you would have to have an official mirror listing so nobody would be able to just "host the files"...Simple PHP/MySQL could be coded to do so.
    I'm an Assistant Admin over Minecraft Monday' and we're still on 1.0 as we only use RB, We're patiently waiting, might consider shutting down out main server and do a "Temp Map", but all users are against it.
    We're patiently waiting and hope for you over there at Bukkit to sort the issues out and hopefully soon we'll here some good news related to Bukkit.
     
    peddermaster2 likes this.
  20. Offline

    diffedge

    How is expecting 15 mins tops every 3 days to give an update unreasonable?

    It seems your saying incredibly basic customer service skills may be expecting to much from the bukkit crew. I think they are a bit more capable than you are giving them credit for.
     
  21. Offline

    lokpique

    I think you were the first person to bring up the RB release date in any form.
     
  22. Offline

    moose517

    Sad to see it locked down as i liked looking at the commits each day to see what kind of features were added. maybe make it so that plugin devs that have been flagged(using bukkitdev having released plugins) can still view it like normal but the general admins/players can't get to it, as its more useful to us than them anyways.
     
  23. Offline

    Gravity Retired Staff

    I think you are failing to realize that the bukkit team is built of people who have LIVES outside of Bukkit.
    They are not paid for their trouble, nothing they do here is returned to them financially. They are doing this work for FREE for you AS WELL as attending school, working, and various other tasks that do not revolve around the internet.

    While they are working on the project, it is much more worth the time to be looking at pull requests, fixing exploits, pushing out new releases, and not sitting on the home page writing updates for people who cant join IRC and ask a question, look at other people's posts, or get a clue as to how open-sourced online projects are run. You would not be enjoying or running minecraft servers with such an expansive amount of plugins, support, a great and dedicated team, and many other things that other distros of the Minecraft server software do not provide without the Bukkit team, and you would be best to remember that.

    You can still look at commit notes and commits -> https://github.com/Bukkit/CraftBukkit

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 23, 2016
  24. Offline

    Cory_

    That's why I'm not pushing the issue, We're patiently waiting...and not going to follow everyone and use Dev Builds...
     
  25. Offline

    chiisana

    Again, as I mentioned in the other thread: if this is the cause, simply load up a cache system for ci.bukkit.org. If the problem is caused by F5 spam, CloudFlare (free, easier, no hassle, but will cache only pages and css, probably not bukkit jar files) or MaxCDN (need to setup Pull Zone; but will cache bukkit jar files if < 10mb) will resolve the problem in no time.
     
  26. Offline

    Gravity Retired Staff

    Jenkins is a dynamic system, pages are always changing. While cloudflare does cache pages it still needs to access the page, its not just some magical system that takes load of your servers, if there are that many people trying to connect it will make the system unstable and crash. Multiplay, Curse, and the Bukkit team are very knowledgeable of how to run their website, I would suggest that it is useless for you to give them tutoring sessions on what to do.
     
    peddermaster2 likes this.
  27. Offline

    HighBomber

    I was thinking the same thing.
     
  28. Offline

    marcusant

    Step 1. Learn to compile your self
    Step 2. Compile from the CraftBukkit source code
    Step 3. ???
    Step 4. Profit
     
  29. Offline

    norsac321

    Whats the Bukkit Repository?
     
  30. Offline

    marcusant

  31. Offline

    Doc

    Thanks for the update.
    Here are some personal observations:

    on the repo site we saw builds in this notation
    craftbukkit-1.1-R1-20120120.195601-80.jar

    While it allowed server operators the ability to download specific builds it did not show the traditional noatation for
    build 1771 (an example)

    This is VERY hard to match plugins that need a specific bukkit build
    Its easier when its >= or <= for build numbers.

    When you do find a nicely cached solution, please find a way to include the traditional build numbers into the list.
    Also, being able to read the comments on builds is helpful for those of us who are truly trying to help you track down bugs etc. Which is why I am sure most people were pounding F5 or clicked on auto refresh.

    Thanks again for your work that you all do for the MC community!
     
    Tanite, Technius, Darkone003 and 2 others like this.
Thread Status:
Not open for further replies.

Share This Page