What happened to CI / Jenkins?

Discussion in 'Bukkit News' started by EvilSeph, Jan 21, 2012.

Thread Status:
Not open for further replies.
  1. Offline



    Three days ago we were forced to lock down our build server, ci.bukkit.org, due to high load from all of you eagerly smashing F5 to see if a new Recommended Build had come out. While it is awesome that we have such a following that this is even possible, unfortunately CI runs on the same box as the rest of our sites and took down our site and forums whenever CI was under heavy load.

    As a result, I had to make the difficult decision of stopping all public access to our build service in order to restore stability and reliability to our website and forums. Once we had done this and the site was mostly reachable and usable, we immediately went on a hunt to determine what the cause of the poor performance was on our servers. Although the site is relatively stable, we're still seeing problems crop up now and again and are doing whatever we can to get down to the bottom of things. As part of this effort, we'll be experiencing minimal downtime on Monday as we'll be updating various aspects of our main server and it will require a reboot. After doing so, we'll continue to monitor the situation and see what we can do to continue to improve things.

    So what are we doing to solve these issues? We're actively working on developing a better way for the community to find and acquire our builds and releases and hope to have a suitable solution ready within the next few days.

    While we fully intend to continue to utilise Jenkins to provide us with continuous integration and reactive build compiling, we feel that Jenkins isn't a suitable solution for providing a user-friendly download experience for our community. As a result, we've been working on an easy-to use solution better designed to meet the needs of both our developers and our server admins. By combining the invaluable information usually provided by Jenkins (build numbers, changes and associated commits between builds, and so on) with a better user experience and intuitive design, we'll be able to provide an optimised, well cached site that can withstand the level of traffic we see day to day while still providing the community with up-to-date information about the latest builds and more.

    Thanks for your patience and understanding while we investigate the cause of the issues we've been having and make changes to address them, your continued unwavering support means the world to us.
  2. Offline


    Copied from CraftBukkit repo...

    We use maven to handle our dependencies.
    • Install Maven 3
    • Check out and install Bukkit
      • Note: this is not needed as the repository we use has Bukkit too, but you might have a newer one (with your own changes :D)
    • Check out this repo and: mvn clean package

    peddermaster2 likes this.
  3. Offline


    Even with dynamic systems, CF will detect mostly-static contents correctly, and cache them when possible. Prime example for this would be static caching of previous builds, and change logs. There are also things Bukkit team can do to modify headers of pages that are rarely modified to have a longer cache, which will then trigger CF to cache them so the physical server does not get hammered. Based on information provided by both thread OP's posts, the lag issue is coming from load caused by excessive requests. As such, the suggested mechanisms are able to address -- let it be fully or partially -- the issues with minimal modifications.

    I have did a rough background scan of your twitter accounts, and your website. And I have come to the conclusion that based on what I've observed, and my previous experiences, I am more experienced in this field than you are. As such, while we are on the subject of suggestions: I would suggest you do the same, and then pass judgements as to whether or not I know what I am talking about.

    PS: I do not do tutoring gigs.
    Jeremy Stuck and h31ix like this.
  4. Offline


    Just please make sure there is an RSS feed! :O
    spydercanopus likes this.
  5. Offline


    yay communication from the mothership!
  6. Offline


    I have been wanting this for a while, but it would be nice to get a push notification of some sort when a new release comes out. Don't need any fancy update (although that may be cool), but just a simple email to the account you signed up with would be nice. And it would cut down on people refreshing the site and whatnot. Currently I just used an xml survive that emails you when a new item has been added, don't remember the name, but there are a ton of them out there. Not sure how that is on server load, but a built in solution would be better.
  7. Is there a site or such yet which we can get devbuilds from? My current one is spamming up to 9 errors every second and its getting bloody annoying to monitor what exacly my players are up to as these errors increase my logs hugely and make me uncapable of following plugin/player actions... Either way i need a new devbuild! Lol.
  8. Offline


    • Not acceptable.
    God your stupid

    See this article to find download links for CraftBukkit: http://goo.gl/kJ37D
  9. Offline


    RIP Jenkins. I hope your replacement is half the butler you were. :(
  10. Offline


    Jenkins was a overworked and underpaid butler. You all drove him into quitting on us :(
  11. Offline


    I didn't spam f5 on jenkins, but I did use it for checking the changelog and, very occasionally, downloading a specific build (as opposed to just "latest")

    I hope whatever you choose to do allows for those use-cases
    Tanite, Doc, Technius and 1 other person like this.
  12. Offline


    I did this too. Recently I've taken to checking for recent changes on GitHub and compiling old builds myself, but it's a bit of a pain compared to the convenience of Jenkins.
    Doc likes this.
    Feartide likes this.
  14. Offline


    Do your thing, Bukkit Team. I can't speak for everyone, but I'll wait as long as it takes. Your goal is to make the best experience possible and I think that's worth the wait. Keep on truckin'!
    peddermaster2 likes this.
  15. Offline


    Just follow @craftbukkit on Twitter...
  16. Offline


    Several developers run their own instances of Jenkins. If you could get them to volunteer as mirrors, using the build publisher plugin, you could spread the load out some.
  17. Offline


    Alternatively; if they just need some bandwidth to go around, I'd be more than happy to donate some. I have a 10mbits unmetered line, which while isn't much, can be used to act as a secondary mirror. I can setup rsync or ssh access for bukkit team, and they can push jar files and checksums over for distribution.
  18. Offline


    Id being willing to help host the CI/Jenkins but i doubt my x10hosting account will handle the traffic
  19. Offline


    Firstly, the only way I've found to see anything about your progress in the past was http://ci.bukkit.org/job/dev-CraftBukkit/changes. All other things I've found weren't useful in comparison. I'll echo the general tone I've seen in this thread which is, we want to know how things are going. While "Stop asking" is understandable, so is people giving up on the project when that's the answer we get.

    I'm a minor, insignificant Bukkit admin with no customers, no managers, no pressure and lots of patience, but I've come to the point where I just shut down my bukkit server and throw up a vanilla one when a minecraft patch comes out because god knows when Bukkit will be ready, and everyone who plays on my server updates instantly because they want to try out the new stuff. If my server isn't up to date, there's simply no reason to run it because nobody connects.

    I had hopes that the delays in getting Bukkit out for a new minecraft release were temporary issues and that, as you got used to the work needed after a release, you'd get faster at. Instead, things seem to be going the other direction. Other mod systems seem to get running after a new release much faster. I'm not sure why, but I'm guessing that it's part due to the complexity of Bukkit, and part because instead of trying to just get a working build out, you're pushing forward with planned new features and changes and releasing a recommended build when they are ready. I understand taking this tact, but it's getting to the point where there's no reason to work with Bukkit because it rarely works with the latest minecraft.

    My recommendation, and I apologize because I suspect it's been given before, is to have a system by which access to non-recommended builds is more useful. Try to push out a working build soon after Minecraft updates and have a way for builds to get flagged as usable. You could possibly use the community for this by creating a system by which the server admins and mod developers could do the work of testing and rating the build, something simple like a 1-5 rating scale with 5 being works perfectly and 1 being fails to function at all, then let us see the ratings on the different builds and talk about them.

    I don't know if this would be prohibitively difficult or time-consuming to manage, but I know I personally would use it, and it would go a long way toward making me feel like Bukkit was the one true way to do things.
    peddermaster2 and Technius like this.
  20. Offline


    Nothing is stopping you from using a development build if you'd rather not wait for us to mark a build as recommended. I chose to delay the RB promotion because we'd be promoting a stability RB, then shortly after promoting a feature-filled RB which will annoy server admins as they probably just got around to updating their server by the time we got the feature-filled RB out. We've done this before and it has just resulted in an army of angry server admins, so now we're doing it a different way.

    The development builds are usable and mostly stable but it is entirely up to you and at your risk to run any builds we make available, recommended or development, it doesn't matter. Nothing is stopping you from using whatever build you want. Nothing will stop you and nothing has ever stopped you.

    If a Recommended Build was something we rushed to get out, it wouldn't be recommended - it'd completely undermine the system. It'd just be an update we mark as working. Recommended Builds take a lot of time, effort and ridiculous amounts of boring, repetitive tests to ensure that we catch whatever issues we can so that your server isn't affected. If you don't appreciate the amount of work my team and myself put in to making sure our Recommended Builds system has a purpose, then fine, use a development build. But many admins out there do appreciate the work we do, do value what the Recommended Build system has to offer and do feel safer knowing they are running a well-tested build on their production server.

    Until that number reaches 0, we will continue doing as we have been doing because we understand that downtime and problems on a server are unacceptable; that you'd be torn apart by your community for the smallest problem and it is these people that we offer the Recommended Builds system to. For those of you who don't care about being on the edge, you have the option of using a non-Recommended Build.

    Just because you and some other people think we should rush a Recommended Build out, doesn't mean there aren't many people out there who see the value in our process. I really don't understand where you are coming from when you complain about delays from us: we are almost always the first to get a compatible build out for our mod whenever a Minecraft update. We've gotten an update out as early as several minutes after it was released and as late as a few hours or more.

    The delay isn't on our part, it's on yours. It is completely up to you if you want to run a development build or not.
  21. I'm fine with the waiting. The only thing I've notice we can look forward to is bow enchantments Whoopee. And Creative Mode spawn eggs. Aah... We can use the /spawnmob command anyway.
    But thanks to Spoutcraft, my members can flit to and from each version at will. So I know they're cool with the wait, also. Take your time, guys. It'll take a while for plugins to catch up anyway...

    And this, kids, is what we mean when we use the analogy "pot calls the kettle 'black'", and a great example of Skitt's Law.

    Perhaps you could be a little more constructive as opposed to, y'know, a douche. The community spirit lies in helping others and pooling knowledge, not beating someone down if they haven't kept up with what's been going on.
    peddermaster2, robinjam and Feartide like this.
  22. Offline


    Bukkit, I have a proposal.
    And by Bukkit, I mean the Bukkit community, developers, anyone who reads threads here.

    Instead of tearing each other apart, wasting time calling each other idiots or hating on someone else over a subject that will never benefit anyone who argues it, how about we do something constructive for once. This is the way open-source software is meant to work. The developer chooses to support you as a community with free software that they spend the equivalent of years working on when they could be doing something else, and you, as a community using their software, return the favor.

    You may or may not be able to code something for them, but you can sure as hell support them. Financially, or just through some good, old fashioned compliments. Yep, that still works over the internet. "Thank you" is one of those great phrases that will never cease to have meaning. There will always be kids on the internet, unfortunately Minecraft tends to draw more of them than usual, but that does not mean that you have to insult them or exclude them. By calling each other "idiots" and other derogatory names you are not only proving to the world that you are a self-minded waste of a human being, you are also degenerating the quality of the Bukkit community, thus essentially telling everyone who is TRYING to support you "Screw you, I don't need your work, I don't even like you or your community" and that is DEFINITELY not supporting them.

    Let's make this a mutual relationship guys. I'm no hippie, but saying thanks to someone is just as good as contributing to the Bukkit code. Especially when you have such a well-thought-out, dedicated team of individuals such as the Bukkit developers, who clearly run their project at a much higher standard that any other Minecraft server project out there, or for that matter, most other open-source projects, you should think twice before contributing to the trash pile here. I wish we could send PEOPLE to the LAWL section sometimes.

    So drop by you're favorite plugin's page, give them a thanks for the work they do, visit the bukkit team's profile and drop them a friendly comment; its a relief from the eyesore that is the current status of those profiles.. And THAT is mutual-ism.
  23. Offline


    Thank you! Bukkit is awesome. You guys rock.

    All the plugins still work fine... The bigger ones have dev builds for 1.1, others work or have been 'fan updated'
    Running 1.1 on my server and only 2 plugins (of 68) required updating to work..
    h31ix likes this.
  24. Offline


    Unless the plugin isn't using the Bukkit API (which it, and all good plugins should be using), it shouldn't require much updating.
  25. Offline


    I dont see how us (the community) throwing out suggestions on ways to fix their current problem ie. Website crashing which translates to them having to waste time fixing the website which translates to more delay in bukkit releasing.

    The solution I recommended is to keep the community updated. Writing a few lines every couple days on the progress would cut the website traffic considerably. There were many others. Few were negative.

    I think we all appreciate the bukkit guys... and personally the only person I am pissed at is notch for doing such a terrible job finishing minecraft by wasting time making "the end" and a horrible version of the millinare mod. Then committing himself (and his team) to several more platforms. When he could have made a decent API and everyone would be alot happier right now!
  26. Offline


    My comment was not directed just at those trying to offer ideas, but at the same time I stand by what I said earlier, the Bukkit team and their partners have more than enough brain power between them to be able to fix this and it's not really helpful for people to be offering solutions on how to balance load. That being said, it can't hurt, so it's not worth arguing.

    As for the "keeping the community updated" do you REALLY think writing a few words every day on the front page would stop the influx of kids spamming the jenkins? Think about it for a second, that's not the way the internet works. Even when something is posted on the front page of the entire website you still get people making posts about it, asking questions about it on IRC, nobody who will spam the jenkins page looking for a new RB will be looking to the front page for updates, and then stop doing what they are doing when actually taking a moment to read the posts on the front page. that's not how those people are. If you want updates, follow the twitter.
  27. Offline


    YES I do.
    Nox13last likes this.
  28. Offline


    Here's the thing - why on earth would you keep the CI on the same server as other websites? It's not a terribly bright idea...
    IMHO if you guys use Atlassan's JIRA Studio would it be so hard to add Bamboo to it as well? :eek:
  29. Offline


    90% of the traffic was probably caused by that CraftBukkitUpToDate plugin.
    About every server I see has it installed.
    peddermaster2 and TopGear93 like this.
  30. Offline


    • Warned for using all capital letters
    rymate1234 and h31ix like this.
  31. Offline



    In all seriousness though, I hate it when people use full caps.

    I would thank bukkit with a monetary donation, but I'm poor :(. So thanks bukkit! You're awesome!
Thread Status:
Not open for further replies.

Share This Page