[WEB/ADMIN/API] BukGet: The Unofficial BukkitDev JSON API

Discussion in 'Bukkit Tools' started by SteveMcGrath, Dec 4, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    Sushi

    That's for md_5 to answer.
     
  2. Offline

    SteveMcGrath

  3. Offline

    md_5

    Something like when I want to run static analysis on all the plugins, which at the moment I have to:
    1. Get your entire list of slugs
    2. Get your page on each slug
    3. Scrape the DBO download page for each project using the link in part 2, since you don't provide a download link.

    The other core reason I have is if I wanted to write a search engine and needed low latency + complex queries, or do mass analytics on the description or some other aspect of every plugin. For this a full dump is absolutely crucial.
     
  4. Offline

    SteveMcGrath

    I finally got a workable version of the API up. Here are some progress screen shots:

    Screen Shot 2012-12-13 at 1.08.04 PM.png Screen Shot 2012-12-13 at 1.08.19 PM.png Screen Shot 2012-12-13 at 1.08.40 PM.png Screen Shot 2012-12-13 at 1.09.05 PM.png
     
    lol768 likes this.
  5. Offline

    SteveMcGrath

    Ok, I'm going to start contacting Devs that I know are using the API to start testing against the development build thats up now. I expect there to be some issues, however I hope everything by and large works w/o breakage. The only things that are significantly different are:

    * Commands and Permissions are normalized out
    * GenInfo Ids are now referencing the MongoDB document, and are not integers
    * GenInfo actually shows changes now. ;)

    I'll be posting the URLs up on the OP
     
  6. Offline

    SteveMcGrath

    API Docs are now posted on the website.
     
    hawkfalcon likes this.
  7. Offline

    SteveMcGrath

    Major code shift. the API and the parser are now separate modules. Furthermore, BukGet now has an organization in Github \o/
     
  8. Offline

    Bertware

    tyzoid, hawkfalcon and Sushi like this.
  9. Offline

    Sushi

    Lmao
     
    lol768 likes this.
  10. Offline

    lol768

  11. Offline

    Bertware

    that's it :)

    SteveMcGrath
    {"description": ["PlayerIP is a plugin with commands..."], "plugin_name": "PlayerIP", "slug": "minecraftplayerip"},

    on http://dev.bukget.org/3/plugins/bukkit/

    Description is array. Is this a bug in the parser? I suppose it is, because I don't know why description would be an array.
    If it isn't a bug, I can alter my parser, but otherwise I thought this might be good for you to know :)


    Edit:
    Except for the bug mentioned above, updating the GUI went flawlessly :)

    Wrong version data for : dev.bukkit.org/server-mods/abitofrealism
    hmmm, http://dev.bukkit.org/server-mods/activator/ too.
    Seems like the version parsing doesn't fully work yet.

    Is dev.bukget relyable to use already (except for the version parsing), or is there a possibility to release more stable beta builds to api.bukget.org/3 ?

    Edit 2:
    Small suggestion:
    I'm using
    http://dev.bukget.org/3/plugins?fie...ption,versions.version,versions.game_versions
    But is there a way to only show the last version and game version? Now it shows all versions, which results in quite a big page.
     
  12. Offline

    SteveMcGrath

    Ill look into adjusting that. Generally speaking, the reason for this is bad plugin.yml, however I can fix it in the parser to just return the string value of the array. For example, here is that plugin.yml:

    [​IMG]

    I just checked into this, the issue is the plugin dev. They have never updated their versions in their Jar files. I will contact them about the issue and hopefully they won't be soo lazy next time.

    BukGet is parsing the version in the plugin.yml over the version on the site now. There are a lot of reasons to do this, however mainly for reliability. So this is not a bug in the parser, just lazy devs who dont update their plugin.yml files properly. I'm continuously amazed by the amount of badly coded plugin.ymls that exist.

    dev.bukget.org will be moved into production in mid Jan. as I have been coding fixes for the new API code, I jave been automatically updating it. As the codebase is a ground-up rebuild, I can't easily graft on API3 to current stable without replacing api2 and api1. All of the APIs have been coded into the new codebase however.

    Not from the plugin listing no. However you could always just accept only the first item in the versions array for each. The Dev API is now keeping stats on the plugins, so I don't think we would impliment this in a single call, mainly because the stats updating is hooked into the plugin details calls.

    So I wrote a quick little script to help me manually fix these errors. If you see any more inconsistencies, then let me know.

    Also Double-check the API, those should report properly now.

    Fixed the link. sorry about that :-p

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 30, 2016
  13. Offline

    Bertware

    The API result for the plugin list now works without problems :)
    http://dev.bukkit.org/server-mods/accident-tnt/
    Also displayed wrong.
    However, the ones that are wrong are consistent in the version field. Always 0.1. Maybe you could use that to run an extra check on it? If you search the filenames a second time using regex for
    Code:
    \d{1,5}(|.\d{1,5}){1,4}
    , or similar, wouldn't it work?
    This regex would match x to x.x.x.x, each x between 1 and 5 digits. You might need to add allowance for some letters (b, #, beta). But you could use it if the first check/parsing didn't work.

    http://regexpal.com/
     
  14. Offline

    SteveMcGrath


    you would assume it to be that easy..however here is the regex I was using: https://github.com/BukGet/bukget/blob/master/bukgen/bukgen/bukkit.py#L30

    The problem is that a LOT of people don't conform to using proper versioning. this led to a lot of issues with the parsing. One of the many reasons why it was switched to parsing the plugin.yml instead. Either way, there will be issues that require cleanup. I also need to hope that the BukkitDev staff are checking for things like these.
     
  15. Offline

    SteveMcGrath

    Ok, well Bertware, I can't ignore this. There are some 484 plugins that are in the database like this, which is unacceptable. I'll have to figure something out, probably add in that logic. I might as well have a dbo_version that uses the regex and is copied to version if this condition is detected.

    Ok, so I ran a quick little script to see how many plugins have 3 or more versions and only 1 version number...155 plugins came back. Here is the pastebin of the json and the code. The problem I can see with performing a simple check is that even if we patch the issue now, if plugin devs don't realize whats going on, it'll get worse later. BukkitDev staff really need to check to see if the plugin version noted on the upload matches the plugin.yml version IMHO.

    JSON Output: http://pastebin.com/0LKKFxGj
    Code: http://pastebin.com/n4G8eqLJ

    EDIT1:

    After working on it for a few hours, I have some code I'm testing in the repository right now that solves this issue I think satisfactorily. I'll update this thread when I have a better idea.

    EDIT2:

    Code now support this favoring of DBO_Versions under specific conditions. Everything appears to be working.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 30, 2016
  16. Offline

    ImminentFate

  17. Offline

    SteveMcGrath

    I fixed the issue. It had something to do with the stats monitors that I added. Everything works as intended now ;)
     
  18. Offline

    Bertware

    Seems like most of them are fixed now :)
     
  19. Offline

    SteveMcGrath

    Yayz! I'm running a full generation as we speak to pick up any stragglers. It should complete sometime this evening.
     
  20. Offline

    ImminentFate

    thanks! Here i was thinking i couldnt read instructions correctly:p
    I think many people underestimate just how useful and easy to use this API actually is
     
  21. Offline

    SteveMcGrath

    Thanks for the kind words. The API has been through a few iterations to get everything right and has taken a lot of Man-hours of coding and work, and it's always nice to hear that people find it useful.
     
  22. Offline

    SteveMcGrath

    Just bumping this for everyone.
     
  23. Offline

    Antariano

    We all greatly appreaciate what you do. I'm sorry, I should have said this in a much more clear way. What you did is amazing :)
    I wish I could help you out better then how I am currently, but I promise that I will do so as soon as I've got the means to!
     
    Bertware likes this.
  24. Offline

    Bertware

    I have to agree with Antariano about the awesomeness and amazingness of the API, without that there wouldn't be plugin managers that can show this much details etc (versions, names, authors, ...) nicely formatted :)
     
  25. Offline

    SteveMcGrath


    Appreciate it man, You guys were one of the first larger projects to utilize the API and helped to promote it to the usage it's at now. ;)

    Appreciate it!

    To be honest, I'm continually amazed at how popular BukGet got in 2012 and am glad people are finding it useful. Now I just need to find a server to play on and just relax a little ;)
     
    Antariano likes this.
  26. Offline

    Antariano

    I hope that with SpaceCP we'll be able to use it much more efficiently than we are currently :) Keep it up!
     
  27. Offline

    SteveMcGrath

    Well im happy to report that after about a week of usage, the new codebase has proven to be rediculously stable. There are still a few things that need to be cleaned up (mostly on the updater) however everything is running well, no memory leaks, and the incoming requests are all being handled well within tolerances (actually better than the old code!)

    I do have some things on my plate to start working on:

    * JSONP support for API3 (Yes, callbacks. Been asked for this a couple of times)
    * Ability to delete items from the database that have been deleted in BukkitDev
    * Support for multiple services (python venv + better config file handling)
    * Code cleanup and dopcumenting (common.py is badly documented, among others...)
    * GET MORE PYTHON DEVS ONBOARD!!!!!
    * Get a JavaScript person onboard to write a simple GUI into the API.
    * More stuff that I can't think of at the moment.
     
  28. Offline

    SteveMcGrath

    Popularity is starting to get phased into the API. More details when we have enough metrics to fill out day/week/month.
     
  29. Offline

    Puseidr

  30. Offline

    SteveMcGrath

    It's not something thats currently parsed, however could be added. Generally speaking however the only thing that I have found to be truly unique is the slug.

    ADDED

    Here is the commit for main to start being tracked. It will organically load up on newly changed plugins until friday, when I kick off a full index to get the updated field into all of the plugins.

    https://github.com/BukGet/bukget/commit/5b48594c1c19e49c53a3e2d5b9a46a43d8967450
     
    Puseidr likes this.
Thread Status:
Not open for further replies.

Share This Page