[WIP] Guardian - the final replacement for BB, LogBlock and HawkEye

Discussion in 'WIP and Development Status' started by oliverw92, Aug 27, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    mbaxter ʇıʞʞnq ɐ sɐɥ ı

    This is my example of a large database. A BigBrother bbdata table with over 150 million rows. With indexing, 14 (fourteen) GB. An individual player with over 20 million rows. This from only tracking blocks (placement and breaking) and nothing else. Occurred in a period of 2.5 months.

    This tends to make BB cry and then lag the server. I'd like LogBlawk to wimper less in these situations. Yes, of course, this is an exceptionally large count and at this point you run into issues with MySQL getting slow as well. But it's a good example of the scale you should reach toward :p
     
  2. Offline

    Raycrash

    maybe delete some of the 1month+ entrys ?
     
  3. Offline

    xcanner

    If you are running a big (multiworld) server, removing 1 month+ entrys is the same as let griefers do what they like.
     
  4. Offline

    but2002

    When you get to this point it would almost be a good idea to create seperate tables for date ranges
     
  5. Offline

    Raycrash

    I mean the entrys which are one month and older not delete more than a month :p
    But if you say so I'll believe


    Edit:
    logdoc
    blockdoc
     
  6. Offline

    Deren909

    :O U WANT IDEAS!!! LOL UR ASKING THE RIGHT GUY :D:

    • Simple Yet Complex (OFC u already do that with LB)
    • Broadcasting a Fire Warning (i.e. A FIRE HAS STARTED IN 'world' BY 'environment' AT '722,24,653'
    • If A Sign Was Placed Where Someone Placed Lava i.e. Ur Walking And U See A Sign On The Ground That Says: "DiddiZ Placed Lava Here" And Its Protected OFC And Only Admins Can Destroy And Something That Helps Against Someone Spamming With Lava Thus Spamming Signs.
    • A Criminal History Maybe With /lookup DiddiZ and you can write the hisory too (I'm not going to say i.e. anymore xD" by doing /writehistory DiddiZ Jailed Once In May, For Griefing Pixel Art.
    • Maybe Some Way To Link Servers Criminal History!!!! :D
    • World Edit Support, Logging World Edit'd Areas: DiddiZ Replaced Block SAND with Block GLASS
    • Colors I Love COLORS
    • MENU BAR SUPPORT PLEASE!! :) (Helps Client Keep The Commands Organized And Ease Of Use)
    • If You Want More Ideas Just Ask But, I Feel Like Thats All I Got. :p
     
  7. WarHawk, your best tool in the war against griefers? :)
     
    tips48 likes this.
  8. Offline

    zcleaver

    Sounds great!
     
  9. Offline

    N3X15

    I can
    I'm not really a fan of AJAX (JQuery/JSON), since many people disable Javascript to reduce security risks, and they place a larger load on the server, not to mention all the weird incompatibilities between browsers regarding support for such systems. However, I can use them if required. I also have plenty of experience in CSS2/3, and write stuff that is XHTML4.1-compliant (still working on HTML5).

    I also have database design experience (BB was a disaster that got dumped onto my lap) and can help secure the web plugin from security vulnerabilities (SQL injection, XSS, etc).

    For the database engine, I propose using ADODB, which has many helpful utilities and database drivers. We use it on 7chan to allow our backend software to switch between MySQL, SQLite, and Postgres comfortably and standardize database access methods.

    Also, a name needs to symbolize the all-seeing eye this system will represent. Since Big Brother is taken, I suggest Eagle.
     
  10. Offline

    untergrundbiber

  11. Offline

    but2002

    jQuery is really good at keeping cross-browser compatibility and have had no problems with it. (Other than Webkit not liking simultaneous HTTP requests)
    In my honest Opinion AJAX, when used correctly can reduce server load because it can use far less bandwidth than sending a full web page back to the client, and JSON data is very easy to parse with jQuery. <3

    You're correct with people disabling Javascript, but a simple way to resolve this would to have Javascript create a cookie, and have the server check for that cookie. If the server sees the cookie send a Javascript compatible page. If it doesn't exist, then have the server set another cookie saying there's no JS, and then serving out an HTML only version.
     
  12. Offline

    Kaikz

    >BaldEagle
     
  13. Offline

    rcfreak0

    What about EagleEye? It is a name of a movie but it fits..
     
  14. Offline

    Vhab

    [​IMG]

    Comes to mind :)
     
  15. Offline

    DiddiZ

    Would you provide this data for stress testing porposes? :D
    Don't think so, as we are the actual devs of LB,BB and HE, so these three will get replaced by this.
    Year, that's rather similar to the archive table, I have at the LB todo list.
    The main problem is, that for really large tables it take ages to regenerate the indexes, so it may be useful to have a smaller table containing only the last few days and a larger where the log is tranferred once per day.
    Hm, that's MCBans

    But I added some of your other ideas
     
  16. Offline

    oliverw92

    :)

    OH that tool thing is pretty cool. Although personally I much prefer the idea of users being able to create their own tools using a /tool bind command. At the end of the day everyone has their own preferences. It would be cool if we did what WorldEdit does with allowing you to bind to anything e.g. /tool bind rollback <parameters> or /tool bind lookup <parameters>.

    My HawkEye server has 90 million records and queries take less than 2 seconds. Server is 100% responsive through out. A rollback of 55,000 edits took just over 2 seconds with the server, again, being 100% responsive. Redo'ing it was a similar situation.

    Broadcasting a fire - job of another plugin
    Sign thing - job of another plugin
    History mode - job of another plugin
    WorldEdit support - something we want to do but not possible at the moment because WorldEdit has no event links.
    Colours - yeah of course :p
    Menu Bar - possibly, althouh we might do a spout GUI instead.

    I like WarHawk!

    Have you tried out the HawkEye interface? It works perfectly across all browsers now (if you need access drop me a pm) and uses JQuery to make the UI sleek and interactive. If the server had to reload for every search you did it would suck ass.

    How big is the ADODB library? We don't want to be packaging over-sized software with a pretty small interface. What are the advantages of ADODB over PDO?

    Eagle seems a bit to simplistic for me. EagleEye perhaps, not sure.

    The HTML only version would be pretty cool. It would be a LOT of work though. We would need two versions of the interface too - one for JSON, one for plain text

    Yeah I really like EagleEye personally

    We should probably keep all the feature stuff in this thread at the moment rather than start mixing it up with the github issues. If stuff needs adding to the op just tell me and I'll stick it up.

    @DiddiZ what do you think of EagleEye?
     
  17. Offline

    DiddiZ

    They can :D
    My system supports both now.
    It remembers me of unreal tournament :D
    but i'm not sure
     
  18. Offline

    oliverw92

    WatchDog
    GuardDog
    BullDog
     
  19. Offline

    DiddiZ

    The Guardian :D
     
  20. Offline

    oliverw92

  21. Offline

    JukkaTibbs

    How close is this to being done? I'm soo excited for it!
    I just got griefed during an open house. and a guest build a statue of himself. That could've been me on my main world.
    But without a quick backup on the folder and level dat I have no way of replacing that statue, I think I just lost an expirenced member. :\
    But I cant get bukkit to load the Get Plugins tab at the main page. also :\
    so I cant get to download LB or HE or somthing at least. This is a GREAT idea guys, work hard and get it done.
     
  22. Offline

    oliverw92

    No-where near to being done. Coding won't start for possibly another week.

    Was just thinking we are going to need a really good import utility. Was thinking it would be cool if we made a Java GUI import utility. Small window with boxes to put your mysql details, then a drop down to select what plugin you are importing. The drop down determines what table names you need to supply.

    Anyone good at GUIs?

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 19, 2016
  23. Offline

    DiddiZ

    Year, would be difficult to write all imports in SQL, especially LB wouldn't work without user modifications.
     
  24. Offline

    oliverw92

    Yeah i found that when trying to make an import script from LB to HawkEye :p I could only do like 5 things
     
  25. Offline

    Sevenos

    My needs are probably quite special, but I will put them anyway.

    A document oriented database like MongoDB would fit perfectly for this logging purposes, no need for weird sub-table structures. The problem is it doesn't work for people who try to host a server with only FTP access. But maybe you can abstract things enough so it's reasonably easy to replace the mysql with a MongoDB (or whatever database one with) backend. This should be quite simple if you work with real objects, not directly with table (objects).

    Example:
    QueryObject: Holds the query parameters, like player name, block location+radius etc (NO SQL at all in that)
    LogEntry: ORM for log entries, getting an array of *LogEntry objects with LogEntry#find(...) method. LogEntry itself contains the basic stuff and gets extended with the special stuff (-> ChatLogEntry, ChestLogEntry, DeathLogEntry, ...). (LogEntry will need more methods for getting sums and such which directly returns HashMap and stuff ofc)

    Then, if you split the Plugin into 3 seperate packets it's really easy to replace parts of it:
    1. Data management: Manages the connection, database schema (if any) and provides the LogEntry (, ChatLogEntry, ...) classes for handling the data
    2. Logging and Query API: Does the logging (bukkit listeners) and has a query-api which does things which are not database-specific so the data management plugin can be as small as possible
    3. Ingame frontend: Handle all the commands, rollbacks and such

    Well this is just a quite rough idea, but it would allow to use at least a part of the plugin if you have special needs. Examples:
    - Instead of the next totally different logging plugin, we will only have a new command interface which use the logging and data parts of this
    - Or if you just want to rollback stuff automatically matching some criteria (like inactive players) it could be a really simple plugin using query api + data management
    - Using postgres to it's full potential (different indices, stored procedures, ...) would just need to replace a small part of the system. The multi-DB support in BB was incredible bad as a comparison
    - Using nosql databases for full awesomeness is nearly as easy as using another SQL database
     
  26. Offline

    oliverw92

    YOur example of a query object and different log entries is exactly what I did in HawkEye. It worked pretty well actually.
     
  27. Offline

    DiddiZ

    I really like this idea, allows to concentrate on different dbms, but I think it's not important to differ between logging and commands, so it would be more like the core and then different database bridges.

    EDIT: That would allow to support SQLite without any drawbacks.
     
  28. Offline

    oliverw92

    @DiddiZ as i said before - let's not use Github issues just yet. There is no point having the same conversation on two separate sites.


    I dislike your idea of having the commands in a separate plugin. I think this over-complicates things for users.
     
  29. Offline

    N3X15

    Let's do the importing system once the coding's done, because then we can just include BB/LB/HE as a library, load LogBlawk as a lib as well, and then load up the database using the native functions.
     
  30. Offline

    DiddiZ

    Didn't saw that.
    It's just a kind of list ot keep some things in mind, I don't want to read trough the whole tread later to search some idea someone brought up. It's no discussen yet there.
    Year, users don't like that.
     
Thread Status:
Not open for further replies.

Share This Page