DB vrs Flatfile?

Discussion in 'Plugin Development' started by feverdream, Jan 7, 2011.

Thread Status:
Not open for further replies.
  1. Offline


    I'm seeing a lot of threads talking about DB's, etc for configuration. While DB's are good to a point, I also feel very striongly that flatfiles should be supported by all plugins/bukkit itself.

    The simple fact of the matter is that forcing people to use a database is wrong. It adds complication, makes it harder to use bukkit and thus limits its growth, and in the end can actually cost the person running the server more money due to the need to install or run additional software.

    Somebody, please tell me bukkit is going to be sane about this.
  2. Offline


    Data goes in a database. Doing otherwise requires lots of I/O operations which, especially on larger servers, is slow. Flatfiles can be used for config, but using flatfiles for large amounts of data is stupid.
  3. Offline


    This can in no way cost someone money. On Ubuntu/Debian all you need to do is sudo apt-get install mysql-server php5-mysql, on other Linux/BSD distro you can get a RPM package or compile it from source. On Mac OSX you can get a .dmg on Solaris 10 you can get a PKG and on Windows you can get a .msi. Point is, a free download can't cost you money. Period. Also, wizjany is right about saving large amounts of data in flatfiles; it makes no sense if a database is available. Go get MySQL here.
  4. Offline


    SQLite is a flat file. If by flat file you mean plain text, those are best left for configuration for a number of reasons also stated. SQL powered databased are also relational, so they support a lot of cool features that would have to be re-coded for plain text databases.
  5. Offline


    Yes, it can cost people money. Some people use servers that are monitored on a usage basic, so billing is much like a utility. You use more, you pay more.

    The user should not be forced to use it, period. Let them decide.

    Key Value Pairs should be enough for anybody for basic configuration like the server.properties style file; such files are usually small and only need t be singleton objects anyway.
  6. I know of noone (personally or from looking up servers) running a usage based server with minecraft, given that minecraft hogs CPU and RAM therefore usage plans would be too costly to run miencraft.

    On Topic: I advocate Databases, since organising flat files requires OS level handling on bukkits part which will slow it down compared to using a library such as SQLite
  7. Offline


    Give me perf numbers that proves a DB is better and takes up less RAM. Lets look at this logically.
  8. Offline


    What do you mean by basic usage (like cloud based if you use 1 GB of ram per hour blah blah blah) if anything once you start to get anywhere near a long usage server of 1 month+ or having a 20-30+ activeish players using flat file is raving mad as it costs far more resource to do it otherwise on MySQL and if you have heard it uses up 2 billion gigs of ram they are sprouting rubbish as the amount of times I have heard sql being mentioned as inefficient is mind boggling.

    I can search through 81,355 records in less that 0.01 seconds (Showing rows 0 - 81338 (81,355 total, Query took 0.0928 sec))
    I can also note it uses minimum amount of ram with about 16MB in usage with minecraft.

    Trying to do that in flatfile is crazy.

  9. Offline


    I agree that data is best stored in a databsase if there is a LOT of data, but no MC server I know of has to worry about 81,355 configuation values.

    In the end the amount of data being stored is too small to make using a database worth it, and it adds complication an RAM usage that otherwise could be spent on a MC server.
  10. Offline


    Thanks for writing that so I didn't have to lol.
  11. Offline


    On my current server now its at 16MB
  12. Offline


    Wait, what? 16 mb for your world data, right?

    Thats handled by minecraft itself, nobody has 16mb of hand crafted configuration values, right?
  13. Offline


    If you are truly worried about RAM usage you would not run a minecraft server as it takes up loads of RAM, also, if you are already paying for that much RAM why not waste an extra 2-20MB?
  14. Offline


    No my mysqls ram usage is 16MB
    I should mention my server size is 40 slots
  15. Offline


    Exactly, and I highly doubt your configuration data is equal to that much data.

    Code Bloat is a slippery road, and one best not traveled. Then again, I'm speaking as a guy who does embedded device development.. so I may be biased.
  16. Offline


    Its way more actually if you were to take a look at my LogBlock database with 1.2 Million Entries.
  17. Offline


    I use a database so i can grab stats for my site, much easier to grab it from a db then flatfiles.
  18. Offline


    Yeh I'm setting up something familiar also
  19. Offline


    This sort of thing is a rare corner case, and I can see why it would need an external DB. But one for simple configuration of a few Key/Value pairs? Not needed.
  20. Offline


    For simple config files (users.txt, groups.txt, severconfig...), I like flatfile too, but for big plugins like Logblock, I use a database.

    You can't handle so much data with a flatfile/txt-doc. My LogBlock DB has just 6mb ;)
  21. Offline


    around 60-70k records?
  22. Offline


    We run our 400+ members using flatfile, it works.. Of course mysql is faster, and switching to bukkit will finally mean we will have the chance to switch.
  23. Offline


    Okay, I've been advocating my side of this. Lets explore the real cost of doing flatfiles.
    First lettuce (sic) look at some basic assumptions:
    • We will do writes on a semi regular basis
    • Writes will be Atomic
    • Writes will be Consistent
    • Writes will be Isolated
    • Writes will be Durable
    These are the minimum requirements that any storage system must have to be sane. This is the foundation of any storage system.

    In order to do this one must lock the writer, which in turn must have exclusive access to the file. Then one must marshal and unmarshal all data to be persisted. Once the data to be persisted has been marshaled that (potentially) binary string must be slotted into the storage. This may require reading into memory large segments of the configuration file and writing them back out so that ordering is maintained or so that old values can be erased. The alternative (nullifying the old data) is not possible with human readable flatfiles.

    Once the data to be written to disk is in place it must be written and then flushed to disk to be safe. This is a blocking operation.

    If the frequency of writes is low and the existing config file is in memory constantly and using something like Ropes (heavyweight strings that are pieced together from smaller pieces, such as an insert-able list of some sort) we can guarantee everything without overloading the server.

    This will require and extensive CRUD system for the users that can handle bad data. It will also require a minimum of O(N/2) average time to write even 1 byte of data to the config file. This can get out of hand fast, as modern filesystems have very strong ideas about consistency of their own. Sync is not a trivial call to make.

    Overall this adds up to a lot of developer time and a lot of skill needed that some people just do not have. No minecraft server should require MySQL to be installed. In fact only the basic JVM should need be present. However, with SQLite available as a lightweight JAR (running native calls, no less, on windows and linux) and MySQL so abundant it is not difficult to see the benefits in using a system already in place designed and perfected for years for storing complex and numerous data.

    Not everyone will need it. However, it should be clear that save for the most user intimate or important data one should at least consider using an external database. I'm getting sick of saying it.

    Do you disagree on any particular point? Have you a fully ACID flatfile storage system? Have you profiled or measured it's speed? Where do you draw the line? What does a DB need to do or have to be acceptable to you? Shall I write a GUI using reflection for my key value store? Or perhaps a generic one so that all can share in it?

    How can we please you and stop you from using flatfiles?
  24. Offline



    I want the OPTION of using flatefiles for small servers where we don't have 400 users or an external myswl db is not an option (like 99.9% of the servers in the community!). Because lets face it, such HUGE servers are the MINORITY and as such they should not be the gold standard used when thinking about the COMMUNITY.

    How can you make me give up my desire to use flat files? Make it DEAD EASY and so simple that even a 10 year old can still set it up and configure it over a ssh connection, because right now its possible to do just that CAN using flatfiles and as most server hosts only give you SSH for your shell/etc we should CONTINUE to support the ACTIVE COMMUNITIES that run on exactly there sort of setups so they they dont have to LOSE THEIR IPs and CHANGE to MORE COSTLY HOSTS and FORCE their COMMUNITIES to DIE.
  25. Offline


    Shouting the loudest isn't going to get you your way. I don't like the fact that you opened yet another thread about this, it's like you're trying to distract attention away from all the posts that have already disagreed with you.

    You already know that not having MySQL is not a valid point. SQLite should be available to anyone who can already run Minecraft.
  26. Offline


    I did that to bold or otherwise highlight words, in response to a question I was asked. If I had really been "shouting" as you suggest, the entire thing would have been in caps.

    Yes I know its a javalib, etc.. but what bothers me is the administration aspect of it. there are no tools to do so over a ssh connection. If you cant set it up, or edit settings, its useless.
  27. Offline


    Even on a windows server there are installer packages to get php/mysql up in 5 minutes. If it's server 2008 it's also pretty dang secure from the get go. I'm all for DB'd plugins.
  28. Offline


    FWIW, my understanding is that one of the goals of Bukkit is to eventually move from a single thread to multi-thread. If that happens, flatfiles can become a huge bottleneck, especially if they are to be manipulated in game. I believe that MySQL should pretty much be required to avoid this, while still giving those would like to run a webserver easy access to data.

    As far as tools to manipulate a db over ssh, there are plenty of generic implementations. If you are looking for something Bukkit specific, I am sure they will come.

    I just really hope Bukkit's performance doesn't get nerfed just because some people want to keep legacy flatfiles.

    @skuggi - Nice to see someone else from bby on here.
  29. Offline


    I'm not a big MySQL fan and I actually prefer PostgreSQL, though I accept that wouldn't be practical here. I wouldn't be entirely opposed to MySQL but unless there was a significant performance penalty, I think SQLite is definitely the most practical option.

    I'm not an expert but I don't think there would be performance issues, even though SQLite only operates on a flat file. I gather from the Firefox/SQLite/fsync debacle that it can be configured to not synchronize on every transaction when things get busy.

    But hey, we're probably talking JDBC here, right? Why not let the user choose? I doubt we'll be writing anything so complex as to require DB-specific SQL.
  30. Offline


    SQLite is really fast. In fact on small databases it can be faster than MySQL. However, during a read the DB is locked globally. Since we should only have one DB connection open this isn't a bit deal. Just be sure to close your result sets as fast as possible (get them into memory and close it! No processing steps between.)

    That is an extremely good point. There might be a few rare instances that it is needed, but with that in place everyone who wants to use a DB is free to choose which one they prefer. Any subtle exceptions (like table creation) can be handled by nicely exposing which DB is in use [ backend.getSQLServerType() returning a string like "postgresql", "sqlite", "mysql").

    With something simple like that, and perhaps a few supported Databases, the pressure would be on the plugin writers to make sure that their things work everywhere. I think mostly the table and index creation will require specifics, though some things like LIMIT have small differences everywhere. Those would need to be accounted for too, ideally by creating something that can spit back the correct string for the DB where possible, but that is getting a little out of scope for this thread (and pushing the edges of where I feel confident to suggest anything without doing a lot of research first).
Thread Status:
Not open for further replies.

Share This Page