Persistence - what would you use it or need it for?

Discussion in 'Plugin Development' started by EvilSeph, Feb 14, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    EvilSeph

    The Bukkit Project is currently looking toward implementing a way to persist and/or handle data and we believe the design of such a thing is best done through the examining of use-cases. That being said, we'd like to hear from all the Developers in our community on what they need persistence for and what they plan to do with it.

    Once we have persistence and data handling implemented, we can implement groups and users, negating the need for any permissions systems outside of our own. We apologise for the delay, but we lost contact with the developer who tasked themselves with handling it.

    Now's the time to help us mold our persistence implementation to provide you with what you need, so let the discussion begin.
     
    Prgr and Canownueasy like this.
  2. Offline

    Cogito

    Some straightforward things I should be able to persist are
    • Groups and permissions. I want to be able to check if a user or group of users has a permission on the fly, and be able to update it (done in memory). When I restart the server those permissions etc should be the same (done by persistence). What happens if the server crashes? The permissions need to be regularly persisting, and this will apply to a lot if not all cases.
    • Player specific data. This can vary widely, but simple things like custom spawn positions, logs of activity/chat, friend lists...
    • Server data. Message of the day, transportation information (say by a waypoint system), lists of towns and their members...
    Player specific data should be linked against a player. I should be able to get the data for a player if they are online or off. It would be great to retrieve lists of players who all share a common attribute - for example, all players who belong to a certain town.
    Potentially, a lot of this functionality would be provided by plugins. That is, instead of asking persistence for all the players from a certain town, I ask TownPlugin for the players. TownPlugin would have all the players and which towns they belong to already in memory, and so wouldn't need to query the persistence engine for them. All that TownPlugin needs to do is load and store the lists of who belongs to what town when it changes.

    A plugin I am working on allows messages to be sent to players. I would like to be able to send messages to offline players, and persists those messages until they come online. We would need a way of knowing if a player exists, as well as if they are online for this to work.

    I have more ideas, but later perhaps.
     
  3. Offline

    Xon

    Being able to assosiate data with blocks or items would also be important.

    A classical case are the portals which link to other servers or worlds. Or items which give buffs/stats to thier owner.
     
  4. Offline

    Zenexer

    Essentials would ultimately benefit from something high-performance, possibly relational. SQL would probably be the best option performance-wise, but is certainly a heavyweight implementation. JSON documents with querying would be a solid alternative, such as mongodb. In fact, I would probably prefer mongodb to SQL, due to its similarity to YAML in implementation (thereby making the transition much easier). Supposedly it also has better performance than SQL.

    Scopes of persistence include objects attached to chunks, player sessions, player lifetimes (forever), worlds, servers, commands, and plugins. Inter-plugin persistence is a bad idea: plugins should opt to query a central data plugin instead, such as is the case with iConomy and Essentials.
     
  5. The thing that would make my day is to have a player.GroupIS(); and so on so that there is a nice way to find out if people should be able to use commands, something that would make me love bukkit to my hearts content is a player.canUseCommand(); :) other than that i think more experianced plugin developers need to say what they need i am just steping in the shallows but i will keep bringing ideas if i think of things.
     
  6. Offline

    Mixcoatl

    I intend to use this functionality for entity persistence. (Not org.bukkit entities, but conceptual entity classes.)

    I've been exploring entity persistence (using EJP, ORMLite, ActiveObjects, and even Hibernate) as a solution.

    I'm presently using EJP with PostgreSQL for this purpose. EJP is much cleaner than the others, but it lacks some very basic features. One-to-many and Many-to-many relationship mapping being two. ActiveObjects forces you to define your entities as interfaces, which, while cute, takes a lot of firepower away from the programmer (in that you can't easily provide your own implementation of the interface). ORMLite uses especially long names for the annotations, which makes it unsightly. Hibernate is extremely powerful but it may require an EE container (still reading) and uses complicated configuration files that would not be trivial for an unexperienced coder to setup.

    My guess is we should pick a tool and stick with it.
     
  7. Offline

    Cogito

    I guess the hard thing to do is to abstract what needs to be done in persistence. Determining if a player has permissions is the responsibility of permissions - not persistence. The method with which persistence persists data is not what this discussion is for. Hence there is no real need to mention SQL, YAML, or anything else here. We need use cases to try and help make those decisions!

    Being able to store information about different blocks is persistence.
    Being able to store messages for offline players is persistence.

    Finding out when a block was last placed is NOT persistence. A plugin would handle that - and that plugin may use persistence to store its information.

    Persistence should probably only ever give access to persisted data to the plugin that persisted the data in the first place. Distribution of that data would then be the plugins job.

    Some data might be distributed by core - for example a players last position or login time could potentially be made available through the Player class, which uses the persisted data to serve this information.

    Accessing the data outside the game is a different story though: should I be allowed to change the persisted data from a web interface? The answer is 'most of the time' here I think. This is not the key design feature here, but worth considering.

    Finally, we should think about ensuring that duplicate data is not stored. Unfortunately, for the most part this depends on plugins communicating effectively. It would be bad however if two different plugins were storing the x,y,z of different blocks, and then attaching extra data to that as well. They should be able to persist a 'block' with extra data - and this would store the block only once.
     
  8. Offline

    Mixcoatl

    I think it's possible to foresee what uses the Bukkit team might have for persistence in the near future, but there is no way (short of time travel) to predict what the community will do with it, even in the short term. In my humble opinion, the only safe assumption is that any class whose state can be serialized should also be able to be persisted. (I'm not advocating binary serialization, I'm just expressing the sort of contract I would expect from the persistence engine.)
    In my humble opinion, this is asking for trouble. There is no way the persistence mechanism could understand, preemptively, the extent and variety of schema needed by developers in the future. Requiring developers' schema to use certain tables or fields to enforce partitioning would be heavy-handed at best.
    Data should be available outside the game if exposed by the backing store. There's nothing one Bukkit could realistically do to prevent a system administrator from using data in their MySQL database, for example. The official position, in my humble opinion, should be: it's your data -- if you mess it up, it's your problem.
    You can only guarantee this for Bukkit-specific data. I don't see any reliable way you can prevent developers from doing whatever they please with their own data. Moreover, the safest way to do this is probably just to make certain that Bukkit database tables use proper indexes, foreign keys, and check constraints.
     
  9. Offline

    LRFLEW

    n00b time :p

    What is Persistence? What does it do really? I've never had to deal with it, so I'm kind of ignorant in this department. Nothing here seems to make any sense to me.
     
  10. Offline

    Mixcoatl

    Marshaling Java objects to storage and un-marshaling Java objects from storage.
    It stores Java objects, usually in a file or database, and provides access to those objects using (usually) common programming interface.
    If you've ever had to store player information or settings, for example, in a properties file or a text file you've implemented a very simple form of persistence.
     
    acuddlyheadcrab likes this.
  11. Offline

    NathanWolf

    Put simply, the ability to save your data.

    So, let's say you have an object that holds some player-specific data (a nickname, maybe) and you write a plugin to let players set their own nicknames, and see each other's nicknames.

    Well, you'd probably like those nicknames to survive a server reboot- so that's where persistence comes in.

    A persistence engine will take your objects and save them away for you, loading them back up (more or less) as you left them next time Bukkit reboots.

    You could just use a properties file, or YML, or some other flat-file source- but this really only good for basic plugin configuration info- something no more complex than a series of key/value pairs.

    While this fits the overly simplified "player"/"nickname" use case, technically, what happens if you want to store a home location for a player? What about a home world? How do you store all that in your flatfile, what happens when you want to add new fields, later, etc.

    A persistence engine takes care of all of this for you and, ideally, lets the admins also decide how to store your data, while giving them a universal level of control and access that we can take advantage of in some really cool ways.

    Something like this helps tremendously when writing a plugin that has complex object structures that you don't want to lose between reboots. And having a persistence engine built in to bukkit means we can all share the tools- the web ui for accessing and modifying data, the server/console commands, etc. It also means no more worrying (potentially) about who's using what sqlite.jar, or whether you need to support MySql to make your users happy, etc. It Just Works :)
     
  12. Offline

    LRFLEW

    Cool. I'm down for it.
     
  13. Offline

    Cogito

    Determining the contract to be used when persisting data is a good idea - hopefully this discussion can illuminate that. Serializable is probably a bad idea however, as a serializable object necessarily needs to serialize other objects which it references (and are not transitive). This can quickly become an issue, though perhaps if its dealt with intelligently it would be ok. My preference would be for restrictive methods that persist specific types of data (potentially against other persisted data). For example, you could store a string and link it to a persisted Player. The ability to link against a persisted data object is very similar to serializing the linked objects of a serializable object - but I think we should make the contract explicit as to what is allowed. This is a hard thing to examine, so excuse me if I have rambled.

    I merely mean that a plugin should not ask the persistence engine questions like "How much money does this player have?" - they should ask a banking plugin that question. The plugin would then consult its own memory/persistence if necessary. If you need to ask a complex question, say of multiple plugins, then get the data from those plugins and then ask the question yourself on the data you receive. There is no need to think about persistence as a database that can be queried, and in fact we need to make sure we separate how the data is stored (storage methods) to how it is accessed (persistence).

    I agree with this completely. Should a plugin be able to persist encrypted data so that the admin can't see it though? For example, should we be able to persist a password which has been encrypted? A plugin could handle that itself, or persistence could handle it - not sure where I lie on this issue.

    This very much is the issue I was discussing about serialization earlier. The default Java serialization stores one
    'instance' and then references that instance anywhere it is used. If we can provide similar behaviour that would be optimal.
     
  14. Offline

    Mixcoatl

    I was not advocating using Serializable, necessarily, but I was advocating a contract of some sort by which object relationships can be discovered and persisted.
    If create an class that holds a List of references to another class, it is the responsibility of the persistence engine, in my humble opinion, to follow those references and perform the appropriate inserts, updates, deletes, or other modifications as required. This should done as atomically as possible, or at least enclosed in a transaction -- for the backing stores that support it.
    Building the object graph, itself, is actually very simple. Mapping the objects in the graph onto a storage model of some kind is the complicated part. If any element of the graph is unmappable and does not specify the transient access modifier or an equivalent, the entire graph should be discarded and an error raised.
    I've thought about this: creating a Settings class of key-value pairs that can be stored one per player, per item, per mobile, per world, or per location. In the end I abandoned it because this would force developers (just myself in the case of my own musings) to roll their own persistence mechanism to support more complicated data structures.
    If developers are limited in what they can persist or what they can use as a key (a player's name for example) the persistence mechanism in Bukkit will only be used for small plug-ins that don't store much data.
    I see. That's not at all the sort of question persistence can resolve.
    It's not Bukkit's job to figure out whether your data should be encrypted or not. If a developer collects private data or data that should not be exposed, it falls to the developer to guarantee that it is not. The Bukkit team could not every hope to account for all the kinds of data a developer might hope to collect, store, collate, or what have you.
    This is actually very easy to do using reflection. I could provide an example of how to reflectively recurse the entire object graph and record a unique ID for each object, if the Bukkit team needed it.
     
  15. Offline

    croxis

    There is also the issue on attempting to support multiple but similar plugins. For example if I want to make a plugin that leverages basic economy (the player's bank account), I would have to support the interface for each base economy plugin I want to support. If, however, there was a money field attached to the core player storage, I can access that directly.
     
  16. Offline

    Cogito

    This was one simple question that, if persistence was done in what I consider the wrong way, could be asked of the persistence engine. I can not think of any examples of questions that would necessitate being asked of the persistence engine, except by the owner of the data (plugin, core etc). If you have some use cases, please show them!

    the persistence engine could potentially provide 'encrypt' methods that encrypt data as it is being stored. This need not be encryption per se, but just making the data not visible to server admins. For example, I want to store the bank account details of players - however I don't want the admin to be able to change those details directly. Can I use the persistence engine to do this? should I? this is a use case that illustrates that it might be useful - however it might not be worth it in the end.

    What if someone tries to persist the server? This could be by accident, but if I keep a reference to what server a player is on, and try to persist that I am going to run into problems. We need to ensure that objects being persisted are 'sane' and doing so is not necessarily easy.
    One option is to force anything being persisted to implement a Persistable interface, and ensure that every reference in that class or instance (that is not atomic or String) is also Persistable.
     
  17. Offline

    Mixcoatl

    The problem is ultimately that a money field is not appropriate for every plug-in that deals with economy. Several of the available economy plug-ins support multiple denominations of currency (ex. gold, silver, copper). Others, such as one that I have slated but haven't gotten to writing yet, uses in-game object (gold bars and blocks) exclusively. Others use a hard-coded currency type or denomination.
    The strength of the Bukkit model has been that individual server administrators can pick and choose the parts they want, and not just economic models, while omitting others entirely.
     
  18. Offline

    Cogito

    This is a key example of how persistence could go wrong. Plugin authors should be relying on other plugins to provide data like how much money a player has. The plugin I have developed has accounts for players (that store a BigInteger to keep track of money) and allows for transfers between accounts (that have to be verified by the debitor). Any plugin that wishes to use my banking system should ask my banking plugin how much money etc a player has.

    They should not ask the persistence engine. Ever.

    The persistence engine is for persisting data. How that data is used is the job of the plugins. Asking questions of the data is using that data, hence the job of the plugins.
     
  19. Offline

    Mixcoatl

    I would expect it to fail. And it should not fail because there's an "instanceof" check in there. It should fail because building and walking the object graph turns up classes that the persistence model can't figure out how to persist. Any object for which a persistence description is configured should be persistent.
    As I indicated before, if the persistence engine cannot determine how to persist an object and the object is not marked to be ignored (for example: using the transient access modifier) the entire object graph must be discarded.
    If the least of your problems is that someone accidentally tries to persist a server reference you'll have it easy. Consider that certain objects simply cannot be persisted: threads and sockets are good examples. In Java, these are not marked with the Serializable interface to force the serializer to fail when it encounters them. The persistence model should have something similar (for example: having no persistence configuration available for that class.)
    The algorithm I mentioned above is the sanity check. It guarantees that for every element in the graph, there is a description that provides the information necessary to map the object onto the backing store. To check anything more than this would require Bukkit have an inappropriate amount of knowledge about the internals of yet unwritten plug-ins' persistent information.
    Yes. Another would be to decorate persistent classes with annotations. Yet another would be to store persistence descriptions for each persistent class in a .yml file in your Bukkit directory, or perhaps include some portion of it in your plugin.yml file.
    Let me draw your attention to an interesting caveat: even with a Persistent interface, annotations, or persistence meta-information, a developer could still mark a class persistent that holds references to non-persistent information (such as threads or sockets or server references). Regardless of the mechanism used to mark classes for persistence, the algorithm I described must still be performed to guarantee the sanity of the object graph (as you put it).
    --- merged: Feb 16, 2011 2:20 AM ---
    I see this as an example of how plug-in development can go wrong. This is not a problem any persistence engine can solve. However, once a persistence engine is in place, I think it will be obvious to programmers using it which problems it is capable of solving. (You're 100% spot on, btw.)
     
  20. Offline

    ShoTro

    This concept has caught my attention! I haven't used persistence using serialized objects except in a few large team projects and I wasn't the architect, but I hope I can shed some light. I have only seen it used in J2EE applications, btw (@Mix). It has been suggested that a form of persistence with a API interface to an undefined data-source might be a possible way to go. I know this idea is throwing more complicated methodologies into the application, but would possibly fix a few issues that people are having in terms of picking what method they would like their data to persist.

    I admit I have had a long day and am a bit tired, so if this doesn't make total sense I will explain this in better detail if needed tomorrow morning, but if the plug-in developers provide models for the data they would like to persist and the server administrators can, at their discretion, utilize the databases or flat-file solutions they would like, via plugins that do the connections and type handling for the values coming from the persistence API then I think everyone could make this work without having to constantly throw the Bukkit team under the bus every time someone wants a new storage solution supported. I have seen it done, but it takes some very serious planning.
     
  21. Offline

    Mixcoatl

    Again, I am not advocating using serialized objects for long-term persistence. Java serialization, in the context it was mentioned, was an example of the sort of contract persistent objects should have with the persistence provider, whatever it may be.
    In my 10+ years doing java in the EE and other environments, the only thing I've used binary serialization for is sending objects over the network, where the footprint is irrelevant and binary serialization provides a quick shortcut. In nearly every other case binary serialization has proven to be a detriment, for one or more of the popular reasons.
    This is the very sort of thing we've been discussing. Correctly done, a plug-in developer would be able to access persistent storage for his or her long-term information without having to know anything at all about the underlying storage mechanism.
    Optimally, the developer would describe the data using annotations or configuration files and the Bukkit persistence layer would dynamically convert the description into the appropriate mappings. The developer should not even need to specify a JDBC connection string (although the Bukkit server would need one somewhere.)
     
  22. Offline

    Cogito

    Apart from the nuts and bolts of what the internals to a persistence engine looks like, what else should it have?

    Should the API it provides simply provide generic store and retrieve functions? How would I, for example, store data alongside a player - just by having a reference to a player in my persisted objects?

    If we can nut out some more concrete examples, in particular thinking about the API visible to plugins and to core, hopefully we can move forward with implementing the nuts and bolts.
     
  23. Offline

    Mixcoatl

    Here's the interface I implement in my storage API:
    Code:
    public interface Storage<T> {
      void close() throws StorageException;
      int count() throws StorageException;
      int count(T example) throws StorageException;
      int delete() throws StorageException;
      int delete(T example) throws StorageException;
      boolean insert(T object) throws StorageException;
      Collection<T> query() throws StorageException;
      Collection<T> query(T example) throws StorageException;
      T queryOne(T example) throws StorageException;
      boolean update(T object) throws StorageException;
    }
    A class that implements this interface is provided by a factory based upon a class token that's passed into it. I would guess that this is the minimum possible set of methods.
    --- merged: Feb 16, 2011 4:02 AM ---
    I suppose that I should add that holding a reference to a player guarantees nothing about how data is accessed for other persistent classes. I would guess that player-specific information would use the player's name as a key. Let me give you an example from my own code:
    Code:
    final Storage<RoleplayUser> storage = StorageFactory.instance().getStorage(RoleplayUser.class);
    
    // Get a RoleplayUser using player's name:
    RoleplayUser user = storage.queryOne(new RolelayUser().setPlayerName("Mixcoatl77"));
    
    // Get all the RoleplayUsers in  storage:
    Collection<RoleplayUser> users = storage.query();
    
    // Add a new RoleplayUser to storage:
    // Note: this automatically generates a primary key (id) and populates it back into the object.
    storage.insert(new RoleplayUser().setPlayerName("Mixcoatl77"));
     
  24. Offline

    Axle

    I've been fiddling with a book shelf plugin for a bit now. One of the issues that I'm dealing with involves how I would store the information. If I restart the server or disable/enable the plugin, I want to make sure that the books maintain their data, as well as appear in the correct inventories. While I'm certain I can create a clever file system, having a set of persistence functions would greatly improve my solution.

    (The intent with the plugin is to make all bookshelves act like a chest; as well as permit users to fill the books with written content)
     
  25. Offline

    Yogoda

    The question was what we need, so here is what I would use persistance for the most :

    - Storing block locations and extra data attached to a block
    - Storing area information and data attached to an area
    - Storing additional informations on players, mobs, items, ... any game object.
     
  26. Offline

    ShoTro

    Well, neither am I. I prefer simpler the better approaches to things. I might have just not been as clear as I would have liked.

    I don't have a huge amount of time to work on something right now (this week), but would love to assist someone in putting together some example code and work on this methodology. I think Mixcoatl has the right idea. Now, I am not familiar with Bukkit as it stands, I need to get better acquainted with the code first, I am working on my first plug-in now... struggling with some of the bad concepts people seem to be using to get things done. But yes. If there is a code-base and further discussions I would like to lend a hand.

    Again, optimally there should be no defined database or data objects in the persistence layer.
     
  27. Offline

    Afforess

    As I understand it, persistence is something that should be used when a variable from some object or class needs to carry over between sessions. Possibly, something like what the user's last command entered was, or who created this minecart. Having an API to do that for me is always nice.

    The big thing I noticed when coding is that I had to create wrappers for almost every entity in the game I deal with (players, minecarts, even chests) because there is no easy way to store additional data to them. For every one, I added a concurrentHashMap for volatile data storage, and it made my life much simpler.
     
  28. Offline

    Kane

    Time to use @NathanWolf and move on before Minecraft 2 comes around <3
     
  29. Offline

    NathanWolf

    Well, I appreciate the support, but that's not really the goal with this thread :)

    To be perfectly honest, I'd prefer that people in this thread didn't even know that the Persistence plugin existed- I'd rather they come in with a fresh outlook, their own ideas about how they might use object persistence, what an API might look like for that, etc.

    You can see already we have a wide range of backgrounds coming in here, and that's really great to see. Some people have never heard of a persistence engine, but they get the idea, certainly- it's a simple concept.

    Others come in here and basically asked for the API I've put together in Persistence, almost to a level of suspiciousness, in fact- I promise I don't have any alternate accounts!

    Seriously, though, this is not hugely surprising- and of course the kind of thing I love seeing. I've followed a pattern set by many great ORM libraries before me, and this is all a pretty well-trodden path, for the most part.

    I (and the Bukkit team devs) are very interested in hearing what you all need, in terms of data storage. This system is for us, not for them - at least, primarily. From what I understand, Bukkit actually has no need (at this time) for an object persistence engine, internally. So, they don't really need to put anything like this together- if they were to, it would be (like everything they do) basically as a "gift" for us- "here you go, devs! A save system!"

    However, before they just hand us something, they want to make sure that it fits the bill. That it can handle at least the majority of use cases we, the devs, can come up with- and that it isn't going to be the kind of thing that no one uses, favoring to go off and do their own thing.

    That being said, an object persistence engine with a normalized relational database as the backed is a Pretty Big Deal to implement. Given everything the devs have to work on that's actually Bukkit-related (this really has nothing to do with Bukkit, CB, or mcserver- Peristsence has no Bukkit code in it outside of the plugin wrapper.....), I'd say that it's very understandable that they haven't exactly had time to just throw something like this together for us.

    So, all that being said- please ignore me and Persistence, if you can, while you put your thoughts up in this thread. And, then, yes, if you're interested- please go check out my Persistence plugin after you've done that :D
     
  30. Offline

    Joshua Burt

    I really enjoy working in systems that provide object/relational query languages for their application to data layer mappings. Regardless of the mechanism used to mechanically store the data within the data layer (i.e. databases, flat files, registry, etc.), the application programmer would use SQL-like query structures that operate against/with object sets rather than tables directly (in most cases). The result sets would most likely be java/bukkit objects or other result set types.
     
Thread Status:
Not open for further replies.

Share This Page