Solved Memory Safety - Storing [click if you wanna know]

Hey, I'm currently developing a plugin which requires data stored for each individual player. For this I'm using GSON to convert the contents of a "json"-file to Java objects. All players' data are kept in an Array full of PlayerData (which has the methods necessary). Whenever a new instance of the object is created, I save the object in a HashMap, in order to be able to get it for later use. However, when the file starts to get big (it creates one for every player), the HashMap will also get bigger.

Let's say I have 3 users in my file, then 3 PlayerData objects will be created on startup of the plugin, then another player joins the server, another instance of the PlayerData object is created and the HashMap gets a size of 4, until the plugin decides to save, it will then take the hashmap and serialize it and save it to the file, then whenever the plugin restarts, it will load the four objects into the HashMap again.

I tried having a HashMap filled with these PlayerData objects on a size of 1 million (1,000,000)-- then shit started slowing down.

I know Bukkit loads your Player object when you join, and removes it from the server whenever you're not on. But then, I couldn't access it for offline players (which I must).

Would you say this method is good?


EDIT:
tldr: Method listed above is not good. @Tecno_Wizard suggested CacheLoader by Gueva, and this is yet the simplest and most elegant way of easily and efficiently caching and retrieving data without having to constantly read from files. It took me only a few questions, but I got it fully working now and it's amazingly simple.

 

All the time. I need the methods such as

getKills
getLives
getUuid
getName

and like 5 more methods.

 

Depends, if the player decides to check their lives every 10 seconds, then that'll be alot.

 

Hmm, that's true, but the command also allows for you to be able to check another player's lives, an offline player.

 

How do you suggest I clean it?

 

@DoggyCode™
The easiest way is googles guava cache.

Code:

Cache<UUID, PlayerObject> cache = CacheBuilder.newBuilder().expireAfterAccess(20, TimeUnit.SECONDS).build();

This will expire the objects 20 seconds after the last access.

You can use this to delete the key, when the player leaves:

Code:

cache.invalidate(<key>);

 

But if I want to delete it from the map, then why would you add it in the first place? How would it work if you want to access it multiple times?

EDIT. Ah, I understand, but what happens when it "expires"?

 

@DoggyCode™
It is removed from the cache. If no other reachable reference exists, it is garbage collected.

 

But what would be the reason to put it in Cache, when it for later use, won't be accessible?

 

@DoggyCode™
You want to cache offline players to not read them again and again. At the same time, having too many cached slows the server down. This cache prevents this by only storing them for a while after they were retrieved, making subsequent requests extremly fast.

You can also use "maximumSize(int)" to set the maximum amounts of elements to cache.

The cache is to not load from the file every time, while also not keeping all objects in memory.

 

But if the objects does not exist in the cache anymore, how would i be able to retrieve their data?

 

@DoggyCode™
Load it from the file and add it to the cache again.

 

@DoggyCode™ want to make it even easier? use a LoadingCache. You pass instructions into the cache with how to load a player if it isn't, which makes it so amazingly simple in use I can hardly believe it exists. Just call get, and Guava does the rest. You don't have to think twice about whether or not it is loaded.

This is how I do ALL of my player files now, online or not. I tell the cache how to load a player, to immediately discard after the cache exceeds 110% of online players, discard if the file hasn't been accessed in 10+ min, and I make my own event to discard when a player logs out and a discard listener to save any file changes.

From then on, all I need to do is LoadingCache#get(UUID), and literally nothing else. It's heavenly.

 

@Tecno_Wizard

But you are right, I should have linked him to the LoadingCache. Have a look here @DoggyCode™

 

I tried something like this:

PHP:

cache = CacheBuilder.newBuilder()
                .maximumSize(1000)
                .expireAfterWrite(10, TimeUnit.MINUTES)
                .removalListener(MY_LISTENER)
                .build(
                        new CacheLoader<UUID, PlayerData>() {
                            public PlayerData load(UUID key) throws Exception {
                                return new PlayerData(key);
                            }
                        });

You reckon it will work? And when do I create it?

 

@DoggyCode™ as long as that removal listener saves and you clear the cache before the plugin shuts down, you're done. Incredible, isn't it? lol.

 

That's yeh, incredibly amazing. But let's say I have maximumSize(1000), would that mean that when I load over 1000 PlayerData objects, it won't work anymore to load more? How will this be efficient in case of having to load 2000 objects if 2000 have been saved to a file.

And also, how would I go about creating the removal listener efficiently? Aand, finally, if I want to loop through all the values of the cache, to get for example a PlayerData through getName, would I just use the asMap function on the cache and it would work smoothly?

 

@DoggyCode™ It will work. It just discards the oldest entires as you request new ones. The max size is really going to depend on what you need. All the removal listener has to do is take the removed mapping's value, which seems to be a PlayerData object, and write its contents to its file.

If you need to loop through everything, it's probably safer to just use the Bukkit API and get that name's UUID.

My favorite part of all of this? It's thread safe! Call it from wherever you want.

 

But I'm using JSON, and using:

PHP:

try (FileWriter writer = new FileWriter(plugin.getUserDataJson().getFile())) {
  gson.toJson(userData, writer);
} catch (Exception e) {
  Bukkit.getLogger().log(Level.SEVERE,
  "nCore - SEVERE - Could not save data to \"user_data.json\". See stacktrace below and contact developer!: " +   e.getLocalizedMessage());
  e.printStackTrace();
}

Which basically takes all the objects from the cache, and writes them to the file on intervals, and when the plugin shuts down. However, this doesn't replace changed objects, it basically takes the entire file, meaning having maximumSize(1000), it would take everything and only save 1000 PlayerData objects to the file. I've tried to google "How to replace edited objects in json file", however, could only find how to write and replace the entire damn thing. And for the removal listener, I have no idea where to find an example of how to set one up (methods needed and stuff).

EDIT: I managed to setup my removal listener, and I see the onRemoval method, but i still dont know how to just save the edited object to the file, without deleting everything else whilst writing.

 

@DoggyCode™
Why are you wanting to write the cache to file? The idea was that the file would be the only complete registry of all the data, while the cache would only keep the most recent ones in memory, and if they aren't in memory, have guava load the from file into the cache.

 

But if I can't write the updated data to the file, how am I gonna save anything?

 

@DoggyCode™
The point was to save that data to the file directly.

 

Yeah. I tried this

PHP:

public class MyRemovalListener implements RemovalListener<UUID, PlayerData> {

    @Override
    public void onRemoval(RemovalNotification removalNotification) {
        PlayerData pd = (PlayerData) removalNotification.getValue();
        Gson gson = new GsonBuilder().setPrettyPrinting().create();
        String json = gson.toJson(pd);
        System.out.println(json);

        try{
            FileWriter fw = new FileWriter(NCore.USERDATA_FILE);
            fw.write(json);
            fw.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}


But when this saves, instead of saving like

Code:

{
  "6756c255-3c00-466c-b40b-d3e022ab9b72": {
    "name": "DoggyCode",
    "uuid": "6756c255-3c00-466c-b40b-d3e022ab9b72",
    "lives": 0,
    "kills": 0,
    "deaths": 0,
    "deathbanStamp": 0,
    "firstLoginStamp": 0,
    "lastLoginStamp": 0
  },
  "cf0794e8-cfb0-4e16-84ba-8e1fbd90a85d": {
    "name": "Exp",
    "uuid": "cf0794e8-cfb0-4e16-84ba-8e1fbd90a85d",
    "lives": 0,
    "kills": 0,
    "deaths": 0,
    "deathbanStamp": 0,
    "firstLoginStamp": 0,
    "lastLoginStamp": 0
  }
}

It saves like

Code:

{
  "uuid": "0e5ff250-4c32-45ac-a077-87f4d16382e0",
  "lives": 0,
  "kills": 0,
  "deaths": 0,
  "deathbanStamp": 0,
  "firstLoginStamp": 0,
  "lastLoginStamp": 0
}

And completely removes the older one. And disformats it comletely too.
@Tecno_Wizard

 

@DoggyCode™
Well, that's the nature of Java's FileOutputStreams, they completely rewrite the file (unless you set them to append mode). Just read the existing file contents, add the data to be saved, then write to file again.

 

Yup, done and good

EDIT:
@Tecno_Wizard Is there any method to have the onRemoval called without actually removing anything? In order to save stuff on intervals, I want it to save all cached objects to the file, but without really invalidating them or removing them from the Cache.

EDIT2:
Found it. Call refresh() on all keys in the CacheLoader map.