Efficient Chunk Storage

Discussion in 'Plugin Development' started by BlueMustache, May 5, 2014.

Thread Status:
Not open for further replies.
  1. Offline


    Hello Community,

    I am currently working on a project that has to do the following:

    - Save a CHUNK into a .mchunk in a folder, with name of the file being the X and the Z coordinates.
    - Load the CHUNK into a chunk in the reverse order of the saving to the file.

    I know how to name the file, and already have a save method I use in another one of my plugins.
    Here is example save and load code.


    1. public void loadData() {
    2. File file = new File("pingdata");
    3. try
    4. {
    5. String l;
    6. while ((l = br.readLine()) != null)
    7. {
    8. String l;
    9. String[] args = l.split("[,]", 2);
    10. if (args.length == 2) {
    11. String ip = args[0].replaceAll(" ", "");
    12. String p = args[1].replaceAll(" ", "");
    13. this.playerData.put(ip, p);
    14. br.close();
    15. }
    16. }
    17. }
    18. }
    19. catch (IOException e) {
    20. }
    21. }


    1. public void saveData() {
    2. File file = new File("pingdata");
    3. if (!this.playerData.isEmpty())
    4. try
    5. {
    6. for (String p : this.playerData.keySet())
    7. {
    8. bw.write(p + "," + (String)this.playerData.get(p));
    9. bw.newLine();
    10. }
    11. bw.flush();
    12. bw.close();
    13. getLogger().info("Data saved.");
    14. } catch (IOException e) {
    15. e.printStackTrace();
    16. }
    17. else;
    18. }

    Ignore the various oddities in the code, this was copied straight from my code without editing.

    Anyways, I know that I could iterate through all the blocks in the chunk.
    Save the block id to a hashmap at the location.
    Save the block data to a hashmap at the location.
    Save the block BlockState to a hashmap at the location.

    That's all fine and dandy, but it's in efficient.

    My thesis: Is their any way to efficiently store a chunk and read a chunk, in a plain text file.

    Can anyone give me some code/ pseudo code on how I might accomplish this.

    Bottom line: Saving/ Loading a chunk from/to a plain text file. (While maintaining block data and BlockState)

    Please respond.

    - Blue
  2. Offline


    Since you can probably expect that you are going to have a full, contiguous array of data over the entire chunk, you probably should ditch the HashMaps because those are just wasting memory holding locations, when the data's position along the file should be well enough to determine where it is. You can probably use a CSV file or something alike, where you have all the block IDs separated by commas, a semicolon, then all the block data separated by commas, a semicolon, and then finally all the block data separated by commas. I should note that BlockState also happens to contain IDs and data as well, so you should make sure to either skip those when storing block states, or just store only the BlockStates. Then when reading back from the file, you can use a CSV library or just split each line by commas.

    It would probably be much easier though to use a NBT file like WorldEdit/MCEdit does. You can read up on how WorldEdit loads schematics on their github page, and either use their system, or you can choose to extend it by adding NBT tags for the block states. Remember that block states are unnecessary for a lot of blocks and can be represented entirely by their data bytes, so you may want to skip it entirely and just have special storage for chest inventories, furnace inventories, etc instead.

    Either way, you need to convert a 3d array of blocks into a 1d array. This is easily accomplished by combining the X, Y, and Z coordinates into a single index by lining up all their bits. For a 128-block tall chunk, you need 7 bits, and for X and Z lengths of 16 you need 4 bits each. This comes out to a total of 15 bits for indexing your 1d array, which implies a length of 32768. It would work something like this:
    int index = (y * 256) + (z * 16) + x;
    This is basically how WorldEdit indexes values in the 1d array in schematics, which I would actually highly recommend over a CSV file now. Even if you don't use schematics you can still use a NBT data structure that would work very efficiently.
  3. Offline


    there are special streams for binary data with much better performance, check oracle for more info.

    i will suggest to use memory mapped file wrapped by bigmemory array, it will allow to write and read data from arbitrary location of file.
  4. Offline


    Hmm, I will consider all these. (I like them, I will try them all.)
    Any other ideas however?
  5. Offline


    Bump. Maybe CHUNK to NBT and NBT to JSON? And vice versa.
Thread Status:
Not open for further replies.

Share This Page