Setting blocks from bytes obtained through socket conncection

Discussion in 'Plugin Development' started by RustyDoorknobs, Mar 4, 2013.

Thread Status:
Not open for further replies.
  1. Offline

    RustyDoorknobs

    I would like to be able to update a 15x15x15 region in a world from a 15x15x15 array of bytes that I will get through a socket connection. I will run the socket in a separate thread. My problem is: how can I update the blocks without corrupting the world by accessing the Bukkit API from a different thread? Should I set up some sort of mediator class that will store a list of socket responses which I can poll every tick from the main thread? Can I set up some sort of event system? I only need one extra thread for one socket connection, I believe. The plugin will connect to a central server and retrieve the 15x15x15 array of bytes which will contain the data for a 15x15x15 region of blocks.
     
  2. Offline

    Comphenix

    Unfortunately, you will at some point have to change the chunk data on the server thread as Minecraft is not thread safe.

    Changing thousands of block thorugh the standard Bukkit API is not fast (due to lighting updates and notifications), so you might consider accessing the underlying NMS classes directly and do this (benchmark). You can also split up the conversion process into smaller segments - i.e. convert for 10 ms, wait until next tick, and continue where you left off.

    You could also send the chunk update directly to the players (use the map chunk packet) while this processing is going on. Also, if you don't actually need the update to save, you could skip the first step entirely.
     
  3. RustyDoorknobs
    You can create bukkit tasks inside an async task or thread, the task will run in the main thread.

    Comphenix
    Is there a PR for that to be added to the Bukkit API ? It really should be, they want us to use the API and pulling this will enforce that.
    This and the packet sending API as well :p and all other stuff that people are using CB code for should be made as API and PR'd too.
     
  4. Offline

    desht

    Digi I haven't submitted a PR for this, since it's not really complete. What it needs is a way to control how lighting is done: never (i.e. plugin will statically light the whole region), immediate (heavy on CPU), or deferred (run lighting over the next several ticks, in a way that doesn't cause "server can't keep up!" messages). "never" & "immediate" are easy, "deferred" not so simple :) One day I'll get some time to work on it...

    Arguably the lighting optimisation check I've added (if the changed block's light emission or translucency is the same as the previous block, skip the lighting recalculation) should be in the NMS World class directly, but I'm not sure such a fundamental change would be accepted without considerable testing - it has the potential to go wrong in interesting ways...
     
  5. desht
    Hmm, how about making a queue-like class ? Create the class, add blocks to it then call a finishing class which sets all blocks then recalculates lightning for all of them and sets chunk update packets to clients if they exceed a certain block count for each chunk.

    It would be like a block.setType/Data/etc() for multiple blocks without recalculating on each block, but instead recalculating on groups of blocks by chunks. (or individual blocks if it's not worth sending a whole chunk for 3 blocks for example)
     
  6. Offline

    desht

    Digi yeah, I had a queue in mind too. There are potentials ptifalls, though - e.g. if you're doing deferred lighting updates, and the block changes again between the original update and when lighting is recalculated, then you need to be aware of that. The lighting recalculation needs to be done over multiple ticks, and the number of ticks needs to depend on both the number of blocks to relight, and the overall server speed - some dynamic calculation of the time used is needed so the recalculation task can yield execution until the next tick. It's doable, but certainly not trivial :)
     
  7. Offline

    RustyDoorknobs

    How does world edit do it? Also, I'm still having trouble understanding how I would actually update the blocks in the world using a byte array from a different thread. Should I just poll a list of received byte arrays in a different thread from the main thread? How can I get the byte array from the socket thread into the minecraft world? I am not very experienced in thread programming. Also, would anyone be able to give an offhand estimation on how long it would take to update those couple thousand of blocks not counting the time to download the data? Thanks to everyone for your replies
     
  8. Offline

    Comphenix

    How about something like this (download):
    Code:
    package com.comphenix.example;
    
    import java.util.Queue;
    import java.util.concurrent.ConcurrentLinkedQueue;
    
    import org.bukkit.Location;
    import org.bukkit.plugin.Plugin;
    import org.bukkit.scheduler.BukkitScheduler;
    
    /**
     * Represents a task for importing chunk data on the main thread.
     * 
     * @author Kristian
     */
    public class ImportChunksTask {
        /**
         * The chunk we are importing. 
         * 
         * @author Kristian
         */
        public static class PendingChunk {
            // Origin block
            private final Location origin;
            
            private final int widthX;
            private final int widthZ;
                    
            private final byte[] blocks;
    
            /**
             * Construct a new pending chunk operation. A chunk is a NxMxH cube of blocks.
             * <p>
             * Note that the height is implicitly deduced from the full length of the block array.
             * <p>
             * The origin block is defined as the first block in the block array
             * @param origin - absolute position of the origin block.
             * @param widthX - the width (N) in the x-axis.
             * @param widthZ - the width (M) in the z-axis.
             * @param blocks - array of block IDs.
             */
            public PendingChunk(Location origin, int widthX, int widthZ, byte[] blocks) {
                this.origin = origin;
                this.widthX = widthX;
                this.widthZ = widthZ;
                this.blocks = blocks;
            }
    
            public Location getOrigin() {
                return origin;
            }
    
            public int getWidthX() {
                return widthX;
            }
    
            public int getWidthZ() {
                return widthZ;
            }
    
            public byte[] getBlocks() {
                return blocks;
            }
        }
        
        private static class PendingOperation {
            private final Location lastLocation;
            private final int lastIndex;
            
            public PendingOperation(Location lastLocation, int lastIndex) {
                this.lastLocation = lastLocation;
                this.lastIndex = lastIndex;
            }
    
            public Location getLastLocation() {
                return lastLocation;
            }
    
            public int getLastIndex() {
                return lastIndex;
            }
        }
        
        /**
         * This is a queue of chunks that are waiting to be processed. 
         * <p>
         * It is unbounded, but you could use a different concurrent queue that has a limit or
         * better performance characteristics.
         */
        private final Queue<PendingChunk> pendingChunks = new ConcurrentLinkedQueue<PendingChunk>();
        
        /**
         * This is the chunk we are currently processing.
         */
        private PendingChunk lastChunk;
        
        /**
         * Saved state from the last tick.
         */
        private PendingOperation lastOperation;
        
        // Used to make mass block updates
        private MassBlockFactory updateFactory;
        
        // Maximum number of nanoseconds to consume per tick
        private long maximumTickConsumption;
        
        // Current task
        private int taskID = -1;
        
        public ImportChunksTask(MassBlockFactory updateFactory, long maximumTickConsumption) {
            this.updateFactory = updateFactory;
            this.maximumTickConsumption = maximumTickConsumption;
        }
        
        /**
         * Enqueue a chunk for import on the main thread.
         * @param chunk - the chunk to import.
         */
        public void queueImportChunk(PendingChunk chunk) {
            pendingChunks.add(chunk);
        }
        
        /**
         * Retrieve the maximum number of nanoseconds to consume per tick.
         * @return Maximum number of nanoseconds.
         */
        public long getMaximumTickConsumption() {
            return maximumTickConsumption;
        }
        
        /**
         * Start the import chunk task.
         * @param scheduler - the Bukkit scheduler.
         * @param plugin - the owner plugin.
         */
        public void start(BukkitScheduler scheduler, Plugin plugin) {
            if (taskID < 0) {
                throw new IllegalStateException("Task has already been started.");
            }
                
            // Start and save task ID
            taskID = scheduler.scheduleSyncDelayedTask(plugin, new Runnable() {
                @Override
                public void run() {
                    processTick();
                }
            });
            
            if (taskID < 0) {
                throw new IllegalStateException("Unable to start task.");
            }
        }
        
        /**
         * Stop the task, if it is running.
         */
        public void stop(BukkitScheduler scheduler) {
            if (taskID >= 0) {
                scheduler.cancelTask(taskID);
                taskID = -1;
            }
        }
    
        private PendingChunk getPendingChunk() {
            if (lastChunk != null)
                return lastChunk;
            
            PendingChunk chunk = pendingChunks.poll();
            
            // Reset state
            if (chunk != null) {
                lastOperation = null;
            }
            return chunk;
        }
        
        /**
         * Invoked when we process the current tick.
         */
        private void processTick() {
            long startTime = System.nanoTime();
            
            // Process chunks until we're done
            while ((lastChunk = getPendingChunk()) != null) {
                lastOperation = processChunk(lastChunk, lastOperation, startTime);
                
                // If the processor saved its state, it's a cue for us to exit
                if (lastOperation != null) {
                    return;
                }
            }
        }
        
        /**
         * Process a given chunk, using the previous state if provided.
         * @param chunk - the chunk to process/import.
         * @param operation - state from the previous operation, or NULL if this is the first tick.
         * @param startTime - time we started processing.
         * @return State to save until the next tick, or NULL if we completed it.
         */
        private PendingOperation processChunk(PendingChunk chunk, PendingOperation operation, long startTime) {
            byte[] blocks = chunk.getBlocks();
            
            // Absolute location of origin
            Location origin = chunk.getOrigin();
            int aX = origin.getBlockX();
            int aY = origin.getBlockY();
            int aZ = origin.getBlockZ();
            
            // Relative location from origin
            int dX = 0, dY = 0, dZ = 0;
    
            // Current index
            int index = operation != null ? operation.getLastIndex() : 0;
            
            // What will update all the blocks
            MassBlockUpdate updator = updateFactory.construct(origin.getWorld());
            
            // Load a previous location?
            if (operation != null) {
                dX = operation.getLastLocation().getBlockX() - aX;
                dY = operation.getLastLocation().getBlockY() - aY;
                dZ = operation.getLastLocation().getBlockZ() - aZ;
            }
            
            // Process as many blocks as possible
            for (; index < blocks.length; index++) {
                updator.setBlock(aX + dX, aY + dY, aZ + dZ, blocks[index]);
                
                // Increment location
                dX++;
                
                // Check bounds
                if (dX >= chunk.getWidthX()) {
                    dX = 0; dZ++;
                }
                if (dZ >= chunk.getWidthZ()) {
                    dZ = 0; dY++;
                }
                
                // Check timeout
                if (System.nanoTime() - startTime > maximumTickConsumption) {
                    // Save this for the next tick
                    operation = new PendingOperation(
                        new Location(origin.getWorld(), aX + dX, aY + dY, aZ + dZ),
                        index + 1);
                    break;
                }
            }
            
            // We are done
            if (index >= blocks.length) {
                operation = null;
            }
            
            // Lighting calculation has to be done here, since other blocks may be modified in the future
            updator.notifyClients();
            return operation;
        }
    }
    I made a couple of assumptions regarding how your data is stored, but this should be relatively easy to adapt to your specific needs.

    It makes use of the mass block updater by desht, along with a simple factory interface:
    Code:java
    1. /**
    2. * A factory for creating a mass block update for a given world
    3. *
    4. * @author Kristian
    5. */
    6. public interface MassBlockFactory {
    7. public MassBlockUpdate construct(World world);
    8. }


    You have to provide an implementation of this factory to the class, for instance one that constructs CraftMassBlockUpdate.

    EDIT: Fixed the example. It should now properly exit when the work is done.
     
    desht likes this.
  9. Offline

    RustyDoorknobs

    Thank you for the reply. I think I understand how to do everything now. My main problem was working between 2 threads. I believe that I can just schedule a delayedSync task. My understanding is that a delayedSync task will not corrupt the world. It will still run concurrently, right?

    Also, the size of the array will be fixed, so to reconstruct the "cube" I will simply follow a preset building pattern, i.e. in order of ascending x, ascending z, and ascending y; the code you have processes cubes of varying dimensions. I plan to compress the byte data before sending it through a socket by converting "chains" of bytes longer than three into three bytes of the form
    Code:
    [chain-byte][block-byte][chain-length]
    . Not sure if it's worth it, since I'm only sending about 4KB max.

    I'm going to go ahead and tell you how I plan on using this feature in production since you seem very wise, and you could tell me if this is plausible. The idea is that players will construct "houses" in predefined "pods" that are 16x16x16. They will then travel to a different server (using BungeeCord) and their house will be reconstructed there, and they will be placed inside. All of these other servers will have predefined 15x15x15 regions where houses & players can spawn. Once a player logs out of the server, his house is removed. I plan on storing the houses in a mysql database on a central server, from which all of the other servers will request the byte array that represents a player's house. I will have to make 2 types of requests on the servers: downloadPlayerHouse and uploadPlayerHouse, which will be called when the player logs in and logs out, respectively. downloadPlayerHouse will retrieve the byte array that represents the player's house, uploadPlayerHouse will upload a new byte array to the central server that represents the changes the player made. I would be likely be making one of these requests every few seconds at the slowest. Do you believe this will put excess pressure on the server? Is this practical? Thank you very much for what you've shown me already; this thread has been very helpful.
     
  10. Offline

    Comphenix

    It will run on the main thread, so it's safe to call methods from the Bukkit API, yes.

    I process the blocks in order of ascending x, z and y, but it can handle a cube of any size. That shouldn't be a problem though - just supply a widthX and widthZ of 16.

    Ah, a rudementary run-lenght encoding. It's not a bad idea, but you could compress the data even more if you use zlib like Minecraft. But it's not as effective on small data sizes - it might not be worth the effort.

    I doubt it - after all, it should only occur when a player is logging in or out. Especially if you subdivide the task as I've done in my example.

    But have you thought about how to save tile entities (chests, furnances)? Their content is not contained in a block ID or data.

    No problem. :)
     
  11. Offline

    RustyDoorknobs

    Ok, thanks a ton. I plan on restricting which blocks the players can use in their home - the suite of plugins I am working on is for a custom game type, so furnaces and crafting tables are not necessary/do not function the same.

    My final question is: If I open a socket connection with the bukkit scheduler in a delayedSync task, downloading the data will not backup/freeze the server, correct?
     
  12. Offline

    Comphenix

    No, it's the other way around.

    Anything in a delayed sync task will be executed on the main thread, and since socket connection and associated read will block until all the data has been read, you shouldn't download data inside a delayed sync task.

    Instead, schedule an asynchronous task, download the data, and THEN use delayed sync task to process the result. You can also use my ImportChunksTask and call queueImportChunk, as it is thread safe.
     
  13. Offline

    RustyDoorknobs

    Ok, so the delayed sync task finds the an appropriate time in the main thread and then updates the blocks, so that no two threads are accessing the blocks at one time. I will likely use that class you coded as a starting point. Thanks once again for all your help; it's amazing you took the time to code that.
     
    Comphenix likes this.
Thread Status:
Not open for further replies.

Share This Page