Library [Thunderbolt 2] - Lightning Fast Data Storage

Discussion in 'Resources' started by The Gaming Grunts, Jan 13, 2015.

?

Do You FInd This Useful?

  1. Pretty neat! Will definitely use this!

  2. It seems pretty cool. I don't know if I wil use it, though.

  3. Pshh... seriously?

Results are only viewable after voting.
Thread Status:
Not open for further replies.
  1. [​IMG]

    What is Thunderbolt?
    Want an easy and fast way to store your data? Well, look no further! Thunderbolt is a Java file library designed from the ground-up to be as fast and efficient as possible, in addition to using a JSON (JavaScript Object Notation) backend for file storage.

    Thunderbolt operates almost exactly the same as Bukkit's YamlConfiguration and has very similar methods, thus retaining the familiarity that most Bukkiteers are used to. The main differences, however, are significantly increased speed and that Thunderbolt can be used in any Java project, not just in Bukkit.

    Library Download: https://dl.project-x.me/files/Thunderbolt.jar
    Don't worry, that's a domain I own :)

    Documentation & Source Code
    Thunderbolt is completely open source and licensed under GNU GPL 3.0. All documentation can be found here on the project's GitHub page, including examples.

    Comments, Questions, and Concerns
    If you have any questions, comments, or concerns, I encourage you to let me know! :) If you have any suggestions feel free to comment below or open an Issue on the project's GitHub page, which I'd prefer. If you find any bugs, like with suggestions, feel free to comment below or open an Issue on the GitHub page if possible :)

    Also, feel free to contribute to the project! If you've got the know-how and have something that you want to add, create a pull request and submit your code!

    Updates

    • 4/13/15 4:10pm - Commit - File data is now stored directly in a JSONObject rather than in a Map and then being stored in a JSONObject
    • 1/15/2015 10:24pm - Commit 1, Commit 2 - General improvements thanks to @Totom3
    • 1/13/2015 9:28pm - Commit - Thunderbolt#load() now returns a ThunderFile object, thus speeding things up if you want to use the object immediately.
     
    Last edited: Apr 29, 2015
    ChipDev likes this.
  2. Offline

    ChipDev

    What is this different than dataAPI except for the awesome name? :p
    good job!
     
  3. @ChipDev
    Honestly, I've never looked into DataAPI really (Sorry @Skionz :( ) However, based on a quick peek it seems that, in a nutshell, Thunderbolt always stores contents in JSON, whereas DataAPI simply takes a map and writes it to file.
     
    ChipDev likes this.
  4. Offline

    Skionz

    No problem :p
    The real difference is that mine reads the file line by line with buffered streams and parses them using the ':' and saves them to a map. I don't bother with Lists and comments because it slows it down, but I'm not sure about you. The main difference is that I believe you use JSON, which I have used like once so I don't know much about about it.
     
    The Gaming Grunts likes this.
  5. @Skionz
    Correct. My goal was to take what Bukkit's YamlConfiguration already had to offer, but improve on it, which includes adding all of it's current methods, including lists. Comments are basically impossible with JSON, however there are some "tricks" that can be done, which are really not worth doing.
     
    Skionz likes this.
  6. Offline

    Skionz

    @The Gaming Grunts Comments are a mess with Bukkit's YamlConfiguration too. The issue I found is that you can easily implement comments, but you sacrifice speed in the process. Basically the issue is that it is simple to load the file's keys and values it into a Map ignoring the comments, but how do you save that Map back to the file without overwriting them? I tried a number of different solutions, but no dice. When implementing comments I got rid of my save method and modified the file every time I used my set method which was perfect, but about 10x slower. When making plugins the only solution I found is to make a classic README.txt file which explains everything.
     
  7. @Skionz
    It's a bit of a lose-lose situation, sadly.
     
  8. Offline

    xTrollxDudex

    Ehh "fast" got proof?
     
  9. Offline

    Totom3

    Looks cool but there are a few things that I have to point out:

    1.
    Snippet of the methods of FileManager (open)
    Code:
    ThunderFile tf = get(name);
    if (tf != null){
        // Do stuff....
    }else{
        System.out.println("[Thunderbolt 2] The file '" + name + ".json' isn't loaded and/or doesn't exist.");
    }

    This is ridiculous. The method may fail and the server admin will be notified instead of the plugin developper... wut? Just throw IllegalArgumentException if there is no such ThunderFile.

    2. You are catching IOExceptions and simply print the stack-trace to the console log. Once again, you are notifying the server admin and not the plugin dev that something went wrong. I suggest you let the IOExceptions flow. Let the plugin dev catch them and recover from them.

    3. Just stop printing info messages, it's most likely going to be useless. If it is important to let know the admins that the file has been created, let the plugin dev print the message.

    4. At the time I am making this reply, I don't see the point of keeping the loaded files in a Map, it could potentially cause memory leaks.

    5. In FileManager#load(String, String), use Map#entrySet instead of keySet if you need both elements.

    6. A suggestion for FileManager#create(String, String) : it could return the newly created ThunderFile.

    The following points discuss some methods in ThunderFile.

    7. Instead of making 500 duplicate methods for primitive types, how about the following : getNumber(String path) and getNumberList(String path). If not, keep in mind that casting might throw NullPointerException if the object is null. <-- Nope. Nevermind.

    8. Why are you removing the character " from the Strings in getStringList() ?

    9. Any reasons for interrupting the thread in save() ?

    Alright, now I'm done criticizing. :D Sounds like it could be useful, I don't really like YAML. I'm also interested in seeing how well it performs compared to SnakeYAML & the Bukkit API.
     
    Last edited: Jan 18, 2015
    Avygeil likes this.
  10. @Totom3
    #s 1, 2, 3, 5, 6 and 7 are nice suggestions and I'll definitely fix those.

    #4 - Keeping the object in a map will significantly reduce latency. I don't see how a memory leak could occur. Not saying it can't happen, just that I don't see how.

    #8 - Oops. That shouldn't be there. Must've forgotten to remove that from when I was debugging things.

    #9 - Now that I think about it, not really :p

    Thanks for the criticizm :) I don't always think of everything and sometimes there's silly little things I do that I miss :p
     
    Totom3 likes this.
  11. Offline

    Totom3

    @The Gaming Grunts The error would be on the side of the plugin dev. Imagine that he's using a ThunderFile. He saves the file at some point after he's done using it, but forgets to unload. The GC will not be able of getting rid of the file with all it's data since your Map is holding a reference to it. The chances of it happening are very small indeed, and you could probably ignore it, but it's important to remind the users that they HAVE to call the unload method when they're done using a ThunderFile.
     
  12. @Totom3
    Good point. Will add that in the documentation :)
     
  13. Offline

    RawCode

    I found exactly zero benchmarks and tests on provided github, probably you should provide comparison to atleast GSON before posting such "tool".

    I found no native or ever unsafe component in posted library, so it perfectly valid to say - it a lot slower then GSON that use unsafe and byte[] under single API layer.

    Lets check class by class:

    https://github.com/TheGamingGrunts/...projectx/thunderbolt2/models/ThunderFile.java

    l 36 - lack directory check, reread file IO documentation and oracle file documentation.
    l 39 - use damn Logger with "debug" or "warn" priority.
    overall - useless wrapper.

    For other classes, i have only one question, why wrappers and strings, native memory layout is byte[], there are no strings or longs exists when you read file from disk or socket, data layout is byte[].
    Same there is no longs or strings when you write file, you write byte[].

    Conversion to and from byte[] is costly procedure that involve memory allocations and garbage collection activity.

    Normally when you load data structure from file, you parse byte[] into effective memory structure that allows access to values you need.
    When you save file to disk, you convert effective data structure into human readable json.
    Loading JSON into memory and keeping it inside ineffective form is very very strange.

    Also, if you state "performance" or "speed" as main feature, you should load and store data inside binary files, JSON not for speed.
     
  14. Working on it. I'm away from my computer atm, so I can't post it right now. I already saw @xTrollxDudex 's post and I'll post it as soon as I can.

    As for everything else, feel free to make a PR with such features. I did it the way I know how, so I'm sorry if it's not the best way. If you (or anyone else) thinks they know how to improve the project, you're more than welcome to contribute.
     
    TigerHix likes this.
  15. Offline

    xTrollxDudex

    Agreed. While "lightning" fast is an overstatement for using a human readable file format, the performance of JSON is fine without wrapping it, as said, conversion between types cause GC pressure and is high level, making it much slower. I don't believe this should be performance oriented, since IO should not be performed at runtime anyways. IO competes for disk writing and file handles, access controllers, etc. It's not a good idea to implement "lightning" fast IO.
     
  16. Offline

    Yekllurt

    Hi Guys i tried the Thunderbolt2 Library and i got these results for puting data and getting data:

    Puting Data: (Puting a String)
    773352 Nano Seconds
    773 Micro Seconds
    0 Mili Seconds
    0 Seconds

    Getting Data: (Getting a String)
    703623 Nano Seconds
    703 Micro Seconds
    0 Mili Seconds
    0 Seconds
     
  17. Offline

    xTrollxDudex

    What is your benchmark code...?
     
    Avygeil likes this.
  18. Offline

    Yekllurt

    Hi xTrollxDudex this is how i made it:^
    I assume in the benchmark that u already created a new ThunderBolt Object and loaded it

    //Setting
    Code:
    public static void main(String[] args) {
          
            Thunderbolt t = new Thunderbolt();
          
            t.load("test", "C:/Users/Steven/Desktop");
          
            ThunderFile file = t.get("test");
          
          
          
            long start = System.nanoTime();
          
            System.out.println("Start " + start);
    
            file.set("Value", "SetValue");
            file.save();
          
            long end = System.nanoTime();
          
            System.out.println("End " + end);
          
          
            long r = end - start;
          
            long microSeconds = TimeUnit.NANOSECONDS.toMicros(r);
          
            long miliSeconds = TimeUnit.NANOSECONDS.toMillis(r);
          
            long seconds = TimeUnit.NANOSECONDS.toSeconds(r);
          
          
          
            System.out.println("Difference " + r + " Nano Seconds");
            System.out.println("Difference " + microSeconds + " Micro Seconds");
            System.out.println("Difference " + miliSeconds + " Mili Seconds");
            System.out.println("Difference " + seconds + " Seconds");
          
        }
    //Geting
    Code:
    public static void main(String[] args) {
          
            Thunderbolt t = new Thunderbolt();
          
            t.load("test", "C:/Users/Steven/Desktop");
          
            ThunderFile file = t.get("test");
          
          
          
            long start = System.nanoTime();
          
            System.out.println("Start " + start);
    
            file.get("Value");
            file.save();
          
            long end = System.nanoTime();
          
            System.out.println("End " + end);
          
          
            long r = end - start;
          
            long microSeconds = TimeUnit.NANOSECONDS.toMicros(r);
          
            long miliSeconds = TimeUnit.NANOSECONDS.toMillis(r);
          
            long seconds = TimeUnit.NANOSECONDS.toSeconds(r);
          
          
          
            System.out.println("Difference " + r + " Nano Seconds");
            System.out.println("Difference " + microSeconds + " Micro Seconds");
            System.out.println("Difference " + miliSeconds + " Mili Seconds");
            System.out.println("Difference " + seconds + " Seconds");
          
        }
     
  19. @Yekllurt
    You can also do

    Code:
    ThunderFile tf = t.load(String name, String path);
    since the load() method returns the file. Also, you should include the load time in your benchmark :p

    Sadly, I can't do much atm because I'm on a train for the next 7hrs.
     
  20. Offline

    Yekllurt

    Good idea im going to do that later. Then il make a more detailed benchmark.

    Load, Unloading and Deleting of Files. Link (Diagram)
    Save, Put and Get. Link (Diagram)

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Jun 13, 2016
  21. Offline

    Totom3

    @Yekllurt Your benchmarks are invalid. You are wasting time on printing the start time, and you benchmark 2 things at the time: set a value and save to a file. Also, the code is only ran once, which suggests JIT wouldn't have made yet it's optimizations. Finally, it's irrelevant to check how long it takes to set and get a value since it only consists in a call to HashMap#put() and HashMap#get() (except for getting Lists, which require O(n) operations in his case).
     
  22. Offline

    xTrollxDudex

    @RawCode
    Unsafe appears to be a tiny bit slower than using NIO:
    Code:
    package net.tridentsdk.server.bench;
    
    import com.google.common.collect.Lists;
    import com.google.common.collect.Maps;
    import net.tridentsdk.plugin.cmd.PlatformColor;
    import sun.misc.Unsafe;
    
    import java.io.*;
    import java.lang.reflect.Field;
    import java.nio.ByteBuffer;
    import java.nio.channels.FileChannel;
    import java.nio.charset.Charset;
    import java.text.DecimalFormat;
    import java.util.Arrays;
    import java.util.List;
    import java.util.Map;
    import java.util.concurrent.TimeUnit;
    
    // Run #main(String[]) using JVM properties:
    // -server -XX:CompileThreshold=2 -XX:+AggressiveOpts -XX:+UseFastAccessorMethods
    // Results: https://paste.ee/p/LFqor
    public class IoTest {
        private static final DecimalFormat df = new DecimalFormat("####0.000");
        private final File file = new File("file");
        private final int amount;
    
        private BinaryIo io;
        private UnsafeIo unsafeIo;
        private String str;
        private byte[] bytes;
    
        public IoTest(int amount) {
            this.amount = amount;
            System.out.println("Strings to be written: " + amount);
        }
    
        public static void main0(String[] args) {
            IoTest test = new IoTest(10000);
            test.setUp();
            File file = test.file;
            UnsafeIo.writeData(test.str).to(file);
            System.out.println(UnsafeIo.read(file));
            // BinaryIo.writeData(test.str).to(file);
            System.out.println(BinaryIo.read(file));
        }
    
        // REMEMBER: This does NOT measure the average IO for NIO/JIO
        // This ONLY should measure the amount of time to IO a specific
        // amount of data to a flatfile and compare between those values
        // You CANNOT use this to average the performance of NIO/JIO between
        // data written because it WILL change depending on the amount of
        // data chunks written
        public static void main(String[] args) {
            int initial = 1000;     // Initial amount of strings to measure
            double step = 50;          // Measurement iterations for each string IO
            double cycles = 5_000_000; // Amount of strings to IO test max
    
            System.out.println("===== BEGIN TEST =====");
    
            // Inform of string length
            String r = Long.toHexString(Double.doubleToLongBits(Math.random()));
            System.out.println("String length: " + r.length());
            System.out.println();
    
            // Measure IO of varied amount of strings
            for (int i = initial; i <= cycles; i += 1000) {
                Map<String, Long> sink = Maps.newLinkedHashMap();
                IoTest test = new IoTest(i);
                Timer timer = new Timer() {
                    @Override
                    public long mark() {
                        return System.currentTimeMillis();
                    }
                };
    
                // Warmup each IO operation a few times just to be sure
                for (int j = 0; j < 5; j++) {
                    test(test);
                }
    
                // Measure
                timer.start();
                for (int j = 0; j < step; j++) {
                    Map<String, Long> collected = test(test);
                    test.sink(collected, sink);
                }
    
                // Remove UNSAFE nodes created by memory copies
                for (int j = 0; j < 5; j++) {
                    System.gc();
                }
    
                // Control file handle access
                // wait for the channels to be finalized
                long millis = TimeUnit.MILLISECONDS.convert(timer.elapsed(), TimeUnit.NANOSECONDS);
                long waitMillis = TimeUnit.MILLISECONDS.convert(1, TimeUnit.SECONDS);
                long wait = waitMillis - millis;
                if (wait > 0) {
                    synchronized (test) {
                        try {
                            test.wait(waitMillis - millis);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                    }
                }
    
                // Print the average value for the amount of string IO
                for (Map.Entry<String, Long> entry : sink.entrySet()) {
                    System.out.println("Time of " + entry.getKey() + " for " + i + ": " +
                            df.format(entry.getValue() / step) + " ns/op");
                }
    
                // Print the percentage of completion
                System.out.println(df.format((i / cycles) * 100) + "% completed");
                System.out.println();
            }
    
            System.out.println("===== END TEST =====");
        }
    
        // Add the values of the from map to a sink
        public void sink(Map<String, Long> from, Map<String, Long> sink) {
            for (Map.Entry<String, Long> entry : from.entrySet()) {
                Long l = sink.get(entry.getKey());
                if (l == null) {
                    sink.put(entry.getKey(), entry.getValue());
                } else {
                    sink.put(entry.getKey(), entry.getValue() + l);
                }
            }
        }
    
        // Run the test cycles and measure them to a data map
        public static Map<String, Long> test(IoTest test) {
            test.reset();
    
            Map<String, Long> data = Maps.newLinkedHashMap();
            Timer timer = new Timer() {
                @Override
                public long mark() {
                    return System.nanoTime();
                }
            };
    
            long elapsed;
    
            timer.start();
            test.writeNio();
            elapsed = timer.elapsed();
            data.put("NIO Write time", elapsed);
    
            timer.start();
            test.readNio();
            elapsed = timer.elapsed();
            data.put("NIO Read  time", elapsed);
    
            test.reset();
    
            timer.start();
            test.writeUnsafe();
            elapsed = timer.elapsed();
            data.put("UIO Write time", elapsed);
    
            timer.start();
            test.readUnsafe();
            elapsed = timer.elapsed();
            data.put("UIO Read  time", elapsed);
    
            test.reset();
    
            timer.start();
            test.writeJio();
            elapsed = timer.elapsed();
            data.put("JIO Write time", elapsed);
    
            timer.start();
            test.readJio();
            elapsed = timer.elapsed();
            data.put("JIO Read  time", elapsed);
    
            return data;
        }
    
        // Reset the file and setup a new session for a new test group
        public void reset() {
            file.delete();
            setUp();
        }
    
        // Creates a new set of strings to write, also creates a new file if it
        // does not exist and sets the new IO unit and default bytes to write to
        // the file
        public void setUp() {
            List<String> strings = Lists.newArrayList();
            for (int i = 0; i < amount; i++) {
                String random = Long.toHexString(Double.doubleToLongBits(Math.random()));
                strings.add(random);
            }
    
            if (!file.exists())
                try {
                    file.createNewFile();
                } catch (IOException e) {
                    e.printStackTrace();
                }
    
            str = Arrays.toString(strings.toArray());
            io = BinaryIo.writeData(str);
            unsafeIo = UnsafeIo.writeData(str);
            bytes = str.getBytes(Charset.defaultCharset());
        }
    
        //////////////////////////////////////// TESTS ////////////////////////////////////////////////
    
        // NIO = Non-blocking IO, or New IO
        // JIO = Java IO, vanilla
        // UIO = Unsafe IO
    
        public void writeNio() {
            io.to(file);
        }
    
        public String readNio() {
            return BinaryIo.read(file);
        }
    
        public void writeUnsafe() {
            unsafeIo.to(file);
        }
    
        private String readUnsafe() {
            return UnsafeIo.read(file);
        }
    
        public void writeJio() {
            try {
                FileOutputStream stream = new FileOutputStream(file);
                stream.write(bytes);
                stream.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    
        private static final byte[] buffer = new byte[65536];
    
        public String readJio() {
            try {
                FileInputStream stream = new FileInputStream(file);
    
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                int len;
                while ((len = stream.read(buffer)) != -1) {
                    baos.write(bytes, 0, len);
                }
    
                stream.close();
                return new String(baos.toByteArray(), Charset.defaultCharset());
            } catch (IOException e) {
                e.printStackTrace();
            }
    
            return null;
        }
    
        //////////////////////////////////////////// END TESTS //////////////////////////////////////////////////
    
        // Stop watch using system TSC nanos
        private abstract static class Timer {
            private long start;
    
            private Timer() {
            }
    
            public abstract long mark();
    
            public void start() {
                start = mark();
            }
    
            public long elapsed() {
                return mark() - start;
            }
        }
    
        // NIO unit for writing bytes to the data file
        private static class BinaryIo {
            private static final Charset CHARSET = Charset.forName("UTF-8");
            private final ByteBuffer data;
    
            private BinaryIo(ByteBuffer data) {
                this.data = data;
            }
    
            public static BinaryIo writeData(String data) {
                return new BinaryIo(ByteBuffer.wrap(data.getBytes(CHARSET)));
            }
    
            // Usage: BinaryIo.writeData(string).to(file);
            public void to(File file) {
                try {
                    FileOutputStream stream = new FileOutputStream(file);
                    FileChannel channel = stream.getChannel();
    
                    channel.write(this.data);
                    stream.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
    
            private static final byte[] bytes = new byte[65536];
            private static final ByteBuffer buf = ByteBuffer.wrap(bytes);
    
            // Usage: String string = BinaryIo.read(file);
            public static String read(File file) {
                try {
                    FileInputStream stream = new FileInputStream(file);
                    FileChannel channel = stream.getChannel();
    
                    ByteArrayOutputStream baos = new ByteArrayOutputStream();
                    int len;
                    while ((len = channel.read(buf)) != -1) {
                        baos.write(bytes, 0, len);
                        buf.clear();
                    }
    
                    stream.close();
                    return new String(baos.toByteArray(), CHARSET);
                } catch (IOException e) {
                    e.printStackTrace();
                }
    
                // This is simply an empty string, it's not a Java library
                return PlatformColor.EMPTY;
            }
        }
    
        public static class UnsafeIo {
            private static final Charset CHARSET = Charset.forName("UTF-8");
            private static final byte[] bytes = new byte[65536];
    
            private final UnsafeBuffer data;
    
            private UnsafeIo(UnsafeBuffer data) {
                this.data = data;
            }
    
            public static UnsafeIo writeData(String data) {
                UnsafeBuffer unsafeBuffer = new UnsafeBuffer();
                unsafeBuffer.write(data.getBytes(CHARSET));
    
                return new UnsafeIo(unsafeBuffer);
            }
    
            // Usage: UnsafeIo.writeData(string).to(file);
            public void to(File file) {
                try {
                    FileOutputStream stream = new FileOutputStream(file);
                    stream.write(data.array());
                    stream.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
    
            // Usage: String string = UnsafeIo.read(file);
            public static String read(File file) {
                try {
                    FileInputStream stream = new FileInputStream(file);
    
                    ByteArrayOutputStream baos = new ByteArrayOutputStream();
                    UnsafeBuffer b = new UnsafeBuffer();
    
                    while (stream.read(bytes) != -1) {
                        b.write(bytes);
                        baos.write(b.array());
                    }
    
                    stream.close();
                    return new String(baos.toByteArray(), CHARSET);
                } catch (IOException e) {
                    e.printStackTrace();
                }
    
                // This is simply an empty string, it's not a Java library
                return PlatformColor.EMPTY;
            }
        }
    
        private static class UnsafeBuffer {
            private static final Unsafe unsafe = obtainUnsafe();
            private static final long arrayBase = unsafe.arrayBaseOffset(byte[].class);
    
            private static Unsafe obtainUnsafe() {
                try {
                    Field field = Unsafe.class.getDeclaredField("theUnsafe");
                    field.setAccessible(true);
                    return (Unsafe) field.get(null);
                } catch (Exception e) {
                    e.printStackTrace();
                }
                return null;
            }
    
            private int cursor = 0;
            private byte[] buffer = new byte[65536];
    
            // copyMemory (from, fromBeginningOffset, dest, destBeginningOffset, length)
            public void write(byte[] values) {
                int len = values.length;
    
                // Check the array length, if the buffer is not large enough,
                // grow it to the inserted value length plus the length header
                if (len > buffer.length)
                    buffer = new byte[len + 4];
    
                writeInt(len);
                unsafe.copyMemory(values, arrayBase, buffer, arrayBase + cursor, len);
    
                cursor += len;
            }
    
            public byte[] array() {
                reset();
    
                int len = readInt();
                byte[] values = new byte[len];
    
                unsafe.copyMemory(buffer, arrayBase + cursor, values, arrayBase, len);
                cursor += len;
    
                return values;
            }
    
            public void clear() {
                write(new byte[65536]);
                reset();
            }
    
            public void reset() {
                this.cursor = 0;
            }
    
            private void writeInt(int value) {
                unsafe.putInt(buffer, arrayBase + cursor, value);
                cursor += 4;
            }
    
            private int readInt() {
                int value = unsafe.getInt(buffer, arrayBase + cursor);
                cursor += 4;
    
                return value;
            }
        }
    }
     
  23. Offline

    Yekllurt

    Hi Totom3 i did not print anything out while i was runing it for the diagram benchmark, that was the old one u saw.
     
  24. Offline

    xTrollxDudex

    Nevertheless, you're benchmark is almost completely incorrect.

    Basically, you don't measure anything. You have to use nanoTime correctly in order for it to work even remotely correctly. Your unload and delete measures thread creation, scheduling, affinity assignments, monitor management, resource locking, bus trafficking, etc. nanoTime depends on the fact that performing x operation(s) will execute in order, and the frames are pushed onto the same stack before the ending time stamp is recorded.

    Unfortunately thread creation occurs on the same stack, but the performed action occurs on a different stack... So you don't measure anything but the thread creation.

    Your save, put, and get tests also don't do anything. Save is performed on a different thread, see the above explanation. Put and gets don't measure anything either. As said, you are making a (terrible) benchmark upon HashMap, and your results don't matter since its useless to compare the write of (non-concurrent)HashMap to any particular gain above another library.

    The closest you've gotten is the load test, however there are more problems with that. As mentioned, you've iterated once only. If it occurs before or during JIT optimizations, your results are skewed. If GC decides to run for no reason (or to clear the references created in the parsing section), your results will be skewed.

    Side note on ThunderFile/FileManager: Not only is the class itself non thread-safe and not suitable for use of multithreaded IO, but the thread design allows for unbounded thread creation. It should be thread safe and IO should be performed within a thread pool or malicious code/idiots will crash the server by OutOfMemoryError. Your synchronization policies are inconsistent, and the design choices are quite baffling, especially making an abstract FileManager and switching between PrintWriter and a buffered stream. Additionally, you have no createIfNull type method, so you will have to check yourself:

    Code:
    String file = // ...
    Thunderbolt b = new Thunderbolt();
    ThunderFile f = b.get(file);
    if (f == null)
        f = b.load(file, path);
    Of course, this assumes it is thread safe, you'd have more to worry about other than this if it is not. And it isn't thread safe.

    Edit: looks like I was ninja'd.

    @Totom3
    You can't get NPE from type casting null AFAIK.
    See this: http://ideone.com/dY9AtZ
     
    Last edited by a moderator: Jan 18, 2015
  25. Offline

    Totom3

    Oh god. My entire life was a lie... :eek:
     
  26. Offline

    AdamQpzm

    @Totom3 Although, when you think about it, it does make sense: You can set any (Object) type to null, so you can cast null to anything without error. The only problem is when you try to actually use the methods on that null value :p
     
  27. Offline

    teej107

    Cast a null value to a NullPointerException. That'll be ironic.

    @The Gaming Grunts What would be a reason to use this over the current YML library?
     
    AdamQpzm likes this.
  28. It's been a while, but I made a couple of small updates. Due to school, I don't have time at the moment to look through everything and make improvements. If you have suggestions, please submit a pull request or comment here and I'll try to get around to it eventually.

    Edit: Also, I'm working on better multithreading & thread safety at the moment, in addition to changing some other things. I'll post the updates shortly.
     
    Last edited: Apr 13, 2015
    TigerHix likes this.
  29. Offline

    CraftBang

    Hey, your download link doesn't work but you can easily find it at the gitHub page!
    And the example ussage on the gitHub:
    Code:
    Thunderbolt t = new ThunderboltManager();
    Is now
    Code:
    Thunderbolt t = new Thunderbolt();

    Thanks a lot I'll be using this:)

    EDIT:
    I may sound incredible stupid now but how the .... do I use this?
    Like, where do I put the Jar? I did import it as an external jar in my properties but if I put it in my plugins folder it won't see it as a plugin (Because it probably isn't... xD)

    EDIT2:
    I found a way to add it, but it's with some hacky code which isn't allowed for public plugins.. Mine is private though.

    A thing, maybe a function for a ThunderFile could be unload?
    Right now I'm using unload(<name>) but what if we could do <thethunderfile>.unload()
    Maybe a suggestion? Not sure if really usefull..

    And thanks a lot, it works like a charm!

    EDIT3:
    Weird error don't know why it happens?
    Code:
    [18:53:14 INFO]: [Thunderbolt 2] Created new file test.json at plugins\DCData\Filter
    [18:53:14 ERROR]: null
    org.bukkit.command.CommandException: Unhandled exception executing command 'dc'
    in plugin DrugCraft v1.0
            at org.bukkit.command.PluginCommand.execute(PluginCommand.java:46) ~[spi
    got2.jar:git-Spigot-1649]
            at org.bukkit.command.SimpleCommandMap.dispatch(SimpleCommandMap.java:18
    1) ~[spigot2.jar:git-Spigot-1649]
            at org.bukkit.craftbukkit.v1_7_R4.CraftServer.dispatchCommand(CraftServe
    r.java:767) ~[spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.PlayerConnection.handleCommand(PlayerCon
    nection.java:1043) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.PlayerConnection.a(PlayerConnection.java
    :880) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.PacketPlayInChat.a(PacketPlayInChat.java
    :28) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.PacketPlayInChat.handle(PacketPlayInChat
    .java:65) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.NetworkManager.a(NetworkManager.java:186
    ) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.ServerConnection.c(ServerConnection.java
    :81) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.MinecraftServer.v(MinecraftServer.java:7
    34) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.DedicatedServer.v(DedicatedServer.java:2
    89) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.MinecraftServer.u(MinecraftServer.java:5
    84) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.MinecraftServer.run(MinecraftServer.java
    :490) [spigot2.jar:git-Spigot-1649]
            at net.minecraft.server.v1_7_R4.ThreadServerApplication.run(SourceFile:6
    28) [spigot2.jar:git-Spigot-1649]
    Caused by: java.lang.NullPointerException
            at me.DrugCraft.CommandHandler.onCommand(CommandHandler.java:22) ~[?:?]
            at org.bukkit.command.PluginCommand.execute(PluginCommand.java:44) ~[spi
    got2.jar:git-Spigot-1649]
            ... 13 more
    
    CommandHandler.java:22
    Code:
    tf.set("Type", "Gold"); 
    Using this code
    Code:
    ThunderFile tf;
                    try {
                        tf = load("test", Main.dataLocation + File.separator + "Filter");
                        tf.set("Type", "Gold");
                        tf.set("Amount", "0");
                        tf.set("MaxAmount", "0");
                        tf.set("Level", 2);
                        tf.save();
                        Bukkit.broadcastMessage(tf.getInt("Level") + "");
                        unload("test");
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
    
    Weird thing is the file is created but it's empty.
    I don't know why setting this data causes a problem?
    This only happens when the server only started up. If I reload the server (/reload) everything works fine..
    So I added tf = get("test"); after
    Code:
    tf = load("test", Main.dataLocation + File.separator + "Filter");
    and it fixed the bug. Still weird though that if I reload I don't need tf = get("test");

    And yes I deleted the file everytime.
     
    Last edited: Apr 29, 2015
  30. @CraftBang
    1. Fixed the download link. I forgot to update it.

    2. All the errors you're experiencing will most likely be fixed by the latest version, which I forgot to upload (oops...).

    3. Thanks for the ThunderFile#unload() suggestion. I'll look into it.

    4. Full updated documented documentation can be found in the JavaDocs as well as on the project's GitHub page (though it may not always be up to date)
     
Thread Status:
Not open for further replies.

Share This Page