[TUTORIAL|ADVANCED] How to develop plugins using MCP for NMS via ASM remapping

Do you have a legitimate need to hook into native Minecraft code but find it difficult to navigate the maze of obfuscation? Frustrated by your carefully crafted code becoming a write-only mess of unreadable "a.a()"'s all alike with confusing non-descriptive field and method names?

This tutorial presents an alternative for writing plugins using NMS (net.minecraft.server in CraftBukkit and mc-dev), by coding against the comprehensive deobfuscations provided by the Minecraft Coder Pack. The technique came to me while reading a tutorial by Icyene, where comparing mc-dev and MCP side-by-side was described as helpful to understand obfuscated code. So I figured, why not make use of a couple clever symbol remapping tricks to allow you to write code directly using MCP's comprehensive symbol mappings? Now, you can.

WARNING: This tutorial is considered advanced as NMS is an advanced topic with notable drawbacks which should be considered. When possible, using the Bukkit API instead of NMS is recommended for maximum safety and compatibility. Plugins using NMS will have a higher maintenance burden and require continuous updates for each new major version of Minecraft (the technique described in this tutorial is not a "safeguard bypass"). But sometimes you have no other choice. Use at your own risk, no warranties, etc.


With that out of the way.. here is a trivial example of how your plugin source will look using MCP mappings:

Code:

package agaricus.plugins.SamplePluginMCP;

import net.minecraft.util.IntHashMap;
import net.minecraft.network.packet.Packet;
import org.bukkit.plugin.java.JavaPlugin;

public class SamplePluginMCP extends JavaPlugin {
    @Override
    public void onEnable() {
        IntHashMap map = Packet.packetIdToClassMap;

        for (int i = 0; i < 255; ++i) {
            if (map.lookup(i) != null) {
                System.out.println("Packet "+i+" = "+map.lookup(i));
            }
        }
    }
}

Notice the new class packages and complete absence of obfuscation. Here's the identical plugin written against NMS:

Code:

package agaricus.plugins.SamplePluginMCP;

import net.minecraft.server.v1_4_R1.IntHashMap;
import net.minecraft.server.v1_4_R1.Packet;
import org.bukkit.plugin.java.JavaPlugin;

public class SamplePluginMCP extends JavaPlugin {
    @Override
    public void onEnable() {
        IntHashMap map = Packet.l;

        for(int i = 0; i < 255; ++i) {
            if(map.get(i) != null) 
                System.out.println("Packet " + i + " = " + map.get(i));
             }
          }
    }
}

Without further ado, lets begin. To start you'll need the following:

Prerequisites
minecraft_server.jar: The "vanilla" Minecraft server jar straight from Mojang. Heavily obfuscated. You can officially download this file from minecraft.net, for this tutorial we'll be using: http://assets.minecraft.net/1_4_7/minecraft_server.jar

minecraft-server-1.4.7.jar: Bukkit's "internally-renamed" server jar, partially deobfuscated for CraftBukkit development. Not linking to this jar directly per forum rules but it can be found in Bukkit's maven repository, artifact name minecraft-server. Just build CraftBukkit yourself and you'll see where it downloads from, should not be too difficult to find. This jar is decompiled to create the mc-dev repository.

Minecraft Coder Pack 7.26a: A toolkit for decompiling and deobfuscating Minecraft. The complete MCP distribution can be downloaded from Ocean Labs and is extremely useful in its own right, but for our purposes, we only need the "mappings", which translate the obfuscated symbols (classes, fields, and methods) to human-readable names, and can be found in the "conf" directory. You have two choices here: use the original mappings included with MCP, which place all classes in a flat net.minecraft.src.* package (analogous to mc-dev net.minecraft.server), or the repackaged mappings from FML/MCP where the classes are logically arranged hierarchically. I'll be using the latter. In either case, save them to a directory named "mcp/conf".

SpecialSource: The Swiss army knife of symbol remapping, SpecialSource is a library and tool using the ASM Java bytecode manipulation framework for generating mappings, applying remappings, and so on. You can download the latest build from the Jenkins or build it from source, creating a file named "SpecialSource-1.3-SNAPSHOT-shaded.jar". For brevity I'll be referring to this file as SpecialSource.jar. It can be executed from the command-line to show help as follows: java -jar SpecialSource.jar.

Knowledge: You should know your way around Bukkit, CraftBukkit, mc-dev, NMS, OBC, Maven.

A Java decompiler (optional): Not strictly necessary, but useful to understand the remapping process. You can use JD-GUI online, or the Fernflower analytical compiler (now included with recent versions of MCP).

Step 1: Generating the Remapped Library
To get setup, we'll first create a remapped Minecraft server jar similar to Bukkit's, except with MCP mappings instead of mc-dev. The --in-jar and --out-jar flags specify the input jar (obfuscated) and output jar (remapped), respectively, and the --srg-in flag provides the mappings to use. Specifying MCP's configuration directory will read the packaged.srg, methods.csv, and fields.csv appropriately, and applying the renames to the vanilla jar then writing a new jar as output. Here's the complete command:

Code:

java -jar SpecialSource.jar --in-jar minecraft_server.jar --out-jar minecraft-server-pkgmcp-1.4.7.jar --srg-in mcp/conf/

You now have a new server jar identical to Mojang's, but with MCP's deobfuscation.

Step 2: Generating CB Mappings
The previous step applied obfuscated-to-MCP mappings, supplied by the Minecraft Coder Pack. Now you need to extract a missing piece: obfuscated-to-CraftBukkit (or more accurately, mc-dev) mappings. SpecialSource again to the rescue:

Code:

java -jar SpecialSource.jar --first-jar minecraft_server.jar --second-jar minecraft-server-1.4.7.jar --srg-out obf2cb.srg

--first-jar/-second-jar will compare the two jars and generate a mapping file. The jars are otherwise identical, differing only in symbol names - that's the generated "mapping" written to obf2cb.srg. If you're curious, open this file in a text editor and take a peek. Each class, field, and method will be listed on a line with the old (obfuscated, in minecraft_server.jar) and new (mc-dev/CB, in minecraft-server-1.4.7) names:

Code:

CL: a net/minecraft/server/CrashReport
FD: aaa/a net/minecraft/server/ChunkSection/yPos
MD: aa/a (Ljava/lang/String;)V net/minecraft/server/ICommandListener/sendMessage (Ljava/lang/String;)V

Note you only need to create this mapping file once per Minecraft version; you can save and reuse it for multiple plugins.

Step 3: Compiling Your Plugin, with the Remapped Library
This remapped server generated in step #2 can be run as usual, but the real value is in adding it as a library to your projects in order to develop against. You can add it in Eclipse, NetBeans, IntelliJ, or whatever IDE you use, but I prefer working with the Maven build system, and add it as an artifact as follows:

Code:

mvn install:install-file -Dfile=minecraft-server-pkgmcp-1.4.7.jar -DgroupId=org.bukkit -DartifactId=minecraft-server-pkgmcp -Dpackaging=jar -Dversion=1.4.7

Then in your plugin's pom.xml, add:

Code:

    <dependency>
      <groupId>org.bukkit</groupId>
      <artifactId>minecraft-server-pkgmcp</artifactId>
      <version>1.4.7</version>
      <type>jar</type>
      <scope>compile</scope>
    </dependency>

or configure your project to use the minecraft-server-pkgmcp-1.4.7.jar as a library however you usually do it.

Note you'll still want to keep the Bukkit API or CraftBukkit dependencies (if you need OBC); this new dependency is only an alternative to NMS.

At this point, you should be able to compile your plugin (or the SamplePluginMCP), with no errors, creating "SamplePluginMCP-1.0.jar". But its not ready yet..few more steps.


Step 4: Reobfuscate
The plugin you just compiled cannot run on any real CraftBukkit server out there, since it is built against MCP mappings which are not used in the final binaries. If you try to load it on your server, it'll just cause ClassNotFoundException errors. To fix this, first remap MCP-to-obfuscated:

Code:

java -jar SpecialSource.jar --in-jar SamplePluginMCP-1.0.jar --out-jar obf-SamplePluginMCP-1.0.jar --reverse --srg-in mcp/conf

The --reverse flag as its name implies reverses the mappings loaded from --srg-in. mcp/conf maps obfuscated-to-MCP, so reversing it maps back MCP-to-obfuscated. As an exercise, try decompiling obf-SamplePluginMCP-1.0.jar, and this is what you'll see:

Code:

public class SamplePluginMCP extends JavaPlugin {

   public void onEnable() {
      jz map = ef.l;

      for(int i = 0; i < 255; ++i) {
         if(map.a(i) != null) {
            System.out.println("Packet " + i + " = " + map.a(i));
         }
      }

   }
}

"jz", "ef", "a", etc. are obfuscated symbols, matching what Mojang ships. But these aren't used in the CraftBukkit, jar either..but we're almost done.

Step 5: Remap to CB
Finally, we'll remap obfuscated-to-CB:

Code:

java -jar SpecialSource.jar --in-jar obf-SamplePluginMCP-1.0.jar --out-jar SamplePluginMCP-1.0-cb.jar --srg-in obf2cb.srg --out-shade-relocation net.minecraft.server=net.minecraft.server.v1_4_R1,org.bouncycastle=net.minecraft.v1_4_R1.org.bouncycastle

--srg-in obf2cb.srg will load the mappings you created in step #2. However, although CraftBukkit is built with the mc-dev minecraft-server-1.4.7.jar, the build process performs its own remapping: specifically, the Maven shade plugin relocates NMS into versioned packages (it uses ASM, too). The complete list of relocations can be found in CraftBukkit's Project Object Model, pom.xml, but this one is the most relevant:

Code:

                <relocation>
                    <pattern>net.minecraft.server</pattern>
                    <shadedPattern>net.minecraft.server.v${minecraft_version}</shadedPattern>
                </relocation>

To apply the same relocation in SpecialSource, the "--out-shade-relocation net.minecraft.server=net.minecraft.server.v1_4_R1" flag can be used. This substitutes the package name on output when loading the mappings, so the correct symbol names are remapped. Decompile SamplePluginMCP-1.0-cb.jar to verify:

Code:

package agaricus.plugins.SamplePluginMCP;

import net.minecraft.server.v1_4_R1.IntHashMap;
import net.minecraft.server.v1_4_R1.Packet;
import org.bukkit.plugin.java.JavaPlugin;

public class SamplePluginMCP extends JavaPlugin {

   public void onEnable() {
      IntHashMap map = Packet.l;

      for(int i = 0; i < 255; ++i) {
         if(map.get(i) != null) {
            System.out.println("Packet " + i + " = " + map.get(i));
         }
      }

   }
}

This completes the process. Your plugin SamplePluginMCP-1.0-cb.jar is now ready to be loaded onto a vanilla CraftBukkit server.


Summary
You now know how to develop non-API Bukkit plugins using MCP mappings and remap to NMS. To recap:

• Build your plugin against MCP-remapped server jar
• Remap MCP-to-obfuscated
• Remap obfuscated-to-CB

The exact steps in this tutorial are only a first cut at a new and experimental development technique. I welcome any refinements. Someone more familiar with Maven or Ant could probably automate the entire process.


Analysis
So now that you know how to do this, should you?

Benefits

• Generally more readable code, with thorough deobfuscation (vs mc-dev partial deobfuscation)
• Code can be cross-referenced with net.minecraft javadocs from MCP (vs mc-dev no docs)
• Hierarchical net.minecraft is easier to browse to find relevant classes
• Mappings can be updated by submitting to MCPBot if you know a better name (vs mc-dev updated once per release)
• Source is more resilient to obfuscation changes, making it less version-dependent (though at least you still need to recompile)

Disadvantages

• Adds new requires dependencies
• New build process may be unfamiliar to other developers
• Symbol names in source code will differ from CraftBukkit and most other plugins code, may be confusing
• Experimental; there could be bugs (but if you find any, let me know)

 

Very, very nice.

I forsee myself updating my NMS-accessing plugins to use this most handy tool in the near future. Definitely makes updating a couple-thousand-lines-long plugin that uses NMS every other line a whole lot less painful to maintain. Thanks!

I do have a couple of questions as to how it works, though. The remapping seems to be a relatively straightforward CSV-based ASM remapper, I'm interested in how you generate the mappings in the first place. I imagine applying them is a good deal easier than actually generating them! I know MCP comes with its own remappings, but I think directly remapping plugins based on those would mess up plugins which use fields/methods/classes whose identifiers differ from those in MCP remappings (i.e. Bukkit-refactored ones). I see you've generated separate remappings for those, (and PCMP et al.), yet I'm intrigued as to how you did this. I assume you didn't do those by hand: that would be very painful.

As a more esoteric question: any specific reason you chose the .srg (MSOffice) file extension for the files? Frans Willem probably knows, but I'm asking you in case you do too.

Also, on an offtopic note, I feel better now knowing I'm not the only one who uses Lombok in their projects . As you mentioned with SpecialSource, it is an extra build-time dependency, but it is so worth it.

Regardless, excellent work!

 

Appreciate the kind words . Wasn't sure it would be worthwhile to develop this process (I was using SpecialSource for some other purposes, and trying to find more uses for it), but I'm glad I did. Maybe I or someone else will come up with more uses in the future.

SS has a built-in mechanism for generating mappings via jar comparison. Given two otherwise-identical jars, it will iterate the archived class files (in order, so the class names can be matched up), comparing each of the fields and methods. The input to this process (in step #2) is the vanilla server jar from Mojang and Bukkit's internally-renamed server jar. The Bukkit team generates their internally-renamed server jar by running a script to rename/deobfuscate selected symbols in the vanilla jar, so the jar compare essentially reverses this and produces the mappings as output.

Oh no, I do nothing by hand when I can avoid it . Basically, quite a few different mappings can be generated by "chaining" through other mappings. This is what steps #4 and #5 are effectively doing, but any mappings can be chained as long as their is a commonality to use as a reference point. Here, the obfuscated symbols are shared by both mappings; mathematically speaking: obf2cb + mcp2obf = mcp2cb

Its true, problems can arise from remapping across CB-refactored code. The obf2cb mappings are more accurately obfuscated-to-"mc-dev", not obfuscated-to-CraftBukkit. mc-dev as in the git repo and mvn artifact, of the server jar partly-deobfuscated by the Bukkit team, but with no other code changes. CraftBukkit can and does change method signatures, and when they do they won't be remapped. Field type changes are fine, since the class name + field name uniquely identifies each field to the remapper, but if a parameter is added or removed to a method, or if its return type changes (btw: did you know the JVM allows overloading methods on return types?), then for the purposes of remapping it will be considered a completely distinct symbol.

CB signature changes are usually annotated with // CraftBukkit comments in the source, but they can also be detected automatically in bytecode with reasonable accuracy using JavaBytecodeDiff. Here's the signature changes from comparing an older build: https://gist.github.com/agaricusb/4449702 (search for 'MD:SIG'). Not too many, but if you do want to refer to these methods in your code using deobfuscated names, the mappings can be added by hand (or compatibility bridge methods can be added to CB, which will then be remapped accordingly).

Yeah I ought to learn more about Lombok, the usages in SS are all md_5's doing. Seems like a neat library though, for sure.

Not 100% positive but I believe the .srg extension is a contraction of "Searge", the name of the founder of MCP (my other guess was "source RetroGuard", but I'm pretty sure its name after the founder). It was originally used to refer to his numeric symbol names, defined in .srg files shipped with MCP and intended to be version-independent. For example, field_73527 is Packet23VehicleSpawn.h, the vehicle type field, even if the obfuscation changes between versions. These numeric "srg" names are the first pass of remapping in MCP, followed by the descriptive "csv" names. The mapped "csv" names are intended to be human-readable, and can be submitted by anyone to MCPBot on their IRC channel. Not all symbols have csv names, but after a release many will be named by various contributors as they begin to use the symbols in their code and want something more descriptive than the numeric "srg" names. I submitted quite a few a while ago.

But to me, .srg is now a generic mapping file format. FransWillem pioneered this usage with SrgTools, a collection of various useful tools for extracting, applying, chaining, reversing, and checking abstract methods of the mappings. SrgTools is a neat tool, I updated it a while ago with a few small fixes, and was using it for some time, but a couple months ago switched to SpecialSource as it had a better design and more active development. There's some overlap between the two tools but SpecialSource can do much more well. It also supports a new "compact srg" .csrg mapping file format, similar to .srg but smaller, easier to parse, and with fewer redundancies.

In any case, that's SpecialSource, the binary remapper. It is much more involved but you can also remap Java source code, also check out Srg2source if you're interested.

Thanks!

 

Looked into this some more, it can be partially addressed by remapping the complete CraftBukkit jar instead of the Minecraft server jar (either vanilla from Mojang or mc-dev from Bukkit or otherwise). The CraftBukkit jar will include the signature changes not present in vanilla, although those symbols will remain unmapped, you would still be able to access them in your plugin as usual, just not through MCP's deobfuscated names. Remapping the CB jar would also prevent erroneously accessing removed/changed symbols present in vanilla, which would compile fine against the MC jar, but cause missing definition exceptions when used on a CB server. So I haven't tried remapping CB yet but it should be fairly simple, just changing the path to point to a CB build instead of MC, if anyone wants to try it out and let me know how it works.

I went ahead and developed a new Maven plugin, which allows fully automating these steps. Leaving the original post as-is because it can be instructive to see the details of the inner workings of the process, but if you want to actually use it in practice, there's now an easier way - check out my sample plugin at https://github.com/agaricusb/SamplePluginMCP (specifically the pom.xml). With this configuration, you can simply type "mvn package":

Code:

SamplePluginMCP $ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building SamplePluginMCP 2.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- specialsource-maven-plugin:1.0:install-remapped-file (install-remapped-minecraft-server) @ SamplePluginMCP ---
Using cached remapped artifact /Users/admin/.m2/repository/org/bukkit/minecraft-server-pkgmcp/1.4.7/minecraft-server-pkgmcp-1.4.7.jar
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ SamplePluginMCP ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/admin/minecraft/1.4.x/SamplePluginMCP/target/classes
[INFO]
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ SamplePluginMCP ---
[INFO] Building jar: /Users/admin/minecraft/1.4.x/SamplePluginMCP/target/SamplePluginMCP-2.0.jar
[INFO]
[INFO] --- specialsource-maven-plugin:1.0:remap (semireobfuscate) @ SamplePluginMCP ---
Using cached file /var/folders/hx/qt_w7z395m554lsys821xp0h0000gn/T/SpecialSource.cache/https___raw.github.com_agaricusb_MinecraftRemapping_master_1.4.7_pkgmcp2vcb.srg for https://raw.github.com/agaricusb/MinecraftRemapping/master/1.4.7/pkgmcp2vcb.srg
Adding inheritance /Users/admin/.m2/repository/org/bukkit/minecraft-server-pkgmcp/1.4.7/minecraft-server-pkgmcp-1.4.7.jar
[INFO] Replacing original artifact with remapped artifact.
[INFO] Replacing /Users/admin/minecraft/1.4.x/SamplePluginMCP/target/SamplePluginMCP-2.0.jar with /Users/admin/minecraft/1.4.x/SamplePluginMCP/target/SamplePluginMCP-2.0-remapped.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.928s
[INFO] Finished at: Sat Mar 09 22:04:25 PST 2013
[INFO] Final Memory: 15M/209M
[INFO] ------------------------------------------------------------------------
SamplePluginMCP $

and the plugin's dependency is remapped to MCP, the plugin is compiled, and 'semireobfuscated' from MCP to CB mappings, ready to run on a vanilla CB server.

 

Would this still work?