[MISC] CraftProxy 0.2.0 - Reduce bandwidth use by caching chunk data [1.6]

CraftProxy - Reduce bandwidth use by caching chunk data
Version: 0.2.0

Description

This plugin and local client allows chunk data to be cached locally by players. This reduces the bandwidth required for hosting the server. It also helps users who are on slower connections.

I am not sure what the status of this system is. It doesn't count as a plugin, since it has 2 parts.

Even when running the plugin, players who don't use the client proxy can still connect. However, they will use the full bandwidth.

The system can reduce bandwidth by 70-90% (after the 2nd login).

Setup

Server

Add the plugin file to the plugins folder

Client

Start minecraft client and login
Double click on the client jar file
Enter login details
Enter the server location/port in the GUI
Press start on the GUI
Connect to localhost on the minecraft client

Stable Builds

None yet

Dev Builds

Warning: These may not be stable

Client
Plugin

Had it get the compress/decompression gain backwards.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

So I can remove craftproxy now to just use spout?

 

Both actually, it caches chunks and also allows auto-redirects.

I wasn't planning to add more features to this plugin anyway, but may have time to keep it from breaking.

Spout is also a more efficient solution. CraftProxy has to decompress and then recompress all packets and also parse the protocol stream. With Sprout packets can be processed before they are compressed by the server and after they are decompressed by the client.

Also, you don't have to explain to your users how to setup the local proxy server, they just connect with the Spout client.

Well, once the they actually release the recommended build.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Hmmm, one of the things I like with my current configuration is that I only need to expose the CraftProxy Liter port and all the connections are then routed locally. I might need to test to see if Spout can pass off the server connections using Transporter.

 

Actually, thinking about it, another benefit is that you don't have to re-authenticate every time you change servers. If you were connected to a server and then minecraft.net went down, you could still reconnect between servers.

 

Using CP 107 and CB 1046

Code:

# ====================================================================================
#     Proxy Parameter
# ====================================================================================
    CP_NAME="CraftProxy"
    CP="-Xmx256M -server -jar craftproxy.jar "
    CP_PORT="20000 25565"
    CP_OPTIONS="auth_off quiet staticlocalhost compression_level 9"

Ram Usage is too high, maybe a memory leak?
CP 105 used only 100mb Ram.

 

Not sure, does this happen with 1 person on the server (i.e. is it easy to reproduce)? Also, does it happen with CB 1045?

If you could track it down to a particular CB version change, it would make it much easier for me to find.

 

CP takes all memory it gets from -Xmx
It happens every day, they are 0-5 players online

I have to investigate the problem to give more detail - may take up to 2 days

And another issue:

If a player is connected by local network and other players by internet, the lan player is causing network troubles for the internet players - this means:

> higher latency for some internet players
> disconnects for some internet players

This happens actualy only with 1 player on my server.
Are network users prefered by craftproxy? ( i know its senseless, because data compression isn nessesary local )

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Is that new? All players are handled by separate threads, but I guess a local player would naturally have a better connection.

That was an issue way back with the fairness manager, but I bypassed that.

I could add a bandwidth limiter, so that no single player goes over say 50kbps.

 

This needs to be made into Spout so I can enforce my players to automatically get it....

 

It is.

 

It is? If they have to start minecraft with a craftproxy client, they cant start the spout client to connect to minecraft servers?

 

No, I have pretty much implemented the caching directly into spout. It will be included in the next RB of Spout.

You just run Spout plugin + Spout client and it should work. Press f3 in game to see cache info.

Anyway, I added bandwidth limiting to the proxy, add

bandwidth_limit 100

and the bandwidth will be limited to 100kbps per user for download.

I think 2*<upload speed>/<number of users> might be reasonable here.

 

New information about the memory leak:

Its solved by removing the "Xmx" tag for startup
It seems CraftProxy acts weird with this parameter

-- > I also removed the parameter "-server"​

​

Some additional info ( complete server restart, without mem leak ):

• after startup
  

• after 30 sec - first player joined
  

• after 1 min
  

• second player joined

• third player joined

• fourth player joined

• after ~ 6 hours - 5 player joined

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

When the memory drops again, was that you restarting the program? 14GB seems high, to say the least.

You can't eliminate a memory leak by changing parameters. Memory leaks are caused by data that isn't properly released. If I didn't release it, then command line changes can't suddenly make it be released.

Parameters can change how java handles garbage collection though.

Hmm, I could force gc to happen every 10 mins.

Xmx is supposed to set the max memory allowed. What had you it set to ? Maybe you had set it to a really high number.

I think what might be happening is that the garbage collector doesn't actually anything unless you are running out of memory. Since there was spare memory, it just didn't do anything. Did Craft Proxy crash with an out of memory exception?

 

@Raphfrk Would it be possible for you to make a client mod version of this that doesn't require a separate proxy app? I have a custom client for the players on my server and if I could just bundle a mod into it it would be a lot easier for people than if they have to start a separate app each time they want to play.

 

Its not crashing , the result is a high latency and a stressed cpu.
I figured out that the memory leak doesent depend on Xmx - perhaps CP is acting weird when available ram is very low ( < 100 mb ) and swap is deactivated.

my server has 8gb of ram - 4 gb for minecraft - 2 gb for a ramdisk - 2gb for system


picture of today: ( hope it helps )

 

Hmm, also, have you tried with just the -server parameter removed?

What is that supposed to do?

The problem could be something like increased disconnects are causing more threads to be created, and it snowballs.

 

-server parameter was removed

 

Gonna try this out when I'm not on vacation.

Is there any way for the server to discern whether or not the client connected is running CraftProxy? It would be nice to be able to provide my players with some sort of iConomy incentive to run the thing.

 

Btw, were you using Spout, there is a bug in Spout that keeps connections open?

 

No iam not using spout

 

Can you start the proxy with

-XX:+HeapDumpOnOutOfMemoryError

This should cause it to do a heap dump when it runs out of memory. Actually, does it actually throw an out of memory exception?

 

Ill try it, thanks!

 

@Raphfrk Was messing around earlier and found out about ModloaderMP for craftbukkit and wanted to try using it with the proxy. I didn't expect it to work, and it didn't and I don't expect support for this, but here's the log if you want to take a look at it. The modloaderMP server is running ModloaderMP and Planes mod for bukkit.

Code:

Connection from 192.168.5.4/64420
[9:35:24 PM] 192.168.5.4/64420 (AgentKid): Connecting to : fusioncraft.org 26565
[9:35:24 PM] 192.168.5.4/64420 (AgentKid): Attempting to connect to: fusioncraft.org:26565
[9:35:24 PM] 192.168.5.4/64420 (AgentKid): Connection successful
[9:35:24 PM] 192.168.5.4/64420 (AgentKid): Connecting using proxy to server connection format
[9:35:25 PM] 192.168.5.4/64420 (AgentKid): Server login successful
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Kicked with: [Redirect] please reconnect to: fusioncraft.org:26571
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Redirect detected: fusioncraft.org:26571
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Closed connection to server
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Connecting to : fusioncraft.org 26571
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Attempting to connect to: fusioncraft.org:26571
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Connection successful
[9:37:19 PM] 192.168.5.4/64420 (AgentKid): Connecting using proxy to server connection format
[9:37:20 PM] 192.168.5.4/64420 (AgentKid): Server login successful
 Unknown packet Id e6
 Unknown packet Id e6
16 1 0 34 67 0 0 28 0 32 4 0 0 67 0 0 29 0 59 30 0 0 67 0 0 2b 1 9 40 0 0 67 0 0 2c 0 4 3e 0 0 *e6* 11 41 11 f8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1f 0 0 0 9 0 1 0 1c 0 0 0 a 0 60 0 43 0 c1
[9:37:20 PM] 192.168.5.4/64420 (AgentKid): Downstream link
[9:37:20 PM] 192.168.5.4/64420 (AgentKid):  Unable to read packet
[9:37:20 PM] 192.168.5.4/64420 (AgentKid): Packets: [103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103]
[9:37:20 PM] 192.168.5.4/64420 (AgentKid): Closed connection to server
[9:37:20 PM] 192.168.5.4/64420 (AgentKid): Closed connection to client

 

Yeah, has an unknown packet. If you can get the packet format I can (possibly) add support.

CraftProxy supports the Spout custom packet system for example.

 

Mr. Raphfrk: hey it's me again! I'm still using your awesome plugin.

Could you explain how to use the new bandwidth_limit parameter? No matter what I set it to, I (or anybody else) can't play. It just loads a few chunks around me and that's it, it doesn't show my chats or anything else. In fact, it does this even without specifying the option... version 107 works just fine.

 

Hmmm, doesn't look like the Modloader MP system is open source and I'm not sure how to find it without the source :/ I'll keep searching, though.

 

Hmm, maybe I uploaded the wrong version. Does it work with just 1 player online (since that is how I tested it).

 

Ive got some news:

with "-XX:+HeapDumpOnOutOfMemoryError" same effect

should i reroll to v105?

 

It is supposed to generated a HeapDump file if java runs out of memory.

The file would be something.hprof and in the directory that you ran the server in.

However, it only happens if you have an out of memory exception.

Worth a try, it is like trying to find a needle in a haystack. If there was 1 build that had it and one that didn't I could focus on the differences.

Another option would be to try and find out which version of CB causes the issue.