Can cactus Cause lag if not what else could be causing this

Hello all on my server I have a tower with four levels

Steep drop of death
Maze full of spiders (spawns about 8 max)
cacuts forest (lots of cactus they stop growing as they can only get two high before hitting the roof they don't drop anything tho)
Lava jump (lava is stationary no flowing lava)

Now I no lava can cause lag but the towers diameter is not that big 30 at max.
Now I have a pretty good server host (now) and with 6 people on it uses about 250-350 mbs of ram (I have two gigs)

but as soon as I go to this tower it shoots drains about half of my memory. any ideals on why?

 

Has someone died inside of it? EXP orbs can cause a massive amount of lag if there are a lot of them in one area. There's a fix around somewhere that will allow you to delete them from the map and make it so they don't drop them on death. Google around and you might find it.