[INACTIVE][ADMN/SEC/WEB] DataLog - replaced by HawkEye [1000]

16/08/2011 - DataLog has been massively rewritten and re-released under the new name of HawkEye. You can find HawkEye here:

​

You can upgrade straight from DataLog to HawkEye. See this wiki page: https://github.com/oliverw92/HawkEye/wiki/Importing-from-other-logging-plugins

 

Seems like something isn't threaded right, i'll have to look into it.

 

Latest version of datalog? Yes.

CraftBukkit 953
Minecraft server 1.7.2
DataLog 1.3.0

According to the mysqltune script, it's claiming that something accessing the database isn't cleanly shutting down it's connections.

Looking at the process list of the server, there are a lot of connections from datalog that have 'sleep' as the command and appear to be just 'sitting' there doing nothing until the server kills it due to inactivity (I've got a 10 minute limit set on mine).

One other thing I've just noticed... the cleanup utility.

Code:

12:18:31 [INFO] [DataLog] Deleted 0 row(s) from database
>reload
12:22:54 [INFO] [DataLog] Version 1.3.0 disabled!
12:22:54 [INFO] [Permissions] (Yeti) saving data...
12:22:54 [INFO] [Permissions] (Yeti) saved all data.
12:22:54 [INFO] [Permissions] (Yeti) disabled successfully.
12:22:54 [INFO] mcSimpleMail[0.89] is disabled
12:22:54 [INFO] [Administrate] - Disabled!
12:22:54 [INFO] Lockette: Closing all automatic doors.
12:22:54 [INFO] Disabled DefaultPermissions v0.3.1
12:22:54 [INFO] [HeroicDeath] 1.8.1.156 disabled.
12:22:54 [INFO] Goodbye world!
12:22:54 [INFO] [MobArena] disabled.
12:22:54 [INFO] mcSimpleMail: Recognizes the "Permissions"plugin
12:22:54 [INFO] [Permissions] (Yeti) was initialized.
12:22:54 [INFO] [DataLog] Starting DataLog initiation process...
12:22:54 [INFO] [DataLog] 1 rule(s) out of 1 loaded from config file
12:22:54 [INFO] [DataLog] Using Permissions for user permissions
12:22:54 [INFO] [DataLog] Started cleanse thread for logs older than 2011-07-05 12:22:54
12:22:54 [INFO] [DataLog] Running cleanse utility...
12:22:54 [INFO] [DataLog] Version 1.3.0 enabled!
12:22:54 [INFO] [Permissions] version [3.1.5] (Yeti)  loaded
12:22:54 [INFO] mcSimpleMail[0.89] is enabled
12:22:54 [INFO] [Administrate] - Permissions hooked using: Permissions v3.1.5
12:22:54 [INFO] [PorteCoulissante] plugin version 1.2.1 by Captain_Chaos enabled
12:22:54 [INFO] Backup v1.6 was sucessfully loaded!
12:22:54 [INFO] [LoginMessage] 0.6.61 enabled
12:22:54 [INFO] [LoginMessage] Attempting to fill in local fields...
12:22:54 [INFO] [LoginMessage] Unable to fill in local fields.
12:22:54 [INFO] Lockette version 1.3.8 is being enabled!  Yay!  (Core version 1.0)
12:22:54 [INFO] Lockette: Detected craftbukkit build [953] ok.
12:22:54 [INFO] Lockette: Linked to Permissions plugin version 3.1.5
12:22:54 [INFO] Lockette: Ready to protect your containers.
12:22:54 [INFO] Enabled DefaultPermissions v0.3.1
12:22:55 [INFO] [HeroicDeath] 1.8.1.156 enabled.
12:22:55 [INFO] Whitelist: Trying to load whitelist and settings...
12:22:55 [INFO] done.
12:22:55 [INFO] Whitelist version 2.7 is enabled!
12:22:55 [INFO] [MobArena] v0.92.3 enabled.
12:22:55 [INFO] Reload complete.
12:22:55 [INFO] [DataLog] Deleted 6029 row(s) from database
12:25:15 [INFO] [DataLog] Running cleanse utility...
12:25:16 [INFO] [DataLog] Deleted 0 row(s) from database

That seems to be a lot of rows removed for just a four minute span when the next run three minutes later doesn't find anything. It could be my config, but with this other DB thing, I'm not so sure, especially when one looks at the relevant config:

Code:

general:
    max-lines: 0
    tool-block: 17
    cleanse-age: 7d0h0s
    delete-data-on-rollback: false
    log-ip-addresses: true
    debug: false
    max-radius: 100

mysql:
    port: 3306
    username: $FOO
    hostname: $BAR
    password: $ALPHA
    database: $BETA
    max-connections: 10
    datalog-table: datalog
    player-table: dl_players
    world-table: dl_worlds

Max connections is 10. A line count on the number of SQL processess running for datalog on the sql server is 99.

Something 0ff on the DB connection handling side of the plugin perhaps?

EDIT: I also changed the debug entry to true (all lower case) and restarted the server. I'm not seeing anything different on the console. Is the debug output put into a file somewhere, or is it supposed to output more to the console?

 

I re-installed phpmyadmin. My VPS had some problems lately so i think the sql got 'hit'. Its all good now...no more errors. thanks

 

Thanks for the bug report, i'll look into it.

 

Hi, I am having the same problem when played around with the rollback. It is able to track blocks that are placed but it wouldn't rollback for some reason.

I am running on:
mysql 5.5
CraftBukkit 953
DataLog 1.3.0

 

This did nothing after a creeper exploded right next to me. I then issued the tool command and it told me it was active but left-clicking with the log did nothing. Then I reloaded plugins and the log worked, but I still get 0 from the creeper rollback attempt. I didn't see any errors in the server window but it seems something is quite wrong.

 

If the tool worked the second time but not the first then maybe DataLog hadn't started up correctly the first time before you reloaded. Try another explosion (just use tnt) and then try rolling that back.


I have fixed the rollback issues people have had with it not rolling the entire player back. I've also fixed the conflict between block filter and rules. Should have an update out today.

I put a message on your profile with a potential fix for the connection issue

If anyone wants the ultimate regular expression for swearing filter:

Code:

(\b(hell|a(ss|\$\$)|crap|(god?)?damn?)\b|(a(ss|\$\$)\s?(hole|holes|es)|bitch(y|ing|ier)?|clit(oris)?|(cock|cunt|dick)\s?(sucker|sucking|licker|lover)?|anus|fag(got)?|vagina|bar?star?d|penis|f(a|u|o|e)c?k(er|ing|ed|s)?|nig?g(er|a)(s)?|piss(ed|ing|off)?|pussys?|blowjob|jizz|shit(e|ier|tier|balls|ty)?|tits?|boobs|twat(waffle)?))

My moderators and I just spent a few minutes creating and testing that

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Cool, thanks. I put it in place and will see what I see. As for the PM, I think I fixed that in my profile. Sorry about that.

 

@oliverw92 I have this situation:
I have a greefer joined the server, he crushed several of houses and lava bucketed alot.

1. Why isn't there a way to rollback everything on his nick? Don't tell me that /dl rollbacik p:nickname will, cus it says zero blocks rollbacked.
2. So first I go to fix where he broke a glass and a lava leaked everything cool. Then I go there he lava bucketed on the ground and the type /dl rollback p:nickname r:100 and it says 800 block rollbacked but the leaking lava is still there, on retyping /dl rollback p:nickname r:100 it again says 800 blocks rollbacked and again nothing.

When I used BigBrother it worked to rollback entire nickname with 1 command. And I have never problems rollbacking anything. The reaseon that I was looking for an alternative was that I thought that BB was causing lagg becouse of too much info but still I am not sure about that.

I am forced to go back to BB now, except if you don't tell me I am a complete morron and I didn't read your guide how to work with the plugin.
The web browser of the data is best thought, really like and, and the fakt that it records the IP of the people that connect with XXX acc and when and so on, but this issues when rollbacking lack the plugin off.

Also I still couldn't fix the PermissionsEx issue where my Moderators, that don't have /give like ops/admin don't have the /dl tool command even if I added it to the permissions.

 

Ok, initial day. Server mostly idle.

Considerably less connections, but still more than I would expect to see. Also, the cleanup thread still doesn't appear to be actually purging entries unless the server is restarted.

 

1. As i said literally 3 posts up, rolling back an entire player is fixed in the next version.
2. DataLog won't rollback all the lava placed - it will only rollback the source block that was placed.

I have no idea about your issue with PermissionsEx. I use it and it works 100% fine with DataLog. My suggestion would be set up a test server with JUST datalog and PermissionsEx and see if you still have the issue.

How many connections?

I didn't do anything about the cleanse utility in that version i linked you.

Is that when you are shutting down the server?

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Hi Oliverw92.
I recently updated Datalog from a older verison and now the panel tells me Error! Unknown column 'player_id' in 'where clause' when I go to search. Is there a fix I can use on the database to correct this or do I have to reset it?

 

Make sure you updated both the server DataLog.jar and the entire web interface. It sounds like you haven't updated either part of it.

 

Both were updated. The panel before was blue and white, now it's black with more options. The older DataLog.jar retrieved the mysql details from the bukkit.yml file, now the mysql info is in DL's config.yml file with more options.

I went ahead and renamed the old mysql table for archival so the old panel can use it. And have since let DL create a new table which is working now. I'm guessing there was a change in the database format at some point preventing the new panel from grabbing info from the old db. I'm not sure which version of DL was running before.

Thanks for your assistance.

 

Run these MySQL commands if you want to import your old data from the 'datalog' table (make sure the old table is called datalog)

Code:

CREATE TABLE IF NOT EXISTS `dl_worlds` (`world_id` int(11) NOT NULL AUTO_INCREMENT, `world` varchar(255) NOT NULL, PRIMARY KEY (`world_id`), KEY `world` (`world`) ) ENGINE=MyISAM;

CREATE TABLE IF NOT EXISTS `dl_players` (`player_id` int(11) NOT NULL AUTO_INCREMENT, `player` varchar(255) NOT NULL, PRIMARY KEY (`player_id`), KEY `player` (`player`) ) ENGINE=MyISAM;

CREATE TABLE IF NOT EXISTS `datalog2` (`data_id` int(11) NOT NULL AUTO_INCREMENT, `date` varchar(255) NOT NULL, `player_id` int(11) NOT NULL, `action` int(11) NOT NULL, `world_id` varchar(255) NOT NULL, `x` double NOT NULL, `y` double NOT NULL, `z` double NOT NULL, `data` varchar(255) DEFAULT NULL, `plugin` varchar(255) DEFAULT 'DataLog', PRIMARY KEY (`data_id`), KEY `player_action_world` (`player_id`,`action`,`world_id`), KEY `x_y_z` (`x`,`y`,`z` )) ENGINE=MyISAM;

INSERT INTO `dl_players` (player) SELECT DISTINCT `player` FROM `datalog`;

INSERT INTO `dl_worlds` (world) SELECT DISTINCT `world` FROM `datalog`;

INSERT INTO `datalog2` (date, player_id, action, world_id, x, y, z, data, plugin) SELECT datalog.date, dl_players.player_id, datalog.action, dl_worlds.world_id, datalog.x, datalogy, datalog.z, datalog.data, datalog.plugin FROM `datalog`, `dl_players`, `dl_worlds.` WHERE dl_players.player = datalog.player AND dl_worlds.world = datalog.world);

DROP TABLE `datalog`;

RENAME TABLE `datalog2` TO `datalog`;

 

As of right now, I'm seeing 18 for the datalog user account via phpmyadmin. About 50% of them have 1000+ values in the time column and the other are around 80.

 

Interesting, thanks for that. Could you tell me the exact numbers of connections? If there are exactly 10 in the low numbers then that is telling me something important

 

Sure, happy to help. Direct cut and paste of the connections:

Code:

mysql> SHOW PROCESSLIST;
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
| Id    | User        | Host                     | db          | Command        | Time  | State              | Info             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
|  6801 | mc-log-user | $server:42293            | minecraft   | Sleep          |   177 |                    | NULL             |
|  6840 | mc-log-user | $server:49526            | minecraft   | Sleep          |   177 |                    | NULL             |
|  7115 | mc-log-user | $server:44707            | minecraft   | Sleep          |   177 |                    | NULL             |
|  7279 | mc-log-user | $server:48338            | minecraft   | Sleep          |   177 |                    | NULL             |
|  7434 | mc-log-user | $server:58019            | minecraft   | Sleep          |   177 |                    | NULL             |
|  7595 | mc-log-user | $server:36198            | minecraft   | Sleep          |   177 |                    | NULL             |
|  7757 | mc-log-user | $server:42018            | minecraft   | Sleep          |   177 |                    | NULL             |
|  7920 | mc-log-user | $server:36566            | minecraft   | Sleep          |   177 |                    | NULL             |
|  8084 | mc-log-user | $server:38074            | minecraft   | Sleep          |   177 |                    | NULL             |
|  8244 | mc-log-user | $server:47814            | minecraft   | Sleep          |   177 |                    | NULL             |
| 12006 | mc-log-user | $server:52399            | minecraft   | Sleep          | 15178 |                    | NULL             |
| 12047 | mc-log-user | $server:45567            | minecraft   | Sleep          | 14906 |                    | NULL             |
| 12089 | mc-log-user | $server:54994            | minecraft   | Sleep          | 14578 |                    | NULL             |
| 12127 | mc-log-user | $server:54995            | minecraft   | Sleep          | 14290 |                    | NULL             |
| 12170 | mc-log-user | $server:48062            | minecraft   | Sleep          | 13978 |                    | NULL             |
| 12211 | mc-log-user | $server:37524            | minecraft   | Sleep          | 13688 |                    | NULL             |
| 12254 | mc-log-user | $server:50408            | minecraft   | Sleep          | 13404 |                    | NULL             |
| 12295 | mc-log-user | $server:36278            | minecraft   | Sleep          | 13318 |                    | NULL             |

Let me know if there is anything else I can turn up, plugin wise or server wise. I own the servers (software and hardware), so I can do what I want with them.

 

Awesome, there are 10 exactly. Helps me track it down. Could you do these things for me:

• Try killing the processes with the stupidly long times. Then wait and see if they come back
• Change the max-connections in the config file and see if the amount of low-time connections increases to that number (might happen over time)

 

Long time pids killed. Current max connections for the server as a whole is 300 (it was originally at the default amount, which IIRC is 150). The user itself isn't limited at this time, but I could do that if it would help any and report back with the results...

 

No need to set connection limit - you shouldn't have to. If were to force you to do that then it isn't a good plugin!

 

Ok then, I'll leave it as it is and report back with the results in about 24 hours.

 

Cheers!

 

Ug, sorry for the delay, had to finish a project that took longer than expected (imagine that, eh?). Anyway, here are the current stats:

Code:

mysql> SHOW PROCESSLIST;

+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
| Id    | User        | Host                     | db          | Command        | Time  | State              | Info             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
|  6840 | mc-log-user | $server:49526            | minecraft   | Sleep          |   113 |                    | NULL             |
|  7279 | mc-log-user | $server:48338            | minecraft   | Sleep          |   113 |                    | NULL             |
|  7595 | mc-log-user | $server:36198            | minecraft   | Sleep          |   113 |                    | NULL             |
|  7920 | mc-log-user | $server:36566            | minecraft   | Sleep          |   113 |                    | NULL             |
|  8244 | mc-log-user | $server:47814            | minecraft   | Sleep          |   113 |                    | NULL             |
| 14273 | mc-log-user | $server:40722            | minecraft   | Sleep          |   241 |                    | NULL             |
| 14370 | mc-log-user | $server:58853            | minecraft   | Sleep          |   113 |                    | NULL             |
| 14456 | mc-log-user | $server:36130            | minecraft   | Sleep          |   241 |                    | NULL             |
| 14537 | mc-log-user | $server:44500            | minecraft   | Sleep          |   113 |                    | NULL             |
| 14625 | mc-log-user | $server:47440            | minecraft   | Sleep          |   241 |                    | NULL             |
| 14707 | mc-log-user | $server:59760            | minecraft   | Sleep          |   113 |                    | NULL             |
| 14794 | mc-log-user | $server:42120            | minecraft   | Sleep          |   241 |                    | NULL             |
| 14873 | mc-log-user | $server:49484            | minecraft   | Sleep          |   113 |                    | NULL             |
| 14955 | mc-log-user | $server:32843            | minecraft   | Sleep          |   241 |                    | NULL             |
| 15033 | mc-log-user | $server:40661            | minecraft   | Sleep          |   113 |                    | NULL             |
| 15116 | mc-log-user | $server:55461            | minecraft   | Sleep          |   241 |                    | NULL             |
| 15276 | mc-log-user | $server:41798            | minecraft   | Sleep          |   241 |                    | NULL             |
| 15440 | mc-log-user | $server:59779            | minecraft   | Sleep          |   241 |                    | NULL             |
| 15608 | mc-log-user | $server:33404            | minecraft   | Sleep          |   241 |                    | NULL             |
| 15764 | mc-log-user | $server:34108            | minecraft   | Sleep          |   241 |                    | NULL             |
| 28682 | mc-log-user | $server:42527            | minecraft   | Sleep          | 17367 |                    | NULL             |
| 28687 | mc-log-user | $server:37163            | minecraft   | Sleep          | 17043 |                    | NULL             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+

 

Hmmm. That is confusing. At least it is less than before! If you could let me know of the stats again tomorrow that would be great!

 

Sure, can do. Right now I had to restart the server as a whole for another issue, so I'll check it daily. I'm looking at the list now and can see that there are no connections open from datalog.

EDIT: Just noticed that it appears that it's not logging anything (well, I did see my joining the server) now with the build you linked to me. However, I'm willing to keep testing things and working with you to see if this can be figured out. Just let me know what else I can turn up or try.

Craftbukkit build 953
MC server 1.7.2
DataLog max connections is set at 15
SQL server max connections is set at 300

 

Yeah i just found that bug myself - when i fixed the block filter/rules order, i broke something. Here is another test version you could try - should fix the logging bug (it only affected block place/break): http://dl.dropbox.com/u/24279633/Bukkit/DataLog/DataLog - block ifx.jar

 

Doh, probably the most important call as well. Seems to be working as expected now, thanks.

Ok, current stats:

Overnight:

Code:

mysql> SHOW PROCESSLIST;
+-------+-------------+--------------------------+-------------+---------+-------+-------+------------------+
| Id    | User        | Host                     | db          | Command | Time  | State | Info             |
+-------+-------------+--------------------------+-------------+---------+-------+-------+------------------+
| 32218 | mc-log-user | $server        :50221    | minecraft   | Sleep   |    15 |       | NULL             |
| 32380 | mc-log-user | $server        :34551    | minecraft   | Sleep   |    15 |       | NULL             |
| 32540 | mc-log-user | $server        :45685    | minecraft   | Sleep   |    15 |       | NULL             |
| 32697 | mc-log-user | $server        :42584    | minecraft   | Sleep   |    15 |       | NULL             |
| 32866 | mc-log-user | $server        :45165    | minecraft   | Sleep   |    15 |       | NULL             |
| 33033 | mc-log-user | $server        :55384    | minecraft   | Sleep   |    15 |       | NULL             |
| 33196 | mc-log-user | $server        :60436    | minecraft   | Sleep   |    15 |       | NULL             |
| 33362 | mc-log-user | $server        :57578    | minecraft   | Sleep   |    15 |       | NULL             |
| 33525 | mc-log-user | $server        :42794    | minecraft   | Sleep   |    15 |       | NULL             |
| 33692 | mc-log-user | $server        :48026    | minecraft   | Sleep   |    15 |       | NULL             |
| 33859 | mc-log-user | $server        :43145    | minecraft   | Sleep   |    15 |       | NULL             |
| 34023 | mc-log-user | $server        :52962    | minecraft   | Sleep   |    15 |       | NULL             |
| 34189 | mc-log-user | $server        :41968    | minecraft   | Sleep   |    15 |       | NULL             |
| 34361 | mc-log-user | $server        :41864    | minecraft   | Sleep   |    15 |       | NULL             |
| 34527 | mc-log-user | $server        :48079    | minecraft   | Sleep   |    15 |       | NULL             |
+-------+-------------+--------------------------+-------------+---------+-------+-------+------------------+

after restart with new block place fix:

Code:

mysql> SHOW PROCESSLIST;
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
| Id    | User        | Host                     | db          | Command        | Time  | State              | Info             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
| 35443 | mc-log-user | $server        :48652    | minecraft   | Sleep          |    24 |                    | NULL             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+

after login, breaking/placing a few items, using the log tool, and generally being a nuisance :

Code:

mysql> SHOW PROCESSLIST;
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
| Id    | User        | Host                     | db          | Command        | Time  | State              | Info             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+
| 35443 | mc-log-user | $server        :48652    | minecraft   | Sleep          |   338 |                    | NULL             |
| 35448 | mc-log-user | $server        :40423    | minecraft   | Sleep          |    28 |                    | NULL             |
+-------+-------------+--------------------------+-------------+----------------+-------+--------------------+------------------+

I will check it again tonight when I get back from work.

 

Ok thanks

 

Ok, tonights results:

Code:

[SIZE=12px][FONT=Helvetica]mysql> SHOW PROCESSLIST;[/FONT][/SIZE]

+-------+-------------+--------------------------+-------------+---------+------+-------+------------------+
| Id    | User        | Host                     | db          | Command | Time | State | Info             |
+-------+-------------+--------------------------+-------------+---------+------+-------+------------------+
| 35448 | mc-log-user | $server        :40423    | minecraft   | Sleep   |  224 |       | NULL             |
| 35963 | mc-log-user | $server        :55776    | minecraft   | Sleep   |  224 |       | NULL             |
| 36286 | mc-log-user | $server        :50532    | minecraft   | Sleep   |  224 |       | NULL             |
| 36612 | mc-log-user | $server        :39195    | minecraft   | Sleep   |  224 |       | NULL             |
| 36947 | mc-log-user | $server        :54913    | minecraft   | Sleep   |  224 |       | NULL             |
| 37281 | mc-log-user | $server        :58807    | minecraft   | Sleep   |  224 |       | NULL             |
| 37616 | mc-log-user | $server        :47911    | minecraft   | Sleep   |  224 |       | NULL             |
| 41332 | mc-log-user | $server        :37193    | minecraft   | Sleep   | 2925 |       | NULL             |
| 41370 | mc-log-user | $server        :54738    | minecraft   | Sleep   | 2627 |       | NULL             |
| 41411 | mc-log-user | $server        :52250    | minecraft   | Sleep   | 2325 |       | NULL             |
| 41455 | mc-log-user | $server        :50768    | minecraft   | Sleep   | 2061 |       | NULL             |
| 41496 | mc-log-user | $server        :51364    | minecraft   | Sleep   |  224 |       | NULL             |
| 41580 | mc-log-user | $server        :51062    | minecraft   | Sleep   |  228 |       | NULL             |
| 41624 | mc-log-user | $server        :59500    | minecraft   | Sleep   |  228 |       | NULL             |
| 41705 | mc-log-user | $server        :42615    | minecraft   | Sleep   |  546 |       | NULL             |
| 41708 | mc-log-user | $server        :42616    | minecraft   | Sleep   |  503 |       | NULL             |
+-------+-------------+--------------------------+-------------+---------+------+-------+------------------+

This is with max-connections set to 15 in the config file.