Just a test post.
There is no "offline" version of the site. To do what you are talking about would require a second server with identical hardware and operating system, and the server we are running on now was just over $10,000. If we loaded TWC into a normal machine that I threw together for $1000 if would puke its guts up and fall on its face trying to upgrade the database. The last time I loaded TWC into a smaller machine I think it was an 8 core with 16 gigs of RAM and it took like 6 days just to import the database, and that was in 2014 or 15.
The database server that was running database stuff only is currently down. It had a RAID failure that I haven't had time to fix yet, and realistically cannot tell you when that will be fixed. We do not actually "need" it but it was nice to offload a bit of work to a second machine, I will get it back up at some point but time is a luxury I have not had for a while now.
For a privately owned site (not a company site with a huge budget) TWC has some serious overhead in both hardware and monthly expenses.
Its in downtown Denver on a 100 x 100 fiber connection. Though now I have fiber at my house and was told I could get a business class dedicated fiber line added to my home connection. I will have to think about that. For a few hundred a month plus utilities it might be worth it, though first I have to get a generator. The winters here can get pretty rough and I need to get one anyways. The house already has a port for it, fire it up plug it straight into the panel.
64 core Opteron
64 gigs of RAM
Ubuntu
lighttpd
mysql
php 5.6 (soon to be 7.2)
Thor (database server mentioned above)
32 core Opteron
32 gigs of RAM
Somewhere in this process an option in the admincp to update thread views immediately got turned off. I just turned it back on. I suspect its a default setting on this version of vBulletin.
After the thread view fix I was tempted to post the 'I see you' scene from avatar but that might get misunderstood
Yeah lets not go there. I love you but not that much!
For those that are interested, here is the current load on Odin. As you can see we have 9 or 10 free CPUs out of 64 at the time of this shot. That actually updates every second, so it goes from 0 free to 15-20 free depending on whats going on. There are 64 php-fpm (fast process manager) processes running and each is using 360ish megs of RAM. I think I have it capped at 400 or so, that's one of the things (along with upload size) that I still need to go look at. Anyways, every time you view a page it takes between 350 and 380 mb of RAM, depending on the page and the number of custom groups you are in and a lot of other things. That's about 24 gigs of RAM for forum views.
You can also see 3 mysql process at the bottom in red that have 22.8 gigs of RAM loaded. Some of that is from normal forum use, some is from rebuilding the search function. Since we only have 64 gigs total, there is some page swapping going on which is part of the speed problem. That stuff goes away when the index is rebuilt.
Took me a while to find it, but here is a wiki page that Simetrical made long ago and promised to update "slightly more often than never". True to his word, it hasn't been updated since 2010. You have to hit the Edit button to view the text right now until we update wiki software, but its still readable.
http://www.twcenter.net/w/index.php?...fo&action=edit
There is also a bunch of stuff in here, some way out of date as well. There are some private forums with a lot more info, but then they also have a lot of info we do not want public. Is there something specific you are curious about?
http://www.twcenter.net/forums/forum...chnical-Bazaar
Hi. Very slowly the pages began to open. Much slower than it was before. Firefox Browser.
And I noticed that when I publish a post in some thread, it does not appear on the main forum page as the last post in the thread. That is, the posts and the title of the thread are not updated on the main page.
I hope that explained everything clearly, sorry if something is not clear.
Thank you for your work.
This post (from this very thread) explains why pages are loading slowly at the moment.
I'm not one of the tech experts, so I don't know why the indexing doesn't work properly at the moment - but I wouldn't be surprised if GED and Squid came along and said that was also happening because the search index hasn't been completely rebuilt yet.
Thank you very much for the info.
Interesting specs on the server. I just love Ubuntu, don't you ???
I am glad that the site will be going to HTTPS at some point.
I'm just overall curious on the back-end of the site. It's awesome that the site is being runned off Linux with 64 GB of RAM. I look at servers all day and I'm always curious to know what is inside a box and what it is running.
Thanks for the INFO
Proudly under the patronage of Tone
Roma Surrectum Local Moderator
I really appreciate the V-bulletin message when the site is down.
Spoiler Alert, click show to read:
"No problem can withstand the assault of sustained Dufferism"
Um yeah well for the typical site out there you can do that. I have another server that is about 1/4 as powerful as the TWC server and it has probably 6-8 sites running on it, and could run another 500 just like them. It all depends how much traffic you get, and how much processing is done to show the page. A straight html page (no interactive, just static content) probably takes less than 1mb of RAM to process. A single page on TWC takes about 400 mb of RAM to process and we process thousands of them an hour. A typical website on a shared hosting platform might get 100 views a day if they are lucky.
Consider it like this. Shared hosting is like putting out a fire with your garden hose. Sure, you can put out your campfire. The forest fire, probably not...
By contrast TWC gets 10-20 views per second on average, and on a busy time 30-50 per second. And we have a TON of custom hooks in our vBulletin setup. And I do mean a ton. All our custom code probably triples our system requirements.
TWC is so much larger than the average website its actually a bit hard to grasp. Its nowhere near the size of Google or Yahoo or anything like that, but its still ranked 22,061 in the US by Alexa. What that number means is that out of the billions of websites out there, there are 22,060 that get more traffic from the US than we do. So that's probably in the top 1%. Probably in the top .5%, for traffic.
For income, not so much. We do not charge, and I do not allow the most profitable types of ads (popups, video, all the annoying crap) and a LOT of our members ad block us anyways. Lots of them are in 3rd world countries on dial up and the site alone is bad enough on that kind of connection, throwing in ads makes it worse, so I get why people block ads here. For the ones that allow ads through, they are in a demographic that doesn't generally pay that well in a category that doesn't pay that well.
TWC's average per month is $1100-1200. Some months we make a few hundred. Some months we lose a few hundred. And that's fine, the goal was never to become rich off it.
We had a few nesting issues, not sure if thats related to editing or to the indexing. I bumped the specs on that but it wont go live until the next php restart.
Its a tool. I don't love it, nor hate it.
Waste of time. If I knew who to slap at Google he wouldn't see straight for a month.I am glad that the site will be going to HTTPS at some point.
The cores are the most important part, though lots of RAM always helps having 64 cores (with an additional 32 available when Thor is up and running) is the important part. We can handle 64 page views per second assuming a 1 second processing time which we should be able to get really close to once this indexing is done. So if a thread view needs 400 mb of RAM and we have 64 at once, then the web services need 25,600 mb or 26 gigs of RAM. mysql uses another 15-30 gigs normally which puts us under our 64 so we don't have much swapping going on to slow things down.I'm just overall curious on the back-end of the site. It's awesome that the site is being runned off Linux with 64 GB of RAM. I look at servers all day and I'm always curious to know what is inside a box and what it is running.
Thanks for the INFO
Thanks for your work and passion for this site.
Last edited by Morticia Iunia Bruti; April 06, 2019 at 03:38 PM.
Cause tomorrow is a brand-new day
And tomorrow you'll be on your way
Don't give a damn about what other people say
Because tomorrow is a brand-new day
You look great today.
Well first you have to define the words "cheap" and "free". It's like free healthcare, its not really free.
Technically its "free" as it it doesn't cost anything to get an SSL cert from a place like LetsEncrypt. Free except for the time Squid and I have top spend to set it up. Free except for the overhead it adds to the process of actually serving a web page. And it doesn't really do anything to provide you security.
Name one major security breach that happened because someone didn't have SSL. You cannot. Is there an off chance that you are sitting in an internet cafe someplace that also has some script kiddie trying to watch your traffic? Yeah, there is. Its about .005%. Real hackers don't spend time trying to get the info of one person, or even the 50 people connected to a particular hotspot. There is no real money to be made on that. Can it be done? Of course it can. And that is why banks or other sites with sensitive information can and do use it.
And the argument that you might use the same password here that you use at your bank? Technically that is true, people do that. Its stupid, but hey that's people. But that doesn't even matter. Your password here is encrypted already, its NOT sent in plain text. So when you type "password" in the password box what it sends is more like "482c811da5d5b4bc6d497ffa98491e38" and then there is an additional layer on top of it called a Salt. So even if they get it they cannot possibly use it at another site, because those sites are salted too. The password "password" will have a different value here than it does at your bank, and another value at your job, and another value at Amazon.
When a true hacker is trying to get into something, they aren't trying to do it through your traffic, they are trying to do it through the site interface, and its poor code that allows it. Whether there is SSL on it or not. They aren't using any password, other than the one they created to login to the site in the first place. They are trying to find weak spots in the code that lets them run commands you wouldn't normally be able to run. When they can do that, they can download the entire user database then run a brute force tool against it on their own machine to try and crack as many passwords as they can. SSL is literally 0 protection from that because now everything is happening on the hackers machines and not the site you visited, and it happens all the time. Every breach you can name: Home Depot, Target, etc, was done this way.
But Google in their infinite wisdom has decreed that every site that has a login section has that page SSL encrypted. Not just the login section, the entire page. They will not allow mixed SSL content, where part of the page is encrypted and part is not. So this text you are reading right now? That has to be encrypted, or you get the Not Secure notice in your browser.
On TWC that is every single page. Every single page on TWC has a login location. And every single thing on that page has to be encrypted to avoid getting the Not Secure notice. That stupid little pixel that represents TWC Reputation? Has to be encrypted. That text by your name that says you joined in 2008? Has to be encrypted. That part that says Today at 06:40 PM? Yep, that has to be encrypted too. So to encrypt the 1% of the site that might actually matter, we have to encrypt the other 99% of the site which is a performance impact of somewhere between 2 and 20% depending where you look at benchmarks and how complicated a site is. That is hardly free.
With a new site, encrypting the entire site is actually a lot easier than encrypting a site as old as TWC. Because every single item on the site must be encrypted to avoid that notice, now I have to worry about what you link to or what you post in [img] tags. And not just what you link to today, I have to worry about what some fool linked to 12 years ago. So every time we process a page for you to view we have to add an additional check to see if there is any non-SSL enabled content on it and if there is run it through a separate process and pull it with a proxy and then encrypt it on the back side. Thanks Google.
In Google's world, wearing the seat belt in your car with an airbag is not good enough to run down to the corner store and buy a soda. You have to wear a full blown NASCAR racing suit with helmet and have the ambulance driver watch you.
Last edited by GrnEyedDvl; April 06, 2019 at 08:35 PM.
Under patronage of General Brewster of the Imperial House of Hader.
How to make Morrowind less buggy for new players - Of course every player may find it useful.