Server Updates

[Squid]

Things are going to be somewhat erratic as the upgrade continues. The upgrade has partially been completed I'm hoping that the scripts finish running at some point today after which I'll bring the new file versions online.

[sumskilz]

Something new for me as of today, I can no longer see new posts within threads. I can sometimes see that there are new posts from the main forum page, but I'm not actually able to view them in the thread or get to the last pages of a thread if it contains posts I haven't viewed.

[Sar1n]

I don't see some latest posts. When I'm at main formu page, I see that latest posts in some threads are later, different from those that appear after clicking on the thread. For example, when on main, I see latest post in this thread is by sumskilz at 6:14 PM today (my timezone), but latest post I see her when writing is by Van Zandt, 2:03 AM.

[sumskilz]

Funny enough I can't even see what I just posted, although from here I can see that it did actually go through.

[Cookiegod]

I commented here, but my comment disappeared. The same happened in another thread as well. So this forum is officially dead for the time being?

[Squid]

Just as a note, in all likelihood after the upgrade is complete there will be some site features, especially those provided by third party plugins, that may need to be temporarily disabled until they are made compatible with all the changes.

[Kilo11]

So this is just a report of another issue I have noticed, in case such a report proves useful for the ladies and gents taking care of the site updates.

When I go to the "My Account" page, which is my default page when I open Firefox, I notice that under the "Subscribed Threads with New Posts" area there is rarely anything listed, even when there are new posts in threads I've subscribed to. This only came to my attention because I am also subscribed to some forums, and in that area of the My Account page it showed new posts, and so I clicked those, and saw that they were posts in threads to which I am subscribed. Wanting to know exactly what was going on, I also checked in my subscriptions folder, and there it also seems to not be registering the most recent posts in many threads. So it seems there is a general issue with the overall site architecture not registering when there are new posts in threads (to be clear, the posts are there, but they don't seem to be registered in the database of new posts, if that makes sense).

Again, no complaint or wish to see this remedied in the next 20 minutes or anything. I am just letting you folks know what I've seen, in case that might be of use while you are revamping all the background machinery of the site. Keep up the good work team!

[GrnEyedDvl]

All the major work on the forums is done, we still need to address the wiki. The site will be slow over the weekend as its rebuilding the search index, which takes bloody forever and is using a ton of resources.

[Skotos of Sinope]

Still can't see any posts in this thread after April 3rd. And many other threads as well.

EDIT: Correction. I can now finally see page 8, but it looks like some of the latest posts still aren't visible. For example, GED posted at 10:27 PM my time, and the latest post I can see in this thread is Gigantus's post at 10:12 PM.

[Gigantus]

Still can't see any posts in this thread after April 3rd. And many other threads as well.

Quoting works again


Edit: so does editing, and everything slow as promised

Edit2: confirming Skotos' observation - got a subscription notice that GED made a (latest) post, but can't see it.

[GrnEyedDvl]

What I don't get: Wouldn't it have been better to simply test those updates out on an offline version of the forum, and then simply dish an update out once it was done?

There is no "offline" version of the site. To do what you are talking about would require a second server with identical hardware and operating system, and the server we are running on now was just over $10,000. If we loaded TWC into a normal machine that I threw together for $1000 if would puke its guts up and fall on its face trying to upgrade the database. The last time I loaded TWC into a smaller machine I think it was an 8 core with 16 gigs of RAM and it took like 6 days just to import the database, and that was in 2014 or 15.

The database server that was running database stuff only is currently down. It had a RAID failure that I haven't had time to fix yet, and realistically cannot tell you when that will be fixed. We do not actually "need" it but it was nice to offload a bit of work to a second machine, I will get it back up at some point but time is a luxury I have not had for a while now.

For a privately owned site (not a company site with a huge budget) TWC has some serious overhead in both hardware and monthly expenses.

Hello,
Where is this site being hosted from ?
A datacenter? someones house, In the united states, Great Britain ?
What Operating system is it running under ?
Sorry, being a network technician you tend to get curious and since i've been coming to this site since 2004 I would like to know its inner guts - LOL

Thanks

Its in downtown Denver on a 100 x 100 fiber connection. Though now I have fiber at my house and was told I could get a business class dedicated fiber line added to my home connection. I will have to think about that. For a few hundred a month plus utilities it might be worth it, though first I have to get a generator. The winters here can get pretty rough and I need to get one anyways. The house already has a port for it, fire it up plug it straight into the panel.

64 core Opteron
64 gigs of RAM
Ubuntu
lighttpd
mysql
php 5.6 (soon to be 7.2)


Thor (database server mentioned above)
32 core Opteron
32 gigs of RAM

I have noticed that this page...

http://www.twcenter.net/forums/forum...itical-Academy

...does not update or show/list any new posts.

- A

Still can't see any posts in this thread after April 3rd. And many other threads as well.

Somewhere in this process an option in the admincp to update thread views immediately got turned off. I just turned it back on. I suspect its a default setting on this version of vBulletin.

[Prodromos]

There is no "offline" version of the site. To do what you are talking about would require a second server with identical hardware and operating system, and the server we are running on now was just over $10,000. If we loaded TWC into a normal machine that I threw together for $1000 if would puke its guts up and fall on its face trying to upgrade the database. The last time I loaded TWC into a smaller machine I think it was an 8 core with 16 gigs of RAM and it took like 6 days just to import the database, and that was in 2014 or 15.

The database server that was running database stuff only is currently down. It had a RAID failure that I haven't had time to fix yet, and realistically cannot tell you when that will be fixed. We do not actually "need" it but it was nice to offload a bit of work to a second machine, I will get it back up at some point but time is a luxury I have not had for a while now.

For a privately owned site (not a company site with a huge budget) TWC has some serious overhead in both hardware and monthly expenses.

I had no clue it cost that much to run a website. I thought it was like $20/year.

[Skotos of Sinope]

Somewhere in this process an option in the admincp to update thread views immediately got turned off. I just turned it back on. I suspect its a default setting on this version of vBulletin.

GED, from the bottom of my heart: thank you, thank you, thank you. (And same to Squid as well.) That bug was really freaking me out.

[GrnEyedBot]

Just a test post.

[Gigantus]

After the thread view fix I was tempted to post the 'I see you' scene from avatar but that might get misunderstood

[GrnEyedDvl]

After the thread view fix I was tempted to post the 'I see you' scene from avatar but that might get misunderstood

Yeah lets not go there. I love you but not that much!


For those that are interested, here is the current load on Odin. As you can see we have 9 or 10 free CPUs out of 64 at the time of this shot. That actually updates every second, so it goes from 0 free to 15-20 free depending on whats going on. There are 64 php-fpm (fast process manager) processes running and each is using 360ish megs of RAM. I think I have it capped at 400 or so, that's one of the things (along with upload size) that I still need to go look at. Anyways, every time you view a page it takes between 350 and 380 mb of RAM, depending on the page and the number of custom groups you are in and a lot of other things. That's about 24 gigs of RAM for forum views.



You can also see 3 mysql process at the bottom in red that have 22.8 gigs of RAM loaded. Some of that is from normal forum use, some is from rebuilding the search function. Since we only have 64 gigs total, there is some page swapping going on which is part of the speed problem. That stuff goes away when the index is rebuilt.

[Kilo11]

Glad to see the updates have gone through well enough, and again, a million thanks to GED and Squid!

[Alexander78]

Hi. Very slowly the pages began to open. Much slower than it was before. Firefox Browser.
And I noticed that when I publish a post in some thread, it does not appear on the main forum page as the last post in the thread. That is, the posts and the title of the thread are not updated on the main page.
I hope that explained everything clearly, sorry if something is not clear.
Thank you for your work.

[Caillagh de Bodemloze]

Hi. Very slowly the pages began to open. Much slower than it was before. Firefox Browser.
And I noticed that when I publish a post in some thread, it does not appear on the main forum page as the last post in the thread. That is, the posts and the title of the thread are not updated on the main page.
I hope that explained everything clearly, sorry if something is not clear.
Thank you for your work.

This post (from this very thread) explains why pages are loading slowly at the moment.

I'm not one of the tech experts, so I don't know why the indexing doesn't work properly at the moment - but I wouldn't be surprised if GED and Squid came along and said that was also happening because the search index hasn't been completely rebuilt yet.

[pannonian]

This post (from this very thread) explains why pages are loading slowly at the moment.

I'm not one of the tech experts, so I don't know why the indexing doesn't work properly at the moment - but I wouldn't be surprised if GED and Squid came along and said that was also happening because the search index hasn't been completely rebuilt yet.

When you post, vBulletin does a number of processes so that everything is consistent. It hasn't been doing that, possibly because one or another process has been timing out.