Threaded Mode

[GrnEyedDvl]

Ok I think I figured something out here.


I was looking through the code for the 10 thousandth time, and it hit me that the showthread.php error we keep getting on line 1288 is inside a couple of different IF statements, and one of them is this:

if ($threadedmode == 2 AND empty($vbulletin->GPC['postid']))

threadedmode 2 is Hybrid.

My Account > Edit Options > Thread Display Mode

What that option does is show not only 40 posts per page, but also a threaded view of every post in the thread, unpaginated. So on a really big thread it will list the header of every post in a scrollable bracket at the top, no matter how many posts there are in the current thread.

So I changed my option to Hybrid went looking for big threads with current activity. I found the Picture War thread, and voila instant 503 error. I forced a lighttpd restart, changed my mode back to Linear, and the thread opened just fine.

The reason its so intermittent is that we have very few users that have their threadedmode set to Hybrid. Querying the database for that gives me this, ignore the hit on my name as I switched back:

mysql> select username, userid, threadedmode from user where threadedmode = '2';
+---------------------------+--------+--------------+
| username                  | userid | threadedmode |
+---------------------------+--------+--------------+
| GrnEyedDvl                |  21866 |            2 |
| mckain666                 |  45543 |            2 |
| ztrain909                 |  48345 |            2 |
| Rifugio                   |  78940 |            2 |
| Ulari                     |  85771 |            2 |
| Eärnur                    |  88913 |            2 |
| Buci210                   |  99294 |            2 |
| infernus mortalis         |  99387 |            2 |
| leonkennedy residentevil4 | 101820 |            2 |
| Hagen.von.Tronje          | 103624 |            2 |
| D0dge_182                 | 116499 |            2 |
| alarioninglorion          | 123753 |            2 |
| Thrillology               | 124217 |            2 |
| LordHoern                 | 133573 |            2 |
| leonardo davinci          | 144603 |            2 |
| Benzeto                   | 151394 |            2 |
+---------------------------+--------+--------------+
16 rows in set (0.37 sec)

So basically we have 15 users (didnt check current activity on those users) that can throw a 503 error by viewing this type of thread. The real solution is to rewrite the code so its paginated, but we are far too close to vB4 to start hacking up vB3 more.

For now I closed the Picture Wars thread and told them to start another one, there are probably other threads out there that can cause this issue, but this one is getting quite a bit of activity.

So without performing a rewrite on vBulletin code, we have 2 choices:

1. Find the huge threads out there, close them, start new ones.

2. Force everyone into Linear mode and remove the 2 Threaded options from the User Control Panel.





Another thing I happened to find in the showthread.php code by comparing to the original file is this:

  if(!empty($db->explain)) $db->timer_start("construct_postbit");

A database explain function is basically used to log info for the developer, it as no use to the average forum user and doesnt display anything, its dumped in a log via includes/class_database_explain.php.

You can see the timer_start and timer_stop functions starting at line 113 in that file. Basically it compares the amount of memory requested to the amount of memory available, and the number of milliseconds involved.

After having a conversation with one of the guys at our host and showing him the files, he suggested that we comment out all of the explain functions in showthread.php as all they are doing is wasting resources. Its pretty CPU intensive to run that function every single time someone views a thread.

I havent done that yet as I want Squid's input as well.

[GrnEyedDvl]

I looked through the 15 users above, only 6 have logged on in the last month, and 2 are brand new users, 1 hasnt posted yet at all.

So basically we have 4 regular users who can cause this problem by viewing very large threads.

[Bolkonsky]

Only two of those users have been active in the last week.

124217 and 151394

EDIT: Ninja'd.

OK I'd say we should just set them to normal, and remove the option from the UserCP. I really don't see how it's super-useful anyways.

[GrnEyedDvl]

That last little 503 was me in Hybrid mode viewing the You Are Banned thread to confirm the theory. Try it if you want.

[pannonian]

Solution 2 is the cleanest.

[GrnEyedDvl]

Yeah I am leaning in that direction as well.

[Augustus Lucifer]

I'd go for Option 2 as well, hardly the end of the world to view the forums in linear mode, even if it had to affect substantially more users.

[Squid]

Given so few users are affected by this, is this what's really causing all the slowdown? I'll agree its responsible for a good chunk of the errors in the log, but many people who aren't on the list are being hit by the slowdown. Even if all those users were tying up fast-cgi threads there should be still be enough left over to avoid slowdowns.

[SigniferOne]

I can give myself as an example, I'm hit by the slowdowns (without the Threaded View on).
Also, my slowdowns are intermittent, sometimes I can get the page to show up quite quickly.

EDIT1: This very post timed out :p

EDIT2: Posting edit1 took 0.5 seconds.

[GrnEyedDvl]

Given so few users are affected by this, is this what's really causing all the slowdown? I'll agree its responsible for a good chunk of the errors in the log, but many people who aren't on the list are being hit by the slowdown. Even if all those users were tying up fast-cgi threads there should be still be enough left over to avoid slowdowns.

We are getting 50-150 requests per second on average. I forget what the max number of cgi process is (128?) since we have been changing stuff around a bit, but if we tie a couple of those up for 30 seconds then things can start backing up. If we are already at our max and we tie a few more up...

There could also be some database locks that prevent read/write until it either finishes or gets killed.

Even if this isnt responsible for the all of the slowdown issues, its responsible for some. I dont doubt that there are still some configuration we can do to lighttpd processes and fasg-cgi processes. I found this on optomizing lighttpd, and the thing that hits me right off is the formula for backend processes. Notice its much different than ours is setup. We have one backend process with a ton of children, their example has 10 backends with fewer children per backend. That might make a difference too.


Last but not least, we should be able to get some CPU back if we comment out the explain code I posted above. I didnt want to test that over the weekend, but we can now.

[GrnEyedDvl]

Ok I found this in the lighttpd optimization article.

Also, when configuring lighty to manage php-fcgi processes, it is better to have more processes and less children than less processes and more children. For example:

  fastcgi.server = ( ".php" =>
      (( "socket" => "/tmp/php-fastcgi.socket",
          "bin-path" => "/usr/bin/php-cgi",
          "max-procs" => 10,
          "bin-environment" => (
              "PHP_FCGI_CHILDREN" => "10",
              "PHP_FCGI_MAX_REQUESTS" => "500" 
          ),
          "broken-scriptfilename" => "enable" 
      ))
  )

Is better than:

  fastcgi.server = ( ".php" =>
      (( "socket" => "/tmp/php-fastcgi.socket",
          "bin-path" => "/usr/bin/php-cgi",
          "max-procs" => 2,
          "bin-environment" => (
              "PHP_FCGI_CHILDREN" => "50",
              "PHP_FCGI_MAX_REQUESTS" => "500" 
          ),
          "broken-scriptfilename" => "enable" 
      ))
  )

because, when a backend "dies", all the children from that one FastCGI process become unavailable. Having many processes means that if one dies the rest can share the load. If you have 2 processes and both are loaded equally then if/when one dies the other one will suddenly become overloaded and die itself, causing lighty to throw 500 errors.

The red explains how the slowdown affects the entire site, we only had a single parent process.


Notice that both of those configurations above you end up with 101 total fast-cgi processes. 2x50+1 and 10x10+1. The number of parents is larger in the "better" example, and the number of childfren is smaller.

When Sim originally setup lighttpd, we had a single parent and 64 children, for a total of 65 cgi processes. I bumped the children to 128 when we added RAM last month. This is the wrong way to do it.

I just changed our setup to:

 
fastcgi.server = ( ".php" => ((
        "socket" => "/var/run/lighttpd/php-fastcgi.socket",
        "bin-path" => "/usr/bin/php-cgi",
        "broken-scriptfilename" => "enable",
        "min-procs" => 1,
        "max-procs" => 10,
        "allow-x-send-file" => "enable",
        "bin-environment" => (
                "PHP_FCGI_CHILDREN" => "18",
                "PHP_FCGI_MAX_REQUESTS" => "500"

So we now have 10 parents instead of 1, and 18 children, for a total of 181. We will see what that does.

Sorry if the lighttpd restart hosed anyone up, it needed to be done.

[GrnEyedDvl]

I am on a machine that has never visited TWC before (new install) and the site is running great. Any observations from anyone else?

[Bolkonsky]

Occasionally great, occasional 5+ seconds loading time.

[Magicman2051]

Haven't noticed it being slow in the past few hours ^^

[SigniferOne]

The red explains how the slowdown affects the entire site, we only had a single parent process.

My question is, why was the slowdown happening now? What changed?

You bumped the children to 128 with extra RAM, but that wouldn't make the whole site slower than when it had half of that, in terms of cgi processes.

[GrnEyedDvl]

My question is, why was the slowdown happening now? What changed?

You bumped the children to 128 with extra RAM, but that wouldn't make the whole site slower than when it had half of that, in terms of cgi processes.

The new members using the hybrid mode on those big threads, most of them just recently joined. Its been steadily getting slower for a few months, its just recently been a huge problem.

[GrnEyedDvl]

We only had 10 errors on showthread.php yesterday, which is a good sign. I havent removed the option for threaded or hybrid, but I switched a few people. They havent complained so..


We are still getting some of the other errors though. After some digging I expect we will find that most of these are somehow related to huge threads as well. The file not found ones are most likely links that to files that no longer exist, which isnt a huge issue for performance.

--------------------- php-cgi Begin ------------------------

php-cgi errors, ordered by number of occurrences:

16 PHP Fatal error: Maximum execution time of 30 seconds exceeded in /mnt/data/www/forums/includes/functions.php on line 657
10 PHP Fatal error: Allowed memory size of 209715200 bytes exhausted (tried to allocate ### bytes) in /mnt/data/www/forums/showthread.php on line 1288
7 PHP Warning: Cannot modify header information - headers already sent by (output started at /mnt/data/www/w/includes/OutputPage.php:1593) in /mnt/data/www/w/includes/WebResponse.php on line 16
6 PHP Fatal error: Maximum execution time of 30 seconds exceeded in /mnt/data/www/forums/includes/functions.php on line 1813
2 PHP Fatal error: Allowed memory size of 209715200 bytes exhausted (tried to allocate ### bytes) in /mnt/data/www/forums/includes/functions_threadedmode.php on line 23
1 PHP Warning: copy(/var/local/userfiles/downloads/39801-TM8.zip) [<a href='function.copy'>function.copy</a>]: failed to open stream: No such file or directory in /mnt/data/www/forums/downloads.php on line 754
1 PHP Fatal error: Maximum execution time of 30 seconds exceeded in /mnt/data/www/forums/includes/class_bbcode.php on line 511
1 PHP Fatal error: Maximum execution time of 30 seconds exceeded in /mnt/data/www/forums/includes/class_bbcode.php on line 2079

---------------------- php-cgi End -------------------------

[Squid]

The file not found warning are from the downloads plugin. There's a line that tries to copy the file before the check to determine if the file exists. I just haven't fixed it.

[GrnEyedDvl]

As long as we know what it is, dont bother. That one can wait for vB4.

[Magicman2051]

Is it possible that errors could occur in threads will dozens of videos per page?