Any experience debugging HHVM falling over frequently?

Trojan

I've set up a server with HHVM. It's falling over every 3 or 4 hours (consistently).

Has anyone any recommendations on where to start debugging this? (I've already tried a couple of options with php.ini).

amen

have you contact the HHVM developers ?

Colonel Panic

Is it dumping anything to a log?

Trojan

I've been going through the error log, knocking off a few things here and there (quite a few warnings, and an error about database collation charset). Not sure I've found the true cause, and I'd prefer not to rebuild it with debug turned on, though I may have to.

I haven't contacted the devs. Know if they're on IRC anywhere?

Trojan

I've given up on HHVM for the moment. Posting it up in case it's useful to anyone else out there experimenting with this.

Problem summary:

1) HHVM is falling over regularly (every 2-3 hours)
1.1) monit and a full reboot sometimes don't bring HHVM back up

2) WordPress won't allow a user login (not wrong password, seems to force a reauth): http://www.example.com/wp-login.php?redirect_to=http%3A%2F%2Fwww.example.com%2Fwp-admin%2Fprofile.php&reauth=1

3) Incorrect HTML is being output from WordPress (in a consistent way)

================================================================================

1) From /var/log/hhvm/error.log:

I'm seeing no error messages printed to /var/log/hhvm/error.log immediately prior to it falling over. However, there are a lot of filesystem calls with warnings or errors:

[Fri Mar  6 22:57:27 2015] [hphp] [32415:7f6830bff700:4:000011] [] f_is_readable/1013: No such file or directory
[Fri Mar  6 22:57:27 2015] [hphp] [32415:7f6830bff700:4:000012] [] f_is_readable/1013: No such file or directory
[Fri Mar  6 22:58:11 2015] [hphp] [32415:7f6823fff700:4:000156] [] f_is_dir/1095: No such file or directory
[Fri Mar  6 22:58:11 2015] [hphp] [32415:7f6823fff700:4:000157] [] f_mkdir/1776: Permission denied

I don't know what file it's trying to write. Is there any way for me to find that out without recompiling HHVM with debug on? Is this definitely the cause of the crash?

I noticed wp-content/debug.log was 644, I made that a+w. Could this be a partial cause of the issue, and turning on debugging made it worse?


2) It's not possible to log in, keeps looping back to login.

http://www.example.com/wp-login.php login with correct password redirects to here:
http://www.example.com/wp-login.php?redirect_to=http%3A%2F%2Fwww.example.com%2Fwp-admin%2Fprofile.php&reauth=1

http://www.example.com/wp-login.php login with incorrect password shows incorrect user/pass error as expected.

Things I've done that haven't worked:

• Resetting user password with WP-CLI
• Removed all plugins
• Changed to a default theme
• Created a new user with WP-CLI
• Checked database siteurl option
• Forced a core update
• Recreating wp-config.php from the sample file,
• Replaced all WordPress core files manually via FTP.

After all that still not possible to log in. I'm guessing this would point at some kind of database corruption, but testing the database locally it seems fine... which leaves nginx & HHVM.

3) This one presents strangely.

Instead of the following:

<ul class="sf-menu">	
<li id="menu-item-2798" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-2798"><a href="http://www.example.dev/">Home</a></li>
<li id="menu-item-2801" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-2801"><a href="http://www.example.dev/podcast/">Podcast</a></li>
<li id="menu-item-2806" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-2700 current_page_item current_page_parent menu-item-2806"><a href="http://www.example.dev/blog/">Blog</a></li>
<li id="menu-item-3229" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-3229"><a href="http://www.example.dev/resources/">Resources</a></li>
<li id="menu-item-3228" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-has-children sf-with-ul menu-item-3228"><a href="http://www.example.dev/services/">Services<span class="sf-sub-indicator"><i class="icon-angle-down"></i></span></a>
	<ul class="sub-menu">
	<li id="menu-item-3728" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-3728"><a href="/">Program X</a></li>
	<li id="menu-item-3786" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-3786"><a href="http://www.example.dev/services/ServiceA/">ServiceA</a></li>
	<li id="menu-item-3297" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-3297"><a href="http://www.example.dev/services/ServiceB/">ServiceB</a></li>
	<li id="menu-item-3305" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-3305"><a href="http://www.example.dev/services/ServiceC/">ServiceC</a></li>
	<li id="menu-item-3308" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-3308"><a href="http://www.example.dev/services/ServiceD/">ServiceD</a></li>
	</ul>
</li>
<li id="menu-item-2898" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-has-children sf-with-ul menu-item-2898"><a href="http://www.example.dev/about/">About<span class="sf-sub-indicator"><i class="icon-angle-down"></i></span></a>
	<ul class="sub-menu">
	<li id="menu-item-2803" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-2803"><a href="http://www.example.dev/contact/">Contact</a></li>
	</ul>
</li>
</ul>

I'm seeing the following:

<ul class="sf-menu">	
<li id="menu-item-2798" class="menu-item menu-item-type- menu-item-object- menu-item-2798"><a>Home</a></li>
<li id="menu-item-2801" class="menu-item menu-item-type- menu-item-object- menu-item-2801"><a>Podcast</a></li>
<li id="menu-item-2806" class="menu-item menu-item-type- menu-item-object- menu-item-2806"><a></a></li>
<li id="menu-item-3229" class="menu-item menu-item-type- menu-item-object- menu-item-3229"><a></a></li>
<li id="menu-item-3228" class="menu-item menu-item-type- menu-item-object- menu-item-3228"><a></a></li>
<li id="menu-item-3728" class="menu-item menu-item-type- menu-item-object- menu-item-3728"><a>ServiceA</a></li>
<li id="menu-item-3786" class="menu-item menu-item-type- menu-item-object- menu-item-3786"><a>ServiceB</a></li>
<li id="menu-item-3297" class="menu-item menu-item-type- menu-item-object- menu-item-3297"><a></a></li>
<li id="menu-item-3305" class="menu-item menu-item-type- menu-item-object- menu-item-3305"><a></a></li>
<li id="menu-item-3308" class="menu-item menu-item-type- menu-item-object- menu-item-3308"><a></a></li>
<li id="menu-item-2898" class="menu-item menu-item-type- menu-item-object- menu-item-2898"><a>About</a></li>
<li id="menu-item-2803" class="menu-item menu-item-type- menu-item-object- menu-item-2803"><a>Contact</a></li>
</ul>

This is consistently output. Some kind of caching where it reads from somewhere that a write failed?

This code is generated by the same theme, the same database and the same plugins, one on HHVM with batcache/memcache, the other locally on WAMP:

<div class="content-inner">
	<a target="_blank" href="http://amdsoft.com/quick-backup-script-with-wp-cli/">
		<div class="link-inner">
			<span class="link-wrap">
				<h2 class="title">One for the techie readers</h2>
				<span class="destination"> http://amdsoft.com/quick-backup-script-with-wp-cli/</span>
			</span>
			<span title="Link" class="icon"></span>
		</div><!--/link-inner-->
	</a>
</div><!--/content-inner-->

vs:

<div class="content-inner">
	<div class="article-content-wrap">
		<div class="post-header">
			<h2 class="title"> <a href="http://www.example.com/backup-with-wp-cli/">One for the techie readers </a> </h2>
			<span class="meta-author"><span>By</span> <a href="http://www.example.com/author/admin/" title="Posts by Alastair McDermott" rel="author">Alastair McDermott</a></span> <span class="meta-category">| Uncategorized</span> <span class="meta-comment-count">| <a href="http://www.example.com/backup-with-wp-cli/#comments">
			No Comments</a></span>
		</div><!--/post-header-->
	</div><!--article-content-wrap-->				
</div><!--/content-inner-->

===============================================================

Approach to try resolving these issues:

a) HHVM is falling over so often, could be causing DB and FS corruption, going to replace that with PHP5-FPM until I can debug it properly.

b) If still issues after removing HHVM, do a completely clean WordPress install with blank slate filesystem and known good local database.

c) Leave memcache disabled for now.

===============================================================

Action taken:

0) Disabled batcache/memcache (actually this was done as part of earlier troubleshooting)

1) Re-enabled PHP-FPM for all sites.

This fixed user login and incorrect HTML output.

Obviously the site is much slower now with no HHVM or caching, but at least seems to be stable.

I'll address performance issues later, maybe revisit HHVM with a different approach.

Talisman

I haven't used HHVM in about 12 months - things may have changed since then but at the time I found it a disaster when using it for WordPress and frequent crashes with Laravel. It certainly wasn't anywhere near production ready.

There's a note in the first line of this tutorial: Using HHVM with PHP-FPM Fallback

HHVM doesn’t play with all WordPress themes and plugins. Test your site properly before using this in production.

Trojan

Well now in fairness nothing plays well with "all WordPress themes and plugins" (not even WordPress itself).