Drupal 6.x and the HTTP request status fails message

As we were developing this site, one day all of a sudden, the news aggregator stopped working and our status page started to report that the site could no longer check for updates. It was the dreaded "HTTP request status fails" message that many Drupal users have been struggling with lately (http://drupal.org/node/227791 and http://drupal.org/node/245853).

After some digging around in the code, I realized a couple of things. One had to do with the site hosting service I'm using at the moment (godaddy.com free hosting). The other was how Drupal 6 handles outgoing HTTP requests.

It works like this: Drupal is happy enough sending out requests to other services, until there is any type of failure in the request. Once that happens, a flag is raised and Drupal then checks whether it can connect to itself. In case it can't, it stops even trying to connect to remote servers. This where the godaddy.com free hosting issue comes in. This service places some js on each and every page. So when drupal connects to itself, it doesn't see what it was expecting.

In the code, it has to do with the error recovery in drupal_http_request in common.inc for version 6.

The code in question is:
if (!$self_test && variable_get('drupal_http_request_fails', FALSE)) {
$self_test = TRUE;
$works = module_invoke('system', 'check_http_request');
$self_test = FALSE;
if (!$works) {
// Do not bother with further operations if we already know that we
// have no chance.
$result->error = t("The server can't issue HTTP requests");
return $result;
}
}

which tries to recover in the event that a request has failed previously. This snippet then calls check_http_request:

function system_check_http_request() {
// Check whether we can do any request at all. First get the results for
// a very simple page which has access TRUE set via the menu system. Then,
// try to drupal_http_request() the same page and compare.
ob_start();
$path = 'admin/reports/request-test';
menu_execute_active_handler($path);
$nothing = ob_get_contents();
ob_end_clean();
$result = drupal_http_request(url($path, array('absolute' => TRUE)));
$works = isset($result->data) && $result->data == $nothing;
variable_set('drupal_http_request_fails', !$works);
return $works;
}

which tries to do a http request on admin/reports/request-test and there-in the real problem lies, at least for me on my hosting service. So when this recovery routine tries to kick in because of some past failure, it never recovers because of the following clause: $result->data == $nothing.

I was able to correct my problem by removing the failing check ($result->data == $nothing). But this brings up whether or not checking admin/reports/request-test is a reliable solution to the problem being solved.

If you feel like hacking the code for yourself do the following:

Edit common.inc and in function drupal_http_request() comment out the recovery code. That is:
// Try to clear the drupal_http_request_fails variable if it's set. We
// can't tie this call to any error because there is no surefire way to
// tell whether a request has failed, so we add the check to places where
// some parsing has failed.
/*
if (!$self_test && variable_get('drupal_http_request_fails', FALSE)) {
$self_test = TRUE;
$works = module_invoke('system', 'check_http_request');
$self_test = FALSE;
if (!$works) {
// Do not bother with further operations if we already know that we
// have no chance.
$result->error = t("The server can't issue HTTP requests");
return $result;
}
}
*/

This will prevent the recovery code from running, but will also allow all http requests to be attempted. In my opinion there is no harm in doing so.

Then add the following line so that the flag gets cleared:
variable_set('drupal_http_request_fails',FALSE);

You *could* add it right after the code that was commented out, but that would defeat the purpose of the flag. Probably a better place for it would be in the success path of the code in this function. Try putting it before these lines at the end of the function:

$result->code = $code;
return $result;
}

So you would get at the end of the function something looking like:

variable_set('drupal_http_request_fails',FALSE);
$result->code = $code;
return $result;
}

That should clear up the majority of problems people are seeing related to the HTTP request status fails message.