HTTP status in the server header
The HTTP response status code in the server header indicates to user agents (bots and browsers) the status of HTTP response following the HTTP request of URLs. Some common HTTP status codes are:
- 200 (OK),
- 301 (Moved Permanently) - for permanent redirects, the server header has also a Location field with the URL destination of the redirect,
- 302 (Moved Temporarily) for temporary redirects, the server header includes the location of the URL destination of the redirect,
- 401 (Authentication Required), for password protected files and folders,
- 403 (Forbidden), used as good practice for URLs of folders without an index page,
- 404 (Not Found), for URLs that do not correspond to content on the server,
- 500 (Internal Server Error),
- 503 (Service unavailable) - indicating that the server is temporarily unavailable, used for example during server maintenance.
It is important for a website to return correct HTTP status response, for example HTTP status response 500 (Internal Server Error) in the server header when there is a server error at generating the requested URL. If a URL that causes a server error is returning HTTP status 500 (Internal Server Error) in the server header, it is not indexed by search engines, but if it returns HTTP status response 200 (OK) it can be indexed in search results, and usually it will contain mainly an error message, that would be of no use to people visiting the site. A correct HTTP status response at server error can be implemented with a comprehensive error catching algorithm in the server-side scripts used by a website to generate content.
Some problems in the way search engines deal with HTTP status response of URLs can be caused by the way the HTTP request and HTTP response deal with the type of HTTP protocol, HTTP/1.1 or HTTP/1.0. For example Google bots make usually HTTP/1.1 requests, Yahoo HTTP/1.0 requests, some MSN bots make HTTP/1.1 requests and others make HTTP/1.0 requests, and there might be some difference in which they deal with HTTP status response codes specific to HTTP/1.0 or HTTP/1.1 protocols, like HTTP status response 410 (Gone), that is specific to the HTTP/1.1 protocol.