Server Ignoring PHP 301 Header: IIS Known Bug

This is a little more back-end/server-side than front-end, but I happened upon an absolutely aggravating bug found with IIS/the FastCGI module when moving a client over to a new site structure. Like any good development company, we wanted to make sure the old site structure URLs redirected permanently (301 Redirect) to the correct new URLs so that off-site links continued to work. Unfortunately, I ran into some roadblocks setting it up.

The Problem and Roadblocks

This is a large site; something on the order of 2500 pages needed redirects for the new structure. Or at least, 2500 URLs. Many of the pages were dynamically generated from a CMS with redirects already set up to resolve SEO-friendly URLs to the server-based variable-laden ones. The fastest and most convenient—not to mention maintainable and scaleable—method of doing this seemed to be writing a quick PHP script (as it is the scripting language used by the site and CMS) to map the old URLs to the new. We’d simply redirect all requests to that script and then let it handle the mapings with a header statement:

header("Location: ".$newURL,true,301);

Trivial, quick, workable. At least on Apache. I traced my headers though, and IIS was sending a 302 (Temporary) redirect despite the explicit 301 call. I threw out some very naughty words at that point.

Why use a 301 redirect instead of 302? We want the search engines to start throwing out their record of the old URL and start storing/caching the new one as an existing page. If we use a 302 redirect, it thinks that we’ve only temporarily moved the page (maybe for a promotion, or holiday, or who knows why), but we’re saying thate eventually, we’ll go back to using that old URL and whatever content is there, so continue storing that URL. A 301 redirect, however, means that we’ve permanently moved to the new URL, and there isn’t anything at the previous one, so start replacing it in memory with the new one. Presumably, someone stored that old URL and wanted to continue to access the content, so rather than simply changing the URL, we’re providing a way for them to continue to access it without getting a 404: Page Not Found error.

Roadblock #1: Using a Microsoft Server

Ok, so, maybe I’m being a little flippant. I suppose there’s nothing wrong with IIS. It’s just that I, personally, prefer LAMP development if I have to do back-end/server-based stuff.

But facetiousness aside, IIS really is the cause of our problems in this case. Ultimately, two issues were affecting what should have been a simple issue. Specifically, a bug in the IIS FastCGI module and a size limit on web.config files.

Roadblock #1.1: IIS/FastCGI and PHP 301 Redirects

Two dirty words: “Known Bug.” Essentially, IIS/FastCGI module somehow lose track of the fact that you said “I want a 301 redirect” and handles it as a 302. Every. Single. Time.

So, our quick, easy, maintainable, scalable script idea goes out the window.

Roadblock #2: web.config filesize limit

Evidently there is a max size for your web.config filesizes. Don’t ask me what it is though: I can’t figure that out. My Google-fu isn’t strong enough to find anything about it, evidently.

I found this out when moving to Redirect Mapping Plan B: Spit Out web.config Redirects for Every URL Mapping In One File. That, my friends, spit out a wonderful 500: Internal Server Error error when I uploaded my new web.config file.

Guess it was time for a Plan C.

Our Workaround

Yuck.

I don’t even want to mention it. It’s depressing. Hopefully if you run into this, you can do something better.

We wrote a script that generated the URLs we needed and the mapping to the new URLs. Then, we formatted them for web.config redirects (everything to this point was actually part of Plan B. Moving on:) and made individual web.config files for each. Individual. Folder. under the old structure. This meant leaving old, useless folders online with nothing by web.config files in them. Like I said. Yuck.

“Why not use another language, like ASP to get around the bug?,” you ask. In theory, that would work. But it would also take more time. See, we needed to interface with the CMS to get the correct URLs, so not only would we have to author the quick mapping script, but we’d have to write a more involved script to actually retrieve the correct URLs from the PHP-based CMS, and that was determined to be a non-option. (Hey, it all comes down to business: time==money && faster_workaround == less_time == more_money_per_time.)

Will This Ever Be Fixed?

According to user ruslany (Microsoft employee?) on the IIS forums:

This is a bug in IIS FastCGI module. It will be fixed in Windows 7 RTM. We are also looking into possible ways for making this fix available for IIS 7.

So, short answer, no, not really, unless you upgrade. Maybe they’ll get around to fixing it in IIS 7 with a patch. Maybe. They’re looking. He says. Here’s the full thread.

As for the max web.config file size, it actually makes a lot of sense, so I can’t imagine them ~~fixing~~ changing it. It was just frustrating to hit that unknown amount after the other bug. If anyone reading this happens to know that mysterious byte size cap, though, please let me know. It’d be useful info in the future, I suppose, and I am too lazy to experiment to determine the answer.