When working on a client site recently I encountered a strange issue that had me scratching my head. I took to Google and was unable to find a solution. I eventually sorted things out, solved my problem, and all was right in the world again. I want to share this issue with other SEO’s and Website owners incase something similar happens to you, as well as with ScreamingFrog incase they see this and are able to prevent this from happening to someone else.
Part of my regular SEO routine with client sites is running Screaming Frog and checking for issues with response codes. This time I encountered something I had never seen before. Two internal URLs on the clients site were returning both 200 OK and 404 Not Found codes! I found that this clients necklaces page and their opal rings page were returning multiple statuses. This of course makes no sense. A page can not be both found and not found, yet this is exactly what I was seeing! Screenshot below.
Of course this drove me crazy, but I like a good mystery every now and then!
My investigation into this started with a manual check of the two URLs in question to see if the page was in fact loading, and of course both loaded just fine. My ‘Link Redirect Trace‘ Chrome extension confirmed as well that both returned a 200 OK status.
I next checked a couple random “http header checker” tools that I found with Google. All tools also returned a 200 OK status. Finally I dove into Google Search Console and the Inspect URL Tool. This tool returned nothing out of the ordinary!
It seemed that the issue was either related to a problem with Screaming Frog OR a really odd problem with the website itself.
My next step (which in hindsight should have been my first step!) was to trace how Screaming Frog was finding this 404 error in the first place. By the image above you can see that for the one /necklaces URL they located 682 links to the URL version with the 200 OK status, but only 4 inlinks to the version they were saying was a 404! So I decided to check that out. Why didn’t I look there first! In screaming frog you can click on the address and see more information such as URL details, inlinks and more. When I clicked on the first link I stumbled onto my answer. The URL Encoded Address infact had a %0A appended to the end of it! It seemed that both of these two pages had some blog post links that for some reason were incorrect and contained this extra code.
Had I looked here first I would have found my solution. The actual URL that was returning the 404 error was different than what was originally presented! I found the real problem and it was something that could be easily fixed! Looking at the inlinks tab I was able to easily find the pages that had the links containing this code and correct them right at their source. Problem solved!
What all this goes to show is that not all tools are perfect. Screaming Frog is awesome. It is a critical tool in my SEO arsenal that I use every single day. I would be lost without it or something like it. But sometimes there is room for improvement in even the best tools. Had the address in the initial report contained this %0A, then everything would have been perfectly clear from the very start. Instead I went down this small rabbit hole trying to trouble shoot what was happening.
To be fair, had I simply clicked on “URL Details” first I would have discovered the source sooner, but hey, what can I say, I’m human!