This article came to me while writing the first part of my Deja Vu: Changes in the Internet (or Lack Thereof) Series at Search Engine People. It got me to thinking about some things that also haven't changed - but probably should...
Back before the search engines first started messing around with the notion of crawling dynamically generated sites, we needed to use URL Rewriting rules in order to trick the search engines into thinking it was a static site. As time went on, the search engines got much better at it. I purposely left the URL's in my old Rock-N-Reel movie site dynamic so that I could see the positives and negatives. By the time I took it offline toward the end of 2005, it was well represented in all the search engines and it ranked in the top 5 for many movies and celebrities.
Four more years have gone by and people are still using URL Rewrite rules to make their sites search engine friendly. Why is this? Do we really need to do this anymore? Are there better ways to do this? And... most importantly, if this is the best way to do it still, why? And how can we change that?
The Nature of The URL
Before we can really take a look at why we are using URL Rewrites, we need to look at the nature of a URL in respects to how a search engine looks at it. The truth of the matter is... a well formed URL is as useful for your users as it is for the search engines. Even though this analogy is going to become outdated within the next 5-10 years, I'm going to use it because it's a good one and it makes sense for here and now. (The reason this will be outdated is that phone numbers are now attached to "people" and not locations nowadays... but at least for the moment, we all still understand the principles, so the analogy is sound.)
Think of a telephone number. 555-666-7777. Just by looking at a phone number, you can get certain clues about the person and where they live. The area code 555 tells us what state the person lives in - and in many cases, it will give clues as to the metropolitan area of that state. The exchange 666 tells us which town the person lives in and it can give us some clues as to how long the person has lived there as new exchanges were added. And then the 7777 is the thing that uniquely identifies that phone within the area and exchange.
A well formed URL is no different. Look at: site.com/about/employees/bob-smith.html
Just by looking at the URL you can tell that Bob Smith is an employee of this company. We can make a few more assumptions about this, also. The default page (index.html or index.php or whatever) is very likely going to be a list of employees. And any other page that we find in "/about/employees/" is also going to be an information page about someone who works there.
We could do the same type of thing with products ("/products/widgets/blue-denim.html") or news ("/articles/SEO/dynamic-urls.html") or anything else. A well formed URL will not only give you the location of the page, but it gives you some general information about how it relates to other things. Any other page in the "/articles/SEO/" directory, above, is going to be some sort of article about SEO.
Reasons For URL Rewrites
Once in a while, I hear the claim that the reason we still use URL Rewrite schemes on our dynamically generated web sites such as CMS (Content Management Systems), Blogs, and so on, is because search engines are no good at spidering URLs with query strings and other variables in them. For the first few months of Google's maiden voyage into crawling dynamic URLs, this might have been true. Since then, though, the answer for this is really that the applications themselves are developed in a way that generate URLs that aren't search engine friendly.
This is the one valid reason for URL rewrites. Almost all of the applications out there are developed without URL formatting in mind. They use numbers to designate the unique field for a page (e.g. "article=275" instead of "article=dynamic-urls") and all of the pages in the site are generated off a single "index.php" instead of sub pages for each function, so there are no clues in the URL itself. Plus, the applications don't do any good sorting of the query string values. The application might spit out "index.php?article=5&category=4" one time and the next time it will spit out "index.php?category=4&article=5". This is bad because, to a search engine, these two pages are not the same thing (though there is some evidence that Google is getting much better at spotting the fact that they are the same. It's difficult to prove, though, since there are so few sites out there that leave raw querystrings in their URLs).
What's Wrong With URL Rewrites
The number one bad things about URL rewrites is that they are a pain in the ass. Plain and simple. There are other issues, too, though. It's a whole new set of processes added on top of the internal workings of the dynamic web site, so it adds a certain amount of load on the server. For most people this doesn't make a huge difference. For very busy sites, it very well can.
Another problem with URL Rewrites is that they are an afterthought. SEO is about a strong foundation and consistent organization. If the web application we are doing isn't creating natural URLs that are SEO friendly and useful in giving us visual clues as to how this page relates to other pages in the structure, it can be very difficult - if not impossible - to create a scheme that is going to take this garbage and turn it into something good. It'll make things "better" - but it doesn't really make things right.
It's Time For Applications To Be Developed With SEO In Mind
Search engines accessing dynamic web sites isn't new... it's 8 years old, at least. There are a few applications out there that are structured in a way that they generate good rewritten URLs right out of the gate, but the vast majority of them don't. While it may be the case that you will want your application to do some internal URL rewriting to get the variables positioned in a good way, there is no reason why we developers can't start thinking about how people are going to use our application and get it structured properly in the first place.
Rather than: index.php?section=4&category=5&article=245 we could easily make an application that uses: index.php?section=news&category=seo&article=url-rewrites.
If you were to put some good sorting routines in there as your link generation code ensures that those are always in the same order, you (and the people who use your applications) wouldn't really need to do any URL rewriting in the first place. And, maybe we would still want to do so and make the URL look like "site.com/news/seo/url-rewrites/" or something along those lines - but here's the thing...
If you develop the application so the "number" is the unique identifier, the URL rewrite system needs to add extra overhead to ensure that the keyword rich text that it converts the url into is unique as well.
When it really comes down to it, an application that is developed with site structure and URL formatting and information architecture all in mind at the foundation - a dynamic URL can be even more beneficial than a rewritten one. In the example above, "/news/" is just that. But "?section=news" goes one step further and gives specific context to the "news" - it is a section of the site, it's not just a word that we need to try to infer context from.
So, if you are considering creating a new dynamic web application - a new killer blog, a new social networking application, or whatever it is... take some time and think about the URL structure during the development process. Build it all right into the foundation of your system. As it is now, we are taking an application foundation that is 8+ years out of date and keep adding a heavier and heavier house on top of it. Build us a good, scalable foundation that takes URL format into consideration and you'll be almost a decade ahead of nearly all of your competition in this regard.
URL format is one of the top five considerations SEO experts look at when optimizing a web site - it is something that simply must be done to get it rank. For most applications out there, we need to stack something else on top of the foundation to get this done, or we need to recommend that the client uses another application.
URL format is a Usability concern - a good URL help not only the search engines get an idea of where they are in the web site, but it also can be very handy for users. You can add breadcrumbs to the system so they can see it there, but those breadcrumbs should also be reflected in the URL as well. (Hint: I have often piggybacked off the breadcrumbs system of applications when making URL rewrite schemes... if you have a breadcrumb system in your application, you already have something in there that has generally good formatting - you just need to ensure that it's unique values.)
If developers start to take things like this into consideration and get them into the core routines of their applications rather than having it as an add-on (even if that add-on comes standard with the application) we people on the "front end" side of the web industry will start to take notice. We will recommend these products to our clients. We will use them in our own work. We will write articles about your applications. We will climb the highest mountains and sing the glorious praises of your application and how it has helped everyone to achieve their goals.
Sure, we'll probably always have good reasons to use some sort of URL rewrite in our programs. But if your application can do that natively - as part of how it stores and presents information in a logical way for users and search engines alike, then application technology will finally take a huge step toward keeping up with search engine technology.
Do you have, or know of, a great web application that already has a solid foundation for creating friendly URLs or at least has a URL rewrite function that doesn't make us think for 6 hours on how to create a useful scheme? Let me know!
-
|2009-10-16 10:29:47 Stockbridge Truslow
(It would help if I got the notification set up properly to know when comments are posted here, huh? lol They are fixed now... sorry about that...)
Keywords and density are important in the URL - but not really the "end-all-be-all". Back when I had my movie site I used "movie=45" and quite often outranked many pages that had the movie name in the URL. I'm certain that this was because it was easy for Google (even back in its early days of crawling dynamic sites) that to look at my url: /moviedata.asp?movie=45" and before it even crawls the page it can say, "Every other time I see this URL format, the page is about some movie - when I spider this page, I will find out what movie it is about." It hits the page and it knows, without question, and without keywords in the url that this page is about the movie "Dirty Dancing" (or whatever it is).
So, while it can help, you can capitalize on other things with your URLs as well. Some URL rewrite scheme...
I just read your piece at search engine people, and added you on Twitter ( I am "searchjunkie"). Welcome back to the fabulous world of SEO!
Hey, great piece that covered a lot on URL rewrites. And yes, they are a nightmare. They are hard to "spec" out and determine a cost on until one starts to really get into the guts of a system too. This makes it hard for SEOs to determine a price. You have to charge by the hour on these things.
I would also argue that search engines love to see the keywords in the URL and that once could think of it as a scenario of keyword density of the URL. In other words I am kind of arguing against your point that have "?section=news" in the URL is better than having "/news/..." And I would argue that based on the fact that the URL will have a higher KW density with a rewritten URL. I have no hard data to back that up but it seems logical to me.
Any thoughts?