How long does it take for Google to recognize 301s?

Or Better Yet - 

It's been over a year and Google still doesn't have the new URLs in the Index

Just over a year ago, I started working on this website that had over 900k top level domain files. We changed the structure of the URLs to a more organized hierarchy. The pages content changed slightly, but most importantly instead of all of the site's pages residing directly under the main domain (Let's use a computer broad to longtail term structure for example - like domain.com/computer.html and domain.com/laptop-computer.html and domain.com/500gb-laptop-computer.html) we changed them to a more representative hierarchy directory to file structure (example - domain.com/computer/ to domain.com/computer/laptop-computer/ then domain.com/computer/laptop-computer/500gb.html).

Why the URL Hierarchy?

The quick and simple explanation as to why we did this is that while URLs are fairly dynamic these days,  the bots like to see and understand how a website is organized on a server. Remember your old school folder and file structure back when sites were in html were built?  The URLs you have today should represent that organize file structure as much as possible. I cover this in my SEO Workshop  (slide 23)- but I also found a pretty good article that explains the hierarchy relatively simply and quickly.

The process in setting the 301

The Since the entire 27+ million pages on the site were mostly files located directly under the main domain, it was difficult to understand what pages fit under what category so that we could organize them. I went to our keyword analysis and bucketed each focus term out and then organized the correlating URLs to fit within that bucket. Once that was done, I worked with the Developers to pull the naming from the database (dynamically) into the directory and file structure that fit the buckets. Some of the keywords I knew I eventually wanted to build out with supporting pages, so those got directory levels instead of page levels for future optimization (and limit more 301 redirecting later on).

I mention a bit about breaking the site up into sections for analytics purposes in my previous post "SEO Issues - is it Penguin? Is it Panda? or is it me?" under "Figuring out what was hit by Penguin". The "video" to the left is a quick (and very raw) animation to help explain exactly what we did. Now that the site was organized it not only helps the bots understand the structure, but helps us understand what sections bring in what SEO traffic in Google Analytics.

How Long Does it Take Google to See New URL via 301 Redirect?

This whole undertaking was completed over a course of 2-3 months starting in June 2012 (last year) and finished up with the last of the redesigns and URL changes in August with one last directory change (no redesigned pages) in January of this year (2013). The most important ones are still are showing 550,000 pages in Google's Index (11 months later):

As I Google to see if others have a solution for speeding up the indexing of these old URLs, or if even if anyone has had the same problem I found a lot of questions in various forums (both reliable and unreliable) but no real articles, blog posts, or anything from reputable SEO's. The most common answer in the forums is to just "wait". It's, of course, what I tell others when they ask me "Be patient, Google will eventually hit those pages again and recognize that they have changed then correct the index then." But after nearly a year and so many pages, this is getting ridiculous.

I spoke with my friend (and SEO mentor) Bruce Clay who came back with the suggestion to add an .xml sitemap and submit it to Google with the old URLs we want removed.

It was kinda making sense that because those old URLs are no longer linked to, and there are so many, that Google wasn't crawling them as much anymore. They are just sitting there in the index - and not getting "updated"

Unfortunately getting a sitemap added is not an easy feat. I would have to define the strategy, present it to the powers that be with data to backup the success metrics in order to get the project prioritized. With so many other initiatives needed for SEO, all of which were more important and affect the business in a positive way, it was in my best interest to keep pushing those and not deal with the sitemap.

My work around, though, was about as black hat as I would get (Matt Cutts if you are reading this, I apologize and throw myself at your mercy, but it had to be done). One weekend over a month ago, I grabbed one of my many impulsive purchased domains and quickly set up hosting and an old school html site that consisted of one page. I then exported all of the links on the Google "site:" search through a Firefox plugin called SEOquake that exports the results into a csv file. It's not the prettiest, and there was a lot of work still needed to get to just the URLs, but it was the best solution I could find (note: if any SEO reading this knows of an easier way to do this - please add to the comments for prosperity). I then parsed out the parameters in the URLs in a separate document and used those as the anchor text for each URL. Finally, using excel I then concatenated the URLs and parameters (that were now anchor text) into an html href string.

Then copying and pasting the "string" column into the html code, the page looked like:

The page wasn't the prettiest, and it had thousands of links (the above is just an example) so it was bad all around, but the point was to get those links crawled by Google.

Of course every SEO knows that you can't just build a website and expect it to immediately get crawled - right? 

So I set it up in Google Webmaster Tools and submitted the page to the the index:

I even got more fancy to ensure Google would see the page and crawl all of those old URLs and +1'd it on Google. 

Did it work?

I checked the URLs this evening to see how many Google is seeing and the number has dropped from 550,000 to now only 175.

I took the domain off of the server, and now have it parked elsewhere (back where it belongs) and removed the webmaster tools account. All traces of it ever existing are now gone, and the small moment of my attempt to get those URLs removed has passed.

Thank For the Advice Jenn - Now I'm Going to Try This!

If you have come across this post and you need to do something similar - I'm going to put the same disclaimer they do when a very dangerous stunt is performed in commercials. 

Do not attempt this at home - this stunt was performed by a trained professional on a closed course.

So, don't go adding a bunch of links to a random domain thinking that your attempt just weeks ago to 301 pages isn't working. The links on the external domain were too many for the domain and page, and were extremely spammy. In addition, all those links pointing to pages that were redirecting and were supposed to pass value to the new URLs, now had many spammy links pointing to them from a very spammy domain. If left up too long, or not done correctly, it could actually cause more damage than ever helping.

If you have any questions, or feel you need to try this same strategy, please don't hesitate to contact me. I'm here to help, and want to ensure that your website has considered all possible options before attempting any such trickery.

Some Helpful Links on the Very Subject:

A Year in the Life of 14 Million 301 Redirects - by Adam Sherk

Conference Adventures Part..: Dernier Versement (final installment)

All-in-all the event went as well as could be expected. Financially, I am left with having to put up ~$1,000 to still cover the expenses of the event. The total cost for the event ran me ~$9,000.00 (slightly under) with ~$8,500 (give or take a few hundred) in registrations and no money from sponsors (sponsors this year received their sponsorships in exchange for distributing swag, or offering award prizes).

Why do I do This?

When I started writing my first post documenting the organizing of this conference, I mentioned in my post titled "Conference Adventures Part Un - 18 Days 'til"

Planning and organizing a conference is not an easy feat by any means, and I often ask myself why I keep doing it...

Which is now bringing me back to asking myself once again why do I keep doing it? I honestly can't find a one sentence answer that rationalizes the time, effort, stress, financial strain, or the pressure it all puts on me.

I'm sure you're probably reading frustration in my post here, and probably in my previous posts, but know this - the last few days I have been coming up with ideas, excited that registrations are already coming in, and I almost have a full speaker list for EmMeCon Seattle that isn't happening until June. I have even started the groundwork of organizing a 2 day Search and Social Series, and/or an SEOGoddess 4 hour SEO Workshop in April so that I don't have to wait too long to do another event).

So why this odd addiction to holding events? I got to talking with a few of the attendees and speakers at this last EmMeCon and the one word that kept popping out of my mouth (and repeated back to me) was integrity. My events, though small, and not highly profitable, still have integrity. I haven't sold out with mindless topics, uninspiring speakers, and selling tickets at an insanely low cost just to appease sponsors with more attendees. Even after all these years of organizing conferences, I still constantly remind myself through the entire process what it is that I wanted to do when I started these events. With EmMeCon, I want people to gain inspiration from the amazing people I have been lucky enough to have access to. People like David Evans Ph.D. who has taught at the University of Washington educating Masters students on Psychographic Segmentation and the importance of understanding the minds of the users they are marketing to. Or Gillian Muessig who has guided not one, but 3 children into thriving adults and in the process molded 2 of them into very successful and inspiring SEO's. The list of inspirational individuals that I feel privileged at the ability to pick their brains, gain inspiration from, or have been helped by in some way is a mile long, and ever growing.

Because of this desire to share, I take careful consideration into the details of every event I organize. It may run me rampant and I get flustered and exhausted from it, but reading the tweets, hearing the feedback, and knowing that at least one person (if not many) has gained inspiration from the event is what I deem as success. 

On Thursday night we wrapped up the event with a packed house for the Meetup Group organized by Chase McMichael (CEO of Infinigraph) the tweets were still coming through strong, and the room was full of questions and discussions. After the Meetup wrapped I began packing things up, and while exhausted I was bouncing around with excitement as the folks that lagged behind thanked me for putting on such a great event, and asked me all sorts of questions on how I got into this, how I came up with the idea for the event, and even asked more about the event in Seattle.

It's that feedback that I get that keeps me going. 

I promise that I won't "sell out" and start making this about the money, I won't ever forget what this event (or any of my events) are there to accomplish, and I promise never to lose the integrity that I still hold onto.

If I do - someone please take me out back and put me out of my misery...?

Conference Adventures Part..: en Cours (in progress)

It's the Friday after the event is finished, and I'm finally getting a chance to sit down and relax long enough to to write a summary of how the event went during the week.

How did the event go?

I have been asked the same question several times throughout the course of the week. The first morning of the first day there were only a handful of people in the audience. I had spent the evening before setting up the room and the audio visual until midnight and didn't get to sleep until well after 2am. With just 3 hours of sleep (I spent another 1.5 hours typing up notes for the next day)  I walked up on stage welcomed everyone, explaining a bit about the conference, dove into announcements, and then introduced our first mini-Keynote Irene Koehler. I am always worried about the first speaker of the event. The speaker and the subject chosen is what sets the tone for the remainder of the event. Irene is a seasoned and very respected speaker. She is an expert in her field, and knows more about social marketing on Linkedin and anyone I know.  I think Benj's tweet summarizes the one word used during the remainder of the event that Irene provided us all: "Stalking"

The room filled up as people trailed in throughout the day. The VIP room was a hit, and no one tried to sneak in for lunch. I didn't hear a single complaint, and none of the volunteers said anything to me about anyone being upset. I have to give a special shout out to Tracy, Lydia, Brenda, and my daughter Katie for all their help on the first day. Especially Tracy who's car was randomly dinged by a crazy driver in the parking garage. Poor Tracy was so distraught that she couldn't come back the remainder of the event. 

Everything went so smoothly the first day that one by one, my volunteers said "It looks like you have everything covered, so I won't be coming tomorrow." or something similar. The evening of the event was one of the best nights after a conference I have ever had. The Speaker's Dinner was great (despite our food coming 85 min after arriving and my steak being very, very well done). Bill Leake and Aaron Kronis blew through the wine, and entertained my daughter while I moved onto my Birthday party with my co-workers and a few other friends. All-in-all Day one is going into my record book as a perfect first conference day.

As the sun rose up the next day I was already busy getting ready to head over to the hotel. My daughter got ready for school but then crashed and said she felt very ill. I told her to stay home, and went onto the event. I struggled with the problem of how to check people in when I needed to be on stage most of the day. I asked our hotel rep and she sent one of their staff, but the staff that arrived was clearly very upset that she had to be there. I asked Stephan (who I work with and had a pass to the event) if he would help out by checking people in. We moved the table into the room so he could watch the talks and hand people their badges as they trailed in. It worked out great!

At some point in the middle of the day I was hit by a ton of bricks and could barely get my energy up enough to even introduce the speakers. I hadn't had anything to drink the night before, so had no idea why I was feeling so ill. Hoping it was stress I powered through. The Meetup organizer showed up and I talked to her about getting set up for the talks that evening and then leaving. I asked Aaron Kronis to help her out that evening, and I eventually went home to bed. I really don't remember much of that day - I remember waving to speakers and saying "just introduce yourself" and checking the video camera, taking pictures, and then sitting down to rest until the next speaker went up. 

The last day was the quietest day I have ever experienced at any of my events. Even my workshops had a better turnout than this conference. Was it the free passes that kept people from wanting to come to the event? If they all pay, then will they show up and stay the whole time? Bill and I talked about where this conference should be headed. Cutting it down to 2 days, holding it just in Seattle, and look into other cities (like NY or Vegas). The conference has an "eclectic" (as Bill called it) array of speakers and topics and as a result we get such a mix of attendees it's quite refreshing. It took over 10 years for TED to finally gain some notoriety, perhaps this event is on the same path.