SEO Issues - is it Penguin? Is it Panda? or is it me?

The following story is one that has been several months in the making. It's one that I have lived through one too many times as an SEO, and it is one that I am sure other SEO's have faced. I fought with the thought of writing this for fear that someone from the company might read it and get angry that the story is told. But, it's something I think that not only people out there could learn from, but speaks to so many others in this industry to show them that they are not alone.

It's long, it's a bit technical (I tried to keep it simple), and it has some personal frustrations laid out in words. My only hope is that you get value out of reading this as much as living it has made me a better person (or well, a better SEO).

It Begins

I started working on this website's SEO in May 2012 at which time I was told the site's traffic was declining due to Panda updates. In February of 2012 the traffic from SEO was the best they had ever seen, but soon after that there was a steady decline.

Traffic from February 2012 - May 2012

Before digging into any possible SEO issues, I first checked the Google Trends to ensure that the decline isn't searcher related. Often times a drop in traffic could just mean that users aren't searching for the terms the website is ranking for as they were in the past.

Top Key Terms in Google Trends

Looking at the same time frame as the traffic data, I noticed an increase in searches for the top 3 terms the website ranked for, and there appeared to be a decline around the same time from March to April that the traffic was reflecting. But there was a drop in the website's traffic in April from the 23rd to the 24th and then significantly on the 25th. The website I was working on had two SEO's already working on it: an agency and a consultant. Both had already done a numerous amount of research and some work to get the website on track. Both were stressing that the drop in traffic was due to the Panda updates by Google. I looked at SEOmoz's Google Algorithm Change History and found an update to Google's Panda on April 19th and an update to Penguin on April 24th. Given that the traffic significantly dropped on the 24th my best guess is that it was possibly Penguin related, but still needed further exploration.

Figuring Out What Was Hit by Penguin.

The site is/was broken up into sections by keyword focus. At one point, I could tell that someone really had a good head on their shoulders for SEO, but the strategy that was used was outdated. Perhaps the site was originally optimized several years before, and it just needs some cleanup now to bring it up to 2012's optimization standards. So, understanding Penguin and identifying which part of the site was driving the bulk of the organic traffic was going to be my next step in solving this mystery. Once I understood why, and where, then I could start to establish a what to do to solve the problem.

I broke the site traffic report by sections as best I could in Google Analytics. There was a bit of a struggle as all of the pages of the site resided on the main domain. Without a hierarchy in place, breaking out the sections had to be accomplished with a custom report and a head matching for landing pages. I hadn't had to do this before, so the agency that was working with the site already helped build the first report, and I began building out the other reports from there.

Section 1 over 72% of traffic

Just focusing on April and May I created a Dashboard in Google Analytics focusing on organic Traffic and identifying the sections of the site. Looking at the different sections - Section 1 was the bulk of the traffic with over 72% and Section 2 coming in second with just over 15%. Subs of Section 3 and other one-off pages make up the difference.

Both Section 1 and Section 2 dropped off after the April 24th date, so clearly they were the bulk of what was pulling the overall traffic numbers down. Since Section 1 was the majority of the traffic, I presented to the executive responsible for the site that we address any issues with that page first.

Actual screenshot of Section 1 presented

I took all of the research from the agency and consultant and we quickly reworked the pages to represent a hierarchy in the URL structure, and cleaned up any issues from the outdated optimization that was done.

Soon after Section 1 was addressed, we did the same with Section 2, and then worked on Section 3 (and sub pages, rolling them up into a solid section) and then added a few pages to grab any new opportunity.

Not Quite As  Easy as it Looks

The projects were launched in increments - first URL hierarchy fix to Section 1 and then the page redesign. Next was a full launch of URL fixes and page redesign to Section 2, and then lastly Section 3 and the new Section 4.

Section 1 - Section 2- Section 3 Launch Dates and Organic Traffic

Soon after Section 1 was launched traffic started declining rapidly. I was asked several times why traffic was getting worse, and I started digging some more. Every time I looked at the Impressions of the new URLs from Section 1 they weren't getting any traction, but the previous URLs were still.  I began looking at the history of the website, trying to find out why it was doing so well at one point, but was not doing well at that time. One of the things I noticed was that there was a lack of priority linking to these pages, but at some point there were links to some of them individually from the homepage. Google matches a hierarchy of pages to a directory structure that links are presented on a site. This site had every page on the first level, and linking to those pages from the homepage, which was telling Google that every page was the most important page. It worked at one time, but as Google has been rolling out their 2012 updates these pages were getting hit, and those links on the homepage weren't there anymore. Before the launch of Section 2, I had them put links to the main directory for each section on the homepage. The links would tell the search engines that these are important pages of the website, but not be so obnoxious with a dozen or more links on the homepage to discourage users (avoiding the appearance of spamminess).

But - even after adding the links to the homepage, the traffic to those pages was still declining. Pressure was put on me to figure out what was wrong. In addition, accusations were flying that I single-handedly ruined the SEO for the site, I spent every waking hour looking at reports, and trying to figure out what was going on. I consulted friends in the industry, and read every article I could find to figure out what Panda or Penguin updates were affecting these pages.

Then it hit me - just as the links to these sections would help them get recognized as important pages, so were the other pages that were being linked to from the homepage. In fact a set of them linked to the website's search results with queries attached to them mimicking pages, but showing search results. On those search results pages, there were over 200 links with multiple (we're talking hundreds - possibly thousands) combinations of parameters. The bots were coming to the homepage, going to the links to the search results pages, and then getting stuck in this vortex of links and combinations of parameter generating URLs - not allowing any crawl time for the pages that once were getting rankings. This also explains why the new URLs weren't showing very many impressions in the Webmaster Tools Data - those pages just weren't getting crawled.

There was a project underway that would solve the many links on the search pages, and there was also talk of using ajax to show the results. When this project would launch, the bots would go to the URL from the homepage, but would then essential not go much further. With this project a few months out, I made the case to add the search page to robots.txt to allow the bots to then recognize the Sections as important pages. After several weeks of attempting to convince the powers that be, the URL was eventually added to the robots.txt file.

Immediately after the search page was added to the robots.txt Google Webmaster tools presented me with a warning:

Warning in Webmaster Tools

In most cases, a warning from Google should never be taken lightly, but in this case it was exactly what I wanted. In fact it proved to me that my theory was correct, and that the site was hopefully headed down the right path.

Panic, Questioning, and a Third Party

As with every up in the SEO world, there must be a down. Soon after the search result page was added to the robots.txt the organic traffic to the site dropped, and continued to drop. Throughout those grueling three months there were several Google Panda and Penguin updates. I documented each and every one of them in Google Analytics, and continued to answer questions, gathering data, and dealing with being under close scrutiny that the work I was doing was complete BS.

Organic Traffic from September 2012 - November 2012

I sat in numerous meetings, some of which I walked out crying (I'm not afraid to admit it), being questioned about the road I had taken and why we weren't seeing results. There were people within the company recommending that they roll the pages back to where they were before, and even changing the URLs. I fought hard that they don't touch a thing. I sent an article posted on Search Engine Land by Barry Schwartz citing Google's patent that "tricks" search spammers.

The patent states:

When a spammer tries to positively influence a document’s rank through rank-modifying spamming, the spammer may be perplexed by the rank assigned by a rank transition function consistent with the principles of the invention, such as the ones described above. For example, the initial response to the spammer’s changes may cause the document’s rank to be negatively influenced rather than positively influenced. Unexpected results are bound to elicit a response from a spammer, particularly if their client is upset with the results. In response to negative results, the spammer may remove the changes and, thereby render the long-term impact on the document’s rank zero. Alternatively or additionally, it may take an unknown (possibly variable) amount of time to see positive (or expected) results in response to the spammer’s changes. In response to delayed results, the spammer may perform additional changes in an attempt to positively (or more positively) influence the document’s rank. In either event, these further spammer-initiated changes may assist in identifying signs of rank-modifying spamming.

 But the article and my please fell on deaf ears...

It had gotten so heated and there was fear that nothing was being done while traffic was significantly declining that the company brought in yet another SEO consultant to look at the site objectively.

Just as the consultant was starting his audit, and the traffic hit the lowest I ever thought it could possibly go, the next day traffic went up. The last week in November (roughly 3 months after we blocked the search result page) I saw an increase in traffic in Google Analytics to Section 1:

Section 1 Organic Traffic

I quickly pulled up my report to check the Section's impressions from the Webmaster Tools data, and there was a significant increase as well:

Section 1 Impressions from Webmaster Tools Data

On December 3, 2012 I logged into Webmaster Tools and saw that the warning had gone away:

It was the "halleluiah" moment that every SEO dreams of, and very few get. All the work I had done, the fighting for what I believed in, it all finally paid off.

To this day traffic continues to increase - we can now focus on some of the cleanup still left to do, and then onto projects that will attract new opportunity.

Organic Traffic from November 2012 - January 17, 2013 (day before this post is written)

Quick Note: 
I forgot to mention a post I wrote months ago while going through all of this - SEO - Panda and the Penguins. It helps to give a bit of perspective of some of the linking stuff I didn't get into in this post. 

Conference Adventures Part Quatre - Working with the Hotel

It's now the evening of the same day since my last post (19 hours later). I checked a lot of items off of my list with regards to the logistics of the event, so I'm feeling pretty confident in how smoothly this is going to go (famous last words right?). I gotta say, my favorite part so far of the whole event... I booked a photographer to bring a photo booth to the party and asked for 80's props to go with my little theme.

After every up, there must be a down... This afternoon I felt an illness coming on. In a panic I ran to the Pharmacy across the street from the office and grabbed Zicam, a spray to protect from illness, Vitamin C, and Vitamin B-12 tablets. If I get sick this close to the event, I am screwed...

13 Days Until the Event

Status Update: 
Speakers: All Speakers confirmed and in the agenda
Speaker's Presentations (PPT or other): 0
Agenda: Full!
Logistics: F&B Coordinated, layout of room and AV discussed - Video Recordings, Meetup event details, Party details, and Photographer all checked off (WooHoo!)
Marketing/Promotion: I managed to get all of the speakers on the website (with the exception of Aaron Kronis, but he doesn't' count anyways) and the Agenda is looking pretty complete. I do have some room if someone comes in last minute, but we can go on with it as is as well.  I still need to get speaker emails set up as well as the marketing emails; schedule tweets, Facebook updates, get a Google+ event update schedule figured out (can't schedule those - have to do them manually), update Lanyrd,  finish last years videos, print materials for event (signs, schedules, promotion of next events) and oh so much more.
Master of Ceremonies: Still Me
Volunteers: 1 (Lydia is going to check people in - Yay!  Thank you Lydia) - I still need a Volunteer to help with the speaker's transition and man the video camera.

Coordinating with the Hotel

While the most feared part of organizing an event for me in the past (simply because I had NO idea what I was doing), it has now become one of my favorites. The first time I walked through the plan for F&B and AV with the event coordinator for the resort in Hawaii I got screwed. $20k in the hole later, and after a bail out from my Father to keep Search and Social Hawaii from being canceled, I learned from my mistakes, and have not made the same mistake since. 

I met with our coordinator this afternoon after getting off work. Now that the agenda is set, we went through day by day and discussed numbers for each meal and service. After the disaster of last year at the Hotel Kabuki, I am confident that the Sofitel will have good food, an impressive room (with heat), the coffee will get served (and won't run out), the mic will work for every talk, and the staff will be there if I need anything (did I miss anything?).

After talking with Lisa, I met with the Audio Visual guy. There were a couple of emails going back and forth, and they drafted up an estimate based on that. I'm not going to divulge the total, but I gotta say, I was pretty upset with what they came up with. I now realize why Lisa, and the IT guy both started off (before showing me the numbers) that everything is negotiable, and if needed we can cut some things out.

Well... They have about 90% of it to cut out before I can even come close to being able to afford the bill...

Conference Adventures Part Trois - 2 Weeks 'til

It's 1:30 am and after a couple of hours of sleep I woke up only to find I can't stop thinking about what I still need to do. On top of organizing the conference, I still have my full time job and I finally found a 2 bedroom apartment for my daughter and I to upgrade from the little studio we have been sharing here in San Francisco. We're moving in on February 2nd (right after the conference). There is still a lot to do... Though I tend to love when my life gets like this. So many tasks ahead leading up to a big goal that may seem overwhelming - but just take each day one by one and check the items off the list that need to get done, and it will all be ok. Not to mention that all those items checked off at the end of the day make me feel like I really accomplished something.

14 Days Until the Event

Status Update: 
Speakers: 28 out of 29 speakers confirmed
Speaker's Presentations (PPT or other): 0
Agenda: 1 spots left to fill (almost there)
Logistics: Still need to coordinate F&B (meeting with the hotel on the way home today), layout of room, AV (emails back and forth with the Hotel's IT guys), Video Recordings, Meetup event details, Party details, and Photographer.
Marketing/Promotion: Started on Daily Emails but haven't gotten any further - still need to get speaker emails set up as well as the marketing emails; schedule tweets, Facebook updates, get a Google+ event update schedule figured out (can't schedule those - have to do them manually), update Lanyrd,  finish last years videos, print materials for event (signs, schedules, promotion of next events) and oh so much more.
Master of Ceremonies: Still Me
Volunteers: 1 (Lydia is going to check people in - Yay!  Thank you Lydia)

Social Media Encouragement

I saw this tweet come through on Sunday and it reminds me how buzz like this around the conference makes all the hard work worth it.

I know, to you it seems like just a simple question, but to me it's so much more than that. You see, when you market an event, you don't get much feedback on how successful your marketing is until you see registrations come through. The conference purchasing decision is a process, and tracking that process is difficult - especially on a tight budget like mine (that's no budget really). It goes something like this:

1. The user saw an ad, a post on Facebook, a Tweet, or some form of notification about the event that peaked their interest.
2. The user visits the website for the event and looks at who is speaking, what the agenda looks likes, and generally reads up on the event.
3. The user Google's the event to see if there is more about the event or if anyone is saying anything about it. In some cases, the user might Tweet or Facebook and ask their "friends" if they heave heard of the event, or are going.
4. The user then goes back to work (or just generally goes about their day)
5. Days, or weeks pass and the user sees another mention of the event (ad, Facebook, Twitter, etc) and then talks to their boss (if they need to get approval) or checks their schedule, with spouse, and.or budget (either way, they need to get some sort of approval).
6. The user then goes back to work or day again.
7. Days, or weeks pass and the user remembers the event (or maybe might see another reminder). The user will then ask the boss, or consult their spouse, schedule, etc one more time.
8. The user then gathers whatever they need to justify the spend. Whether it be their boss, spouse, or own personal budget. This process could include a simple breakdown of costs, or it could involve comparing it to other similar events. It can also include items on the agenda that might help state their case (either to the boss, spouse, or themselves).
9. Once the event has been approved, the user will go about planning their trip (or if it is local, getting ready to be out of the office for a few days). The flight will get booked, but perhaps not the hotel quite yet, and certainly not registering for the event.
10. Days, perhaps weeks might pass and the user has let everyone around them know they are going. Facebook posts, Tweets, co-workers, Family, Friends, etc. Everyone knows that the user is going to attend the event... but the event itself...
11. 1-2 weeks prior to the event the user will realize they haven't booked their hotel, or registered for the event. At this point, the user will go to the website and register for the event, then find the hotel information and book their stay. If they are local and there is no stay to book, they usually register a few days before the event.

So, you see there is a lot from the awareness to the commitment that has to happen, and tracking that process is a very tough thing to do. I have my ways, don't get me wrong - I wouldn't be a marketer if I didn't find ways to track the process as much as possible. But numbers don't say nearly as much as seeing a Tweet come through that your event seems to be THE event to attend.