Google Search Advocate, John Mueller, was the guest in this 400th episode of SEO 101 on WebmasterRadio.FM. The show started with a solemn nod to the loss of a fellow SEO, Hamlet Batista. Next, John Carcutt and Ross Dunn asked several questions of John Mueller including how to bounce back from the negative repercussions of a Core Algorithm Update, whether inferred links are part of Google’s ranking algorithm, how to improve the overall indexation of pages that were crawled but not indexed, and much more.
Noteworthy links from this episode:
- Lily Ray’s tribute to Hamlet on Search Engine Journal: Here
- Hamlet Batista’s GoFundMe
- Hamlet’s RankSense Manifesto and his business – RankSense
- Info on Search Off the Record podcast
- Roger Montti’s article on Google’s John Mueller Praises Digital PR
- Digital PR Tweet
- Article by Rand Fishkin: Inferred Links Will Replace the Link Graph
- SEO 101’s Facebook Group
Key Quotes from John Mueller in this Episode
Regarding being stuck in purgatory with dropped rankings and the long wait after fixing the issues between Google’s Core algorithm updates:
John Mueller: It’s annoying if you’re stuck in between. I do get that it’s annoying, but it’s also something where I would not just make some fixes and then wait, but rather, it’s like, you can continue working on your website and continue improving things over time. It’s not something where I’d really tell people, you shouldn’t do anything and just wait and see how Google thinks then, and then make more changes if Google doesn’t like it or not.
Where does Google stand on whether mentions or inferred links have a say in ranking factors?
John Mueller: On the one hand, I have shied away from ever trying to compare these ranking factors because it’s not the case that they’re fixed weights. You can say, up to half or there’s the most important thing and the second most important thing. Those change all the time. They change depending on the query on the situation as well. It’s something where weighing the individual things is tricky and misleading also.
Transcription of Episode 400:
Ross: Hello and welcome to SEO 101 on WebmasterRadio.FM, episode number 400. This is Ross Dunn, CEO of StepForth Web Marketing, and my co-host is John Carcutt, the Director of SEO for Advance Local. Today, we have a special episode for you. We’re interviewing John Mueller, Google Search Advocate. It’s a title I haven’t heard yet—very clean, pretty simple, and says a lot.
John, I know this is your Friday evening, but thank you so much for joining us.
John Mueller: Oh, thank you for having me.
John Carcutt: I highly would recommend, since it’s a Friday evening for you, that you grab a cocktail. This is going to be interesting.
Ross: I know 400 episodes don’t compare to some shows, but we’re pretty proud of the milestone. I understand you listen to the show occasionally. I know you even mentioned that a few years ago. Thank you.
Mueller: Not all 400 yet, but maybe at some point. If you slow down, people can catch up.
Carcutt: I was explaining to someone the other day, it’s our 400th. They’re like, wow, how long have you been doing this? I’m like, no, it’s 400 weekly episodes. We’ve been doing this since 2009.
Ross: Not quite weekly, but we try. Before we get into things, I wanted to first take a moment. The SEO community experienced a significant loss this week, on the show’s recording anyway. We lost one of the brightest minds of SEO, Hamlet Batista. We lost him to COVID-19. He was a much-loved member of the SEO community, and his loss is being felt acutely as evidenced by some tsunami of posts across all of the industry sites. In particular, Lily Ray wrote a tribute to Hamlet on Search Engine Journal, and it made me dearly wish I’d gotten to know him personally. I only had read his articles and respected what he’d written. I’ve since read more of them and just the loss of life is bad enough, but to lose him is, obviously, a loss for the industry as well.
Lily has put together a GoFundMe campaign. You’ll be able to find a link to that in the show notes if you are part of that. If you’re not, just go to seo101radio.com and you’ll be able to subscribe to them, or there will be a link there to the blog as well that has the transcription. I’ll also share the link to Hamlet’s RankSense Manifesto—RankSense is his business—it’s a fantastic read. It really shows his passion for the SEO community, and how he wanted to improve the reputation as a whole through a greater understanding of data science and programming in Python.
It got me pumped, made me want to get into Python actually. I’m not a programmer. Math scares the bejesus out of me. I don’t know how much math is in Python. I’m just excited and even to honor him, I’m just going to give it a shot. It’s a big loss and I thought it was important we say that.
John, this is how we’re going to do this. John Mueller, we’re going to call you John. John Carcutt, I’m going to call you Johnny. I’m going to try anyway and make sure we get this right.
Carcutt: Alright, fine. I will allow that my name be used by the more important John on this call.
Ross: I can call you Johnny-boy if you want.
Ross: John, how have you been doing through all this COVID mess? Is everything okay? Is your family okay?
Mueller: Yeah. I don’t know. What is okay? You survive. At Google, we stopped working in the office sometime early in March. I was back in the office once in between when things were a little bit better here in Switzerland, but otherwise, it’s all working from home. My daughter is studying from home, essentially. It’s just all very weird and different. That’s even excluding seeing the direct effects like with Hamlet when you’re isolating and staying to yourself. It almost comes across as you can get used to it and it’s normal, but you don’t realize that there’s still this big thing happening out there that hasn’t really slowed down much.
Carcutt: We always think of Google as being really innovative about things. Are they doing anything interesting or innovative around COVID for you guys?
Mueller: I don’t know. Not in the sense that we have a secret weapon against COVID or something.
Carcutt: No, not like that. I’m just saying.
Mueller: Working from home is fairly easy with regards to Google because we have all of the tools, and most of the teams are more software and coding-related. It’s not that you have to go out and meet people and do things in person. I know there are some teams that are focused on hardware. Obviously, they have it a little bit harder and all of the data center teams. It’s a bit trickier there, but lots of people can work from home. We have a lot of the tools that make it possible so that you can do all of your video conferencing, and pretty much all of your work from home or from any location, essentially. At least from that point of view, it’s good.
Carcutt: I know in our business, we’re talking about the idea of, does everybody really have to go back to the office after all this is over? I think that’s going to happen in a lot of places.
Mueller: It will be a little bit weird for a longer time. Even that mixed mode of people who want to go to the office or who don’t want to go to the office will be a tricky thing to navigate.
Ross: No doubt, especially with different immune system issues, and who knows. Lots of paranoia. Hey, I get it, really a stressful time. Let’s plow on here. Tell me, what does your role entail as a Google Search Advocate?
Mueller: I don’t know. What should I be doing? Good question. I have to think about this from time. The general idea is to have some people who talk with people externally who make websites, who work in the world that is associated with search. Our goal is to connect those people who are making websites, who are optimizing websites, with the engineers and the teams internally at Google that work on the search side of things.
We have similar ideas. We pull in the same direction. We understand what it is, that on Google’s side we need to do to help you all to be successful on the web and that the people on the web understand what Google is trying to do, and what they could be doing slightly differently, perhaps, to be more successful.
Carcutt: Is your role different than Danny Sullivan’s or do you coordinate or work together? How does that work?
Mueller: We do work together quite a bit. But I see Danny’s role a little bit more on a higher level with regards to policy decisions. Those kinds of things. We’re more on a tactical level in that we actually go out and try to help people one to one.
Carcutt: Who’s we? Is it just you, and Martin, and Gary, or is there more?
Mueller: Daniel is also in my team, Daniel Weisberg. He used to do a lot of analytics stuff. Then, we have a handful of other folks as well, including the people who work on the developer documentation for search. All of that is under one umbrella.
Ross: Nice. How did you get your start in this line of work? These are questions I don’t normally see people ask. I’m just curious.
Mueller: How did I get started? It was a long time ago. Before joining Google, I had a small software company and at some point, things were working fairly well. Around that time, the internet was coming up. We started doing it a little bit more in that direction. I created a sitemaps generator. Sitemaps just came out right about then for Windows and started being a little bit more active in the Help Forum around Google Sitemaps and in general.
Slowly, I got in contact more with people from Google. I went and visited the office in Zurich and I grew out of there.
Ross: Were you recruited by them?
Mueller: Yeah, somehow.
Ross: Were you trying to get a job, or did they try and get you, that’s what I was getting at?
Mueller: They sent me an email to a domain that I wasn’t actively monitoring the email for, and just accidentally, I noticed that it was there. I was like, is this real? Why is someone sending me an email to this domain? It took over a year to go from there to actually work at Google.
Ross: Good for you, very cool. Quite flattering, although I know it’s early in their years, but still would have been quite flattering, I’m sure. What is your favorite and least favorite part of the job?
Mueller: Favorite part of the job. What I really enjoy is seeing the smart and creative things that people do online with regards to search and the things that are unexpected. Those aspects where you look at them and you say, that was really cool. With regards to content sometimes, with regards to SEO techniques, sometimes also just technical details that they do in a really cool way, that’s something I really enjoy seeing. It’s something that, within our role, you run across every now and then because people go and are like, does this actually work for Google or not? You look at it and it’s a pretty cool idea.
That is pretty much something I really enjoyed. Being connected to the really smart people, the creative people makes that really fun.
Let’s see. Least favorite.
Mueller: Podcast, no. Let’s see. Time zones are one of the things that always come up for me.
Mueller: It’s like here now to you. But this time, it’s not too bad. It’s something where since a lot of people at Google are located in California, where the headquarters are, we tend to have more and more meetings during the California daytime. That tends to be later in the evening here. Some days it just piles on where it’s from 5:00 PM until 8:00 PM-9:00 PM, we have nonstop meetings. It’s something where I really enjoy seeing what people are up to and what they’re working on, but at the same time, oh, it’s such a long day. If I could get rid of time zones I think that would make me feel a lot better.
Ross: They don’t let you start later in the day?
Mueller: Yeah. I can start later in the day. It’s not like we have a punch clock to check in and check out. If you have kids, they just get up in the mornings. You can’t be there like, yeah, I’m going to sleep until 10:00 AM. You’d go do your school stuff. It just doesn’t work that way.
Carcutt: I was going to say, Ross, you should know better. There’s no such thing as clocking in and clocking out when you’re doing this work. You’re on 24/7.
Ross: Yeah. As long as they don’t make you do a full day and then you make you work late. I don’t know how things work in these big companies. I never go to all that corporate.
Mueller: Once you get over 80 hours in the week, they tend to let you back off a little. Just a little though.
Ross: You often do Q&As on YouTube called Google Search News. This is work for me and I’m sure other listeners will like this stuff, but do you have a podcast version of that? I try and listen to it while I’m on my walk with my dog, and I’ve got to run the video in my pocket. It’s annoying.
Mueller: The Office Hours you mean, I guess?
Ross: I guess so. It’s on YouTube. It’s called Google Search News, but yeah.
Mueller: The Search News is the short one that we do maybe once every month or every other month. The Office Hours are ones that we do almost weekly, essentially. But it’s not in podcast format. We do have the Search Off the Record podcast though.
Ross: I was going to mention that.
Carcutt: That’s a good one.
Ross: That’s funny. I get the impression Gary is quite the character to work with.
Mueller: Everyone’s a character to work with. Everyone’s unique.
Ross: Martin definitely is. I love Martin. I love the whole thing. It’s great.
Carcutt: Are you saying you don’t love Gary? Come on.
Ross: Gary seems—
Carcutt: You’re stuttering way too much. Gary, we all love you. I’ll work on Ross.
Ross: He’s a quick-witted one. I envy that, but I bet that is tricky for interviews. Let’s get into the questions here. Let’s start with something straightforward. What do you enjoy talking about, John, in this area?
Mueller: In this area, it depends. I like saying it depends.
Carcutt: There’s a very good meme for that, related to SEO, out there. If you haven’t seen it, I’ll send it to you.
Mueller: All over, yeah. Okay.
Ross: It’s all good. I know it’s vague, but I was curious whether or not there’s a particular topic that you get questions on a lot and it’s always intriguing to explain or whatever. It was a last-minute addition, probably put more thought into that one.
I understand that many businesses need to wait until the most recent core algorithm update to see improvements in their rankings if they were hit the last time. For those who suffered as a result of the latest core update, is there really nothing they can do to get back into Google’s good races, assuming that’s what the issue is, until the next one? How does that work?
Mueller: There were multiple levels there with regards to the effect, and it’s not so much. I’d say something where we would look at it as the website has to fix, something like there’s some technical problem that you need to fix. The core algorithms tend to look at it more as we don’t think your current website is as relevant for those queries as we showed in the past. That’s something where you have to take a step back and think about how can I improve my relevance or how can I show Google end users that I am still relevant for these topics? Can I tweak the topics that I write about to match where I really have big strength to work on?
The effects there you would see, on one hand, you would see some incremental improvements over time if you significantly improve your website like with anything. As we recrawl, reprocess everything, that’s something where you probably see some small changes over time, and depending on the effect that you saw from the core update, that might be something that takes a little bit longer. On the one hand for us to re-understand your website, understand how it is relevant. On the other hand, with regards to core updates, if we think it really requires a rethinking of everything, then maybe you do need to wait a little bit longer until the next core update happens.
Ross: I know that there’s quite a gap. Was it nearly a year between December 1st and the last? Maybe not.
Mueller: I thought it was June.
Carcutt: Yeah, I’m not sure if it was a year.
Mueller: It’s annoying if you’re stuck in between. I do get that it’s annoying, but it’s also something where I would not just make some fixes and then wait, but rather, it’s like, you can continue working on your website and continue improving things over time. It’s not something where I’d really tell people, you shouldn’t do anything and just wait and see how Google thinks then, and then make more changes if Google doesn’t like it or not.
Carcutt: What do you say to people who, their philosophy and their practices when they’re doing this, is never to make multiple big changes because you don’t know which one of those changes impacted whatever results you saw, whether they’re positive or negative? You always do one major thing at a time. If they’re doing that based on the core update, it’s hard to keep going until you find out what happened. Do you know what I mean?
Mueller: Yeah. That’s the thing where I would say I would not try to wait for that to settle down because if you know that there are multiple things that you can do to improve, just making one of those changes is maybe good enough. But maybe it’s still not completely good enough and you don’t want to wait for multiple cycles. That’s something where even those incremental gains can be quite valuable and they can also result in indirect effects, as well, with regards to things like conversions, maybe traffic from social media. All of that can play into that as well, where you don’t always just need to wait for that update to happen.
Ross: I’ve heard people talk about how it’s such a long wait that they’ve considered creating a new site and trying to get that ranked while they wait for this to happen, and it’s unfortunate. Hopefully, that won’t be the case for a long time, like on a regular basis that people have to think that way. But it is sometimes a long wait. Four months or five months can be very deadly for a business because it usually isn’t a gradual or small drop if they’ve been hit. We’ve got a couple we’re trying to help right now, and they just went down. They lost 90% of their traffic and to have to run a business is nearly impossible at that point.
It is unfortunate but I guess they don’t really have an option. They can fix what they can of the other one, of the one that’s already established. We’re working with someone on that, but they oftentimes have to start another one and get back up there. I do wish that was a little different. I almost missed the Google Dance. In that sense, at least it happened more often.
Carcutt: You can read my Facebook.
Mueller: My general hope is that we can make it a little bit smoother and that you can react or make it a little bit more real-time. But it is really tricky when it comes to bigger algorithm changes because we’re not just tweaking things slightly. Sometimes we really have to rethink how we do relevance with regards to search results.
Ross: A good segue, actually, here, back in the early days, the Google Search algorithm was reportedly managed 100% by engineers. At that time, when there are questions about the existence of a ranking, I imagine it was easier to outline that, why it happened. Now that Google uses so much artificial intelligence, is it still possible for search engineers to know exactly how a ranking is chosen? Is it more like a black box now?
Mueller: We can still figure out why things are ranking the way they are. It’s not something that I personally do because we tend to focus more on the technical aspects which tend to be more around like, is this page even indexed or not? Then the ranking is a secondary effect there. But it is something where when we run across issues where we see like, this page is not ranking at all or not ranking well and we think it’s actually a fantastic page for this query, we will sometimes pass that on to the search quality team. They do have ways to figure out what is actually happening there. They can look into the individual factors and figure out, oh yeah, we need to tweak this a little bit, or this is a good example for this kind of specific case.
The next time we make an algorithm change that involves that part of the system, then we’ll take these examples into account and try to make sure that they work out appropriately. It is something where we can still figure out why things are the way they are and then work to improve that.
It’s something that, for the longest time, we did hold off with regards to machine learning, with regards to search because we were worried that you would have one big system, and I would just pop out the rankings and nobody really knew why. But a lot of things are smaller systems and they play together. Even if within a small system, we use machine learning to figure out canonicalization. We can still work backward from what was in there and figure out why did the system come to this conclusion, and what do we need to tune with the training data with the weights, all of those things.
Ross: If you keep them separated, it’s a little easier to manage what it is that’s interacting. I see. Until the day when it’s all merged, then everyone is out of work.
Mueller: I don’t see that happening. There are always bigger changes in the web overall which make it so that the thing you were optimizing for in the past might be completely different from what you would optimize for in the future, and that’s something we saw. The first time I really noticed it was everything around Panda when that came up, where people were creating these really targeted landing pages. Our algorithms were like, oh yeah, this is a perfect match for this query. We should show this page how-to-tie-your-shoes kind of thing. Then we had to rethink how we were training the system or how we were working with our algorithms to say, sure, this page is an exact match of that query. But the site overall is just a bunch of thin content, not really useful information.
We should take other things into account a little bit more. Those shifts of almost the philosophy or the expectations on the web is something that’s really hard to automate or probably not possible to automate.
Mueller: Otherwise, it’s easy to end up in a situation where everything is just clickbait where if we train our systems like, oh, people click the number one result, and that’s good. Then the number one result is always just some weird clickbaity article and not really something useful.
Ross: Let’s take a quick break. When we get back, we’re going to talk about crawling. I know John’s got important questions.
Carcutt: Yeah, I want to finally ask some questions.
Ross: Welcome back to SEO 101 on WebmasterRadio.FM hosted by John Carrcut, the Director of SEO for Advance Local, and myself, Ross Dunn, CEO of StepForth Web Marketing, Inc. We’re joined today by John Mueller, Google Search Advocate. Alright, Johnny. Fire away.
Carcutt: Johnny or John. Just don’t call me Jimmy John.
Ross: Don’t say that.
Carcutt: I want to jump into crawling and crawl budgets. I know you probably get these questions all the time and I’m hoping this is one you haven’t heard before. I’ll preface this by knowing that I do understand that you always want to continue to crawl content just to make sure it hasn’t changed. But I know there are sites—big large ecommerce sites—our site, media, news sites—where we have big archives of content. In our case, it’s millions of pages on a single site, and we get a lot of crawler activity back in those pages because I know you’re checking to make sure they haven’t changed, or they haven’t been impacted by malware or anything like that. But I have noticed that it impacts our fresh content.
I get a lot of things in Search Console that say, hey, we found this page but we haven’t crawled it yet. Is there a way without just blocking all that content to get the crawlers to focus on the fresh content instead of that archived content?
Mueller: Good question. I don’t have a good answer for that other than that our system is trying to figure that out. That’s one of those things where what happens on our side is we try to automatically split the site into different parts and to recognize which parts need to be crawled and how frequently. Assuming you have a subdirectory called archive, or even just a subdirectory called 2020, 2019 kind of thing, then our systems would try to recognize that these are separate parts of your website, and we need to crawl and index them individually. Then essentially, what we’ll do is we’ll try to focus on the things that we need to crawl most frequently. They’ll be in the higher priority queue.
It’s not that we have a queue, but more in the sense that they’ll be more important with regards to crawling. Then we’ll say, oh, but we have all of this extra room left where we know the website has the capacity for crawling. We’ll fill that with all of the other stuff—things that we have to refresh to get updated, maybe 404 is where we just want to make sure that there’s actually still no content here—all of those things.
If you have a website that has grown over a longer period of time than that archive area, that 404 section just becomes really large compared to the recent content. If you look at your server logs overall, it might look like Google is crawling too much of this old stuff and not enough new stuff. But actually, we’re still trying to balance that. In the new crawl stats report in Search Console, you’ll also see the difference between the refresh and discovery crawl.
Carcutt: Just to be honest, it’s something I’m considering, and I’d love to get your opinion. Would it help to create dedicated XML files for the archive with lower priorities instead of those URLs?
Mueller: No. We ignore that priority.
Carcutt: You saved me so much time. Thank you.
Ross: You had another follow up there, Johnny?
Carcutt: I have another question that’s a different topic, I’d love to ask. You guys just announced a new alternative to cookies. How are you visualizing a post-cookie world overall? Because people, and software, and browsers, and cookies are going to be around for a long time, how’s this switch going to take place in your mind?
Mueller: I have no idea. I did not notice that we announced an alternative to cookies.
Carcutt: I understand. That makes sense. You got to go back, Ross. I know you got way more questions than I do.
Ross: I did prep.
Carcutt: I like to go with the flow.
Ross: In the tweet on January 23rd, John, you had a discussion about digital PR and you noted how you love some of the things we’ve seen in digital PR. First, how would you describe digital PR? Second, I’m wondering if you remember any of the examples that stood out to you.
Mueller: I don’t have a definition for digital PR. I was just going with what people were showing there. From my point of view, it is just something that we have never talked about in the past. I felt it was also important to bring that up, in that, it is something where, when we talk to people, when you tell them, just make great content and we leave it at that, and it explicitly points people to examples of how you can make great content and use that to do link building essentially. That was the intent there.
It’s not that I had a specific definition of digital PR like, exactly this industry or exactly this amount of this work, but more than the general idea of you’re creating some cool new content and it’s something fascinating where you know it matches what people are looking for, what they’re interested in, and you go out and you present that to other people and they think it’s also cool, they link to you. That was my intent there, to show that instead of going out and buying links, you can actually create something fantastic on your site. You’re not just putting it out there and waiting for people to link to it. You are going and presenting it to people, but it is something that you’re doing with the intent, at least partially, to get links, to get the awareness of your site, of the other things that you’re doing.
Ross: Links are not a bad thing. It’s a positive vote of confidence.
Roger Montti just mentioned it here, I thought it described it really well in his article. Again, we’ve had him on the show before. He’s a great writer. He says, “The short description of the strategy,” and this is digital PR, “could be framed as, create something worth linking to and tell others about it.” It’s exactly what you just said there, John.
Carcutt: Just to be clear though, if you hire a company to do press release distribution, there’s been perception for years now that that could be considered paid linking because you’re paying someone to distribute your content with links in it. Is that an acceptable practice? Do you use those distribution services?
Mueller: If you’re talking about press releases, that’s different from creating link-worthy content where people freely link to your content. Because with a press release, you’re taking a chunk of checks, a chunk of HTML, and you’re giving it to people and say, hey, put this on your website one to one. That essentially goes into the area of guest posts, where you’re providing the link to your website, which is a bit different from other people saying, oh, this is interesting. I will link to your website.
Carcutt: Got you. What kind of digital PR do you think people should focus on? Because what I think digital PR, and I know a lot of people, initially, when they think digital PR are press releases is the first thing that comes to mind. What kind of things do you think they should do?
Mueller: I didn’t bring across association with press releases. It’s about creating something link-worthy or interesting, and telling people about it and saying, hey, look at this cool thing that I created and other people saying, yeah, I really like that. I will link to it.
Carcutt: Socializing, the change or the update as opposed to just writing a description, and sending it out to a bunch of people, and hope they pick it up.
Ross: What I would think, too, just mentioning that hey, this is a great article, I know you write about this article all the time. Maybe you’d consider interviewing me or checking it out, and commenting on it, or anything. I’d be happy to chat with you about it. That sort of thing. Create a discussion, keep the discussion going, that would be my idea of what it would be at least one part of it. I’m sure there are many brilliant ways of doing digital PR.
Unfortunately, I was reading about this when I was doing my background research and I noted that a lot of people were saying, oh no, link builders are going to rebrand themselves as digital PR.
Mueller: Taking out of context will definitely happen as well. It certainly is happening already. It’s also important to bring it out and clearly say, it’s just getting links on its own is not something bad. If you create something fantastic and you show it to people and they say oh, this is great, I will link to it, then that’s it. That’s a good thing from my point of view. It’s not every link that you’re involved in any way is bad.
Carcutt: This actually ties into what we talked about with Stephan Spencer in our last show. Not enough people pick up the phone and try to reach people to talk about what’s happening with their businesses. It’s all email these days. He made a great point about those personal conversations that could have a much bigger impact with that kind of thing.
Mueller: I don’t know. I don’t use my phone.
Ross: We’re talking about that. My wife doesn’t pick up the phone.
In a recent article by Rand Fishkin, he postulates that what he considers inferred links or mentions of the business, for example, carry as much as half the ranking influence of a link. Where does Google stand on whether mentions or inferred links have a say in ranking factors?
Mueller: On the one hand, I have shied away from ever trying to compare these ranking factors because it’s not the case that they’re fixed weights. You can say, up to half or there’s the most important thing and the second most important thing. Those change all the time. They change depending on the query on the situation as well. It’s something where weighing the individual things is tricky and misleading also.
Carcutt: I was going to say, it’s interesting because I’ve been having a conversation and it leads to a follow-up question. I don’t think there’s anything that I can think of that’s black and white when it comes to the algorithms. There are always outliers, always shades of gray. I can find 404s that are still ranking that kind of thing. Is there anything that you’re aware of that’s black and white? It’s got to be either this way or that way, or is it all shades of gray?
Mueller: Things like no-index are pretty clear if it’s a no-index and we see the no-index that’s pretty clear. But even something blocked by robots.txt, obviously, we wouldn’t crawl it but it could still appear in the search results. The section of things that are clearly not possible is certainly there. But everything else falls into that gray area that you mentioned.
Carcutt: How I shake my head when somebody says, this always happens. I’m like, I bet it doesn’t.
Ross: Inferred links or mentions and stuff are considered in the algorithms. Would that be the statement?
Mueller: I don’t know how that would be handled in detail. One of the things that we do with things like the Knowledge Graph is to try to understand what entities are on a page, and to try to understand the relationship between those entities, and to figure out how we should take that into account with regards to ranking. But I’m not aware of anything specific where we’d say this is something you need to do or something that we expect people to do, or that we expect to find on the natural web. My understanding is that a lot of that is just about better understanding the relationship between those entities and figuring out how things are connected on the web.
Ross: That makes good sense.
Carcutt: Could I follow up on the entity thing for a second?
Ross: Please do.
Carcutt: If I was to try to describe someone, what’s the difference between doing entities research versus keyword research? Should I first point him in a one-way direction or the other? Or they basically are close enough or it doesn’t matter? What’s your opinion?
Mueller: For the most part there, they’re pretty similar. But it’s always worthwhile. I have no idea. I don’t do this myself. It’s interesting to look at the specifics. Maybe in Google Trends, you can look at entities themselves and you can see which queries are associated with them. That’s one way to look at it a little bit but you can also go from the queries themselves to see which entities tend to show up more for that. I think both aspects make sense.
Carcutt: I get the question when I’m talking to entities about like, I do this training for all of our reporters and editors across 25 newspapers, and I talk about entities and things. I get the question of, John, you’re an entity, you’re John Carcutt. But you’re also a man. You’re also an SEO. You’re a bunch of different entities. How does Google know which entity that you’re talking about? And I’m going to ask you the same question.
Mueller: It’s hard. Search for my name. It gets tricky when there are multiple entities with the same name. For the most part, when you search for my name, you have a political scientist showing up in the knowledge panel and then you have some social account for me—Twitter, or LinkedIn, or something weird—and there’s a mix of all of these other John Muellers out there as well. I don’t think there is a simple map from this query to exactly this entity.
Carcutt: Tell me if I’m wrong because the answer that I give them is that’s where the AI comes in. They look at the content around the entity to try to get on the page to try to determine which variation. If there’s a page that mentions me talking about how I raised my son versus an entity about me talking about SEO, the entity could be an SEO versus a father, depending on the content around it. Is that a good explanation to give them or is that wrong?
Mueller: That’s reasonable. I have not studied all of this entity stuff and haven’t been in touch with the teams that work on this. My answers are a bit more of what I’ve randomly seen. It’s something where my guess is we try to figure out which entity is the most relevant for specific queries, that kind of thing.
Carcutt: That was my guess, too.
Mueller: That’s sometimes pretty tricky if you have multiple entities with the same name like in my case—political scientists or a person at Google—you can differentiate between the two depending on what else you’re searching for. But if you have someone like George Bush, which entity is George Bush?
You essentially have to differentiate between the two, but finding which one of these, where a lot of the attributes are very similar, is relevant. I imagine that’s pretty tricky.
Carcutt: I agree.
Ross: I recently encountered an issue in a client site where according to Google Search Console, thousands of job listings have been crawled but were not admitted to the index. No more information is seen forthcoming. It’s helpful that that information is there. I love that Google Search Console is there so thank you. But I’m still hunting for clear reasons, other than perhaps the deficit of site authority. I find this issue comes up with other clients as well. What are some of the reasons for this?
Mueller: Usually, when I get these kinds of questions, I’d like to see a little bit more about the context of the site to figure out what exactly is happening there. On the one hand, the various technical things that could be playing a role there and in a lot of cases they are. That could be something like you have a URL structure that is hard to crawl, or you have a URL structure with a lot of duplication with maybe tagged URLs, and parameters, and all kinds of things like that. That’s something that happens quite a bit, where we essentially get bogged down and just can’t crawl everything.
That also sometimes plays a role with regards to crawl budget and that maybe the server is just slow. It’s a very well set up site, but we can’t crawl as much as we would like. We know the content is there, but we can’t get to it in time it ends up in that limbo state.
The other aspect that you hinted at with authority as something around that line could also be playing a role and that if we understand that this website is something where we tend not to show a lot of content from in the search results, then maybe it’s not so worthwhile to make sure that we have millions of pages indexed from that side.
Ross: We have crawled this. It just didn’t determine that it was worthwhile being indexed. It is job content. It’s actually been around a long time, the website. I’m sure like you said, you don’t know enough about the actual situation. But I see it enough that it does boggle me sometimes, and as a troubleshooter, it’s like, oh God, where is the loose strand here? It’s difficult to find. I almost wish Google Search Console at least gave me a vague reason why. I realized that you don’t want to give away anything to spammers, but even a vague reason.
Mueller: One of the tricky parts also with a lot of job content is that it’s just duplicated across the web.
Ross: It is.
Carcutt: You got that exactly. That’s what I was going to say. Ross, I deal with this all the time with jobs, autos, real estate. They’re all inventory-based content that’s duplicated everywhere on the web. It’s hard to get those things to stick.
Ross: Yeah. Making sure the site has more authority will help a substantial amount, I would imagine, and making sure that stuff is not buried too deeply as well. That’s the way I’m approaching it, anyway, at this point.
Mueller: Authority is a vague term and it’s not like we use domain authority or anything like that but it is helpful for us to understand this is a very important website and we should try to get as much index as possible from that website. That’s something we try to pick up in various signals. Working on that aspect definitely makes sense.
At the same time, if you’re recognizing that Google is only indexing a part of the content, you could also take that and say, I will make sure that Google is indexing the most important content or the content I care about most which could be something where you see the highest ROI, or the just the highest value in general, or the highest engagement from users, and bring that a little bit more into the foreground so that Google can really recognize, oh, this is good stuff. Then from there, it goes a little bit deeper and then picks up more.
Ross: How would you do that? How could you make sure because it almost comes back to John’s question earlier about content and ensuring that one gets priority over the other?
Mueller: Making sure it’s relevant within the site structure. It’s something within the internal linking that highlights the thing that you care about most. For example, with ecommerce sites, if you have a new product and you link to them from the home page, then that’s a big sign for us. You’re saying that this is something new and it’s important and you should go check it out, whereas if you put the new content somewhere in the subcategory then maybe Google will find it, and notice it as well. Bringing it closer to the parts where people tend to go more often, the higher-level areas of the site, that helps us to understand you think this is important. We trust you on that and say, we’ll give this a little bit more weight and focus on that more, and then work down from there.
Carcutt: If you created a new section on your site, maybe to hold some of this content that’s not as important as others, because that section is brand new, it’s not going to have much weight built up. Probably the only type of links it will have would be internal links. How’s that going to impact the content you move from a higher trusted area of the site to a lower trusted? Trust might not be the right word, but I think you understand what I mean.
Mueller: If you’re moving it further away, essentially?
Mueller: A little bit like if you have news content and you move it into an archive section. You’ll link to the archive from somewhere within your website, but then from the archive, you find the rest, compared to you find everything from the top level of the website. You’re taking something that’s high up in the pyramid of your website and shifting it a little bit lower level.
Carcutt: I ask that question because this is SEO 101, and I’m just hoping you would, and you did reinforce the idea that the further the content is away from the homepage of the important pages on your site, the less relevant Google may see it.
Mueller: It’s tricky in that it’s not always the homepage. Sometimes there are parts of the site that are just like the most important thing where you’re a bigger company and you have this one sub-brand that is popular and that’s the most important part of your website. But in many cases, it is the homepage.
Ross: In a sense, our users or listeners could focus on the page that has the most traffic as almost it’s another entry page and then make links from there. It’s worth experimenting anyway.
Occasionally, we encounter something, just downright odd and frustrating in SEO and I just hate these things.
Ross: Doing this long enough, we get these issues where, oh my God, I was like, why can’t I make this stuff move? I can’t get visibility. This can all come around to simple questions because I know we can’t look into this instance but in one instance, where there’s a domain, the person had built a lot of authority, too, and then they, for some inexplicable reason, decide to change domains. They redirected it, the old one to the new one. But the old one is still, to this day, showing in Google index, and it still ranks.
This new site won’t get a ranking, for the life of us. It’s the weirdest thing and again, it’s 301 redirected. In these situations, and I imagine there are people listening who have had issues like this, where do they reach out to if they have these problems? Is there a simple way to get attention, especially during COVID? I know there have been issues with manpower at Google, or at least there was, at some point there. What do they do?
Mueller: The most general answer is to go to the help forums and post about it there. There are various people from Google that monitor the help forums and there are top contributors in the help forum that can escalate these issues to Google as well. Usually, what will happen there is you’ll get some feedback from the people that are active in the community. If it turns out that they can’t help with this, then they can generally escalate that onto someone from Google, and sometimes it reaches me, sometimes it reaches other people at Google.
The more direct approach or maybe a bit more intimidating approach sometimes is to join the office hours that we do and ask in person. I generally can’t give direct feedback on specific issues live during the Hangout because it would be weird for me to be able to figure out within one minute where you’ve been struggling with for the past year. Usually, that’s not possible, but I do take all of these escalations and I bring them back to the teams. We do end up looking at them, and trying to figure out, is there something specific that the site owner should be doing differently? Is this essentially how algorithms are meant to be? Is there maybe something on our site that we can improve in that regard?
With site moves, in particular, we had one or two cases come up during the office hours in the last couple of months. I went to the team, it’s like, you can’t tell people to do 301 redirects and everything will be okay. Look at these two cases, everything went badly. What they ended up doing is an analysis of all of the site moves that we’ve done in the last couple of months to figure out where there were significant traffic changes and to figure out what might be causing these.
It turned out that pretty much everything is working well, except for maybe some small thing that we ended up fixing, but for the most part, it ends up working out well. There are some tricky aspects always with regards to site moves, but for the most part, that should be working out well. In some of the cases that I did bring up to the team, they were like, I don’t know. Look at all of these links that they built there and look at the kind of this stuff that they did there. They were saying, it’s understandable that it looks the way it looks.
Carcutt: Nice. I have one more question on my list, Ross. I belong to a ton of open and private SEO groups on Facebook. Some of them have a few dedicated, focused people. But there’s a couple, there were 60,000, 70,000 people of every type of SEO or one of the SEO. Somebody in one of those groups asked an interesting question and got dozens and dozens of different kinds of answers, and I wanted to ask you the same question. If you could change one thing in the SEO industry, what would it be?
Mueller: Oh man. It’s hard to say. On the one hand, one of the things that comes to mind often for people is I want people to stop asking the same question over and over. But from my point of view, that’s almost a good sign. I don’t want that to be changed because if people stop asking the same questions over and over, it means that there are no more new people coming to the SEO industry. If new people aren’t coming into the industry, that’s a sign that it’s not as interesting anymore. I like it when more and more new people come in and even if they ask the same questions as everyone else before.
Carcutt: The thing you’d change is to have everybody ask the same question over and over again. Got it.
Mueller: I think you’re taking it out of context.
Carcutt: I can tell you the most popular response in that thread, and it was actually my response as well, was some certification process for the SEO industry so that people would act ethically and do things the way it should be done to help minimize the negative aspects of the industry that we’ve had since we began. We’ve been doing the snake oil salesman. Is that something that could ever happen? Because I always say that’s not going to happen without the engines coming in and helping with the certification process.
Mueller: That’s tricky.
Carcutt: Is that something that could happen?
Mueller: I don’t see that happening anytime soon. It’s theoretically possible that it could happen at some point. One of the interesting things about SEO is that a lot of people got started by doing all of the black hat stuff. Essentially, you’d grow from there, and you say, I know my affiliate sites rank. I don’t know if you did that, but it is certainly something that I see a lot of people doing in the beginning. From there, you grow into the mindset where I want something more stable rather than kind of these sites that I have to burn down every couple of weeks.
At that point, you have that black hat or sneaky mindset already and you’re doing all of this illegitimate stuff, finding people who know what they’re talking about from a legitimate point of view but don’t have any sneaky background where you can say, we can confirm these are smart people who would never do anything to harm your website, that would be hard to do.
Carcutt: It would.
Ross: Johnny, you’ve been doing this since ‘95. I’m since ‘97. A lot of what we did back then is considered black hat now. You can’t avoid it. It’s just the way it was.
Mueller: It’s an experience that you build up as well, where you know when I did this, this tended to happen, and if you do it for a longer period of time you note as well, it takes a couple of weeks or a couple of months and then search engines figure it out. Someone comes to you and asks like should I be doing more hidden text on my site? You can say, well, that was then.
Carcutt: First of all, more is out of the question. The fact that you’re already doing it, that’s what we need to discuss.
Ross: I still see it sometimes. It just shocks me.
Thank you, John. It’s been a pleasure having you on the show. I know it’s Friday night, and it just makes it all the better that you took the time. Thank you.
Carcutt: It’s time for a cocktail.
Ross: It’s been a great Episode 400. On behalf of myself, Ross Dunn, CEO of StepForth Web Marketing, John Carcutt, Director of SEO for Advance Local, and our special guest, John Mueller, Google Search Advocate, thank you for joining us today.
Remember we have a show notes newsletter that you can sign up for at seo101radio.com. There you won’t miss a single link and you can see the transcription and any links we have, including the GoFundMe for Hamlet Batista. If you have any questions you’d like to share with us, please feel free to post them on our Facebook group, easily found by searching SEO 101 Podcast on Facebook.
Have a great week and remember to tune into future episodes every week on WebmasterRadio.FM.
Carcutt: I have a question for us. This is our first video version of SEO 101. Are you going to post it somewhere?
Ross: I’m going to try. Yes, that’s my intention.
Carcutt: Everybody, look for the video. You get to see all of us in our glory.
Ross: You’ll see us, commercial breaks, and everything.
Carcutt: Yes. Until then, thanks for listening, everybody.