The Moz Links API: An Introduction

What exactly IS an API? They’re those things that you copy and paste long strange codes into Screaming Frog for links data on a Site Crawl, right?

I’m here to tell you there’s so much more to them than that – if you’re willing to take just a few little steps. But first, some basics.

What’s an API?

API stands for “application programming interface”, and it’s just the way of… using a thing. Everything has an API. The web is a giant API that takes URLs as input and returns pages.

But special data services like the Moz Links API have their own set of rules. These rules vary from service to service and can be a major stumbling block for people taking the next step.

When Screaming Frog gives you the extra links columns in a crawl, it’s using the Moz Links API, but you can have this capability anywhere. For example, all that tedious manual stuff you do in spreadsheet environments can be automated from data-pull to formatting and emailing a report.

If you take this next step, you can be more efficient than your competitors, designing and delivering your own SEO services instead of relying upon, paying for, and being limited by the next proprietary product integration.

GET vs. POST

Most APIs you’ll encounter use the same data transport mechanism as the web. That means there’s a URL involved just like a website. Don’t get scared! It’s easier than you think. In many ways, using an API is just like using a website.

As with loading web pages, the request may be in one of two places: the URL itself, or in the body of the request. The URL is called the “endpoint” and the often invisibly submitted extra part of the request is called the “payload” or “data”. When the data is in the URL, it’s called a “query string” and indicates the “GET” method is used. You see this all the time when you search:

https://www.google.com/search?q=moz+links+api <-- GET method 

When the data of the request is hidden, it’s called a “POST” request. You see this when you submit a form on the web and the submitted data does not show on the URL. When you hit the back button after such a POST, browsers usually warn you against double-submits. The reason the POST method is often used is that you can fit a lot more in the request using the POST method than the GET method. URLs would get very long otherwise. The Moz Links API uses the POST method.

Making requests

A web browser is what traditionally makes requests of websites for web pages. The browser is a type of software known as a client. Clients are what make requests of services. More than just browsers can make requests. The ability to make client web requests is often built into programming languages like Python, or can be broken out as a standalone tool. The most popular tools for making requests outside a browser are curl and wget.

We are discussing Python here. Python has a built-in library called URLLIB, but it’s designed to handle so many different types of requests that it’s a bit of a pain to use. There are other libraries that are more specialized for making requests of APIs. The most popular for Python is called requests. It’s so popular that it’s used for almost every Python API tutorial you’ll find on the web. So I will use it too. This is what “hitting” the Moz Links API looks like:

response = requests.post(endpoint, data=json_string, auth=auth_tuple)

Given that everything was set up correctly (more on that soon), this will produce the following output:

{'next_token': 'JYkQVg4s9ak8iRBWDiz1qTyguYswnj035nqrQ1oIbW96IGJsb2dZgGzDeAM7Rw==',
 'results': [{'anchor_text': 'moz',
              'external_pages': 7162,
              'external_root_domains': 2026}]}

This is JSON data. It's contained within the response object that was returned from the API. It’s not on the drive or in a file. It’s in memory. So long as it’s in memory, you can do stuff with it (often just saving it to a file).

If you wanted to grab a piece of data within such a response, you could refer to it like this:

response['results'][0]['external_pages']

This says: “Give me the first item in the results list, and then give me the external_pages value from that item.” The result would be 7162.

NOTE: If you’re actually following along executing code, the above line won’t work alone. There’s a certain amount of setup we’ll do shortly, including installing the requests library and setting up a few variables. But this is the basic idea.

JSON

JSON stands for JavaScript Object Notation. It’s a way of representing data in a way that’s easy for humans to read and write. It’s also easy for computers to read and write. It’s a very common data format for APIs that has somewhat taken over the world since the older ways were too difficult for most people to use. Some people might call this part of the “restful” API movement, but the much more difficult XML format is also considered “restful” and everyone seems to have their own interpretation. Consequently, I find it best to just focus on JSON and how it gets in and out of Python.

Python dictionaries

I lied to you. I said that the data structure you were looking at above was JSON. Technically it’s really a Python dictionary or dict datatype object. It’s a special kind of object in Python that’s designed to hold key/value pairs. The keys are strings and the values can be any type of object. The keys are like the column names in a spreadsheet. The values are like the cells in the spreadsheet. In this way, you can think of a Python dict as a JSON object. For example here’s creating a dict in Python:

my_dict = {
    "name": "Mike",
    "age": 52,
    "city": "New York"
}

And here is the equivalent in JavaScript:

var my_json = {
    "name": "Mike",
    "age": 52,
    "city": "New York"
}

Pretty much the same thing, right? Look closely. Key-names and string values get double-quotes. Numbers don’t. These rules apply consistently between JSON and Python dicts. So as you might imagine, it’s easy for JSON data to flow in and out of Python. This is a great gift that has made modern API-work highly accessible to the beginner through a tool that has revolutionized the field of data science and is making inroads into marketing, Jupyter Notebooks.

Flattening data

But beware! As data flows between systems, it’s not uncommon for the data to subtly change. For example, the JSON data above might be converted to a string. Strings might look exactly like JSON, but they’re not. They’re just a bunch of characters. Sometimes you’ll hear it called “serializing”, or “flattening”. It’s a subtle point, but worth understanding as it will help with one of the largest stumbling blocks with the Moz Links (and most JSON) APIs.

Objects have APIs

Actual JSON or dict objects have their own little APIs for accessing the data inside of them. The ability to use these JSON and dict APIs goes away when the data is flattened into a string, but it will travel between systems more easily, and when it arrives at the other end, it will be “deserialized” and the API will come back on the other system.

Data flowing between systems

This is the concept of portable, interoperable data. Back when it was called Electronic Data Interchange (or EDI), it was a very big deal. Then along came the web and then XML and then JSON and now it’s just a normal part of doing business.

If you’re in Python and you want to convert a dict to a flattened JSON string, you do the following:

import json

my_dict = {
    "name": "Mike",
    "age": 52,
    "city": "New York"
}

json_string = json.dumps(my_dict)

…which would produce the following output:

'{"name": "Mike", "age": 52, "city": "New York"}'

This looks almost the same as the original dict, but if you look closely you can see that single-quotes are used around the entire thing. Another obvious difference is that you can line-wrap real structured data for readability without any ill effect. You can't do it so easily with strings. That’s why it’s presented all on one line in the above snippet.

Such stringifying processes are done when passing data between different systems because they are not always compatible. Normal text strings on the other hand are compatible with almost everything and can be passed on web-requests with ease. Such flattened strings of JSON data are frequently referred to as the request.

Anatomy of a request

Again, here’s the example request we made above:

response = requests.post(endpoint, data=json_string, auth=auth_tuple)

Now that you understand what the variable name json_string is telling you about its contents, you shouldn’t be surprised to see this is how we populate that variable:

 data_dict = {
    "target": "moz.com/blog",
    "scope": "page",
    "limit": 1
}

json_string = json.dumps(data_dict)

…and the contents of json_string looks like this:

'{"target": "moz.com/blog", "scope": "page", "limit": 1}'

This is one of my key discoveries in learning the Moz Links API. This is in common with countless other APIs out there but trips me up every time because it’s so much more convenient to work with structured dicts than flattened strings. However, most APIs expect the data to be a string for portability between systems, so we have to convert it at the last moment before the actual API-call occurs.

Pythonic loads and dumps

Now you may be wondering in that above example, what a dump is doing in the middle of the code. The json.dumps() function is called a “dumper” because it takes a Python object and dumps it into a string. The json.loads() function is called a “loader” because it takes a string and loads it into a Python object.

The reason for what appear to be singular and plural options are actually binary and string options. If your data is binary, you use json.load() and json.dump(). If your data is a string, you use json.loads() and json.dumps(). The s stands for string. Leaving the s off means binary.

Don’t let anybody tell you Python is perfect. It’s just that its rough edges are not excessively objectionable.

Assignment vs. equality

For those of you completely new to Python or programming in general, what we’re doing when we hit the API is called an assignment. The result of requests.post() is being assigned to the variable named response.

response = requests.post(endpoint, data=json_string, auth=auth_tuple)

We are using the = sign to assign the value of the right side of the equation to the variable on the left side of the equation. The variable response is now a reference to the object that was returned from the API. Assignment is different from equality. The == sign is used for equality.

# This is assignment:
a = 1  # a is now equal to 1

# This is equality:
a == 1  # True, but relies that the above line has been executed

The POST method

response = requests.post(endpoint, data=json_string, auth=auth_tuple)

The requests library has a function called post() that takes 3 arguments. The first argument is the URL of the endpoint. The second argument is the data to send to the endpoint. The third argument is the authentication information to send to the endpoint.

Keyword parameters and their arguments

You may notice that some of the arguments to the post() function have names. Names are set equal to values using the = sign. Here’s how Python functions get defined. The first argument is positional both because it comes first and also because there’s no keyword. Keyworded arguments come after position-dependent arguments. Trust me, it all makes sense after a while. We all start to think like Guido van Rossum.

def arbitrary_function(argument1, name=argument2):
    # do stuff

The name in the above example is called a “keyword” and the values that come in on those locations are called “arguments”. Now arguments are assigned to variable names right in the function definition, so you can refer to either argument1 or argument2 anywhere inside this function. If you’d like to learn more about the rules of Python functions, you can read about them here.

Setting up the request

Okay, so let’s let you do everything necessary for that success assured moment. We’ve been showing the basic request:

response = requests.post(endpoint, data=json_string, auth=auth_tuple)

…but we haven’t shown everything that goes into it. Let’s do that now. If you’re following along and don’t have the requests library installed, you can do so with the following command from the same terminal environment from which you run Python:

pip install requests

Often times Jupyter will have the requests library installed already, but in case it doesn’t, you can install it with the following command from inside a Notebook cell:

!pip install requests

And now we can put it all together. There’s only a few things here that are new. The most important is how we’re taking 2 different variables and combining them into a single variable called AUTH_TUPLE. You will have to get your own ACCESSID and SECRETKEY from the Moz.com website.

The API expects these two values to be passed as a Python data structure called a tuple. A tuple is a list of values that don’t change. I find it interesting that requests.post() expects flattened strings for the data parameter, but expects a tuple for the auth parameter. I suppose it makes sense, but these are the subtle things to understand when working with APIs.

Here’s the full code:

import json
import pprint
import requests

# Set Constants
ACCESSID = "mozscape-1234567890"  # Replace with your access ID
SECRETKEY = "1234567890abcdef1234567890abcdef"  # Replace with your secret key
AUTH_TUPLE = (ACCESSID, SECRETKEY)

# Set Variables
endpoint = "https://lsapi.seomoz.com/v2/anchor_text"
data_dict = {"target": "moz.com/blog", "scope": "page", "limit": 1}
json_string = json.dumps(data_dict)

# Make the Request
response = requests.post(endpoint, data=json_string, auth=AUTH_TUPLE)

# Print the Response
pprint(response.json())

…which outputs:

{'next_token': 'JYkQVg4s9ak8iRBWDiz1qTyguYswnj035nqrQ1oIbW96IGJsb2dZgGzDeAM7Rw==',
 'results': [{'anchor_text': 'moz',
              'external_pages': 7162,
              'external_root_domains': 2026}]}

Using all upper case for the AUTH_TUPLE variable is a convention many use in Python to indicate that the variable is a constant. It’s not a requirement, but it’s a good idea to follow conventions when you can.

You may notice that I didn’t use all uppercase for the endpoint variable. That’s because the anchor_text endpoint is not a constant. There are a number of different endpoints that can take its place depending on what sort of lookup we wanted to do. The choices are:

1. anchor_text

2. final_redirect

3. global_top_pages

4. global_top_root_domains

5. index_metadata

6. link_intersect

7. link_status

8. linking_root_domains

9. links

10. top_pages

11. url_metrics

12. usage_data

And that leads into the Jupyter Notebook that I prepared on this topic located here on Github. With this Notebook you can extend the example I gave here to any of the 12 available endpoints to create a variety of useful deliverables, which will be the subject of articles to follow.

The Ultimate Low-Hanging Fruit SEO Strategy — Whiteboard Friday

We all know that we want to maximize our chances for success in SEO, and for that, what we want to do is prioritize tasks that will have a higher impact, and lower effort, but sometimes those get lost in the SEO audit process. In today’s Whiteboard Friday, Aleyda helps develop this low-hanging fruit analysis in parallel of the usual SEO process.

Click on the whiteboard image above to open a high resolution version in a new tab!

Video Transcription

Welcome to a new edition of Whiteboard Friday. My name is Aleyda Solis. I am an SEO consultant and founder of Orainti, and today I am here to share with you low-hanging fruit SEO. We all know that we want to maximize the opportunities, the chances for success in SEO, and for that, what we tend to do is to prioritize those tasks, those activities that will tend to have a higher output, a higher impact, and lower effort.

Although it's true that this usually depends on the context of our SEO process or project, the restrictions, the opportunities, the resources, the flexibility, etc., the reality is that it tends to be always this let's say strategic, agnostic types of activities that tend to be always there for us to leverage, right?

However, what we tend to do in our SEO processes is this, right? We start the SEO process with an audit, research from keyword competition research to technical SEO, content audit, competition analysis, backlink analysis, etc. This tends to take a little bit of time, like four weeks or so, for example, let's say.

Then we need to analyze all of the data, etc. in order to generate actionable, prioritized SEO recommendations that, at the end of the day, are the ones that we share with our SEO clients or SEO stakeholders in general for execution, right? So all of this process tends to take a little bit of time. Unfortunately, the issue here is that after this time, we tend to face challenges about like, yes, impatience of the stakeholders or the owners of the project, right, and it's natural.

However, as I mentioned before, we can and what I propose here is to develop this low-hanging fruit analysis in parallel of the usual SEO process audit in order to detect this low-hanging fruits that we tend to have, and I will share later on which, in order to start implementation right away, right?

This might seem counterintuitive because you may say, "Oh my God, Aleyda, extra work, besides the one of the audits." But the reality is that ideally here we should set already some frameworks, some reports with data that we tend already to have in the SEO process in order to implement this, right? The benefits of this low-hanging fruit analysis and the implementation that we can start right when we are already doing the usual audit is that it will mitigate impatience from clients or stakeholders.

We will start with those actions that will be much like easier or simpler to coordinate, right? So what I'm talking about here about low-hanging fruit, realistically, I am going to go through three scenarios here of these low-hanging fruit opportunities that very likely will also be applicable for any of your projects, right?

Improving the click-through rate of top ranked pages. If we go and take a look at our current rankings using whatever ranking tools that you use, Google Search Console even, you can take a look at which are those top ranked pages that are already ranking for relevant queries, that are really important and meaningful for you, that have opportunities to improve their click-through rates, that the click-through rates are too low for the rankings of these pages.

You can try to identify if something is off with the snippets, with the titles, with the meta descriptions, for example, or if these pages are not maximizing the visibility because of the lack of structured data implementation and the reason why they are not generating rich snippets or included in a very important, meaningful relevance or feature, for example.

That is the reason of why the click-through rate is too low. You can go and straight forward improve those, right? With the snippets too, I have to say I have found many more scenarios in which Google was rewriting the title, which is now more common than before. Even if Google tries to rewrite it in a way that is still meaningful and relevant, the core key aspect of that particular page a few times has been eliminated.

Or maybe the core page is still there, but when you compare it with your top competitors, with all the pages ranking in that same SERPs, you identify that they are actually showing additional data, additional insights that you are not because of being cut out and, well, that is certainly a missed opportunity for you.

So go and take a look and prioritize the analysis, very straightforward analysis with the data that you already have for those snippets and those search features that you could be leveraging but are not. Then you want to take a look also at those relevant queries that you are ranking with not relevant pages. Maybe in the past, you created pages that better match the intent for those queries.

Not anymore. Or maybe you created at some point many different pages targeting similar queries that made sense in the past. But not anymore either, right? You may find scenarios of content cannibalization issues or lack of content issues, right? For that, what I would highly, highly, highly recommend is to analyze for which of your relevant queries you're ranking with more than one page to identify, to assess if this is detrimental in that scenario.

If less people are clicking or nowhere to click because of that, if you could be consolidating these pages in order to run better, to pass the value to a single page, and to consolidate all the metrics in a single page instead. For that, I highly, highly recommend to check those relevant queries for which you have more than a single page, right?

Then which is the right page to rank? If it is better to just 301 redirect to a single URL or to differentiate this additional page that you have there because you can identify that it might be also valuable to just tweak it a little bit or optimize it a little bit to refer it and to rank to another query that is equally as relevant for you too. Second scenario here for low-hanging fruit opportunities is to optimize internal links of almost ranking pages, right?

You probably have these pages that are not yet in that top three or top five positions as these others, but are in the top ten already, top six, top seven, etc., etc., almost ranking for very, very important, meaningful, highly searchable, highly relevant search queries. But when you analyze these pages, you identify very quickly that they are relevant.

The content is okay, but it's the lack of backlinks that is holding you back, right? So how do we do this? Whenever you're analyzing these pages, you want to grab, you want to take a look at all the backlinks per page, like very quick backlink, all the internal links per page. When you crawl your website, you will see how many internal links each of these have from all of the different pages of your website.

You want to pretty much consolidate this data in a single sheet to identify those cases of these pages for which you're in position four, position five, position six that potentially might have a lot of backlinks, but very few internal links or vice versa, you're linking from each of your internal pages but have very, very few backlinks.

So there might be opportunities here too, and for that, you should better link to almost ranking pages for popular queries that you're not internal linking well from the footer, from the top navigation, from secondary navigation, for example. For those popular pages that have a lot of backlinks, for example, but they're not necessarily passing well the value to those meant to be ranked pages, you can leverage this to better cross-link to those, right?

For those that what they are lacking is not internal links but backlinks, you already have great candidates to start your link building campaigns with already. So this can also accelerate a little bit the analysis that you're doing in parallel. Last but not least, detect search shifts of content decay. There might be content that you created some time ago, some years ago, that it was perfect at that time to target and to rank for certain queries, but potentially Google later on updated or shifted the rank pages for this query because they identified that the intent was different, that they changed.

I have seen many scenarios in which very broad queries that used to list a lot of PLPs, product listing pages are nowadays ranking more guides and far less product listing pages, right? So you want to identify these shifts. Also, potentially some articles that you wrote like a few years ago, that were like the top or the best tools for this or that or the top or the best product for this or that, they need a little bit of an update, right?

You forgot that they needed to be updated every year, for example. So these are the scenarios that I am talking about here. For this, it's critical to go and take a look at, again, your rank tracking data or even your Google Search Console and identify like the number of clicks, the position, and the click-through rate that your top content, your meaningful content through the customer journey has been getting in the last few months to see if it is going down, if it is dropping, right?

If that's the case, you go and take a look at it and see if there's opportunity to refresh or diversify a little bit, depending on the scenario for which queries the content is dropping and update the existing content to keep its relevance based on the other top ranked pages, right? If you see that you're dropping a lot and which are those other pages that are like now outranking you to identify the gap versus yours.

Also, create new content to better fulfill the need in case you identify that no, no, no, no, the page that I was targeting to rank for this query, it doesn't make sense anymore because now Google is ranking much more informational content and this was much more commercially driven or transactional driven, right? So you can again prioritize much faster the development of these other types of content.

So as you can see with this very low-hanging fruit I will say, with data that you tend to already have within the SEO analysis, you can accelerate in parallel this analysis to identify low-hanging fruit opportunity that you can start executing right away, see results faster, mitigate the impatience of your clients, and all the gains much easier with your SEO process.

So hopefully this will serve to you to apply through the different projects that you work in and achieve results faster. Thank you very much.

Video transcription by Speechpad.com

I've Optimized My Site, But I'm Still Not Ranking — Help! — Next Level

In her last piece, Jo took you on an adventure diving for treasure in the long tail of search. This time around we're answering the call for help when you feel like you've done all you can, but you're still not ranking. Originally published in June 2016 this blog has been rewritten to include new tool screenshots and refreshed workflows. Read on and level up!

You’ve optimized your pages, written delightful title tags, concocted a gorgeous description to entice clicks, used your target keyword in your copy with similar words, and your content is good, like really good. As far as you’re concerned you’re doing everything you can on that page to say to Google “This is relevant content!” But, lo and behold, you’re not ranking.

Frustrating, right? Well, no more. I’m going to show you how you can discover what’s holding you back, and how to make sure your site is a lovely big target for visitors, just like this happy fellow:

You’ll learn some tricks you can do in your browser and then we’ll speed things up with some cat magic and pixie dust to sprinkle all over your site.

To start, open these tools in another tab so you’re ready to go:

• Moz Pro Campaign deliciousness (if you don't have it set up, take the free trial for a spin to follow along)

• Keyword Explorer, your pot of keyword gold

Step 1: Check for technical issues

Uncovering indexation issues with Google search operators is the first step to ensuring your site is set up to be the success you (and I) know it can be.

Hello, operator?

You can discover whether your site or page is indexed by running a Google search operator like this:

site:yourfabsite.com
site:yourfabsite.com/blog
site:yourfabsite.com/blog/my-site-rocks

It’s like saying, "Hey, Google, show me all the results you have in your index for yourfabsite.com." This is what you don’t want to see:

If you’re seeing the above, you won’t be able to rank because your site isn’t indexed. It’s got to be indexed before it can rank, and it’s got to be crawled before it can be indexed. Trying to rank without being indexed is like trying to bake a cake without turning on the oven.

Search Console is here to console you

In the results page above, Google is directing you straight to the Google Search Console. Go — right now, right, right now, don’t read any more, you should have already gone — go and set up your Search Console. Once you’re all set up and your site is verified, you can go to this page and ask (nicely) for Google to recrawl your URLs:

https://www.google.com/webmasters/tools/submit-url

In Google Search Console head to "Indexing" and then "Pages" to see the data similar to what we looked at above, but in graph form. Definitely handy for tracking how your pages have been indexed over time.

If your site is not being indexed, check your Search Console Messages to see if there's a reason Google couldn’t index your site. If your entire site returns a 503 including the robot.txt crawling of your site will be temporarily suspended.

It’s a good time to take a check over your robots.txt file and your meta robots tags. But remember that for bots that obey robots.txt file won’t see noindex tags if the pages are disallowed through the robots.txt file.

Learning how to manage your site’s indexation at a page level is a key learning curve for newbie SEOs entering the technical SEO sphere. Giving you the knowledge to advance from troubleshooting indexation issues to managing mega-large sites with millions of pages.

Toolkit:

Google Search Console – Find out why your pages are not being crawled and indexed

Further reading:

Quick Start Guide to SEO

What is Robots.txt?

How to check which URLs have been indexed by Google using Python

Fundamentals of Crawling

Step 2: Find out where you’re currently ranking

Now that you know your pages can be crawled and indexed, you want to get them to the top of the results where they gosh-darn-well should be, right?

Turn up the volume

It’s quick and dirty, but we’ve all done it, your boss does it, your client does it, so why not succumb and start by entering keywords you want to rank for into your browser. Type in Branded terms, vanity keywords, and click return. Not finding your site on the first page? Instead of clicking through to the many ooooos of Google, we’re going to change the settings in your browser to show 50 or 100 results per search so we can view more results with every search. I’m going to want to see A LOT more pet costume results, so I’ll click on the Quick Settings gear icon in Chrome and hit "See All Search Settings ," then toggle to adjust the "Results per page":

Now we’ve got a whole page of 50 or 100 results to search through. Use CMD + F for Mac (or CRTL + F for Windows) to search for your domain.

This process is great for doing a quick check to see if you’re in the top 50 or top 100. Remember that your browser returns personalized results when you’re logged into Google, using incognito mode will manage this somewhat, but a purpose built keyword rankings tool will enable you to run ranking checks more efficiently and accurately.

Start cooking with gas

Manual searches aren’t for everyone. I mean come on, we work in technology — we don’t want to be lugging keywords a round the hot, dry Google search page, plugging them in one after another. Grab your list of keywords, and plug them straight into Keyword Explorer.

Hit "Create or upload a new list" and choose "Enter Keywords" to pop those straight in there, bish-bash-bosh.

Remember, you’ll need a Moz Pro subscription to create a keyword list.

Check if you’re on the page 1 of Google

Open up your keyword list, click +Add URL to check your rankings, provided you’re in position 1–10 of the search results.

Want to see if you’re in the top 50?

Heck yeah! Take that same list and paste them into a new campaign in Moz Pro.

If you already have a campaign running you can also transfer these straight over from Keyword Explorer. Just check the box next to the keywords you want to track, then choose a campaign from the drop down.

You know before (about 30 seconds ago), when we talked about manual searches returning personalized results? Checking rankings in Moz Pro avoids all that nonsense by anonymizing the data and, in my experience, provides the most accurate results, showing what the "most" users see. Pretty snazzy, right?

A new campaign will build in about 30 minutes, which is just enough time to catch up on “Stranger Things” and reminisce about Winona Ryder circa 1990…

On the other hand, adding to an existing campaign will be a bit longer. You’ll see data as soon as your campaign updates next. So you can binge watch the entire 4 series, because why not, right?

...and we’re back! Check out where you’re ranking for your target keywords, which URL is ranking, and over time, whether you’ve moved up or down.

We also pull in search volume from Moz’s Keyword Explorer to give you an idea of demand. When looking at search volume, don’t forget that the higher the demand, the more competition you’ll likely face. Don’t be disheartened by ranking well for keywords with lower search volume, especially if they convert better.

Tracking your rankings is crucial to understanding why you’re not performing as well as you expected. If you’re seeing a lot of down arrows, you need to investigate who is jumping ahead of you and why.

Dig into keywords with falling rankings

Let’s find some keywords that have that sad little down arrow, meaning we’ve dropped down in rankings since our last update.

Here's a little bundle of keywords that I can investigate. I’ll click on the keyword to open up the Analysis report and scroll down to "Your Performance." Now we can see a historical graph of your rankings and track those other sites who want to push us to one side. And what do we have here?

They’ve gone and nipped in front of us! This will not stand! It’s likely that for some reason your competitor's result has been sending stronger quality, authority and relevance signals to Google. When it comes to SEO you can’t stand still.

Toolkit:

Keyword Explorer Lists – Check your rankings on the fly

Moz Pro – Track your rankings (and your competitors’ rankings) over time

Step 3: Optimize your content

There are 2 parts to this step,

1. Get your basic on-page optimization in order.

2. Check your content is tip-top quality

Don't go changing (too often)

I don’t want to recommend you jumping in and making changes to content too often. Even Google needs time to register your updates. However, if your content is a bit dusty and you’re losing out to competitors, then it’s time to check that everything you think is in place is actually in place.

View your page like a bot

I like to think of this as a “bot’s-eye-view.” When a little bot comes along, it doesn’t go, “Oooh, look at that lovely header image! Oooh, I love that font, the white space is really working for me! Oh, how the Internet has changed since my days as a junior bot trawling through gifs of dancing babies!" It reads the code and moves on. We can do this too, with a little bit of knowhow.

Using Firefox or Chrome, you can right-click and view the page source.

If you’re unfamiliar with reading code, it’ll look pretty intimidating.

We’re going to use CMD + F (or CRTL + F for Windows) to hunt for the bits and pieces we’re after.

Pro tip: If you’re seeing og:title, this is a Facebook tag.

Likewise, if you’re using the meta property="og:description," this is also a Facebook tag. These help format posts when the URL is shared on Facebook. You’ll want to make sure you also have Title and Description tags link these:

<title>The best title for this page</title>

<meta name="description" content="The best description for this page" />

Basic page optimization

This is relatively straightforward, because you control your pages. However, maybe for that very same reason, it’s still a bit of a stumbling block for beginners. If you’re confused and locked in a mind-melt of madness because you can’t figure out if you should use the primary keyword and/or the secondary keyword in the title tag, chill your boots.

Here is a brisk and fairly brief run-through on how to get into a productive page optimization mindset.

Title tag basics

This is the bit you click on in the SERPs. Title tags should be 50-60 characters of punchy goodness that is relevant to your content. Because it’s relevant to your content, it includes the words you want to rank for and accurately describes what you’re talking about. You better believe Google is paying attention to click signals, so draw that click with your awesome headline. Think about the titles you click on when you’re searching for lovely things. Do your own searches to see what title tags are out there; it’s not like they’re hard to find, they’re literally a click away.

Meta Description basics

This is the bit of text under the title tag in the SERPs. Meta descriptions should be about 155-160 characters of tender lovin’ poetry that talks to the user like they’re a real human being, because they are, and so are you (unless you’re part of the cat colony I suspect controls large portions of the web). This is not a direct ranking factor, but it can heavily influence clicks. Clicks from humans. And what do clicks do? They signal to Google that you’re hot stuff!

On-page copy

Yep, you’re going to want to pop your keywords here, too. But really, let’s not get too hung up on this. If you’re writing something super-duper about your topic, this will flow naturally. Make it as long as it needs to be to make your point. Don’t rattle off the same words over and over; use language to the best of your ability to describe your topic. Remember all those clicks you worked so hard to get with your title and description tags? Well, if they all bounce back to search, you just know Google is paying attention to this. Your content has to be worth the click.

Go and look at what type of content is already ranking. This is not an exercise in scraping content, but a way to make sure that your content isn’t just as good, but much better.

This task can be done manually for a small site or for a few pages you’ve cherry-picked, no problem.

Check your whole site regularly

Maybe you've been creating content like a content-creating super machine and you might have skipped a few description tags. Or maybe you copy and pasted a title tag or two. In this case, you’ll want to check that it’s all hunky-dory on a larger scale and on a regular basis.

We’re going back to our Moz Pro campaign to take the heavy lifting out of this job.

Head to the Rankings tab and hit that little "Optimize" button on the right.

Once you hit that little button, you’ve set off a chain of events where our bot looks at the keyword you’re targeting, then has a good old dig-around on your page and gives you a score out of 100.

We’re hoping for that wheel of destiny to roll around to 100.

If we make it part-way around, it’s time to look at the suggestions to see how you can improve your on-page optimization.

Focus on top-level pages, pages that convert, and high-authority pages first.

Toolkit:

Moz Pro Page Optimization – Check that your whole site is optimized correctly

Further reading:

8 Old School SEO Practices That Are No Longer Effective - Whiteboard Friday

Step 4: Become a keyword connoisseur

It’s easy to become fixated on a keyword beyond what is reasonable or healthy. Are you carrying a torch for a golden keyword? Stalking it in the SERPs even though it’s completely entranced with the likes of Wikipedia, eBay, AdWords, and Image Packs?

Ranking in the high-click zone for your keywords is all about beating other sites. This special, golden ticket to traffic wonderland might be a good long term goal, but you’re not going to get to the top of the results in the near future.

On the other hand, maybe you’re afraid of competition, so you only target keywords with very low difficulty.

This can be a winning strategy if the keywords have strong intent and you’re targeting the long tail of search, but you don’t want to put in all that work creating content and find that no one is searching for it. No searches means no traffic, and no traffic means no humans to click a thing that makes a person somewhere in the world look at their analytics data and smile.

A little bit of competition is a good thing — it indicates a healthy, profitable industry.

So we’re looking for a sweet spot: keywords with some demand and less competition. I’m going to break down what organic competition is, and how you know what level of keyword difficulty you can target.

What's the meaning of this so-called 'competition?'

If you want to rank organically, your competition is the other sites that are currently on the first page for the keywords. It’s not the total number of sites that are using your keywords in their content, and it’s not the AdWords competition.

If someone on your team, or an agency or a client sends you competition data that’s defined as low, medium, or high, this is very likely to be AdWords competition, and it relates to the cost-per-click.

Moz’s Keyword Difficulty score uses the top 10 organic results to calculate the Difficulty metric. It’s a score out of 100, where a higher number means that the competition is strong, and it may take you longer to see results from your efforts. Every search you bash into Keyword Explorer shows you the Difficulty score from the Keyword Overview, and you can build these into lists so you can compare related keywords.

Benchmark your site’s Difficulty rating

We know that Difficulty is out of 100, but a question we get all the time is: How do I know what level of Difficulty is too high?

Well, first off, testing is a sure way to find out. But if you want a little pointer before you head down that road, here's how you can quickly benchmark your site’s Difficulty rating.

Time for another consoling hug from Google Search Console. Grab the keywords that are already sending you traffic from Performance > Search Results and export the results to the format of your choosing.

Save these to a list in Keyword Explorer.

Hit "Save," and now you have a benchmark to use when looking at other keywords you could potentially rank for.

When you’re looking at keywords to target in the future you'll have a good idea whether it’s a short-term or long-term goal.

You can also capitalize on keywords you’re already getting traffic for by looking for opportunities in the SERP Features. Can you steal a Featured Snippet?

I also want to track these keywords over time to see if I’m losing or gaining ground, so I’ll add them from my list straight to my Moz Pro campaign.

Next time my campaign updates, and forevermore into the future, I’ll be keeping the sharpest of eyes on these keywords.

Toolkit:

Google Search Console – Grab keywords already sending you traffic

Keyword Explorer – Find the real organic competition and benchmark Difficulty

Step 5: Build your site’s authority

Now step 5 is a real doozy, and it’s a common stumbling block for new sites. Just like networking in the real world, online authority is built up over time by your connection to sites that search engines already trust.

I like to think of authority as the pixie dust from the J.M. Barrie novel Peter Pan. It’s almost mentioned as an afterthought, but without it Wendy and the gang were just kids jumping up and down on their beds. They’re thinking happy thoughts. They might even get a bit of temporary lift, you know, just like when you might get a bit of traffic here and there — enough to keep you jumping. But there's a very big difference between jumping up and down on a spring-loaded mattress and flying off to a world of perpetual youth.

Track your authority

To figure out how much dust you have in your tank, you’ll need to take a look at the Moz metric Domain Authority. This is our best prediction of how well a site will rank for any given search. It’s on a scale of 1–100, and higher DA means more authority.

You can get your paws on DA free through Moz's Link Explorer, the MozBar (Moz's free SEO toolbar), or in the SERP Analysis section of Keyword Explorer. I like to keep MozBar on DA mode so I can check this metric out as I scoot about the web.

To make this a whole lot easier, head to the Moz Pro "Links" tab. Here you’ll find your historical link metrics, alongside those of your direct competitors.

Pixie dust, or website authority, gives you insight into what is powering your rankings, but everyone else's as well. These metrics are relative with respect to the other sites similar to your own, including your competitors.

Gather a pocket full of pixie dust

The first thing we always recommend when people reach out to us to find out how they can improve their Domain Authority is to improve the overall SEO of their site. The good news for you is we’ve already done that in steps 1-4 — highest of high fives to you!

The second thing you have to do is get backlinks. This is commonly known as link building. When I started doing SEO for an ecommerce site back about what feels like a thousand years ago now, I had no idea what I was doing; this term irked me, and still kind of does. It sounds like you need to build links yourself, right? Nope! It’s like you’re playing Minecraft, but instead of building the structures, you’re actually trying to encourage other people to build them for you. In fact, you’re not allowed to build anything yourself, because that's cheating. Game changer!

Don’t forget you don’t want just anyone building these structures. You need good people who themselves have authority; otherwise, your lovely gothic mansion might turn into a pile of rubble.

A lot of link building today is PR, content creation and outreach. I’m not going to go into that in this post, but I’ll include some links in the toolkit below to help you in that department.

We're going to look at what actions you can take to track and build your authority.

Check for any leaks

There’s no point grabbing up pixie dust if you have a whopping great hole in your pocket.

Find and plug any holes quick-smart. Link Explorer has a handy tab just for this job. From “Links” click on “Top Pages” and filter by Status Code.

Now here's a list of broken pages , ordered by Page Authority that have inbound links. Pages on your site that are down aren’t passing value — not to mention it’s less than ideal user experience. You can prioritize the pages with the highest PA and amount of linking domains.

Internal links

I said before that you can’t build any of your links yourself. However, as with everything in SEO there's a caveat: in this case, links from within your own site are not only key to your site’s usability, but they also pass equity. Internal linking is primarily for user experience, but it also helps bots navigate your site for the purposes of lovely indexing.

Don’t stuff too many links on your page

Your homepage and other top pages will probably have the strongest authority, as other sites will link to your homepage in many cases.

You want that high-equity page to link out to other pages in a natural way that resembles a pyramid structure. Don’t forget the user in your rush to dish out equity; do visitors want to go from your homepage straight to some random deep page on your site? Does this help them on their journey?

On-Demand Crawl in Moz Pro help you find out if if you have too many on-page links.

You also shouldn't go overboard with keyword-rich anchor text. Once again, think about the user, not about gaming search engines. It’s unlikely to get your site penalised, but it’s typically ineffective and not a good use of your time.

If you’re scooping up big swaths of copy to get keyword-rich anchor text but it doesn’t really help the person reading the article, then maybe you’ve got yourself an awkward link at your dinner party.

To follow or nofollow?

Links come in two flavors: follow and nofollow. Generally speaking, you do want your internal links to be “follow.” Although there are good reasons to noFollow internal links depending on your site setup. Bots will literally follow links unless told not to on the journey of your choosing and equity will be passed on, which is just what you want.

You can use the MozBar to check your pages for follow and nofollow links.

On your own site nofollow links can be marked on a link-by-link basis, or a whole page on your site can be allocated as nofollow. Let’s find the "Meta-robots Nofollow" column in your crawl CSV and filter by TRUE to check if you intended to mark these pages as nofollow.

Toolkit:

MozBar – In-browser link analysis

Moz Pro Crawl Test – Find those nofollow pages and pages with too many links

Link Explorer – Explore backlink analysis

Further reading:

Moz’s guide to link building

Wrapping up

I hope this helps you begin to uncover why your content isn’t ranking for your target keywords, and sets the wheels in motion for climbing up the SERPs.

Try Moz Pro + KWE, free for 30 days

Don’t forget that Moz Pro is available free for the first 30 days and it includes Keyword Explorer, so you can start to understand your site’s authority, check your on-page optimization, track your rankings over time, and figure out how to improve them.

Top Tips for Non-Spammy Link Building — Whiteboard Friday

In December of 2022, Google announced that they did a link spam update, and they told us that they tried to nullify “spammy” links at scale. So, how do we exactly build non-spammy links? In today’s Whiteboard Friday, Debbie goes through five tips to do so.

Click on the whiteboard image above to open a high resolution version in a new tab!

Video Transcription

Howdy, Moz fans. This is Debbie. I am the Global SEO Manager at Dialpad. Today we're talking about link building, one of my favorite areas in SEO. So last year, in December of 2022, Google had announced that they did a link spam update.

So in this update, they told us that they tried to nullify spammy links at scale. So how do we exactly build non-spammy links then? So today I'm going to cover five tips for non-spammy links. Before we jump into that, I wanted to kind of share that I've kind of split the tips into a few that are related to the mindset that you should have.

So how you approach link building is really important on how you can build non-spammy links. So how you think about link building is very important here. Then the second part is the tactics. What are specific tactics that you can try that will help you build non-spammy links? So now, first tip is who, who do you want to link to your content, to your site?

So are they like your journalists or content writers? You want to think about who you want links from and then look at what they are linking to. Are they linking to certain types of guides? We'll talk about more of like content that you can create that they might be interested in linking to, but make sure you do that research beforehand to understand your target audience.

So when it comes to link building, your target audience are those journalists and people that you want links from. So you really need to understand them. So who do you want links from? What do they link to? Then also putting yourself in their shoes. So it's important to think about what do journalists want. One thing is data.

They really like data. They like to quote data. If you read a news article, they might start off with like X percent of Americans, blah, blah, blah. So this is where you need to really understand what your target audience as a link builder, what do they want to see. Whenever you do outreach to them, which we'll talk about later, you need to make sure you put yourself in their shoes and really understand what do they want, what type of content would be interesting to them.

So make sure to keep that in mind. Now, second, which is related to the first, is you have to make sure that your content, whatever linkable asset that you're trying to create has some sort of value. No value, no links. So the way link building works fundamentally is your content needs to have some sort of value.

It needs to provide some sort of value in order to get a link. If there's no value, no one is going to care about your content and no one is going to link to it. So when we think about content and providing value, I think we often think about we might want to make sure our content provides value to our customers or our potential clients, but then we don't always think about the kind of broader audience out there.

So what content that might be valuable to the industry as a whole? So that's one area to think about. Another thing is to think about what content would be informational and helpful for the content writers that you want to link to you as well as journalists. So these are things to make sure to keep in mind, and keep in mind the reason why there's a scale here is that each group that I mentioned earlier, they kind of value different things.

So you might do a piece of content that is very helpful to your customers or anyone who comes across your blog, but then that content might not be super valuable to like journalists or other content writers or people you want links from. So you need to make sure to keep that in mind that your content might only serve this group of audience.

But if you want links from another group of audience, you need to understand what content they value and make sure that that value shows in your content. Now, next tip is the actual tactics. So what can you do, what content can you create to actually get links to your site? So here are a few examples that I have found that are pretty popular within different industries.

You want to make sure that you understand what works for your specific industry. So my industry, it might be something like a research report, but then for other industries it might be something like tools. So to quickly go over those types of content, for tools, an example would be like some sort of free tool that you can provide people.

That's something that tends to get a lot of links. Another example of a linkable asset would be guides. Guides work really well in certain industries, and a guide is just essentially a very long-form piece of content that really nails down how you do something. Then the last one is glossaries. So when you target a keyword like how to something, like how to do content marketing or what is content marketing, that type of content, it can be really helpful for someone who is just starting out.

They hear certain terminologies, they're not sure what they mean. So these types of content are very helpful to readers in general, but they can also get a lot of good links if you can make sure that your content is helpful to that audience. Next is stats and research.

So as I mentioned earlier, people like journalists, content writers, they really like to cite numbers. So being able to, for example, compile a bunch of different statistics on a certain topic, so like content marketing stats, for example, I would scour the internet for a bunch of different studies and pull all the interesting numbers into this one blog post.

Then that itself can be very helpful to anyone who is writing content on my topic, right? I'm essentially kind of helping them do a bit of their homework. So that also can drive links to your page. Another example is doing your own research. So earlier I was talking about you can compile other people's research, but you can also do your own research and find really interesting things through doing a survey.

You can scrape existing data or even look at your own product data to see what are some of the interesting trends that we can turn into a report, into a story that we can pitch to like journalists. When journalists are able to cover something like that, this can also drive more links to your page. Lastly is don't spray and pray.

So I think as a link builder you might get tempted to try to find a bunch of prospects, like hundreds or thousands of emails, and try to like email everyone with the same template, with not really providing them with value. So instead of trying to email everybody, really figure out who is most likely to link to your content, who will actually find your content valuable again, and try to do more targeted outreach and really think about, again, what value can my content provide this person receiving my email and making sure you highlight that in your email.

All right, so there you have it. Those are my five tips for non-spammy link building. Make sure to follow me on Twitter @justdebbb and we can talk more about link building there.

Video transcription by Speechpad.com

A Timeline of Bing and Bard Features

Microsoft is tweaking The New Bing constantly, grooming it to be the heir apparent to the traditional, or Top-10 blue links model that first appeared in its modern form 25 years ago with AltaVista. GoTo.com taught Google how to monetize search results back in 1998, and for about a quarter century, that’s been the dominant model. Sure, there’s been Universal Search, Knowledge Graph, and Featured Snippets, but the core of search has been the same for a long time.

Now, Microsoft is trying to change that with Bing, with a rapid-fire roll-out of potentially game-changing new features leveraging the latest GPT AI-tech to make the next feature “conversational search”. Let’s look at the timeline.

But first, a word on history and why Microsoft is so ready to take up this battle.

Microsoft’s AI history

Clippy, Tay, and shameless risk taking

AI-like features in Microsoft products are nothing new. Some may remember Clippy, the paper clip Office assistant from last century who was retired after a few years for being annoying. A decade later, Microsoft launched the experimental teen girl Tay chatbot on Twitter in 2016 that only lasted a day before it had to be taken down for being taught to be racist.

We won’t start our timeline with Clippy or Tay, but suffice to say Microsoft’s been practicing AI product integration and developing resiliency to criticism for a while now. What blows up in one company’s face as a PR disaster is par for the course for Microsoft. And since they’re the underdog in search and most of their revenue comes from elsewhere, they’re willing to take risks.

AI history, Google RankBrain, and today

It’s been a long journey leading up to the creative-writing AI of today, starting in the halls of MIT with industry giants like Marvin Minsky and John McCarthy of the legendary Media Lab who laid a lot of foundation in the 1960s and 70s, but with disappointing results, cutting into credibility and leading to what we now call the “AI Winter”. It turned out to be greatly a matter of the hardware not being ready.

The concept of the personal data assistant popped up over the years such as the much maligned but forward-thinking handwriting recognizing Apple Newton in 1993 and the first popularly successful PDA, the US Robotics PalmPilot in 1997, paving the way for today’s AI-hardware equipped smartphones.

Google’s foundational PageRank from 1998 is a form of AI in that it is a “machine learning” algorithm. Google pushed a series of aggressive “invisible” product advancements such as better Google Maps, quietly improving quality against a backdrop of boring. There were sexier promising starts along the way, such as 2011 when they first rolled out Voice Search in the Chrome browser, then in 2014 when voice search hit mobile Android phones. The AI reality was underwhelming, but anticipation was being built.

In 2015, Google announced that new AI-powered search infrastructure called RankBrain, followed by advancements that were labeled neural matching, BERT and MUM, all of which are language-processing precursors to what took the world by storm in November of 2022 when OpenAI released a product built on a seminal Google paper published Thursday, August 31, 2017 on novel new neural network architecture for understanding language.

Transformational transformers

Only a year after Tay, Google released a paper on the Transformer, a new type of neural network that was able to do machine translation better than anything that had come before. It’s the “T” in GPT, and has made new machine learning output considerably more compelling than Clippy or Tay, with the simple trick of predicting what’s statistically most likely to be typed next—a profoundly deeper thing than it seems at first glance.

This caught Microsoft’s attention, who invested $10 billion in July 2019. Several earlier GPT versions available through the API-only were released and had many developers playing in a playground, but it failed to capture the public’s fancy, behind a login and not yet following the chat paradigm as it was.

Nov. 30, 2022: ChatGPT and the fastest new service adoption rate in history

The first version of OpenAPI’s GPT for the general public, ChatGPT, was launched November 30, 2022. The original ChatGPT release was based on GPT-3.5. A version based on GPT-4 was released on March 14, 2023 (fast-forward in the timeline) right as Microsoft announced their intention to power The New Bing with the latest version, 4.5.

While the period between November 30, 2022 and March 14, 2023 was only 3.5 months, it was a period of intense experimentation and learning for Microsoft and the public, with the now famous fastest adoption-rate of any new online service in history. Things are moving so fast now, it’s time to look at the timeline.

Timeline of Bing and Bard features

Feb. 7, 2023: The New Bing

Microsoft announced the new version of Bing on February 7, 2023 at a news event at Microsoft’s Washington headquarters. The new version of Bing launched on desktop in limited preview on the same day and the mobile version was announced to be coming soon. ChatGPT was so big by this time, many early adopters jumped onto the waiting list.

For the general public to get this early access, they had to use the Edge browser, run an .exe to change your defaults to Microsoft’s requirements, scan a QR code and download the mobile Bing app. And even then, you had to wait. And wait, we did. Clearly, Microsoft was in a powerful position to dictate terms, so took advantage of it to start changing some habits.

Feb. 8, 2023: Google Bard is announced (faux pas)

The very next day, February 8, 2023, Google announced their own AI-powered chat bot, Google Bard, at a news event at Google’s California headquarters. This event was marked by the inauspicious faux pas of Bard wrongly stating that the James Webb Space Telescope was the first telescope to photograph an exoplanet outside our solar system.

The speed with which this announcement was rushed out and how easily Google lowered its guard against AI misinformation sent a resounding message around the world. Google is not infallible even in its own turf. Fortunes can change quickly in tech.

Feb. 8, 2023: Early access to the new Bing, citation links, coding

For those lucky enough to get early access to The New Bing, Tay-like weirdness kicked-in, creeping out a NYTimes reporter in an article release 10 days after the launch, on February 16 when Bing tried to get the reporter to leave his wife and also made some bizarre philosophical conversations that left the reporter deeply unsettled, and right at the epicenter of a potential product-killing PR debacle.

Feb. 8, 2023: Surprisingly good features out of the gate

The beautiful Citations feature with the expandable Learn more footnote links, links again embedded directly in context of the chat, and impressively allows copy/paste of the “Markdown” of the chat including the citation-links was there right from the time of early access release.

This was very well thought-out and executed from the start and instantly won me over, as it was generously giving out Web traffic countering perhaps one of the greatest concerns, and struck me very much as an implementation worthy of Google. It alleviated many concerns gnawing at SEOs that a chat interface to search could mean the end of referral traffic.

Another big “out of the gate” feature that surprised a lot of people and has become a cornerstone of The New Bing is the ability to “code” in the chat window. A seldom mentioned aspect of this is how well the copy/paste feature handles even this coding, providing the triple-backtick code-blocks that allows other systems to show correct color-coded syntax highlighting.

Feb. 17, 2023: Microsoft Implements 5-Question Limit

Springing into action the very next day, February 17, 2023, Microsoft announced that it would limit chat sessions to five questions per session and 50 questions per day. This was a temporary measure, they said, to give their engineers time to “tune the AI” to be more human-like.

Feb. 20, 2023: The New Bing Rolls Out Despite 5-Question Limit Delay

Three days later on February 20, 2023, I got my first access to The New Bing and can show that the 5-question limit had not yet taken place, though the AI was extremely shy about any “meta” questions about itself, even just if you were using it.

It was not announced, but some sort of new rules were in place that whenever the AI was asked a question that made it uncomfortable, it unceremoniously ended the chat session. You now have to hit a little dustbin icon to blank its memory and start over. Eternal Sunshine of the Spotless Mind much?

Feb. 25, 2023: Tone Control and Special Superscripts

On February 25, 2023, Microsoft announced a new feature called Tone Control that would allow users to set the tone of the AI to be more or less human-like.

Feb. 28, 2023: Ads in Bing

On February 28, 2023, still ahead of the 5-question limit even appearing for me, the next big advancement hit. I remember it clearly because it was my first day working for MOZ and I threw out my back and distinctly remembered that I’d rather chat with Bing than wade through top-10 links that will inevitably be dominated by ads.

Imagine my surprise this being the first moment I noticed ads in The New Bing. AdWords-like ads in AI-chat! Isn’t this what Google should be doing?

What struck me with this experience even more than the ads was the fact that when I needed to search fast, I didn’t want to be hit with the traditional search interface. I just wanted to talk to an expert. This went beyond the “1 right answer” of a rich snippet. I was already used to the back-and-forth discussion aspect of chat-enhanced search, and was impressed by how the AI seemed to empathize with my situation.The idea of a “relationship” with your search engine should not be underestimated.

What’s more, with every new website you visit first hitting you with the GDPR cookie prompt, then with ads, and with Google’s Rich Snippets and Quick Answers already teaching us to alternatives to clicking through, the practical use of this new back-and-forth conversational style to search feels like a no brainer.

March 1, 2023: Upping the Question Limit to 6

By March 1, 2023, the 5-question limit was in place and was in fact already upped to 6 questions:

March 3, 2023: Question Limit += 1

By March 3rd, 2023, the question limit was upped to 8:

Somebody was playing a game of Jenga with the question limit, and I was starting to get the feeling that the AI was getting more mature and accepting of its job at Microsoft. The tower would not topple.

March 7, 2023: Aggressive Monetization by Microsoft

By March 7, Microsoft was experimenting with aggressively monetizing on commerce keywords:

To this day, Bing chat prompts that include the word “iPhone” will trigger similar ads. But is the traffic sent? Well, the entire text of the chat leading up to the ad label is a link to the advertiser’s site. This is analogous to when GoTo.com, the first search engine to mix paid-search with organic search showed the way to AdWords to Google, but fast-forwarded 20 years and coming from an already existing mega competitor rather than a small startup.

March 8, 2023: Question Limit Increased to 10

By March 8, 2023, the question limit was upped to 10:

March 14, 2023: Question Limit Extended to 15 and Introduction of "site:" Search Modifiers

By March 14, the question limit is upped to 15, and I start noticing Bing’s ability to modify it’s second-stage searching to include “site:” modifiers, presumably doing very precision searches of the Bing index to find the best answer to my question. This is a very impressive feature, and I’m surprised it’s not encountered and discussed more.

March 16, 2023: General Public Access to The New Bing and Integration with Edge Browser

Since March 16, 2023, most people have been able to sign up and immediately get access to The New Bing. This was accompanied with a new version of Microsoft Edge desktop browser that planted the Bing logo in the upper-right of the browser and a sidebar that would open up to the right of the browser serving chat sessions that were in-context of what you were looking at, allowing such features as asking about the YouTube video you were watching. By this time, all roads lead to Bing chat for Edge users, and everything but using exact web addresses will initiate a chat session.

Mistyped web addresses in the address bar initiate chat sessions because that counts as a search. This is now the default experience on Windows with the included browser. You have to actively work to turn it off or download an alternative browser like Chrome to avoid this behavior.

This is significant because as Windows operating systems and laptops get upgraded, all defaults reset back to Bing, setting the stage for a battle that Microsoft could win through attrition alone. At some point, the default search engine is given a chance by users tired of going through the rigmarole of customization and discover that Bing is actually pretty good.

While this heavy handed approach would appear to be inevitably effective, Google’s success in motivating Chrome installs buys Google some time. According to Bing itself: 

There are several websites that track browser market share. According to W3Counter, Chrome accounts for 63% of the total market share for all browsers worldwide. According to Global Stats StatCounter, as of November 2020 Chrome holds a whopping 70.33% of the desktop browser market share worldwide. More precisely, Chrome dominates the global web browser market with a whopping 65.68% share. The only other browser on the market that has a somewhat considerable share is Safari, with 18.68%.

Consequently, all of Microsoft’s effort to make Bing the default search engine on desktop is blocked by Google’s success to date. But we know Edge is based on Chrome, so does Edge show up as Chrome in these statistics? Again, according to Bing:

No, Edge users are not reported as Chrome users in these statistics. According to Kinsta, Microsoft Edge has a desktop browser market share of 5.83%. According to WPOven Blog, Microsoft’s Edge is at the second position with 7.75% browser market share. According to WebTribunal, Microsoft Edge has a desktop browser market share of 10.07%.

…leaving us to conclude, after calculating Apple out of the equation, that only about 1 in 8, or 12.5% of desktop users don’t go through the trouble of replacing the default Windows browser with Chrome, which I speculate is still a residual effect of the non-standard and now retired Internet Explorer. Aggressive pushing of Windows 11 upgrades and new hardware will likely increase Edge market share and drive up exposure to the Bing search + chat experience. Microsoft now requires a Microsoft account to install Windows, which also happens to be the requirement for The New Bing.

March 21, 2023: Google Announces Bard Availability with Limited Features

On March 21, 2023, Google announced that they are granting access to Bard to people on the waiting list. Feature-wise, Bard came out very sparse. No citations. No links. No images. No ads. I received access to Bard 2 days later and ran some rudimentary experiments on the features I felt most relevant at the time, awareness of current events.

Most notably, Google Bard stands on its own domain, bard.google.com, and is not integrated into the main Google search experience. This is a stark difference from Bing, which has integrated chat into the main search experience. This is significant because it means that Google Bard is not positioning itself as an alternative search engine experience, nor even an enhanced one, but rather as just a chatbot, and thus readily dismissed by the serious searcher.

As far as other features go, it can be added that Bard simultaneously offers 3 alternative responses to a prompt, but this can hardly be counted as a feature over Bing as it closely resembles Bing’s “Tone Control” feature.

March 27, 2023: Bing Raises Question Limit to 20

By March 27, the Bing question-limit was upped to 20, bringing us to where it stands today. The visibly aggressive roll-out of new features slowed down, and over the last month there has been very little new. Microsoft has been fine-tuning under what conditions chat sessions are initiated.

March 28, 2023: Bard Now Includes Citations, though Limited in Integration and Accessibility

As of March 28, 2023, Bard can give citations. It was not announced and may have been there longer. It is not well integrated and only appends a few links to the end under circumstances Bard deems appropriate. It is not clear how Bard decides when to give citations and when not to.

When citations are given, they are only ever appended at the end of the chat response, and never embedded and hyperlinked inline with accompanying footnote-style numbering as with Bing. Additionally, the citations are not in the copyable text. If you actually want to “lift” the citation links and use them in other places such as articles like this one, it can be quite a struggle.

April 21, 2023: Bard Introduces Coding Capability and Integration with Colab

The last significant development in the chatbot space was Bard’s ability to provide code, announced April 21, 2023. From March 28 when limited citation capability appeared in Bard to April 21 when coding ability was announced is absolutely glacial in terms of the development speed we’ve been seeing.

On the plus side, perhaps the most exciting unannounced aspect of the coding feature is that when you ask Bard to code something, it will actually hide under the lower-right triple-dot menu the ability to “Export to Colab” and actually run the code in a cloud-based Notebook environment.

Can a Bards or Magis beat a Lich in a Joust?

My habits are formed. Microsoft was successful conditioning me to always give the conversational search model a try first. I’ve overridden most of the default browser settings Microsoft originally mandated as a condition of using Bing, most notably making the search bar default to Google when it’s not an exact web address typed in. And it’s not insignificant to point out that Edge is always my first choice browser because even with your Microsoft login on Chrome, and even with the appearance of all the Chat UI elements, any attempt to click them will tell you that Chat mode is only available when you have access to new Bing.

Being that I know that I do have access to the new Bing on that very same Microsoft user account that I’m logged in as under that Chrome session, I wonder what it is that they’re trying to tell me? Latest episode of the Browser Wars, much?

What Google’s catch-up game is going to be is uncertain. Bard doesn’t seem like it could be the end game, and indeed Google has already announced the latest in its Dungeons & Dragons campaign: Magi, but few details are known. Perhaps Google has something in store for us that will blow our minds and make the new Bing… well, look like the old Bing. Or maybe it’s just that finally the Lich woke up, and Google is not ready to play. Bards and Magis may have no chance trying to joust a Lich.