Name: AI Visibility: What Actually Matters (And What Doesn’t)
Uploaded: 2026-03-11T19:16:09.135Z
Duration: 1 h 28 s
Description: AI Visibility: What Actually Matters (And What Doesn’t)

Transcript for "AI Visibility: What Actually Matters (And What Doesn’t)": Howdy, everyone. Sorry for our delay here. Just kicking off the morning a tiny bit behind, but, we've got an incredible presentation for you. I'm thrilled you're with us. I believe my colleague Amanda will be joining shortly. Howdy, Amanda. Can you hear me? Yep. I can hear you. Really? Yeah. It's not working. This is my MacBook microphone. I. No one cares. I'm sorry. This has been a morning. How are you? hear you. I hear you. I am gonna kick off right away because we have so much to get through, and I know we're starting a little bit late. So, the two questions we always get, one is, will I get the recording? And the answer is yes. The recording will be sent to your email. If you have colleagues who want that recording, tell them to register at the same link that you did, and we will make sure that they get them. The the recording comes out very shortly after this. We also put it on YouTube, and we've been talking about taking these webinars and putting them on on the blog with the YouTube video embedded. So you you may be able to find it on the blog depending on when Amanda has time. She and. I are very busy writing this. are. Well, Oh, though this Friday, we'll start publishing our old office hours. We're thinking of hosting it on Wistia great. on the website. That way, it is an an ad free experience, and you all deserve the best. An excellent point. Alrighty. Let's see. I am going to present GoldCast made me switch away from Firefox, so I now need to find their new every time we use them, they have put the share in a new place, which is always fun. And that's how you turn off the camera. Where is it? It is surely on the oh, top left where it says screen. Sure. Great. Why not? Let's just put it there. Alright. Yes. Boom. Alrighty, Perfect. friends. Hi, that guy. So as you might know, we recently did some very interesting research, with a company called Gumshoe, who's also based here in Seattle. They're in the AI tracking world, and I was skeptical that AI tracking was even a possibility. Because as as you probably know, if you ask an AI, to give you a response to to virtually anything, it will give you a different one every single time. And this is particularly applicable when you're asking for a list of brands or recommendations. So what how can you possibly track something that gives you a different answer every single time? That's that's the rabbit hole we wanted to go down. My worry was that marketers are getting fleeced because there is a lot of hype around AI. I'll I'll show you in a sec. And I I was just worried, truly, truly worried that people are spending tens of thousands, hundreds of thousands of dollars of budget on tracking their AI presence in a world where it does not make sense or matter and instead instead of investing in marketing things that do matter. And I still believe there's some of that happening, but maybe a little bit less than my original hypothesis was. So let's go kick off. I bet you've seen research like this. I'm not gonna mention the company behind this because I don't believe their data. I think it's sketchy. I I left a LinkedIn comment saying, like, hey. This doesn't this doesn't look real to me. I think you I think you've not made this up, but squinted and made things motivated thinking wise. I bet you've seen research like this. I love this one. It's only gonna be four more years until ChatGPT overtakes Google. Yeah. Sure. That sounds that sounds right. And then they say data source similar web, but you see those dots? Those are all projected. Also, that the little drop in Google, total lie, I don't know why they included that. Or even this one, which I think this one might be true. This is from Ahrefs showing that visitors who have the refer string of coming from an AI tool have four to 23 times the conversion rate. And I believe this because AI tools send one tenth of 1% of the traffic that search engines do per search. And so it is only the one in 1,000 person who is, like, absolutely directly bought in. And oftentimes, you have to say the AI tool, Okay. Thanks for the recommendation. Can you give me a website? And then when it gives you the website, you can say, please turn that into a link for me, and then it will give you the actual link to go. And so I I do sort of believe this, but it's it's very again, a little motivated thinking there. So I wanted to dig into how big AI search is really. Granted, this is desktop. But as you might be well aware, 75% of ChatGPT's activity is desktop, whereas 66% of Google search activity is mobile. So we are way under counting Google in this for sure, and we're almost certainly over counting AI tools. And still, when you look at the share of visits this is visits, right, not not searches, but visits. All AI tools combined to 2.9% of visits and search engines, 34. So, you know, look. There's lots of things happening on the web, but AI is still relatively small when you think about it globally from a visitation perspective. We also did this with searches. So this was recent research that we did with Datos, clickstream data provider. Again, desktop, so it's gonna have an AI bias in their favor. And what we looked at is the number of prompts and number of searches. Now we we treated searches and prompts the same. So if you had a long prompt session where you asked tons of prompts or you had a long search session where you click through tons of results and went to tons of pages, those are treated equally. The answer here is AI tools, they have some searches, but they they don't have as many searches as social networks do or as many prompts as there are searches happening on an Instagram or a Reddit or a Facebook or a LinkedIn, all those places. And yet there's no hype around that. There's no hype around social search. There's very little hype around ecommerce search, commerce platform search, and traditional search engines still dominate. Google, by the way, is, like, 73, 74% of that 80%. So we we we looked at this trend over time to to try and see what was going on here, and you can see Google's comments really skews this. So I I broke it down. Now I've made the the y axis start at 70%. I don't wanna fool you with visuals like lots of people do with charts, but I'm just taking let's take this top 70% here and just look at that. And you can see how, you know, Amazon, Bing, YouTube are growing their share. Google is shrinking their share very slightly, and ChatGPT share is also shrinking because ChatGPT, as you probably know, stopped growing middle of last year. Right? They started plateauing, and then we saw a slight decline in usage. That's probably continuing as people are switching to Claude and Google AI mode and other things. I I have some skepticism about this whole field. So we looked as well at the percent of visitors who performed a search. And And I just wanna point out that when someone shows you a chart like this one, right, so this this, like, here's how many people visited, you should make sure you counter with a 97% of people who go to Google perform a search. 57% of people who go to ChatGPT perform a search. 22% who go to YouTube perform a search. Right? So, like, the the intent there and you might say, wait a minute, Rand. That doesn't make any sense. How is it possible? I'll I'll show you how it's possible. I always get this question. Wait. Let me grab my browser. Yep. That one. Oh, la. That'll make it easy. And in the in the chat, Nikoletta asks, when you say perform a search on ChatGPT, do you mean any prompt? Yes. Any prompt. Right? Any any prompt at. all doesn't right. Mhmm. So if I perform a if I go to my best men's dress shoes, which I've been researching, you probably tell, and then I say at the bottom here, hey. You know what? I want to share this with someone, and I copy the link and I send it to somebody, which my mother-in-law does all the time on WhatsApp. She's like, oh my god. It's alive. Look. I figured out how to make the computer alive. And then, right, she sends it to me. That's her accent, by the way. Perfect perfect representation. So, and she and then that's how you get tons of people visiting ChatGPT without ever she's not she has an Italian accent, sort of, but she sounds more like Ariana Huffington. I don't know why. Like, she sounds like no other Italian. That's why you see that 57% number. Okay. Okay. So a, lot so 57%, a lot of people are sending their chats to other people, like, hey. Check this out. Is that mostly it? And then I guess other people, they might be looking for something on the OpenAI interface or ChatGPT interface. That's possible. Yeah. I would guess that's. a probably a smaller percent. I I. do think, Amanda, I do think there's a large percent of people who go to ChatGPT because they've heard about it on the news or from a friend, and they stare at it. and they go, what am I supposed to do? And then they leave. And you can see in the visitation data that there's tons of, like, one page visits to ChatGPT who don't come back, you know, that month again. So that. that would be my guess. Right? That there's lots of that. There's lots of sharing, all that kind of stuff. Okay. I don't know if you saw this one, Amanda. I I guess you would like I I thought you might like this. This was an NBC poll. You know, they do these every week or every month. Right? And then they they publish the results. They've been doing these for, god, decades, right, since I was a kid. And so they ask how popular certain people are. Right? So, like, they they asked this poll of how popular is Paul Pope John Paul the second in 1998, you know, prior prior to Seanan O'Connor going on SNL and revealing his activities. And you can see that artificial intelligence, AI, minus 20, right, when you take the somewhat negative, very negative, subtract them from the somewhat positive and very positive, That is that is lower than Trump or ICE or either political party in The US. AI is extremely, extremely unpopular. And I am shocked. I think I think this is the most shocking number. Only 1% of Americans said they were unfamiliar. What 1% of Americans are unfamiliar with AI? Like, that's that's that's. way less than are unfamiliar with Iran, the country with 90,000,000 people in it. That's crazy. Alright. So more people know what AI is than Iran? yep. Yeah. Okay. Well, Yep. for America, that sounds about right. Because we're really bad at geography. AI's impact on search, I don't wanna underplay it. I just I'm just trying to show you that the AI tools are quite popular with a very select subset. It's it's the people who are on this webinar in your LinkedIn feed, in your professional world. All of us are sort of obsessed with AI because the executive class of The US is obsessed with AI, and they're spending tons of money on it. But that is not where AI's true influence is being felt by most people. Most of the influence and impact, in in my view and from all of our research, is Google. It is because 18% of Google's search results now show an AI overview. That is that is a fifth of 5,000,000,000,000 searches a year. That is impacting almost everyone on the planet because almost everyone uses Google. Almost every one of us performs, you know, an average of 90 to a 100 searches in a month. And and there you go. Right? 20 of those searches, 18 of those searches contain an AI overview, and you can see by industry. If you're in science or health or pets or, you know, people results, right, how old is Paul Rudd, like, those are they are showing way more frequently. And, you know, maybe if you're in shopping or real estate, less so. So AI, huge impact. And that brings us to our research on AI consistency. We care about this stuff. 50% because our execs are obsessed and 50 because of Google's AI overviews, and maybe 1%, 2% because of ChatGPT. Right? And and Claude and Perplexity and DeepSeek and all the other models. Alright. Let's dig into how these work. Alright. If you were to go to ChatGPT and ask for a list of brand recommendations, what are what are the top chef's knives? A 100 times. How many different answers will you get? That was the that was the core of the question that we were trying to answer so that we could figure out whether you can track AI visibility. And, yes, Robert is correct. The AI is not correct, but Robert Karnes, knife genius, knows what's up. We solicited 600 volunteers. Probably some of you on this call, we truly appreciate your and their labor because I sent email after email in q four to people saying, like, hey. Sorry. I need you to run another, you know, bunch of prompts and, like, will you try this and will you try that? And so we we collected all these through Google Forms. We asked people to essentially run the same prompts over and over and then submit the answers that they got back. Just copy and paste them in here. And then we did well, Christy Christy Morrison, my chief of staff, did an incredible job of normalizing this data, which meant making sure that, you know, if it said the Victorinox Pro, which sometimes it would, we knew that that meant, oh, that means the Victorinox Fibrox Pro, and let's, you know, make sure that those two answers are the same. So normalizing all this data. But you can see the the variety of responses absolutely astounding. Right? So this is essentially when you ask for the top chef's knives, here's all the brands that were mentioned. Here's when they were the first brand to appear, the second, the third, other ranks that they appeared in, the total number of mentions which we sorted by just wild wild amounts of variety. And after creating and distributing and then and then analyzing thousands of these responses, we sent five different surveys across three months. And I'm gonna try and succinctly summarize that for you to show you what's up. So this is the number of unique brand mentions. Right? So this would be, like, the Victorinox or or let's let's use, you know yeah. We'll we'll use chef's knives. That's fine. So the Victorinox versus the Mac versus the Wusthof. Right? That's that would be three different brands that were mentioned. But if if there were seven models mentioned from Wusthof, we we still normalize that as a a Wusthof mention brand mention. So we're not counting individual models. We're counting brands here. And you can see that it varies dramatically based on what you ask. Right? So in LA Volvo dealerships, turns out in the greater Los Angeles area, there's only 11 Volvo dealerships or 11 dealerships where you can buy a Volvo. Only nine of them were ever mentioned by any AI, which I thought was interesting. There's, like, two invisible LA Volvo dealerships to AI in the in this, which, you know, I mentioned to Darren Shaw that might be an interesting case to look at. Like, why are they invisible to why can Google Maps see them and AI can't? And this and then the the pink dot is the average number of mentions. Because remember, you know, unlike this is one of my problems with an AI tracking tool, like a ProFound or even gumshoe or other ones, right, is that when they prompt AI, they say, what are the five top Seattle real estate companies? What are the five best, you know, US politics accounts to follow on TikTok? They they have a number that they assign to get a specific number of responses, but nobody else prompts AI that way. When we looked at how people really prompt AI from the from the open source data that we have well, from from Datas, nobody uses a number. Nobody says give me the top five, top 10. They just ask. So you can see those response number of responses bouncing between two and twelve, you know, just all over the place. And chat GBT, Claude, and Google AI, which we used AI overviews unless the overview didn't show, in which case we went to AI mode, then you you can see what those look like. So, like, for example, nutrition accounts, way more diversity of accounts mentioned. Sci fi novels, incredible amount of diversity, especially by Google. Google gave huge lists, massive lists, more than more than 10 on average, and more than 200 different sci fi novels were mentioned. Whereas LA Volvo dealerships, right, the the numbers were quite small. Alright. There's the there's the numbers for you. How consistent were these? So we we calculated based on all these, you know, data points and all these different ones that we did. We calculated the odds that the same list of brands would show up. Like, if it's the Wusthof, the Mac, and the Victorinox, what are the odds that that will show up in two responses out out of a 100? Well, them odds are low. These these it's not gonna happen. Right? Like, Google AI AI mode, less than 1% shot. Right? So one, you would have to ask Google's AI mode a 124 times to get the same two brands or this the same list of answers twice. I I I want you to take this graph and that stat to all your meetings where people are like, we need to track our AI rankings. Oh, boy. Oh, boy. I I I know this seems crazy, but if you try it yourself, you will quickly see this. Go go ask an AI right now for I don't know. Amanda, what are you shopping for? A new microphone? A new microphone. Yeah. If you go ask a good example. yeah. I bet if you go ask. ChatGPT for, hey. What are the best microphones for webinars and podcasts? And you ask that 10 times, every single one of the answers you get will be different. It'll be different in the brands that are mentioned. It'll be different in the order that's mentioned. The visualization will be different. It's just different. And by the way, if you want the same brands in the same order, 1,429 times before you are statistically likely to get two. Crazy. This is why you have to be careful trusting AI visibility metrics. I'm not say I'm gonna show you some ways to do this, and and that there is a number that I think is reasonable. But, man, if you are just running some searches for example, a lot of agencies tell me that people will email them. They'll send them, like, a ChatGPT, you know, screenshot or or a link to the responses, and they'll be like, hey. You don't show up in AI. And and sometimes this convinces a team that they don't show up in AI. I'm like, dude, just ask again. You'll show up. You you can you can run it until you don't you get or don't get what you want. The AI response consistency, we we put this through the different prompts that we put in. And you can see that there are some where it's more and less likely. So the LA Volvo dealerships example. Right? Like, that prompt consistency is granted still very low, but way, way higher than, you know, home studio microphones, which was which was one of our one of our questions or sci fi novels, right, where it's vanishingly unlikely. Claude, we did the same thing. Google AI mode, we did the same thing. I this is one of the examples that really stands out to me, friends. And I think this is a little bit less about how you use it in your marketing career and a little bit more about how you use it in your personal and and sort of research life, which is that lots of people trust Google's AI for medical advice. Right? Like, we ask we ask Google's AI mode or we ask ChatGPT, and we're like, hey. You know, my friend was just diagnosed with cancer. Where where should they get care? Where are the best hospitals? And you think you're getting a great answer, but what you're getting is a is a somewhat randomized list. If you ask again, you're gonna get a different list of hospitals in a different order. I I'm not sure that's the right thing the the right thing to be asking. Right? It just makes. me very worried. I get nervous. I mean, just to take the kind of smaller stakes example of the whole microphone, I mean, that's going to vary a lot based on which websites you would go to for that opinion. Right? Because people want different things in microphones. If you're looking for a home microphone, you might be prioritizing noise reduction, like external noise reduction. Or maybe you're prioritizing the sound of your voice in the microphone. Or maybe you're thinking about podcasting versus creating music. Like, these all have very, very different use cases. So that's already gonna be different when you look at different websites. But then add to that the, I don't know, the the the lottery machine of LLMs, it's gonna be even more variable and. unpredictable. And the one thing about going to a website is you know who's behind it. They'll they'll say why they have their opinion or they won't, and then you you judge appropriately. They have criteria that they list. The LLM has no criteria. Right? It doesn't say, like, oh, we weighted, you know, survivability of cancer patients versus comfort of care versus number of doctors to nurses ratio, they they don't tell you what their criteria are. There are no criteria. The criteria is did you appear in the training data, you know, this many times and in the citation alongside you know, words alongside other words. It's it just bugs me. Right? It feels like to be honest, Amanda, to me, AI responses on a lot of the stuff feel like Internet one point o, and Google feels like two point o. Right? Like, if AI had come out first in the nineties, think I we'd all be like, oh my god. It's so much better now that I just type one or two words and get, like, a list of more trustworthy, you know, responses that I can go investigate. Mhmm. Like, it. feels it feels like AI to me feels like ask Jeeves in 1999. Like, it really it really does. I know I'm dressed like him, but, you know, let's ignore that fact. Sorry. I don't wanna hold this up, but I do wanna make another point here about intent. So intent when people are searching on these tools because I'm also remembering, you know, after you had your study, which we're gonna link to I'll link to right now, but we're also you know, it's been out. We'll also link to it in our follow-up email. But since your research, I I was then thinking about, like, what if I ask other mom friends a similar question? So this was my my. very, Yeah. very small scale experiment of, like, okay. These people are pretty similar to me demographically. Right? We have similar sort of values. What if I ask them to look up local basketball leagues? Now these are friends near me, but also friends across the country. So and I wanted to see all of their prompts. I have. a blog post on this too. And so what was interesting to me was just the the very different ways in which we framed it. Some friends really prioritized, you know, locality, like, low location. Like like, I want the closest, you know, league to me. Other peep someone else was like, I really want one that's gonna nurture my son. Right? And so people can still get the same answer. Right? These are people who are near each other. Let's say one person really values, you know, closeness. The other person values the the coaching, the quality of the basketball league. Right? They might go to the same league, but they're both still going to have different intents, different values. Right? They're still gonna be like, oh, I chose this league because it's next door to my orthodontist. Someone else is gonna say, oh, no. This is the best one because they really focus on the, you know, coachability and interpersonal interactions with the children. Right? But they could still be the same league. And maybe that league doesn't have better coaching. It's just that Chad GPT said that it did. I I copied know. and pasted. your your research. in there. Yeah. And you can find that on the SparkToro blog as well. Yeah. But just all that to say, like, people have very different intent. People describe things very differently. And in certain settings, they might get the same answer from chat GPT, but they're still going to interpret that differently. I, I. have some research around that coming up. Coming right up. So we asked about how often the most mentioned brands are included in prompt responses. Like, hey. What how how much is their concentration around the sort of, like, top three? And so this this is showing in in these, you know, blue for Google AI, orange for Claude, green for ChatGPT, like that. Is there concentration around a few brands? The answer is kind of. Most of the time, some there is some concentration around those top brands. For example, Smart Sites was the the brand that appeared most frequently in the dataset. I think it was, like, 85 times in 95 Google AI responses for recommended digital marketing consultants in ecommerce. Quite impressive. Right? I that that's a big field. I'm I'm surprised to see them so mentioned that they were the most mentioned of anyone, and they appear, you know, whatever, eight out of 10 times. The least mentioned was the men's fashion Instagram account, Adam Gallagher, who was who was the most popular account appearing there but appeared in less than half the responses. So depending on, you know, which types of questions you're asking, the inconsistency can be extremely high or or it can be much more consistent and contained around the top few. And so we we looked at that consistency rate and average rank, which you can if you wanna dig into the math, you can. We we did pair consistency calculations. Thanks to the folks at Gumshu who have a PhD math student who helped me put this together because I am I am a stats amateur. And and the only thing that was very important to me is I said, you have to explain it to me in a way that I can explain it to other people. And so this is this is essentially the how how often are you likely to get two answers, you know, any any two pairs of brands in the answer and the and then the rank difference between those. It varies. Right? The answer is it varies. And you can see that it's not consistent in the variation either. Like, if you don't if you don't have all the numbers for your specific space or the space you're tracking for a client or or, you know, around a brand or a competitor, you will not know how consistent or inconsistent, how much concentration of the brands there are at the top or or not. Like, seeing this data by running a few, you know, ChatGPT prompts or a few Google AI mode prompts will not get you there. You need to do serious at scale data. There, is. more oh, yeah. oh, sorry. We're curious in the chat. Like, Robert and Twila mentioned this. How like, why why cloud computing was always so highly correlated in that chart? Why is cloud computing. highly correlated? This one. Yeah. Okay. Right. So that that so it's not correlation. Pair consistency calculation. Pair consistency rate is the chance the odds that two that a pair of answers will appear again and again. And so as you might imagine, in cloud computing, if you ask the AI, hey. What are good cloud computing providers for startups? The Google Cloud and Amazon's AWS are gonna get mentioned almost every time. So that is why you see that very, very high or relatively high pair consistency across all three models. Right? Cloud computing for ChatGPT, CloudGPT for Cloud. Like, they almost always gave those. Right? Does that does that make sense? Yes. Okay. I see I see clouds. Got it. Alright. Cool. The one of the important things that we wanted to point out here is that the consistency happens more across the sectors than across the tools. Right? So when you looked at cloud computing here, you saw similarly consistent versus, you know, sci fi novels similarly inconsistent. You you can see that in the when we break it down by the prompt as well. Right? That essentially, like, these these bars tend to be similar when they're grouped by there's the cloud computing bars. Right? That really stands out. Or the nutrition accounts, that really stands out. Right? So that consistency happens more in the sector than it does in the in the model or tool used. And then you can see visibility of top and and second most likely as we talked about previously, City of Hope Hospital, great example here of of a cancer care hospital that came up 69 out of 71 answers. Like, it was almost always mentioned by ChatGPT, but it was the top answer in only a third of those. So which one matters more? Does that mean City of Hope is a better hospital for cancer care because it was mentioned so often or because it's not the number one one? Look. I don't have a perfect answer for you here, but what I'm going to say what I believe after doing this research is that ranking is a bunch of bollocks. You should throw out the idea of ranking. If if an AI puts a thing first or second or third or fifth or twelfth, you should consider them all the same. There's there's no not enough consistency for me to believe that AI tools, when they put a brand or recommendation or a location or product in those results that there is nothing meant by putting one first and one second and one third. Those are random. That's that's just what I'm saying. Okay. And we did this for Claude and ChatGPT and and Google AI. Across all three of them, the average visibility so this is, like, the percent of times that the top three mentioned brands came up quite a bit of variability. You can see the standard deviation's high in all of these, like, near 20%. But ChatGeeBT, it's 64%. Claude, 73. Google AI, 68. So kinda similar. Like, kinda similar. They tend to have the same patterns around this. And that yeah. I think that's that's quite interesting too. You can get a sense that, like, AIs have a consistent amount of inconsistency. Alright. Visibility percentages across numerous prompts run dozens of times actually does seem to be a reasonable metric. I I was not convinced of this. Like, when we started this experiment, if you look at sort of my hypothesis for it, I thought that was not true. That is true. But rankings rankings, bunch of baloney. Right? The the order that things appear in just not meaningful, not useful. I am upset with the AI tools that try and, you know, make that a thing. I think that's I think they're misleading you and everyone else in the industry when they do that. So here's an example. This is Gumshu, the the the company that ran this with us. By the way, if you're wondering why would an AI visibility tool provider work with you on this, What if the results had been much, much worse for them? I also asked that question. I'm gonna tell you the answer. It's because one of the cofounders, Patrick O'Donnell, who's just a lovely guy, plays Dungeons and Dragons with me. And and so I basically managed to convince him. I'm like, Patrick, come on. Let's do this research together. He was like, well, okay. You know, I've been I've been feeding him great meals and, you know, keeping his characters alive for years, so I think I think he owes me. Turns out this research actually made me quite a bit more of a fan of gumshoe. I'm not gonna lie. Patrick knows this, but I was a little skeptical of him starting that company, and I was skeptical of, like, this this whole process. But I I'm I'm converted. I think it is useful if you if you really care about your visibility in AI. I think it is useful to know the visibility percent. So this is essentially saying, if you were to ask ChatGPT is that what we're looking at? No. I think this is Google AI. If you were to ask Google's AI, what are easy to use mark audience research tools for solo marketers? And, you know, and then a bunch of prompts around that and you were to ask those hundreds of times, you would find that SparkToro is mentioned in 16 approximately 69% of those responses. There's no ranking here because ranking, we've already determined. And then you can see that that Audience I'm not totally sure if I'm pronouncing their name right, but Audience and SimilarWeb and BrandWatch and curated by Quantcast, which I've actually never used, and full story, those brands are also mentioned sometimes. And I appreciate that, like, they're showing me, hey. Here's who else gets mentioned along with this, and here's how frequently. Tracking that over time might be useful for us. Right? It might be interesting to see, like, Yeah. who else is popping on the radar. I'm always trying to include other audience research providers in my, you know, presentations because I hate being self promotional. So. And. just to be clear, like, in the case for easy to use audience research tools for solo marketers, I mean, BrandWatch may not want to show up in that, right, because they're not really a tool for solo marketers. Yep. So that's also just to say, like, it's not like we're throwing shade at them. It's just they probably don't wanna be there. right. Right. Yeah. I mean, Yeah. definitely, my my understanding of Quantcast's pricing is that it would be extraordinarily unfriendly to a solo marketer. My, yeah, my my understanding as well. Yeah. Alright. There's one more problem. I know we I know we have last twenty five minutes here, but there's one more problem, big problem, and it goes to, Amanda, what you were talking about when you, asked your mom friends about, you know, prompting. Do any two people ever use the same language in their AI prompts? Like, remember, Yeah. for all this research, every single slide I've shown you, we asked people to ask the exact same question to ChatGPT precisely framed. Right? Like, the the the phrasing was the same every time, copied and pasted. And that's the inconsistency. The level of inconsistency is astounding for exactly the same question. Like, if Google rankings were that inconsistent, you couldn't track them. There'd be there'd be no point. Right? Do two people ask use the same language when they're asked? So what we did is I asked several 100 people. I think I I wanna say, like, a 150 ish filled this out. Our panelists to provide the text that they would use for a simple prompt. Right? So imagine you're seeking to find the best headphone options for a gift to a family member who blah blah blah blah. How would you prompt at the AI tool? And then you can see I I didn't copy in the 140 some odd, 47 whatever responses, but I wanna buy a set of headphones. I am buying a gift for a frequent traveler. What are the best headphones for someone who travels a lot? People ask this question in a million different ways. There was there were no two that were similar. So what we did with these because they're they're long strings of text. So what we did is we calculated a metric called semantic similarity. And semantic similarity, again, I was like, you need to explain it to me so I can explain it to other people. Semantic similarity is a numeric metric that tells you how similar the not the exact words, but the underlying meaning of those words is. Right? So for for example, if you said what's a what's a good example of this? If if you said shoes and footwear, semantic similarity one. Right? Like, that's the same thing. They mean the same thing. It doesn't matter if they were phrased, you know, differently. We're gonna calculate. So so the number 0.05, that means semantic similarity. Both are food, but one's tacos and one's banana bread. K? Like, point one, both are dinner entrees. Both do chicken, but from totally different cuisines. Point one eight, both are chicken, lemon, garlic recipes. You would probably recognize that they're actually similar. And if you're if you're over point five, it's probably a recipe for the same dish with some tweaks in there. So how similar were the answers from these people? 144 responses. There's the semantic similarity graph. The average average pairwise similarities, we're getting that that pairwise number, 0.0809. I'll shorthand it for you. Even with the exact same goal, people ask AIs very, very different questions just as Amanda's small scale research showed too. The similarity of survey takers' prompts to one another is approximately the similarity of, like, kung pao chicken and peanut butter. I know Casey loves kung pao chicken. He I think he likes peanut butter too. Maybe maybe because they have this 0.081 semantic pairwise similarity. That's probably He loves both. there you go. He's he's in the he's in the comment. He loves it for the stats. And and so, essentially, what we're saying is these these prompts, these 140 prompts, not not close. They are not close to each other. The the underlying intent is similar, right, as we know because we we ask them to all have the same intent. But the way they phrase it, completely wildly different. So Gumshoe took all 142 unique prompts through their system, and here's what they found. And before this, I will just say that when I saw that data for the first time, I was like, I told you. AI tracking is bullshit. Like, I I was I was like, yes. You're taking people's money. Like, it's a scam. Then then Gumshu was like, okay. Sure. We'll run all 142 prompts. We ran them a bunch of different times. They ran them through the seven what they call the seven major AI model families and then, you know, did it 994 times. You know what's wild about this, Amanda? The visibility of the brands in this looks almost exactly like if you were to take one of those prompts, anyone, and run it a 100 times, which has me which which got me thinking, oh, you know what? This actually makes sense because what the AI can do quite well is take vastly semantically different, you know, words and prompts and interpret it, essentially, you know, collect that and interpret it as the underlying intent and then give the responses to the underlying intent, which which made me realize that this the the process here, right, whether if you come up with synthetic prompts, right, from a from an AI or you use human prompts, if the intent is correct and you ask enough times, you can get visibility in a sector even despite the fact that the way people ask is wildly different and probably no two prompts are ever the same. So, again, I I had a hypothesis invalidated through real data and experience, which which is kinda cool. We did this again, by the way. We did this again because I was like, no. I don't I don't believe it. I wanna do it again. We did b to b. Again, the prompts were were wildly unique, and the visit visibility percent numbers are much lower no matter the model. But that's because this one, right, imagine you're trying to help a friend's coffee shop find a good brand design agency. This you know, the the set of options is I don't know. There there must be tens of thousands just in The United States alone for brand design agency options versus headphones where there's, you know, maybe 20 options. I don't know. 25 that that ever came up. And you can see here, wow, the percent numbers are much, much lower, and this also matches how the AI responds to synthetic prompts around this. Again, sort of confirming that the process here works out. Okay. Which which brands get mentioned with varying degrees of frequency appears to be stable across multiple runs? So if you run, you know, prompt, like like, time wise. Right? So we ask, hey. Let's run it, you know, this time, this time, this time, this time. That that looks pretty similar. It doesn't fluctuate day to day too much, too badly, hour to hour. That that consistency does say the same. And and so okay. We these are the this is my conclusions from this project. I wanna represent that this is relatively small scale. Like, we asked, you know, a few 100 people to participate. We ran a few 100 different times on dozens of prompts, not thousands or millions. But we know I think we we we can say with relative certainty. AIs rarely give the same list of brands or recommendations. They almost never give the list in the same order. You can think of these tools as probability engines, and you and you probably should. Worse still, users almost never craft similar prompts. If you're counting on, like, your prompt that you came up with to be representative how real people really ask their AI tool things, you're no. That's not. Measuring your brand's presence in AI answers with precision, like like, you know, how many times are we ranking number one, number two, number three, all that. That's fully fully baloney. But you can, with enough prompts, run enough times, get this sort of dartboard like pattern, right, that we this kind of thing, and that is pretty reasonable. Pretty reasonable. I am not gonna spell Bologna, Marjorie. I'm not that's rude to the rude to the city of Emilia Romana. Alright. The good news the good news here is if AI generated synthetic prompts and topics are accurate, then you know what? Like, if they if they really represent how user behavior works, what you need what you would essentially need to get a great metric is you need the search terms. Right? This and the underlying intent that an audience uses. Like, you need to know a lot of information about an audience. So, you know, Amanda, I was like, hey, Casey. After doing this research, I think could we at SparkToro tell people what AI prompt topics the audience is likely to run? And Casey was like, yeah. I can do that. In fact, here it is. You could see we launched it last night. What do knitters prompt AIs about? This stuff. And and if the you know, I think a reasonable question because I have tons of skepticism around this too. A reasonable question is, how do you know? And the answer is, well, we know what websites they visit. We know their demographics, and we know their search keywords. Most importantly, search keywords, like the things that they ask Google. And if you feed those into an AI API call and then say, give me the prompt topics, you you get good results. I don't wanna say they're fantastic. Maybe you might know your audience even better and know exactly what they're, you know, curious about. And maybe you would say, oh, I you know what? Best yarns for sweaters should be higher. That's actually something that more nears I'm not gonna I'm not gonna argue. You might know your audience even better, but this is a an excellent approximation. And we've run, you know, dozens and hundreds of tests around this. We can also do this, which is kinda cool. So if you if you click on the see examples, we will show you those examples of what those are. And you could take this and put it in your prompt tracking platform of choice or whatever you vibe coded to save some money. That's great too. I am just gonna say this this looks this looks pretty good. I don't know. Like, Marjorie said, you would not trust an AI for patterns. Do these look to you? Well, I'll we'll talk later, Marjorie. I don't know. Interrupt the webinar. Ask what purpose. I I found these pretty compelling. I asked about a bunch of topics that I know more about than, knitting hobbyists shopping for fiber arts. But okay. Next steps. I don't wanna talk too much SparkToro stuff. Next next steps. AI tracking tool providers, in my opinion, need to pick up where we left off. We need larger sample sizes. We need people to do more research around this. And, also, marketers and execs, I would really urge you to stop throwing your money at AI tracking products unless they have great answers to these things and research back findings around this. Like, I I think it's pretty embarrassing that we are the ones publishing this research. We don't have a dog in this race. Right? Like, that's it doesn't we're we're not an AI tracking product. We don't care about this stuff. We I just worry about marketers getting fleeced, and I'm still worried about that. I'm still worried that a lot of these products are not doing this in a scientific way, that they're not asking the prompts enough times, that they are not getting the topics from the right kinds of places with with real user data. Hopefully, you know, the search keywords, which is which is most indicative of people's behavior and and interest in topics. And I I'm worried that they're giving you rankings instead of visibility. So after we published this a few weeks ago, some folks did pick up the Slack, and I wanna highlight them because I I love when people do this. So Gumshoe published a methodology piece, which any AI tool tracking tool could copy. Right? Like, here's how many times you need to run a prompt or a set of prompts in a model in order to get visibility percentages that you can trust with, you know, a high, you know, plus or minus this this much per visibility percent. So if you want, you know, plus minus one, you would need to run 10 prompts, seven phrasing, seven models. That means a single pass generates 490 kind of results, you would need to run that 20 times to get plus or minus one percentage point. Or you could only run it once, and you'd have plus or minus 5%, but you should specify that. That's very important. Mike Saunders from Contender did some excellent research. This is not his only research piece. He's done some some other good work there. I I started subscribing to him. He looked at how citations correlate with brand mentions in b two b prompt responses. I found I find a lot of that citation research, like, here's how citations correlate to this really interesting, But I think it's super important. Please treat it the same way you treated ranking factor correlations in SEO. Like, this thing correlates with ranking higher or lower, not this definitely influences that. Be really careful. I I with credit to Britney Mueller here, I I filmed this video about essentially the link between citations and responses. And there's a there's a lot of people who believe that if you take if you were let's say you opened up a new cancer care hospital, and you then got listed in Health US News and UCSF Health and US News Health, that because those are the citations that the AI points out, you would automatically appear in these results on the left. Right? Like, you you would be mentioned here. These citations do not power these results very often. Not always. Lily Ray made some good points about how sometimes they do. But most of the time, they are what's called post hoc rationalization. So that means the AI generates the answer it's gonna give the list of brands. And then after generating that, it gives you it goes and finds a list of citations that support. That's that's post hoc rationalization. That is not these citations are why these things rank. That is or we're gonna come up with the prompt with the answer, then we're gonna find citations to closely match it. And you can you can see if you ask for AI recommendations, by the way my my last point here before we get to q and a, please, dear god, ask 10 times before you trust the list. Right? Like, here's my here's me asking for the best high quality brands of whole cut leather men's Oxfords. You can see wildly different results. By the way, the Verluti is an outstanding choice. But, oh my god, no. Please do not get Beckett Simon or the Tom Fords. I don't even recommend the Santonis, honestly. Xenia's hit or miss. Crockett and Jones. Great. Yeah. Fine. Okay. Thank you. I have three links for you to check out. Spark Toro, you can go test our new AI prompts, prompt topics that Casey built. We'd love your feedback on that. Gumshoe, obviously. And for those who don't know, Amanda has a new website, 0clickmarketing.co, where she has got her new podcast, which is launching tomorrow. No. No. Now, It's out now. launched. Yeah. Amazing. Already live. Soon to be announced book, our our zero click marketing book that we're writing together. And, and Amanda also is doing some consulting work. So, you know, lucky you if you. find some time. I know we've only got a few minutes, but do we have some, some questions we should go through? I don't let's see. Oh, we had some in the q and a. Great. There's let's see. Someone Rich asks, what are your thoughts on tracking visibility percent by topic over time, like month over month? Yeah. So that is I'm gonna see, actually, if I can pull it up. Can you see my screen when new things show? Yes. It looks like you can. Great. So this one of the really nice things that Gumshoe did let me see if I can find his link. Oh, I know where it is. Gumshoe. So they made this free for anyone to see. So you can track SparkToro inside Gumshoe over time and just see how how we're doing, I guess. And I find this pretty useful, like the I I shouldn't say useful. I don't personally care about this. I don't think I don't think AI has much of an impact on whether people use SparkToro or not. But in your sector of space, that might be different, in which case, I probably would be tracking AI visibility. And, unfortunately, there's a lot of people who need to track this not because it's useful to them, but because their boss told them they need it. Right? Their boss is obsessed with it. So if your boss is obsessed with it, I think you can sort of get some some value from seeing how your visibility, you know, changes over time. And, certainly, if you're doing things like the the most effective thing you can do for AI is run a PR campaign, like, mentioned a lot. Right? How people talk about you on the Internet, hopefully, in lots of places, diverse places. That that PR campaign could be somewhat measured, not not attributed, but somewhat measured through AI visibility, especially if you do the PR campaign around a specific set of language consistently because language the the language usage is what what the AIs will pick up on. Other Oops. Let me go back to that. Great. Bridge says shout out to AlertMouse for tracking brand mentions. Heck yeah. That's very kind. Yeah. I've we've had a lot of fun running AlertMouse. Do you know 5,000 people are using AlertMouse now? Like, That's just. yeah. I mean, granted it's free, but it's still Nathan and Adam and I were like, oh my god. We also were like, man, we're losing money on this project. We should we should probably we need to do some more marketing before we run out of money. Alright. Uh-oh. I do wanna there's a question here from Allison. I work in philanthropy and grant and and in the grant making space, trying to figure out how to address showing up or not showing up in AI prompts asking broad questions about niche or specific funding types. Our appearance is so inconsistent, and I'm struggling to know what I'm doing wrong. So I I think the the question around this might be if you're very inconsistently showing. Right? Like, if you're I'll go back to one of the examples here. So okay. So, for example, first off, the nonprofit space is huge. So it really is gonna depend on the prompt intent. Like, what is the intent of the people who are prompting an AI tool? What are they trying to figure out? For example, I was, I think you saw Amanda. I had a call last week with the folks from GiveDirectly, like their marketing team, and I I help them maybe once a year, once once every couple quarters Oh, around. their their strategy stuff. And and they were seeing that because lots of people in the tech world ask what is the most efficient sort of effective, you know, high ROI charity to give money to, Mhmm. they they come up. Right? They come up very frequently in the list of answers. But if you ask differently, right, if you instead say, what are some of the best charities in America? Or what are some of the most well regard highly regarded charities? Or what are the best health care charities? They won't come up at all. Right? They just not at all, but they almost never show up in those results. They'd be more like the, you know, Pearl Fisher in these answers. Right? And so my my answer I'm sorry. What was the what was the woman's question or the woman's name? Allison. Allison. So my my answer for Allison would be I think the thing I would do first is try you you you conceivably could use Spectora for this. Right? Like, if you have a great understanding of your audience and then you get the prompt topics that you know that they care about, that they're likely to use when they wanna find charities like yours or the and the the types of things that you wanna come up for. And then you track those over time. You you could use Gumshoot. You could you could do it yourself. Like, very frankly, a lot of people have used Claude to vibe code their own tracker, and it's not that expensive. Right? And you can track the results in a Google Sheet or whatever. And and then you see how often you come up and then try and do essentially, like, get those words that you want to come up to be next to your brand name as you're doing PR efforts and social media marketing and content marketing efforts and sponsorships and partnerships and, you know, charity events, all that kind of stuff. Right? You want your brand next to those words and phrases that that have the overlap with the intent so that, you know, the both the citations and the core model will over time pick that up and then start to show your brand, and that that helps. Amanda, I think we are out of time. But if. there's yeah. But I do wanna say a huge thank you to everyone who joined us. This was very, very interesting research for me. Just sort of a a pet project that I I couldn't get out of my head, and I'm really glad to have some answers. And I'm even more excited for more folks to do research like this. So if you do research like this or you find amazing stuff, please share with us. We would love to see that. And if you have other questions or feedback about the the new Spark chore feature, drop us a line. And this will be on the blog in sometime in the next few weeks when when Amanda. has some bandwidth. In a couple weeks. But this but, you know, keep an eye out on the blog this week. We're going to on Friday, we will publish, an office hours episode from a couple months ago. So keep an eye out for that. Cool. Alright, Amanda. Hope you have a wonderful rest of your week. Thanks for joining me and being cohost. Thanks to everyone for coming, and we'll, see you again. next time. Thank you, Rand. That was amazing. Okay. Not at all. Bye, friends. Ciao.