Many websites ask the question, “How did you hear about us?” when users sign up or convert. I believe there’s an assumption that most respondents will answer this question correctly, making the data valuable for insights. In this case study, we will look at exactly how accurate the responses to this question are.
Case Study Background
I own a web development company. The core of our marketing website is our contact form, where potential customers reach out to us to inquire about working together on a project. For business purposes, knowing where these people come from is extremely important. Understanding the source of our acquisitions lets us fine-tune our various marketing efforts. This is particularly true for paid marketing channels, where we want to optimize our spending based on our cost per acquisition.
Nowadays, we automatically track where people who fill out our contact form come from. However, before we had automatic tracking setup, our contact form had the question, “How did you hear about us?”. This is meant to give us some insights into acquisition sources for our converters. Mind you, I knew there would be some degree of error in the accuracy of responses to this question. However, I have seen this question in many places on the web and assumed, therefore, that most visitors must respond accurately.
I was wrong.
Comparing Automatic Tracking to Manual Responses
For a period of time after we started automatically tracking the acquisition source of visitors that submitted our contact form, we continued to ask users how they heard about us. The answer to this question was submitted to an Airtable along with our automatic tracking of their acquisition source.
The manual question on our form was a dropdown that had all of our popular acquisition sources as predefined answer options. You could select Clutch, LinkedIn, Google Search, or another acquisition channel from the dropdown to specify where you heard of us. It looked something like the screenshot below.
Our automatic tracking utilizes UTM parameters. These allow us to easily track acquisition sources in Google Analytics. When someone visits our website with UTM parameters set, we record the campaign that they came from, and submit that information to Airtable alongside the response to the manual question. So, if someone clicks on a Google Ad to visit our website, the UTM parameter will tell us they came from Google Ads and record that information to Airtable. This is what our Airtable data looks like:
Having this data side by side allows us to directly understand the accuracy of the manual customer acquisition tracking question. We know that our automatic tracking is working correctly, so we simply have to see how many visitors answered the manual question with the same acquisition source as the automatic tracking.
The Results
During the time period in which we recorded manual responses and recorded the actual acquisition source, we had just over 100 visitors submit our contact us form.
Despite having Cloudflare (captcha) on our contact form, we do get some spam. So, after cleaning spam conversions, we have about 80 form submissions to work with.
And, out of those 80 submissions, only about 20 answered the “Heard of us by” question with the actual source that they found our website from.
Only about 25% of visitors answered the question “How did you hear about us?” accurately.
Summary
As I have discovered time and again in software development, the key takeaway here is simply to never be too confident in guessing at user behavior. Of course, it was expected there would be a margin of error in asking users to tell us how they heard of us. However, I did not realize that the data would be completely useless due to the fact that the margin of error was so high.
Don’t rely on a manual response to this question dictate your marketing strategy and how you allocate money towards your various marketing campaigns.
