Welcome to our blog series on decoding identity resolution. This is a nine part blog that offers an attempt at a friendly, comprehensive view of how to think about the concept of identity resolution as well as how to interpret the way it is represented in marketing and sales materials by different companies across the tech landscape. The other articles in the series can be found here:
Part Five: The Green Checkmark Effect
ML and AI and the “Green Checkmark Effect”
This might come as a shock but machine learning and artificial intelligence are complicated. You could get a computer science degree and come out of it having barely spent any time on the topic.
More broadly, it’s impossible to expect one person looking to buy software for an enterprise company to understand literally everything. You might have an understanding of your business and product but not be able to write a line of code. You might understand databases and ETL but have never spent time learning data science. In general, most companies don’t have an advanced data science team, and the ones that do often keep them in a cool, comfortable basement to protect them from the rest of us.
Marketing and sales teams know this and they use that against buyers in what I call the “green checkbox effect.”
The green checkmark effect happens when a buyer is researching the latest and greatest trends and sees something complicated — they know they want the results the sales pitch offers, but do not have the expertise to understand the technical elements. To come to a decision, they put together a list of features and use cases and evaluate each against a piece of software, giving each item on the list a red X or a green checkmark.
Sales and marketing teams know this and toss around terms they expect will be on the list, secretly hoping you won’t ask for more detail. They hope that you’ll hear “machine learning” and think “this sounds appropriately complex and should be future-proof,” then make a nice big green checkmark on your list.
Legitimacy Testing
You can avoid the green checkmark effect for ML/AI-based identity resolution with this handy list of questions:
“How do you train your ML algorithms?”
If they don’t have a sufficiently technical answer for this, they probably aren’t using actual machine learning (or at the very least, not GOOD machine learning)
“How does your algorithm handle messy data?”
If the piece of software is any good the answer will include details about how the algorithm intelligently handles various combinations of data reflecting real-world situations. It should be hard for the sales team to fit all the details into a one hour meeting and they should offer to do a “deep dive” on it.
“How do you measure the accuracy of the results?”
Again you’re looking for a sufficiently complex answer. A good solution will provide transparency and the ability to tune the confidence level of the algorithm.
With those three questions, you should be able to learn a lot about whether or not an application claiming to use ML or AI-based ID resolution is truly doing something advanced.
“But wait, Caleb,” you may be wondering. “Isn’t asking specific questions and judging the answer based on whether it’s sufficiently complicated basically the green checkmark effect?
Well… yes I suppose it is. Unfortunately, actual data scientists are in short supply so you’re unlikely to have access to anyone who will be able to get that far into the weeds. Here are a couple of ideas for how you can verify ID resolution even if you don’t understand machine learning:
Bring along the most technical person you have access to.
Ask to do a pilot or do a deeper dive into ID resolution results. A good vendor will be able to walk you through their offering step by step.
Add some entries into your pilot data then have the vendor look them up afterward and see if they matched or not. A good example goes a long way.
Ask for any whitepapers or academic papers available on their algorithms. Anyone doing actual ML should be able to provide documentation.
Next up
Next up we will shift gears away from PII-focused ID resolution to talking about digital identity and how to think about online ID resolution. Click here to advance to part six!