How to Tell (or Show) If Someone Is Data Literate?
A 4-steps guide to interviewing Product Managers for this hard-to-check skill
In the product development ecosystem, one of the critical traits to build successful solutions is to make decisions based on data. It is one of the skills required in most products regardless of category or industry.
Nonetheless, it isn’t easy to assess how data literate someone is during an interview (or convincing others you are!). These are my thoughts exploring 4 aspects that can help understand a candidate’s data ability:
1. Ability to resolve data-driven questions
The first step is checking the capability to understand how to measure products and map product problems to data. We can divide it into 3 qualities:
1.1. Data models and frameworks
Having an understanding of the different ways to measure products depending on their type and the problem being solved, is probably the first step to being data literate.
While there are several models, we are not looking for a candidate who simply remembers a few clever acronyms. It is more useful to explore previous experiences and understand what the most important metrics were and why.
1.2. Can map decisions needed to data required
The second step would be to understand what data should be used in each decision scenario. For instance, knowing that to increase the number of sales, we need to search opportunities in the conversion funnel, understanding the drop rate at each step, and using segments to differentiate the behavior of different consumers.
We can prepare a few questions related to our domain and check how the candidate would go about the available data to decide:
- How would you search for opportunities to reduce the churn rate?
- How can we identify our best performing locations?
- What content in our platform is being more successful?
- How did new app users' behavior change from July to August?
1.3. Knows how to dive deeper to get insights
The final step towards getting insights is the ability to be granular in the analysis, exploring deeper layers of data to gain precision and more actionable answers.
Using tools like a KPI tree, segmentation, clustering, cohorts, and others, we can explore variations to find better data for our decisions.
Note that data can be used to answer questions and also browse it looking for opportunities.
In this case, I would again rely on past experiences, asking for scenarios where the candidate found a data-driven insight or found the reason for an enigmatic result.
2. Ability to access data
In an increasingly complex world of tools and data repositories, the ability to unaidedly access the information is highly valued. It would be important for a PM to:
- Know data analytics tools (like Google Analytics, Mix Panel, Amplitude, etcetera).
- Have designed dashboards or reports.
- Being able to “query” information (either with standards like SQL or more tool-specific languages).
In this case, I would not be worried about a PM knowing the specific tool used in my company, but rather being familiar with any of them. The core functionality is usually similar, and the learning curve tends to be small for data-literate users.
Finally, in the real world, data isn’t always clean and ready to analyze. Having some experience putting together and cleaning data-sets is a plus for data accessibility.
3. Solving complex data problems
Probably the most “advanced” skill in this group is the ability to get a complex problem or question and be able to find the right data to answer it.
I can break it down into two scenarios:
3.1. Analyzing a complex question
So far, we have been discussing standard product problems. But there are questions where the answer is harder to analyze. Some good examples may be:
- If we create a new selling channel, how much of the existing channels would it cannibalize?
- How do features usage correlate to future 6 months retention?
- What parameters are more important for our sorting algorithm?
3.2. A problem where data is hard to get
Similarly, there are situations where we don’t have the information. For example, “What conversion rate will we have in this new geography or retail category?”. Gather external information, do clever estimates or even experiment to capture sample info is part of deciding with data.
For both scenarios, we can either use cases (or the infamous brain teasers), or past experiences if the candidate has some.
Depending on your company, PMs may be assisted by data analysts to solve these complex questions. But understanding how to solve it will increase the PM's ability to request this sort of information.
4. Sharing data
Finally, considering the critical role of communication for PMs, being able to share the information with other team members and stakeholders is essential. Embedding storytelling with data, using the right visualization tools, and removing complexity from graphs are the traits we are looking for.
If you are using a case, how the candidate presents the results is a good proxy to understand their ability. Otherwise, you can ask some situations where he/she failed to share data properly, and what lessons were applied in future presentations.
Conclusion
In my case, this list helped me clarify what I’m looking for when I think about a data-driven person. It can also help organize the coaching of product managers that lack some of the listed abilities.
I would love to hear other thoughts about interviewing for this ability! Please connect and share.
PD: I have not included experimentation because I consider it a separate core skill. It’s closely related to data, but it has its own set of interview questions.
This story was originally published in LeanExperimentation.com. If you enjoyed it and want to receive more tools & tips to improve your product practice, you can subscribe here and join +1400 newsletter readers!
