https://ift.tt/dlfzcM2 Tap into the experts in your organization to create a source of truth and train a model against it Working in a bus...
Tap into the experts in your organization to create a source of truth and train a model against it
Working in a business team, I am often asked to quantify a subjective concept for which there is no broadly accepted definition. A typical question looks like: “how many of our stores are successful?” or “how many loyal customers do we have?”
The crux of the issue is that everyone has their own ‘theoretical’ definition for these subjective words (and believe their understanding is shared across the company). So when asked such a question, the main challenge is to actually align all the different point of views to actually come up with an objective definition.
To give a concrete example — let’s say that you work for an important supermarket chain, and have been asked “how many stores are successful?”
You could look into ‘profit’ and start thinking about ranking the different stores based on profitability, but then:
- Assuming that in your industry profit is a function of the size — would a small store with not a lot of profit be less successful?
- What about a store that just launched and is showing tremendous growth but won’t be profitable until next year? Or on the contrary, what about a store with large profits but a negative growth? Which one would you flag as successful?
- What about a very profitable store that has been gathering an increasing number of negative customer reviews? Or that has a very high employee turnover?
Internal crowd-sourcing to the rescue
As we see above, subjective concepts like “successful” or “loyal” can have different definitions across your organization. It is important and can be quite hard to align these definitions into one solid metric.
There are different ways to go about this problem, but one that I find particularly elegant is the following: Build a source of truth in collaboration with a few stakeholders.
Basically — ask several internal experts to review and classify a representative sample of your population, and from there fit a model against the manual classifications. This will allow you to understand what is the ‘data’ definition that best matches the subjective concept you are trying to define.
- This approach enables alignment inside your organization as you bring in stakeholders from different teams with different point of views.
- It can also create meaningful discussions and unlock new learnings, especially when two different experts have different classifications of the same record — this will help you and them understand each other’s rationale.
How to concretely use it
- Pull a list of records that are representative of your whole population
- Ask a few experts internally to help manually classify the different records and review the classifications with them to come up with a final rating
- Congratulations! You now have a source of truth for your new metric, and this is now a much more simple classification problem. You can start fitting a model against the final ratings, and adjust it based on your needs (i.e. how you balance explainability vs complexity or accuracy vs precision, or else).
- Once you have a 1st definition, present it to your audience, and show them what that would mean to the business (with some concrete examples). Iterate as needed.
Start operationalizing today!
The methodology presented above is one of many methodologies that you can use when presented with these kind of questions. It is not a panacea, and it does require you to find some experts willing to spend time transferring the knowledge they have been internalizing for years, but it can work pretty well when you are working on projects requiring a lot of alignment and/or on projects where you are stuck and you don’t even know where to begin.
Thanks for reading!
If you had fun see my other articles:
- 7 tips to make your data analysis more robust
- How to Choose Which Data Projects to Work On
- How to… build a successful dashboard
One Simple Methodology to Turn a Subjective Concept into an Objective Metric was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
from Towards Data Science - Medium https://ift.tt/GtkhxU3
via RiYo Analytics
ليست هناك تعليقات