The first game ESOs play when reporting their metrics is that they measure meaningless metrics to begin with. You can read my post Why Typical Entrepreneurship Support Success Metrics are BS to get a taste for some of the key metrics that nearly every ESO uses that don't actually mean much.
In addition to these universal ones, there are others specific to individual programs, which may include things like:
- - 100% of program graduates will X….when X is a requirement of graduation from the program so it's literally impossible for this stat to not be 100%
- - 80% of program participants will have increased their knowledge of Y, as reported by student surveys….as if students are likely to write on a survey, "I actually didn't learn anything at all"
- - 80% of program participants will feel more confident about Z….which may be useful if the course is about self-esteem but tells you nothing about whether a business owner is actually positioned to be more successful
The second game that ESOs play when reporting their metrics is that they often have very loose definitions for those metrics. One metric that pretty much all ESOs measure is clients served and it's very typical for ESOs to host events, or even speak at events hosted by others, and count every person in the audience as a client served. That's a very loose definition of a client served that would not fly in the for-profit realm.
The third game that ESOs play when reporting their metrics is that they often count the same entrepreneur more than once. Sticking with the clients served metric, many ESOs are unable to get attendance records from events that can easily be cross-referenced with their other clients to eliminate duplicates, meaning that the same person is often counted as two or three clients if s/he attends multiple events and is a training client.
The fourth issue isn't so much a game the ESOs play as a shortcoming in the system and it's the fact that most of the metrics ESOs measure count on reporting from entrepreneurs and are, therefore, incredibly inaccurate. When serving small businesses, it's very difficult to get business owners to spend the time to report data on contracts, loans, employees, etc. because it pulls them away from actually working on their businesses. Additionally, small business owners are notorious for not keeping tidy, accurate records so, even when you do manage to get the info from them, it's likely not 100% accurate.
Add these four key issues together and when you see an impact report from an ESO you're basically looking at an exercise is fudging the numbers, not any realistic picture of the impact that the ESO has had on the community(ies) it's meant to serve.
Source: Cate Costa.