I found this blog a few months ago: NonProfit With Balls (NWB). The blogger is Vu Le and this guy loves data as much as I do. He says that if data were syrup, he’d put it on his soy ice cream. If you know me, I’d say that capsaicin is the edible form of data. Food isn’t worth eating if it isn’t spicy. Similarly, life is a bland meandering mess without data.

Nonprofit With Balls covers the nonprofit universe. Vu spends a lot of time on data-related issues in that universe, and does it from an on-the-streets, hands-dirty perspective.

Vu’s recent blogpost, Weaponized data: How the obsession with data has been hurting marginalized communities, is fascinating because it digs into ways that data can be misused, and the consequences of even the most goodhearted, unintended misuses. As Vu puts it:

WEAPONIZING DATA

Data: Protector or weaponized menace?

Weaponizing data is an intriguing concept, especially how Vu presents it. We don’t intend to use data for evil. Sometimes we mean well. Other times we’re compelled to provide data when we know there’s a backstory that belies what the numbers suggest.

Vu offers an example of measuring the impact of a program, while knowing that you had a positive impact on kids who dropped out mid-way. However, the dropouts can’t be counted in any stats for reporting positive influence.

The consequences can be:

• On paper, the program or organization don’t look good enough to support financially
• Pressure to find ways to keep kids in the program–even those who aren’t benefiting
• The end of the program because it can’t get sufficient funding
• Extensive resources spent scraping together tiny donations because the stats don’t look good enough to large funding sources

The weaponizing of the data is this unexpected and unintended data skirmish that some entities just can’t win for any number of reasons. They look bad on paper. They don’t have the resources to collect the most useful data. Their effectiveness isn’t easily quantifiable.

But Vu offers several ways that we can de-weaponize data. This next section will focus on disaggregation of data.

DE-WEAPONIZING & DISAGGREGATING

Fundamentally, we’re getting to my biggest concern: data literacy. I think Vu’s focus on the bigger picture doesn’t say enough about how we need to empower ourselves to understand data.

Let’s go back into a world I spent a long time in: call centers.

Let’s say there are 5 call center reps and we’re going to measure “who took the most calls.” After 8 hours, Vann trounced everybody: 100 calls, and Ted was in second place with just 69 calls.

317 calls and almost a full one-third was all Vann, all day.

What is Vann’s magic? But be careful, the weaponization of this data could be: “why aren’t you people as good as Vann?” If Vann is some highly-trained, seasoned pro, it’d be unfair to compare everyone to Vann.

But let’s do some disaggregation because the numbers do look a little odd.

Accessing a data dump of all 317 calls we can look deeper into the overly simple count of all the calls. This is why having access to a data dump is important.

What Does The Disaggregation Tell Us?

This data does not tell us any truth about the world. Instead, it gives us more questions to ask, and more work to do. Vann has a high number of short calls and much fewer long calls, compared to the other reps. Next question:

Is Vann fast and supra-efficient, or is Vann doing something diabolical like hanging up on complicated calls or transferring calls instead of handling them?

Digging into the answer also requires being open to all possibilities, including flaws in the data-collection methods.

Unfortunately, in situations like an “answer the most calls” contest, the resulting data is a weapon against inexperienced people, and people whose style just isn’t cranking through calls. We can ask questions about the structure and appropriateness of the contest itself, but overly simple definitions and overly simple use of the data is the weaponization.

PERVERSE DISAGGREGATION

Disaggregation can be used to persuade and confuse. Public schools that are at risk of being closed can be put in a difficult position to disaggregate data so that the school looks better on paper. “Our overall numbers are horrible, but what if we:

• Exclude the performance of the kids who live in a homeless shelter?
• Disaggregate the data as snapshots of before/after we absorbed kids from that other school that was closed?”

On the face of it, this can look repugnant. The deeper problem is the understanding that data is key to loss or victory, and it becomes incentive to start doing weird stuff like hanging up on people or blaming “those kids” for an entire school’s poor performance data.

Next, let’s lighten up a bit and look at the world of sports.

WEAPONIZATION IN SPORTS

In Super bowl XX the Chicago Bears beat the New England Patriots 46-10. The Patriots are often listed among “Teams That Should Not Have Been In The Super Bowl.” This suggests that the teams got lucky along the way, stumbled into the Super Bowl and got a sound thrashing. As evidence, the 46-10 loss, an embarrassingly low 123 offensive yards, 4 lost fumbles, and 2 interceptions.

For anyone familiar with the 1985-86 season, we saw the Chicago Bears defense dominate the whole season. Any team that ended up in the Super Bowl against the Bears defense was in trouble. They were first in the league for fewest points allowed: 198.

Therefore, using data to show the Patriots as victims of luck is a weapon against the real story of the ruthless, electrifying Bears defense that dominated the NFL that whole season.

FINAL THOUGHTS

I appreciate Vu’s blogpost because it reminds us of the strengths and weaknesses of data. For me, it’s more evidence that data literacy is in sore need of attention. More of us who are on the streets and are responsible for data must be more data-conscious and data savvy. Do the numbers look weird? Is the data clean? What are the definitions that lead to the calculations and results? Can disaggregation help or just become another weapon pointed toward a new enemy?

When you get data, be sure you also get access to the raw data so that you can use a data dump to disaggregate the results and dig into relevant questions. Learn to work with data dumps.

A lot of the analysis that many of us have to do is not technically hard. We just need a new way of thinking so that we can disarm, deflect, de-weaponize data. Be sure to check out Vu Le’s other suggestions for de-weaponizing data.