The PASS Business Analytics Conference 2015 was April 20 to 22 and it was an honor to be a speaker, sharing knowledge and getting to know people in the Business Intelligence realm who are outside of Excel. PASS is primarily a SQL community but they recently started to look at the larger business intelligence picture and ask questions about the tension between IT departments and end users.

• Who are Excel users?
• What are their needs?
• How do they see the world?
• How are they different from professional IT personnel?

Conversations with a lot of people around the PASS Conference left me with a few ideas to dig into. Today’s post will explore a statistic that was presented at the Conference:

80% of big data solutions are never finished or are never adopted.

Whether 80% is accurate, or not, isn’t my focus. I’ve been involved in a failed big-data solution, and have heard more failures than successes. After a while, big-data just bores the hell outta me. What is far more interesting and but continues to be missing is empowerment of the end-users.

HOW CAN BIG DATA SOLUTIONS FAIL?

Here’s one way how these things can go:

Titillating Data Solutions sells their multi-million dollar integrated system to CEO, Mr. Means-well.

Things get complicated.

Managers need different kinds of reports.

Other departments see an opportunity to force Customer Service to start collecting data that takes time to collect and has nothing to do with customer service. (“I’m ordering scarf, why are you asking me for my education level?) Customer Service successfully fights back.

The end-users in various departments need reports but also need to override various inputs and calculations due to business processes that are still unestablished.

Finally! The titillating solution is in place. However, at the lowest levels it’s still not handling critical scenarios properly, and people who were empowered before the transformation transmogrification are now reliant on others who have the permissions to work the sophisticated system. For them, this isn’t a sip of smooth sweet centralized data, it’s a mouthful of tepid, snake oil!

Company-wide, the corridors echo with the sound of one person after another spitting out the snake oil. En masse they’re going back to their Excel spreadsheets and guerrilla processes. RIGHTFULLY SO!

WE’RE ALL DATA MANAGERS & NEED TO ACCEPT THAT

Rather than try to relieve end-users of the burden of managing data, we need more people to see that they are data managers. That means getting trained outside of whatever our regular jobs are so that data is a complement to our objectives and not an annoyance. That way, we might be able to reduce the problems that cause us to go looking for complex expensive solutions that are far more likely to fail than succeed.

Excel is a tool that can be almost anything we need. Most folks have it and could learn to get more use from it. They don’t even need mastery. They need to accept that data management is a skill that can be learned and refined.

Here’s one example of where fundamentals can help someone do much better with their data management. Let’s step outside of a corporate enterprise and into the world of a 1-person operation–an area that big data solutions can’t help. Yet, the issues are the same for an end-user in a company, and the consequences are serious.

THE EVENT PLANNER

An Event Planner is tracking costs of a party. Here are a few things to notice about the spreadsheet below:

• The yellow and black cells have significance. A common mistake is to have random colors or just one color. In this spreadsheet:
  • Yellow cells are inputs that the client or event planner can change
  • Black cells are formulas that calculate and pull key data from somewhere else.
• For each of the yellow cells they are the only place that the value needs to be added or changed. Cost of linens, for example, might impact 10 or more formulas. One common spreadsheet blunder is to have multiple places to enter the same value. No! Design a spreadsheet such that there’s ONE place for changes.
• Think of ways that the data can be useful at-a-glance.
  • There’s a huge text box that makes is clear that the client is $1537.15 over budget.
  • It’s also helpful to know more than a total cost so, there are Low and High figures, the difference begin: if fewer tables are rented and guests are packed in tighter, the cost is $8287.15 but if people are allowed maximum space, the cost is $9756.33.

The 246 guests in the previous image (cell J6) is coming from the “Plan For” field, cell G7, on a separate worksheet:

The RSVP list is a whole separate Excel worksheet that automatically updates everything throughout the workbook with changes in the RSVP count or number of guests. I also added a spot in cell G5 where an event planner has the option of estimating the percentage of people who are likely to show up.

Let’s look at the impact of 2 changes:

• Add 7 guests for Amelia Hall
• The client increases the budget to $9000
  Bottom Line: $476.62 UNDER budget.

Everything is automatic!

I am purposefully leaving out the how-to because the internet if full off how-tos already. What’s missing is presentation of what’s possible and how to think, plan and organize. In this scenario, it actually might help some event planners to accept that they are in the data management world. Putting a name to a task, and showing that there are solutions, Excel can be seen as a tool, and the overall job as do-able, rather than an unnamed hassle.

REDUX: Big Data Solutions

We still need solutions for the fact that data is coming at us from all sides. I don’t think that data scientists and expensive solutions are irrelevant, but conversations have to expand beyond all the fancy, intellectual shit. Otherwise, big data solutions will continue to be undermined until, either:

1. One day we wake up to a world where data management can truly be limited to 3 or 4 trained people. Every entity from the international corporations to the teenager with a handwritten shopping list are all operating with clean, centralized data, and established processes. Or,
2. We can empower and make partners of the folks on the ground level of this all-data all-the-time world.

Until one of those 2 conditions is met, Small Data David will continue to triumph over Big Data Goliath.

“with a sling and a stone; without a sword in his hand he struck down the Philistine and killed him.” (Samuel 17:50)