Evaluate your metric

From HMCwiki

Measurement of critical processes are the vital signs of organizational health. Yet many organizations make inferences which drive decisions on such poorly represented data. Here are some things to think about as you create and evaluate your metrics.

Goals

1. Strengthen the value of the measures that currently drive decisions.
2. Assist in knowing what to collect and what to report
3. Guide leadership on how to interpret the measures
4. Become a 10 Commandments of proper measure reporting. Groom or add to it to fit your organization.

Procedure

1. Look at the metrics you use or are creating to make decisions
2. Determine the type of metric and what it is reporting is appropriate for decision-making.
3. Apply the questions below to and adjust accordingly to elevate the metrics’ efficacy

Types of metrics Types of Questions Answered
Current Performance............How Are We Doing
Targets........................How Are We Supposed To Be Doing
Comparative Data...............How Are They Doing
Trends.........................How Have We Been Doing
Stretch Goals..................How Do We Do That
Worst Thing To Do..............Make Decisions Off Badly Represented Data

Purpose Questions

1. Can it answer the question “How do we know we got there”?
2. Can the measure be traced to a strategic goal?
3. Do you have an agreed upon integrity statement (Thou shalt) by all reporting data so transgressions are obvious?

Accuracy Questions

1. Is it representative of the entire population?
2. Is it statistically sound?
3. Is it placed in the appropriate control chart (see Bob Lloyd’s book)?
4. Is data graph data over time if possible?
5. Is it sensitive to important changes so they can be acted upon?
6. How can you insure integrity of the measure over time?
7. Does it indicate trends appropriately? Six successive data points steadily in an upward or downward constitute a trend.
8. Does your metric have value Blind Spots (no see) or Dead Zones (no effect)?

Efficiency Questions

1. Can it be electronically collected?
2. Can a sampling be representative alternative of 100% collection?
3. Can the measure be more useful as a flag in an exception report?
4. Does it display significant digits: rounded whole numbers, decimal points, etc. so that extra specificity doesn’t get in the way of significance?

Frequency Questions

1. How often does it need to re-sampled or rerun?
2. How ‘fresh’ or current does it need to be?
3. When should the measure be discontinued, does it cause action or just provide assurance?
4. Are there parameters for when to stop the measure known?

Effectiveness Questions

1. Does it measure the entire process?
2. Does it measure efficacy, efficiency, effectiveness, or accountability of a system?
3. Is data reporting sensitive to the decisions its supposed to drive: use local collection for local use?

Creation Questions

1. If there is not a direct measure, is there an indirect correlative alternate?
2. Are there at least eight data points to draw an inference.
3. Is the cost of producing the measure justified?
4. Is the measure clear to all, or does it need an explanation in the graph?
5. Is the form appropriate. Pie graphs to show portions, run charts for trends.
6. Can you use a smaller set of data with faster feedback instead of larger data with slo-o-o-o-wer feedback?

Interpretation Questions

1. If it’s a cycle time, are the start and the stop points clear?
2. Does it require interpretation to have meaning, if so whom do they contact?
3. Is it a process component piece of a larger measure?
4. Are special causes (out of limit points) annotated or fully explained?
5. Do the graphical displays tell the truth?
6. Does it represent significant change, does it infer with integrity, does the scale accurately reflect reality?
7. Are the rules for which type graph to use properly followed (Bob Lloyd’s book)?
8. How do you keep managers from ‘tampering’ with the data to look good?

Source Questions

1. Is the data source(s) listed on the graph?
2. Who commits to collect and report the measure and at what frequency?
3. Who needs to receive the measure?
4. When is the last time you checked with the ones using the measure as to the value to them?
5. Do you groom your distribution list or fall into the spam category.
6. Is confidentiality appropriately secured?
7. Can masking the data be more fitting, more acceptable?
8. Are there other people/departments that might gain from the measure?
9. Do those who impact the measure see the results?
10. Can public posting of data be of open benefit? It makes people ask questions. And questions usually help.

If you would like to provide feedback or discuss the content of this article, please visit the discussion page. To be automatically notified by email of any changes or additions to the article or discussions about this article, please click here.