From Davis Balestracci – New results = New conversations (%-rates and Standards: Part 2)

Published: Mon, 04/10/17

From Davis Balestracci – New results = New conversations (%-rates and Standards:  Part 2)

"Thank goodness Davis was entertaining. Otherwise today would have been absolutely brutal! I am going to go back and convert my bar graphs." -- Participant evaluation from 22 February Data Sanity seminar

[~ 950 words: take 3-1/2 to 4 minutes to read over a break or lunch]

View as Web Page

1. A Data Sane Alternative for Percentage Performance Comparisons


Hi, Folks,

My last newsletter demonstrated a common incorrect technique for comparing percentage rate performances – based of course in the usual normal distribution nonsense.  Let's revisit that data with a superior alternative.

To quickly review the scenario:  In an effort to reduce unnecessary expensive prescriptions, a pharmacy administrator developed a protocol to monitor and compare individual physicians' tendencies to prescribe the most expensive drug within a class.

Data were obtained for each of a peer group of 51 physicians – the total number of prescriptions written and how many of them were for the target drug.  During this time period, these 51 physicians had written 4032 prescriptions, of which 596 were for the target drug – an overall rate of 14.8 percent.

The Correct Alternative – a p-chart Analysis of Means

.
The goal of analysis of means (ANOM) is to compare a group of physicians who have what should be similar practice types – a relatively homogeneous "system," if you will.  Each is compared to this system's overall average. Variation is exposed and a group conversation ensues to discuss the variation, then reduce the inappropriate and unintended  variation.

For each individual physician's performance, one calculates the common-cause limits of what would be expected due to statistical variation from the system's 14.8 target prescription rate. Based on the appropriate statistical theory for percentage data based on counts (i.e., binomial), a standard deviation must be calculated separately for each physician because each wrote a different number of total prescriptions, in this case, ranging from 30 to 217. 

The calculation for this situation's p-chart ANOM is as follows (note its dependence on the system average):

The result of the square root is multiplied by three (for "three standard deviations"), then added and subtracted to the overall system average to see whether the actual value for an individual physician is in the range of this expected variation, given an assumed rate of 14.8 percent – "innocent until proven guilty," the best strategy for dealing with physicians from my experience.

Last newsletter's analysis is shown directly below, and the p-chart ANOM is below that.  


Note that what many of you would consider conservative three standard deviation limits are, in the case of the ANOM, comparable to approximately 1.5 standard deviations of the incorrect analysis. Why?  Because the standard deviation is calculated correctly

Another difference: the overall system value obtained from the aggregated summed numerators and denominators of the 51 physicians was 14.8 percent (596 / 4032), which differs from taking the average of the 51 individual percentages (15.8).

In ANOM, anyone outside their (correctly calculated) unique common-cause band is a probable special cause; these physicians are truly "above average" or "below average." Note that:  (1) physicians 48 and 49 could still be indicative of a prescribing process at 14.8 percent because of the number of prescriptions written, and (2) there are five below average performances found by the three standard deviation criteria (there is not even a lower two standard deviation line in the incorrect analysis).

The incorrect analysis and its inappropriate declaration of normality, coupled with the standard deviation criterion subjectively selected could claim to statistically find one or 11 upper outliers, using two or one standard deviations respectively. The ANOM shows eight probable above average outliers with a lot more certainty.

So, what should we conclude from our correctly plotted graph? Only that these outlier physicians have a different process for prescribing this particular drug than their colleagues, 36 of whom exhibit average behavior. These physicians between the red lines are indistinguishable from each other and the system average of 14.8 percent.

Some physicians' outlier variation might be appropriate because of the type of patient they treat ("people" input to their process), while for others it may be inappropriate or unintended due to their "methods" of prescribing – but they don't know it. Maybe collegial discussion (also considering the outliers who are below average?) using this graph as a starting point would be more productive than what has become known as "public blaming and shaming."

I get very positive responses when presenting this approach to front-line physician groups in grand rounds:  This makes sense to their scientific intuition. In fact, many have told me, "If results were presented to us like this, we'd take care of it ourselves."

This gives them back the sense of control they lose when presented with arbitrary, incorrect, judgmental analyses that inappropriately threaten their sense of competence.
==========================================================================
Some of you may have already encountered a similar analysis with the term "Funnel Plot"
==========================================================================
For some reason, academic journals have taken a fancy to ordering results with the X-axis sorted from lowest denominator to highest then labeling the result a funnel plot due to its resulting appearance.

Here is the ANOM above presented as a funnel plot. The same doctor labels from above are shown on the X-axis. My personal preference remains sorting the proportions from lowest to highest as above.


2. For the people obsessed with standards


Integrating ANOM with Standards Thinking

  • If there were a standard of 15 percent, the first order of business would be to study the eight above average outlier physicians and see whether it is appropriate to bring them into the average red zone with their 36 colleagues. 

  • If the standard were 10 percent, there needs to be fundamental change in all physician behavior, but the discussion could begin by dialoguing with the five physicians who are truly below average...if their outcomes are better or at least no different. 

Consider.  In the case of an arbitrary 10 percent standard, is such a level even appropriate? With the current focus on costs, shouldn't linking these discussions with observed patient outcomes also be part of the equation? I would call this a much better analysis for determining where the process should be – for which patients would be thankful.

Until next time...

Kind regards,
Davis
==============================================================================
Chapter 7 of my book Data Sanity thoroughly covers the technique Analysis of Means (ANOM) and is one of the very few available resources for learning it.
==============================================================================

=====================================================
NOTE:  for some unknown reason, U.S. Amazon currently has Data Sanity discounted 26 percent! (free shipping)          
[I honestly don't know for how long]
===============================================
ANOM is a very powerful statistical stratification technique that has become one of my major analyses -- and should be one of yours, too. It mysteriously remains a seemingly well-kept secret.


Data Sanity: A Quantum Leap to Unprecedented Results is a unique synthesis of the sane use of data, culture change, and leadership principles to create a road map for excellence.

One of its major goal is to create a common organizational language for healthier dialogue about reducing ongoing confusion, conflict, complexity, and chaos. ​​​​​​​

  • e-book format for all e-readers, including iBook, Nook and Kindle (available only through publisher MGMA, includes downloadable .pdf)

  • GREAT NEWS for UK readers who want a hard copy.  It is now available on Amazon UK for  £69 with free shipping.

My publisher has informed me that there will also be an option to print on demand in Europe, Canada and Australia.  He has also lowered the price a bit in these countries.
[Any problems or questions, please e-mail Craig Wiberg at: cwiberg@mgma.org]

Click here or visit my LinkedIn profile for a copy of its Preface and chapter summaries.

=============================================================================
Please know that I always have time for you and am never too busy.to answer a question, discuss opportunities for a leadership or staff retreat, webinar, mentoring, or public speaking --  or just about any other reason!  Don't hesitate to e-mail or phone me.

==============================================================================
Please visit my LinkedIn page to listen to a 10-minute podcast or watch a 10-minute video interview where I talk about data sanity.

=========================================================
Was this forwarded to you?  Would you like to sign up?
=========================================================
If so, please visit my web site -- www.davisdatasanity.com -- and fill out the box in the left margin on the home page, then click on the link in the confirmation e-mail you will immediately receive.