How do we Respond to Physicians' Classic Critique to Data?

Patient Safety Quality Monthly

• Email|
• Print

February 16, 2007

Today's column is written by Dr. Mark Smith, Senior Consultant with The Greeley Company. Mark is a practicing vascular surgeon in Palm Springs, California and has been consulting with medical staffs for The Greeley Company for the past three years. He has recently joined The Greeley Company full time and will be a regular contributor for this column. Here are his thoughts for this month.

A major shift in peer review for medical staffs across the country is a lessening of dependence on individual case review and an increase in the collection of rate data regarding physician performance. It is only natural that many discussions among physicians turn to a critique of the data. As more and more data, both internal to the organization as well as external (i.e., Healthgrades.com), is collected and handed to physicians, many physicians look at this data and push back. See if some of these comments don't sound familiar.

"The data is garbage; it's no good."

In many instances, this is may indeed be the case. In one hospital in South Carolina, the hospital gave initial mortality data back to their physicians. One cardiac surgeon looked at his data and found that they had assigned him four mortalities for the year when he knew that he had had only two deaths. He investigated and found out that the other two deaths ascribed to him were neuro-surgical patients who had undergone craniotomy. As a result, all the data lost credibility with this physician.

Of course, we need to subject the data to best efforts at getting it correct, especially in the area of physician attribution. The medical staff can actually lead this effort by creating better ground rules for their hospital on to how to attribute cases. Is the attending physician the admitting physician, the physician who provided the greatest amount of care, or the one who does the discharge summary? One medical staff puts the onus on the physician who completes the discharge summary unless they indicate at that time that another physician should be the responsible party for outcomes assignment.

Another area of debate is hospitalists and intensivists groups--when the care is provided in shifts and many physicians have a part of each patient's care. How do we assign individual cases in these situations? Many medical staffs have chosen the group think concept here. If they take care of the patients as a group, then they are assigned the outcomes as an aggregate group, and they all get the same score.

"My patients are sicker than other physicians."

This, too, may be a valid criticism of the data if it has not been subjected to risk adjustment to take these factors into consideration. However, today there are many companies that provide this type of data analysis and can deliver expected versus observed outcomes. Physician leaders should inquire whether or not their organizations have such a system in place for evaluating aggregate rate data of mortality and complications. If the hospital does not have a risk adjustment program in place, the medical staff should work with the administration to put one into place and not allow this criticism to have validity.

"It's not statistically significant."

At the end of the day, if none of the above can be invoked, this is the great fallback position. If the p is not <.05, then we obviously do not have to pay attention to it. We all learned that in medical school, didn't we? In fact, few individual physician's data ever truly obtains statistical significance due to a lack of sufficient numbers. Even institutions may not get enough volume to draw major conclusions, so we have mega studies like Framingham and meta-analysis to draw many large studies together to amass enough numbers. In fact, individually, at best, we can only look for major trends and the opportunities for improvement that they represent. Rather than looking for statistical significance, it is better to think in terms of signal-to-noise ratios. If a physician's results rise above the noise level, it's time to ask the question, "Why are you different?" In trying to answer this question about the data, the opportunities for improvement will reveal themselves much clearer than trying to get to the right p level.

The really important lesson about data is to make sense out of what one has rather than dismiss the data out of hand for what it isn't.

Sincerely,
Mark A. Smith, MD, MBA, FACS
Senior Consultant, The Greeley Company

Question of the Month | Expert Advice & Guidance

•  I've heard that medication containers have to be labeled in areas other than procedural locations or the operating room. Is this true?

• MM.05.01.09 requires medications to be labeled whenever they are prepared but not administered immediately. The term "administered ... Read More...