The VA Proves Why Rating Systems and Quality Metrics Don’t Work

Want some more proof how the system can be gamed or rigged?  Or how about bogus ratings and metrics?  Well, here is the VA doing its thing again:

When the Department of Veterans Affairs released the annual ratings of its hospitals this fall, the facility in Atlanta dropped to the bottom, while the one in West Haven, Conn., shot to the top. It was something of a mystery as to why.

The Atlanta hospital was downgraded to one star from three on the agency’s five-star scale, even though there had been only a “trivial change” in its quality data from the year before, according to the department. The Connecticut hospital climbed to five stars from three, even though numerous operations had to be performed elsewhere or canceled at the last minute because of problems with sterilization of surgical tools, according to an internal assessment and other accounts cited by Senator Richard Blumenthal in a letter to the agency.

Veterans Affairs set up the rating system in 2012 in the hope of pushing its hospitals to improve, and it has been increasingly aggressive in using the ratings to hold hospital managers accountable. Leaders with low ratings can be ousted, as happened last week in Atlanta, where the chief of staff and heads of the emergency department, primary care and clinical access services were removed because of low scores.

But former senior officials at the agency and experts in health care metrics say the system can be confusing, and so arbitrary that hospitals may gain and lose stars based only on statistical error. More than a dozen hospitals improved care but lost stars; another did not improve and gained one.

What is most worrisome to some experts is the role that the star ratings now play in grading performance of hospitals and their managers. They say it creates an incentive to conceal problems rather than grapple with them, in order to collect bonuses or sidestep penalties.

You can find the rest of the article here in the NY Times.  The question is whether we can truly measure quality?


Will people game the system to make themselves look better?


The article is proof of everything I have been blogging about over the last decade.

Get our awesome newsletter by signing up here. We don’t give your email out and we don’t spam you.)

Douglas Farrago MD

Douglas Farrago MD is a full-time practicing family doc in Forest, Va. He started Forest Direct Primary Care where he takes no insurance and bills patients a monthly fee. He is board certified in the specialty of Family Practice. He is the inventor of a product called the Knee Saver which is currently in the Baseball Hall of Fame. The Knee Saver and its knock-offs are worn by many major league baseball catchers. He is also the inventor of the CryoHelmet used by athletes for head injuries as well as migraine sufferers. Dr. Farrago is the author of four books, two of which are the top two most popular DPC books. From 2001 – 2011, Dr. Farrago was the editor and creator of the Placebo Journal which ran for 10 full years. Described as the Mad Magazine for doctors, he and the Placebo Journal were featured in the Washington Post, US News and World Report, the AP, and the NY Times. Dr. Farrago is also the editor of the blog Authentic Medicine which was born out of concern about where the direction of healthcare is heading and the belief that the wrong people are in charge. This blog has been going daily for more than 15 years Article about Dr. Farrago in Doximity Email Dr. Farrago – [email protected] 

  4 comments for “The VA Proves Why Rating Systems and Quality Metrics Don’t Work

  1. Clif Tinsley
    November 4, 2018 at 4:28 pm

    I’m living this now. I’m at the Hampton VA which just went from a 1 star to 2 stars. I see no difference in Primary Care. I am the 6th provider on this panel in 3 years and they burned me out in 8 months. I asked for help from the Chief of staff and Department Head and they said they would get right on it. I work 11 hour days and I still can’t finish all the work. The assistant to the Chief of staff, Mrs Durio, wanted me to stay late to do a computer class. I told her I already work 11 hours a day. She told me that’s no excuse. I will definitely send her a Be Well email when I leave.

  2. bbneo
    November 4, 2018 at 1:13 pm

    In the book Extreme Ownership, the authors talk about how APPROPRIATE incentives can drive positive changes and how complex, poorly thought out incentives don’t. The VA is all about complex incentive programs that pay lip service to “performance improvement” without actually DRIVING performance improvement. It’s old news… “Same as it ever was… Same as it ever was…”

    The VA just needs to be watched closely and contained. They provide some good service to veterans and that is their ONLY reason for being.

  3. November 4, 2018 at 8:50 am

    Spot on, Doug, spot on!????????

  4. Dr. L
    November 4, 2018 at 8:41 am

    I work for VA and my location is one of the best. I have been to several as a a vet and they are all unique. I realized the bogusness of this system at a recent meeting where we were told we went from 5 to 4 stars because we didn’t improve. We were doing great! Maintaining should equate to a sustained score as well. Another example of bean counters trying to practice medicine. Running a company making widgets is not the same as running a medical establishment.

Comments are closed.