Inflated ratings complicate online nursing home comparison

Nearly two million Americans currently reside in 15,700 nursing home facilities across the United States.

Every year, Medicaid spends at least $140 billion on long-term services and supports. The Centers for Medicare & Medicaid Services (CMS) designed and implemented its nursing home rating system after a congressional hearing in 2007 in which Senator Ron Wyden asked why it was “easier to shop for washing machines than it is to select a nursing home.” Given the lack of alternative information resources on nursing homes, the publicly available CMS rating has become the gold standard in the industry. Recent studies show that after the release of the ratings, the market share of one-star nursing homes decreased by eight percent while the market share of five-star facilities increased by more than six percent. Given its important role, nursing homes have a significant incentive to improve their ratings; however, these ratings may not always reflect true quality. The heavy reliance of the star ratings on nursing homes’ self-reported measures, coupled with the lack of an auditing system to detect and deter inflation, significantly reduces its reliability.

To determine a nursing home’s star rating, CMS first conducts an on-site inspection which determines an initial rating. The nursing home will then self-report measures of its quality and staffing which can add or subtract up to two stars to/ from its initial on-site rating. For example, a nursing home that initially received three stars from the on-site inspection could increase its overall ratings to five stars if it self-reports excellent measures on its quality and staffing. On the other hand, if the same nursing home reports unsatisfactory measures on its quality and staffing, it could lose two stars and downgrade its overall ratings from three stars to just one.

As shown in Figure 1, in 2009 when the system first started, only 11 percent of the nursing homes self-reported excellent performance in their quality measures and therefore received five stars in this domain. This percentage increased to 33 percent in 2013.

Figure 1: Distribution of star ratings from self-reported quality measures

As a result of these self-reported measures, as shown in Figure 2, the percentage of the nursing homes with five-star ratings has increased from 11 percent in 2009 to 25 percent in 2013. Over the same period of time, the percentage of nursing homes with a one-star rating has decreased from 22 percent to only 10 percent.

Figure 2: Distribution of overall star ratings

In a recently released paper, my colleagues and I shed doubt on the legitimacy of self-reported measures, by analyzing publically available datasets about the nursing homes in the state of California.  We show that while nursing homes’ financial incentives increase self-reported measures, such ratings are neither associated with the corresponding on-site inspection ratings in the same year nor the year after. We corroborate our findings by showing that patient complaints reported by independent third parties are predicted by the on-site inspection ratings, which cannot be inflated, rather than overall ratings, which can be inflated. Our estimations show that between six and 12 percent of the nursing homes in the study are likely inflators.

To foster competition among nursing home facilities, patients should have access to reliable information and be able to make informed choices. Performance information has been the missing element in the health care market for a long time. Over the past few years, CMS has spearheaded efforts toward transparency and empowering patients to make informed choices through various initiatives including hospital compare and nursing home compare. The benefits of such initiatives hinge on the reliability of the information that they report. Unless their reported information reflects the true quality of medical providers, patients will soon lose trust in these resources. Given our findings, we recommend CMS to enhance the reliability of the nursing homes rating system by establishing an audit system to detect suspicious self-reported measures and impose penalties to deter it.

Read the full paper “Five-star ratings for sub-par service: Evidence of inflation in nursing home ratings” here.