Fun with numbers – CIHI under fire for hospital rankings

John Wright folded his arms looking more than a little apprehensive.

The CEO of the Canadian Institute for Health Information was about to address a room full of hospital officials, many upset about the Canadian Hospital Reporting Project (CHRP) launched a month earlier. The forum was the May 16 Breakfast with the Chiefs organized by publisher longwoods.com.

CHRP was supposed to be the ultimate benchmark, looking at data from 600 hospitals across Canada and involving 100,000 bits of information.

Wright said that CIHI had already experienced 80,000 hits on the site, which to some may suggest success, to others a quantification of the damage done to the reputation of their hospital.

Knowing the onslaught that was coming, Wright pointed out that “perfection is achieved by slow degrees. It needs the hand of time.”

Kelly Isfan, CEO of the Norfolk General Hospital in Southwestern Ontario, said the purpose of such data was to make you ask more questions.

Looking at her hospital’s mortality rate, they were able to isolate the problems with one of four surgeons doing one specific procedure that he likely shouldn’t have been doing.

She said the CIHI indicators were a jumping off point for a conversation with her community, but said the clinical indicators were more useful than the financial ones.

Ontario hospitals looked particularly bad compared on administrative costs, which were higher than other provinces. Whereas many provinces run their hospitals directly from a regional health authority, Ontario has maintained its hospitals as independent not-for-profit corporations with their own boards.

Given many Ontario hospitals are taking away the ability of communities to participate in ratification votes of newly appointed board members, that cost difference could become a point of debate as questions get raised about the value of local boards that the community has no say in.

The ever-present Kevin Smith, CEO of Hamilton St. Joe’s, and the present provincially-appointed supervisor of the Niagara Health System, was much blunter about the CHRP. Smith complained CHRP often compared apples to oranges, reflected wild data swings, and was difficult to both use and understand.

The Ontario hospitals were particularly upset about the lack of consultation and notice about the release of CHRP, particularly after a Toronto Star story that reflected some particularly peculiar rankings.

“The face validity didn’t make sense,” he said.

Given perception is often reality, Smith said the rankings challenged the morale of hospital staff, who saw the rankings of specialty hospitals fall below small rural facilities with fewer available services.

The surprise release of the report also put many Ontario hospitals on the defensive.

Wright admitted there were many problems with CHRP.

“Low volumes are a huge issue,” he admitted.

While the report made it difficult to compare hospitals, particularly between provinces, Wright said the database was still useful in looking at the ongoing progress of an individual hospital.

He said the Toronto Star article “started ranting in a significant way.”

Responding to Smith’s comments about data swings, Wright said that there were data problems with the reporting hospitals.

“CIHI can’t fix this by ourselves – it is your data,” he said. “Now that we have your attention, can we pull this all together?”

Smith said hospitals should post embarrassing data every time it was true. “It drives performance. But we’re not comparing apples to apples,” he said.

Shalom Glouberman of the Patients’ Association of Canada, asked who CIHI had prepared this report for?

“What do patients want to know?”  he asked, stating CHRP could be far more patient friendly.

Neil Stuart, the Chair of Cancer Care Ontario, has been engaged by CIHI to speak with the hospitals and make recommendations around improving CHRP.

Meanwhile, fresh from that public bloodletting, the media is applauding the first hospital in Ontario to post real-time waits for their ER.

St. Mary’s General Hospital in Kitchener posts online how many people are waiting online and the expected time it will take to see a doctor. The system updates every five minutes, unlike the province’s web site which updates once a month.

The system uses software developed by Oculys, a third-party company set up by the hospital to develop and market the software. Profits from the sale of the software go back to the hospital.

The hospital tells the Toronto Star that the software is based on thousands of hours research to understand the complex algorithm of the ER. They say that within the first month the system had 10,000 hits and 87 per cent of patients were seen in less time than the system had indicated.

They admit that the system is built for the 30 per cent of ER patients who need medical help, but not necessarily urgent care.

Given patients undergo triage when they arrive in ER, it would suggest that the system is built on averages, not necessarily a true indicator of how long you’ll spend waiting.

The hospital did admit the live data has the potential for some embarrassment, or what the roomful of hospital officials last week suggested were challenges of public confidence.

John Wright admitted that the genie is now out of the model. Hospitals have no choice but to find ways to improve their data.

Is it all too much? Is there data fatigue out there? Does the focus on data collection take away from front line care or enhance it? Will funding and other policy decisions be driven by the data? These are questions that will undoubtedly be debated as hospitals open themselves up to have their performance scrutinized.

If you liked this story, pass it on! Click the share button below, post it on your Facebook site, or retweet it to your followers.