Skip to main content

Conventional risk prediction models fail to accurately predict mortality risk among patients with coronavirus disease 2019 in intensive care units: a difficult time to assess clinical severity and quality of care

Abstract

Since the start of the coronavirus disease 2019 (COVID-19) pandemic, it has remained unknown whether conventional risk prediction tools used in intensive care units are applicable to patients with COVID-19. Therefore, we assessed the performance of established risk prediction models using the Japanese Intensive Care database. Discrimination and calibration of the models were poor. Revised risk prediction models are needed to assess the clinical severity of COVID-19 patients and monitor healthcare quality in ICUs overwhelmed by patients with COVID-19.

Dear Editor,

Since the start of the coronavirus disease 2019 (COVID-19) pandemic, intensive care units (ICUs) worldwide have struggled to treat affected patients who require a completely different approach to treatment than other patients [1]. Although many severe cases are admitted to ICUs, it is unknown whether the conventional risk scoring systems that were developed for ICU patients can be applied to patients with COVID-19. With unknown predictive performance, healthcare professionals have faced difficulties in assessing the clinical severity of patients with COVID-19 and monitoring the quality of care in ICUs. New risk prediction models for COVID-19 patients have been developed [2], but most of these were not developed specifically for ICU patients, and it is unknown whether they perform as well in clinical practice as they did in the model development studies. It is also likely that overwhelmed ICUs lack the time to derive and validate novel risk scores. In such circumstances, ICUs must use conventional scoring systems, such as the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score (SAPS). Several recent studies have used APACHE and SAPS to provide information on the clinical severity of COVID-19 [3,4,5]. However, very few reports have examined their validity of applying them to patients with COVID-19. One letter from the UK reported that APACHE II underestimated the risk of death, concluding that the risk scoring systems that were widely used before the pandemic were inappropriate for evaluating the clinical severity of COVID-19 [6]. In Japan, a research group recently developed the Japan Risk of Death (JROD), a prediction model that recalibrated the APACHE III-j model [7]. However, this model may show limited validity in patients with COVID-19 because it was developed using the data collected before the pandemic and it was designed for general use in ICUs. Therefore, we investigated whether conventional risk prediction models, such as APACHE II, SAPS II, APACHE III-j, and JROD, can be applied to patients with COVID-19 and determined their predictive performance.

We obtained data for confirmed cases of COVID-19 admitted between January 2020 and February 2021 from the Japanese Intensive Care Patient Database (JIPAD) [8]. We used JROD to predict mortality in the same way as in the previous study [7], but with a development period of January 2019 to December 2019. This was then applied to predict mortality in the study cohort and defined as JROD2019 predicted mortality. The predictive performances of APACHE II, SAPS II, APACHE III-j, and JROD2019 were assessed using the area under the receiver operating characteristic curves, Brier scores, Hosmer–Lemeshow tests, calibration plots, and standardized mortality ratios.

A total of 444 patients admitted to 40 ICUs in Japan were extracted from the JIPAD for analysis. The clinical characteristics of patients are shown in Table 1. The model performance statistics are presented in Table 2 and Fig. 1. Death at hospital discharge was recorded in 69 patients (15.5%), which was less than half the mortality reported by Stephens et al., although the APACHE II scores were comparable [6]. Using JIPAD data, the APACHE II, SAPS II, and APACHE III-j models overestimated the risk of death, whereas JROD2019 underestimated the risk. The discrimination and calibration of APACHE III-j and JROD were poor compared with those reported in the JROD development study [7]. Although the results are dissimilar to a previous report [6] in terms of the direction of estimated risk (i.e., overestimation/underestimation), we make the same conclusion that the risk models used before the pandemic are not suitable for patients with COVID-19. Of note, even JROD2019, a model that was developed to improve the predictive ability of APACHE III-j, displayed suboptimal predictive performance. Owing to the poor predictive performance, it is difficult to incorporate the predicted mortality calculated using these risk models in quality assessment tools, such as funnel plots and exponentially weighted moving average charts, with high reliability. Consequently, it will be difficult to implement quality assessment and improvement in ICUs, particularly those where patients with COVID-19 occupy a high proportion of ICU beds. Calibration can be improved with simple update methods, like that done in the JROD study, but discrimination can only be improved by updating the coefficients of each predictor and/or adding other relevant predictors [9]. Thus, a revised risk prediction model designed specifically for COVID-19 patients together with logistical support for its implementation in ICUs are urgently needed.

Table 1 Clinical characteristics
Table 2 Model performance statistics
Fig. 1
figure1

Calibration plots. APACHE, Acute Physiology and Chronic Health Evaluation; JROD, Japan Risk of Death; SAPS, Simplified Acute Physiology Score. Note: Observed mortality is plotted against predicted mortality. The study population was divided according to the predicted mortality into 10 equally sized groups, which are presented as a rug plot along the horizontal axis. A natural spline was drawn for the plots. The shaded area indicates the 95% confidence interval. If the calibration is perfect, the plot aligns with the diagonal line

Availability of data and materials

The authors’ agreement with the JIPAD project does not allow us to publish the data used for this manuscript or to share it with others.

Abbreviations

APACHE:

Acute Physiology and Chronic Health Evaluation

COVID-19:

Coronavirus disease 2019

ICU:

Intensive care unit

JIPAD:

Japanese Intensive Care Patient Database

JROD:

Japan Risk of Death

SAPS:

Simplified Acute Physiology Score

References

  1. 1.

    Ma X, Vervoort D. Critical care capacity during the COVID-19 pandemic: global availability of intensive care beds. J Crit Care. 2020;58:96–7. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jcrc.2020.04.012.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328. https://0-doi-org.brum.beds.ac.uk/10.1136/bmj.m1328.

  3. 3.

    Xie J, Wu W, Li S, Hu Y, Hu M, Li J, et al. Clinical characteristics and outcomes of critically ill patients with novel coronavirus infectious disease (COVID-19) in China: a retrospective multicenter study. Intensive Care Med. 2020;46(10):1863–72. https://0-doi-org.brum.beds.ac.uk/10.1007/s00134-020-06211-2.

    CAS  Article  PubMed  Google Scholar 

  4. 4.

    Ferrando C, Mellado-Artigas R, Gea A, Arruti E, Aldecoa C, Bordell A, et al. Patient characteristics, clinical course and factors associated to ICU mortality in critically ill patients infected with SARS-CoV-2 in Spain: a prospective, cohort, multicentre study. Rev Esp Anestesiol Reanim. 2020;67(8):425–37. https://0-doi-org.brum.beds.ac.uk/10.1016/j.redar.2020.07.003.

    CAS  Article  PubMed Central  Google Scholar 

  5. 5.

    COVID-ICU Group on behalf of the REVA Network and the COVID-ICU Investigators. Clinical characteristics and day-90 outcomes of 4244 critically ill adults with COVID-19: a prospective cohort study. Intensive Care Med. 2021;47:60–73. https://0-doi-org.brum.beds.ac.uk/10.1007/s00134-020-06294-x.

  6. 6.

    Stephens JR, Stümpfle R, Patel P, Brett S, Broomhead R, Baharlo B, et al. Analysis of critical care severity of illness scoring systems in patients with coronavirus disease 2019: a retrospective analysis of three U.K. ICUs. Crit Care Med. 2021;49(1):e105–7. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000004674.

    CAS  Article  PubMed  Google Scholar 

  7. 7.

    Endo H, Uchino S, Hashimoto S, Aoki Y, Hashiba E, Hatakeyama J, et al. Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures. J Intensive Care. 2021;9(1):18. https://0-doi-org.brum.beds.ac.uk/10.1186/s40560-021-00533-z.

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Irie H, Okamoto H, Uchino S, Endo H, Uchida M, Kawasaki T, et al. The Japanese Intensive care PAtient Database (JIPAD): a national intensive care unit registry in Japan. J Crit Care. 2019;55:86–94.https://0-doi-org.brum.beds.ac.uk/10.1016/j.jcrc.2019.09.004.

  9. 9.

    Steyerberg EW. Clinical prediction models. 2nd ed. Springer Nature: Cham; 2019. https://0-doi-org.brum.beds.ac.uk/10.1007/978-3-030-16399-0.

    Book  Google Scholar 

Download references

Acknowledgements

We would like to thank all of the hospitals participating in the JIPAD for their contribution.

Funding

This paper was written as a part of the JIPAD project, which was funded by the Japanese Society of Intensive Care Medicine.

Author information

Affiliations

Authors

Contributions

HE, HO, and JK conceived and designed the study. HE analyzed the data, interpreted the results, and wrote the first draft of the manuscript. SU, JH, KH, HI, JK, HK, TN, and MU contributed to data collection and ensured data credibility. YA, TA, EH, SH, JH, KH, NI, HI, TK, JK, HK, HM, TN, HO, HO, HS, TT, ST, KT, MU, RU, and SU contributed to the interpretation of the results and revision of the manuscript. SH organized the JIPAD project. The authors read and approved the final version of the manuscript and agreed to its submission.

Corresponding author

Correspondence to Hideki Endo.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Research Ethics Committee of the University of Tokyo (Approval number: 2020242NI-(1)). Consent to participate was waived because of the anonymous nature of the data.

Consent for publication

Not applicable.

Competing interests

HE, NI, and HM are affiliated with the Department of Healthcare Quality Assessment at the University of Tokyo. The department is a social collaboration department supported by grants from the National Clinical Database, Johnson & Johnson K.K., and Nipro Corporation. The other authors do not have any competing interests to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Endo, H., Ohbe, H., Kumasawa, J. et al. Conventional risk prediction models fail to accurately predict mortality risk among patients with coronavirus disease 2019 in intensive care units: a difficult time to assess clinical severity and quality of care. j intensive care 9, 42 (2021). https://0-doi-org.brum.beds.ac.uk/10.1186/s40560-021-00557-5

Download citation

Keywords

  • Coronavirus disease 2019
  • Risk of death
  • Intensive care unit
  • Risk prediction model
  • Quality improvement