Skip to main content
Download PDF

Development of quality indicators for palliative care in intensive care units and pilot testing them via electronic medical record review

Journal of Intensive Care volume 12, Article number: 1 (2024) Cite this article

Abstract

Background

Patients in intensive care units (ICUs) often require quality palliative care for relief from various types of suffering. To achieve quality palliative care, specific goals need to be identified, measured, and reported. The present study aimed to develop quality indicators (QIs) for palliative care in ICUs, based on a systematic review and modified Delphi method, and test their feasibility by reviewing electronic medical record (EMR) data.

Methods

The current study was performed in two phases: the development of QIs using the modified Delphi method, and pilot-testing the quality of palliative care in ICUs based on EMR review. The pilot test included 262 patients admitted to the general or emergency ICU at a university hospital from January 1, 2019, to June 30, 2019.

Results

A 28-item QI set for palliative care in ICUs was developed based on the consensus of 16 experts. The Delphi process resulted in low measurability ratings for two items: "Assessment of the patient's psychological distress" and "Assessment of the patient's spiritual and cultural practices." However, these items were determined to be important for quality care from the perspective of holistic assessment of distress and were adopted in the final version of the QI set. While the pilot test results indicated the feasibility of the developed QIs, they suggested that the frequency of care performance varied, and certain aspects of palliative care in ICUs needed to be improved, namely (1) regular pain assessment, (2) identification of the patient's advance directive and advance care planning for treatment, (3) conducting an interdisciplinary family conference on palliative care, and (4) assessment of psychological distress of family members.

Conclusions

The QI set, developed using the modified Delphi method and tested using EMR data, provided a tool for assessing the quality of palliative care in ICUs. In the two ICUs considered in this study, aspects of the palliative care process with a low performance frequency were identified, and further national surveys were recommended. It is necessary to conduct ongoing surveys at more facilities to improve the quality of palliative care in ICUs.

Background

Many patients in intensive care units (ICUs) need palliative care [1,2,3] to improve their quality of life as well as that of their families as they all face physical, psychological, social, or spiritual challenges associated with life-threatening illnesses [4]. In ICUs, approximately 20–70% of patients experience physical distress, such as pain, dyspnea, thirst, and fatigue [5], whereas 30–60% experience psychological distress, such as anxiety, depression, and post-traumatic stress disorder [6, 7]. Additionally, patients experience social and spiritual distress, such as loss of social roles, fear of death, loneliness, and loss of self-control [8]. Besides the patients, their families also experience psychological distress, including anxiety and persistent grief disorder [9, 10]. Therefore, quality ICU care should provide palliative care. Basic palliative care for ICU patients can shorten ICU stays, reduce the use of non-beneficial life-sustaining therapies, and reduce psychological distress in patients' families, without shortening patients’ life expectancy [11, 12]. High-quality palliative care can improve the quality of life and symptoms of patients and their families, and may reduce healthcare costs by aligning care with end-of-life goals [11, 13,14,15,16].

Quality indicators (QIs) are statements that define the quality of a service explicitly and quantifiably [17]. Measuring and reporting quality of care using QIs can identify potential problems in care and serve as foundation for further improvement [18, 19]. Previous studies, mainly conducted in the United States, developed and evaluated QIs using electronic medical record (EMR) data [20,21,22]. EMR data have the advantage of reflecting the patient's condition and treatment in an actual medical setting and can be collected without burdening the patients, their families, or clinical staff, as they are accumulated from daily clinical practice [23]. Additionally, discussing and decision-making regarding patients’ values and goals of care is important for quality palliative care [24].

A QI set needs to be based on Donabedian’s theoretical framework, evaluating three aspects including structure, process, and outcome [25]. However, the QI sets developed to date have limitations. Most of them focus on process indicators and do not include all three aforementioned aspects. Second, their feasibility has not been tested during development [26]. Additionally, they are based on a consensus among project teams in the United States [26] and have not been considered for use in other countries, such as those in Asia. Since the roles and functions of ICUs vary according to the healthcare delivery system and culture of each country, it is imperative to develop QI sets for individual countries [27]. Ideally, QI sets should be evidence based; however, evidence of palliative care practices in the ICU is currently limited. Therefore, the present study aimed to develop a QI set for palliative care in ICUs, based on a systematic review and modified Delphi method, which is a formalized process of consensus building by expert groups, and to test its feasibility by reviewing EMR data.

Methods

Study design

The study was performed in two phases, namely the development of QIs using the modified Delphi method and the measurement of quality of palliative care in ICUs by reviewing EMRs.

Development of QIs using the modified Delphi method

The modified Delphi method is a survey technique that involves multiple rounds to reach a consensus and is effective in determining expert consensus even when there is little or no conclusive evidence [28, 29]. The methodology and reporting of the modified Delphi study were based on Recommendations for the Conducting and Reporting of Delphi Studies [30].

Figure 1 outlines the Delphi process conducted between December 2021 and July 2023.

Fig. 1
figure 1

Modified Delphi process. This figure outlines the Delphi process conducted between December 2021 and July 2023

Full size image

In round 1, a systematic review of the QIs of palliative care in ICUs was conducted [26], and focus group interviews were held with eight core study members (palliative care physicians, intensivists, emergency physicians, nurses, and an expert on Delphi methodology).

In round 2, results of the systematic review and focus group interviews were considered, and the National Consensus Project Clinical Practice Guidelines for Quality Palliative Care, 4th Edition [31] was referred to in order to identify the best practices for palliative care of ICU patients and develop candidate QIs based on Donabedian's structure-process-outcome framework [25].

Subsequent rounds (rounds 3–6) comprised questionnaire surveys among expert panels, teleconferences, and e-mail discussions among the core members. One core member (Y.T.) sent an e-mail inviting potential panel members to participate in the study. The panel consisted of 16 members (five physicians, eight nurses, one physical therapist, one pharmacist, and one medical social worker) specialized in palliative or intensive care. Additional file 1 presents the details of the survey's rating by the panel of experts.

Pilot test of QIs for palliative care in ICUs via EMR review

The QIs for palliative care in ICUs, developed in this study, were used to measure the QIs by reviewing EMRs.

The two sites included in this study were a general ICU and an emergency ICU within the same university hospital. The general ICU is a semi-closed ICU system that is primarily intended for patients with acute illnesses, postoperative patients, and patients with chronic illnesses such as chronic respiratory disease, heart failure, and renal failure. The emergency ICU is a closed ICU system that is primarily intended for emergency patients, including those with trauma, stroke, and other conditions requiring immediate medical intervention. The general ICU had 18 beds, and the emergency ICU had 16 beds. Both ICUs have intensivists on duty 24/7. The hospital is the largest in the region, with approximately one million patient visits per year. The hospital's ICUs have been designated as formal training facilities for intensivists.

Eligible patients were defined as follows: (1) patients admitted to the ICU on an urgent basis, and who stayed in the ICU for 48 or more consecutive hours or (2) patients admitted after scheduled surgery, who required mechanical ventilation for more than 48 consecutive hours. These patient criteria were established through a Delphi process to develop the QIs. We reviewed the EMRs of patients admitted to the ICU between January 1, 2019, and June 30, 2019. Data were collected from core-member nurses who were not working in the ICU. To verify the inter-rater reliability, another nurse independently reviewed the QIs and checked agreement with the main reviewer. We randomly selected 10% of the patients from our sample for testing. Data collectors also monitored the time required to measure the QIs per patient.

Each process and outcome QI was assessed at the individual level over the duration of ICU care. Aggregate quality scores were calculated by dividing the number of times palliative care was provided (numerator) by the number of eligible events (denominator). Structural QIs were assessed by reviewing the site policies and interviewing the site staff. Data collected included patient demographic data, survival data, disease severity, and length of ICU stay. A QI measurement manual was developed and used (Additional file 2).

Ethical approval for this study was granted by the Institutional Review Board of Tohoku University (No. 2022-1-1023), and the study conformed to the principles outlined in the Declaration of Helsinki.

Statistical analysis

Summary statistics were calculated, and frequency distributions of the data were stratified according to each ICU. Performance frequency was calculated for each QI value. Continuous variables are presented as mean and standard deviation (SD), if normally distributed, and as median and interquartile range, if not normally distributed. To assess inter-rater reliability, we calculated the Cohen's kappa coefficient and the mean of differences. The data were analyzed using JMP version 17 (SAS Institute, Cary, NC, USA) and R version 4.2.0 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Development of QIs using the modified Delphi method

In round 1, a systematic review was conducted, and 109 QIs were extracted from 5 literature sources [20, 22, 32,33,34]. The results were combined with the opinions from the focus group interviews to develop a draft of 35 candidate QIs.

In round 2, the core members discussed the appropriateness of each QI as an indicator to assess high- or low-quality palliative care in the ICU and narrowed the list to 28 candidate QIs.

The expert panel consisted of 16 participants with 20.0 ± 7.2 years of experience. Response rate of the surveys (rounds 3 and 5) was 100%; none of the responses were missing.

The panel members rated each QI item in rounds 3 and 5 (Table 1). In round 3, there was no item with a median score below seven for appropriateness, but five items (items 3, 5, 7,8, and 13) were rated low for feasibility. The core members mainly discussed feasibility issues (round 4), revised the QI titles, and modified the denominator settings. In round 5, feasibility ratings improved for most items. However, the following two items had median scores below seven: item 5, assessment of the patient's psychological distress, and item 7, assessment of the patient's spiritual and cultural practices.

Table 1 Delphi ratings for the advocated quality indicators
Full size table

Discussions among core members (round 6) resulted in opinions regarding two of the above indicators, such as "the low feasibility of measurement ratings may simply reflect the fact that they are not currently being documented." Another commented, "spiritual and cultural practices are important to understand what kind of person the patient is, and are necessary as a consideration for diversity in the future." Therefore, we agreed that the two indicators would be necessary to improve palliative care in our country in future and decided to adopt the two items: item 5, assessment of the patient's psychological distress, and item 7, assessment of the patient's spiritual and cultural practices. Consequently, we modified the first 28 candidate QIs, and the new 28-item QI set consisting of eight domains became the final version (Table 2).

Table 2 Set of 28 quality indicators for palliative care in ICU
Full size table

Pilot test of the QIs for palliative care in ICUs via EMR review

Pilot testing of the developed QI set confirmed that process and outcome indicators were measurable from EMR data, whereas structural indicators were measurable from a survey of facility policies. Inter-rater reliability of the assessments was evaluated using Cohen's kappa coefficient. Overall, for QIs 5–15, the obtained Cohen's kappa value was 0.92 (95% confidence interval [CI] 0.87–0.97), indicating substantial inter-rater agreement. Additional file 3 presents the inter-rater reliability for each QI. Measurements per patient took a mean time of 32.2 ± 16.8 min.

Patient characteristics

In 2019, a total of 1697 patients were admitted to the ICU. Of these, 850 patients were admitted from January 1, 2019, to June 30, 2019, and 262 (30.1%) were eligible for measurement of the QIs. Of the eligible patients, the patients scheduled for surgeries were all admitted to the General ICU, which accounted for 23.1% of the total number. The mean age of the patients was 63.1 ± 17.6 years, and 42.4% were admitted to the ICU for surgery. The mean duration of ICU stay was 7.1 days (Table 3).

Table 3 Characteristics of ICU patients
Full size table

Structures, processes, and outcomes of palliative care in ICUs

Table 4 presents the frequencies of process performance and outcome indicators for the entire study sample. Of the 15 process and outcome indicators, 7 with performance frequencies less than 50% were as follows: (1) regular pain assessment, (3) reassessment of pain after treatment and/or management, (5) assessment of patient’s psychological distress, (7) assessment of patient’s spiritual and cultural practices, (8) identification of the patient’s advance directive and advance care planning (ACP) for treatment, (9) conducting an interdisciplinary family conference on palliative care, and (11) assessment of family members’ psychological distress. Additionally, there were some differences in the percentage of practices across the sites. The largest differences between the sites were regarding (1) regular pain assessment and (14) pain-free status in the last 24 h of life. “Patient pain-free in last 24 h of life” could not be evaluated due to the low implementation rate of “regular pain assessment” in emergency ICU.

Table 4 Frequency of performance of the measures of processes and outcomes of palliative care in ICU
Full size table

Table 5 presents the results of evaluation of the structural QIs of the two ICUs. The results were similar since the ICUs were located within the same hospital.

Table 5 Performance frequency of measures of palliative care structures in the ICU
Full size table

Discussion

In this study, based on a systematic review and expert consensus, we developed a QI set for palliative care in ICUs that consisted of 28 items across eight domains. Pilot testing demonstrated the set of QIs to be feasible and measurable. Additionally, the pilot test results suggested several potential improvements in palliative care in ICU settings.

This study developed a QI set that followed Donabedian’s structure-process-outcome framework. Additionally, the QI set was based on a systematic review and covered eight domains of the Clinical Practice Guidelines for High-Quality Palliative Care, which provide a foundation for improving the quality and delivery of palliative care in the United States [31]. In the Delphi rounds of this study, two items, namely “assessment of the patient's psychological distress” and “assessment of the patient's spiritual and cultural practices,” were rated as having low feasibility. Low-feasibility indicators are often not documented in EMRs, and hence, the potential need for quality improvement is missed [35, 36]. We included these two items in the QI set, assuming that low feasibility due to non-documentation is indicative of poor quality of care.

This study confirmed the feasibility and inter-rater reliability of QI measurements using medical record data from two ICUs. An average time of 32 min was spent on each patient to measure the process outcome QIs. Since this was a pilot study, the time taken for these measurements may reduce as people become accustomed to the task of measurement. A study testing the quality-of-care indicators for patients with cancer reported that the review of 92 indicators from EMRs takes approximately 2.4 h per patient [37, 38]. Thus, the necessary information may be collected within a reasonable time, but the simplicity and feasibility of the evaluation method should be further enhanced to promote quality palliative care worldwide. In this set of QIs, the evaluator spends more than half of the time on the analysis of the patient's assessment of pain and delirium. In this regard, to reduce the evaluation time, data covering only 48 h after admission to the ICU could be assessed. In addition, natural language processing and machine learning techniques have been demonstrated to handle textual data from EMRs with little burden on healthcare providers or patients in palliative care settings [39,40,41]. This state-of-the-art technology can enable rapid review and feedback of documentation. Therefore, the simplification of QI measurement methods and the use of technology in evaluation are important to increase the feasibility of continuous evaluation.

Pilot testing of the medical record review survey identified the following four aspects that required improvement: assessment of regular pain, identification of the patient's advance directive and ACP for treatment, conducting an interdisciplinary family conference on palliative care, and assessment of the psychological distress experienced by family members.

In this study, the frequency of pain assessment was 46%. Previous studies conducted in the United States reported pain assessment frequencies of 76–87% [20, 21], hence, suggesting the need for improved pain assessment in Japan. The outcome indicator “no pain in the 24 h before death” was 69% in this study, which is not greatly different from 72% in the United States [34]. However, because the frequency of regular pain assessment was low, the presence or absence of pain may not have been accurately assessed, thereby possibly affecting the outcomes. The guidelines emphasize that the presence of pain and need for pain management should not be dismissed in critically ill patients, who are often unable to communicate clearly [42, 43]. Therefore, continuous assessment and recording of pain severity would ensure that pain presence is not neglected in critically ill patients; this would improve the quality of ICU care, including the efficient use of analgesics and sedative medications.

In this survey, approximately 10% of respondents confirmed their advance directive and ACP for treatment, and approximately 30% discussed the patient's quality of life and values at a multidisciplinary conference that included family members. This was similar to the results of a survey of three ICUs in the United States, which reported advanced directive status (31%) and conduction of family conferences (19%) [21]. In a national institutional survey of ICUs in Japan regarding the frequency of multidisciplinary conferences on palliative care, 75% of ICUs reported that conferences were never held, and only 8% responded that they were held four or more times a month [44]. Interventions in family conferences and communication are necessary to provide patient-centered care and increase family satisfaction and trust in healthcare providers [45]. In recent years, the decision-making process for treatment in ICUs has recommended the implementation of discussions with patients and their families about goals of care within 5–7 days of ICU admission and holding of weekly multidisciplinary conferences [46]. Confirming ACPs regarding treatment and conducting family conferences are important for achieving treatment goals that are in line with the patient's wishes, that is, to provide patient-centered care.

A unique element of this QI set is the inclusion of a QI that assessed families’ psychological distress. In ICUs, where patients often lack decision-making capacity or are unable to express their treatment preferences, the patient's family is an important component of palliative care. Patients’ families, who are often surrogate decision-makers in the unique environment of the ICU, also experience psychological distress, since they are in crisis [9, 10]. Therefore, family members of ICU patients should be recognized as beneficiaries of palliative care, and assessment of family members' psychological distress is considered important.

The structure indicator in this QI set consists of three domains, namely "Setup and availability of resources and care protocols," "Support system for patient's family," and "Support system for ICU staff." Support for ICU staff is a distinctive domain. Prevention of burnout through a support system of ICU staff is an important indicator that leads to the continuous provision of high-quality care to ICU patients and their families [47]. This survey could not evaluate the structure of palliative care in ICUs in Japan, since data were obtained from only two ICUs. In future, surveys should be conducted at more facilities to clarify the current status at the national level.

Implications for future

This study developed a QI set comprising QIs that can be measured using data extracted from medical records, allowing easy measurability and sustained and continued use of the QIs, with limited measurement of patient outcomes. Future studies are warranted that evaluate patient-family experience outcomes from questionnaires regarding patient health [48] or assess family satisfaction using the questionnaires regarding intensive care units [49] in conjunction with measurement of this QI set. To improve the quality of ICUs, establishment of a system that, in parallel with educational interventions for staff, feeds the results of quality measurement surveys back to clinical practice in a cycle of improvement, would be required [50]. Kruser et al. conducted a large study that enrolled 68 ICUs in the United States and evaluated the three aspects of structure, process, and outcome to identify unit-level variation. To improve quality of palliative care in ICUs in future, they proposed analyzing the potential characteristics of units that provide high-quality care [34]. Even in Japan, where palliative care in ICUs is still developing, a multicenter survey needs to be conducted and benchmarked to assess the quality of care at the national level. Continuous research utilizing this QI set needs to be conducted to monitor the cycle of change and improvement in clinical practice, owing to the implementation of awareness and educational interventions.

Study limitations

This study had several limitations. First, since the measures in this QI set were based on medical record data, we may have underestimated the actual care provided in the absence of documentation. On the other hand, there are concerns about the risk of the possibility that the measures are not actually being provided even though they are being documented in the EMR and the risk of copy and paste duplication of documents [23]. However, we believe that documentation of pain assessment and discussion of goals of care are important in good-quality palliative care and may be quality indicators in and of itself. Second, since the study was conducted in two units within a single hospital, generalization is limited. Different hospital sizes and regions may have different demographic characteristics and disease coverage. In addition, the way in which medical records are written can vary depending on the hospital's situation and organizational culture. To deal with the issue of representativeness of the pilot test, the study was set up in a large-sized university hospital that serves as a teaching facility for intensive care physicians, and thus requires a certain level of medical care. Furthermore, by validating the study in two different types of units, we endeavored to accommodate variations. As a logical next step in the research, a multicenter study is being considered for further validation of our findings. Finally, although we developed a comprehensive QI set related to patient/family-centered and ICU staff-support domains, evidence associating most structures and processes of palliative care in the ICU with outcomes remains limited [51]. This would need to be modified in future to adjust to the changes in practice and accumulate new evidence.

Conclusions

A 28-item QI set was developed using the modified Delphi method and measured using EMR data, thereby providing a tool for assessment of the quality of palliative care in ICUs. Pilot testing using medical record data from two ICUs confirmed its feasibility and measurability. Our pilot study suggested the aspects of palliative care in ICUs in Japan that need to be improved, further highlighting the importance of conducting a nation-wide multicenter survey. To conduct large-scale surveys in future, we need to test the QI set further, with focus on decreasing the burden of measurement.

Availability of data and materials

The data sets used and analyzed during the current study are available from the corresponding author upon reasonable request.

Abbreviations

ICU:

Intensive care unit

QI:

Quality indicator

EMR:

Electronic medical record

ACP:

Advance care planning

SD:

Standard deviation

CI:

Confidence interval

NRS:

Numeric rating scale

BPS:

Behavioral pain scale

CPOT:

Critical-care pain observation tool

GCS:

Glasgow coma scale

CPR:

Cardiopulmonary resuscitation

DNAR:

Do not attempt resuscitation

IQR:

Interquartile range

References

  1. Kelley AS, Morrison RS. Palliative care for the seriously Ill. N Engl J Med. 2015;373:747–55. https://0-doi-org.brum.beds.ac.uk/10.1056/NEJMra1404684.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  2. Aslakson RA, Cox CE, Baggs JG, Curtis JR. Palliative and end-of-life care: prioritizing compassion within the ICU and beyond. Crit Care Med. 2021;49:1626–37. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000005208.

    Article  PubMed  Google Scholar 

  3. Lanken PN, Terry PB, Delisser HM, Fahy BF, Hansen-Flaschen J, Heffner JE, et al. An official American Thoracic Society clinical policy statement: palliative care for patients with respiratory diseases and critical illnesses. Am J Respir Crit Care Med. 2008;177:912–27. https://0-doi-org.brum.beds.ac.uk/10.1164/rccm.200605-587ST.

    Article  PubMed  Google Scholar 

  4. WHO. WHO definition of palliative care. 2012 Jan 28. https://www.who.int/cancer/palliative/definition/en/. Accessed 2020 Sep 8.

  5. Puntillo KA, Arai S, Cohen NH, Gropper MA, Neuhaus J, Paul SM, et al. Symptoms experienced by intensive care unit patients at high risk of dying. Crit Care Med. 2010;38:2155–60. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0b013e3181f267ee.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Teixeira C, Rosa RG, Sganzerla D, Sanchez EC, Robinson CC, Dietrich C, et al. The burden of mental illness among survivors of critical care-risk factors and impact on quality of life: a multicenter prospective cohort study. Chest. 2021;160:157–64. https://0-doi-org.brum.beds.ac.uk/10.1016/j.chest.2021.02.034.

    Article  PubMed  Google Scholar 

  7. Hatch R, Young D, Barber V, Griffiths J, Harrison DA, Watkinson P. Anxiety, depression and post traumatic stress disorder after critical illness: a UK-wide prospective cohort study. Crit Care. 2018;22:310. https://0-doi-org.brum.beds.ac.uk/10.1186/s13054-018-2223-6.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Timmins F, Naughton MT, Plakas S, Pesut B. Supporting patients’ and families’ religious and spiritual needs in ICU–can we do more? Nurs Crit Care. 2015;20:115–7. https://0-doi-org.brum.beds.ac.uk/10.1111/nicc.12177.

    Article  PubMed  Google Scholar 

  9. Johnson CC, Suchyta MR, Darowski ES, Collar EM, Kiehl AL, Van J, et al. Psychological sequelae in family caregivers of critically III Intensive Care Unit patients. A systematic review. Ann Am Thorac Soc. 2019;16:894–909. https://0-doi-org.brum.beds.ac.uk/10.1513/AnnalsATS.201808-540SR.

    Article  PubMed  Google Scholar 

  10. Kentish-Barnes N, Chaize M, Seegers V, Legriel S, Cariou A, Jaber S, et al. Complicated grief after death of a relative in the intensive care unit. Eur Respir J. 2015;45:1341–52. https://0-doi-org.brum.beds.ac.uk/10.1183/09031936.00160014.

    Article  PubMed  Google Scholar 

  11. Aslakson R, Cheng J, Vollenweider D, Galusca D, Smith TJ, Pronovost PJ. Evidence-based palliative care in the intensive care unit: a systematic review of interventions. J Palliat Med. 2014;17:219–35. https://0-doi-org.brum.beds.ac.uk/10.1089/jpm.2013.0409.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Ma J, Chi S, Buettner B, Pollard K, Muir M, Kolekar C, et al. Early palliative care consultation in the medical ICU: A cluster randomized crossover trial. Crit Care Med. 2019;47:1707–15. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000004016.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Johnson JR, Engelberg RA, Nielsen EL, Kross EK, Smith NL, Hanada JC, et al. The association of spiritual care providers’ activities with family members’ satisfaction with care after a death in the ICU*. Crit Care Med. 2014;42:1991–2000. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000000412.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Ribeiro AF, Martins Pereira S, Gomes B, Nunes R. Do patients, families, and healthcare teams benefit from the integration of palliative care in burn intensive care units? Results from a systematic review with narrative synthesis. Palliat Med. 2019;33:1241–54. https://0-doi-org.brum.beds.ac.uk/10.1177/0269216319862160.

    Article  PubMed  Google Scholar 

  15. Yefimova M, Aslakson RA, Yang L, Garcia A, Boothroyd D, Gale RC, et al. Palliative care and end-of-life outcomes following high-risk surgery. JAMA Surg. 2020;155:138–46. https://0-doi-org.brum.beds.ac.uk/10.1001/jamasurg.2019.5083.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Zante B, Camenisch SA, Schefold JC. Interventions in post-intensive care syndrome-family: a systematic literature review. Crit Care Med. 2020;48:e835–40. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000004450.

    Article  PubMed  Google Scholar 

  17. Farquhar M. AHRQ quality indicators. Agency for Healthcare Research and Quality; 2008

  18. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q. 1998;76:517–63. https://0-doi-org.brum.beds.ac.uk/10.1111/1468-0009.00105.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  19. Curtis JR, Cook DJ, Wall RJ, Angus DC, Bion J, Kacmarek R, et al. Intensive care unit quality improvement: a “how-to” guide for the interdisciplinary team. Crit Care Med. 2006;34:211–8. https://0-doi-org.brum.beds.ac.uk/10.1097/01.ccm.0000190617.76104.ac.

    Article  PubMed  Google Scholar 

  20. Nelson JE, Mulkerin CM, Adams LL, Pronovost PJ. Improving comfort and communication in the ICU: a practical new tool for palliative care performance measurement and feedback. Qual Saf Health Care. 2006;15:264–71. https://0-doi-org.brum.beds.ac.uk/10.1136/qshc.2005.017707.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  21. Penrod JD, Pronovost PJ, Livote EE, Puntillo KA, Walker AS, Wallenstein S, et al. Meeting standards of high-quality intensive care unit palliative care: clinical performance and predictors. Crit Care Med. 2012;40:1105–12. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0b013e3182374a50.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Mularski RA, Hansen L, Rosenkranz SJ, Leo MC, Nagy P, Asch SM. Medical record quality assessments of palliative care for Intensive Care Unit patients. Do they match the perspectives of nurses and families? Ann Am Thorac Soc. 2016;13:690–8. https://0-doi-org.brum.beds.ac.uk/10.1513/AnnalsATS.201508-501OC.

    Article  PubMed  Google Scholar 

  23. Curtis JR, Sathitratanacheewin S, Starks H, Lee RY, Kross EK, Downey L, et al. Using electronic health records for quality measurement and accountability in care of the seriously ill: opportunities and challenges. J Palliat Med. 2018;21(Suppl 2):S52-60. https://0-doi-org.brum.beds.ac.uk/10.1089/jpm.2017.0542.

    Article  PubMed  Google Scholar 

  24. Nelson JE, Puntillo KA, Pronovost PJ, Walker AS, McAdam JL, Ilaoa D, et al. In their own words: patients and families define high-quality palliative care in the intensive care unit. Crit Care Med. 2010;38:808–18. https://0-doi-org.brum.beds.ac.uk/10.1097/ccm.0b013e3181c5887c.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260:1743–8. https://0-doi-org.brum.beds.ac.uk/10.1001/jama.260.12.1743.

    Article  PubMed  CAS  Google Scholar 

  26. Tanaka Y, Masukawa K, Kawashima A, Hirayama H, Miyashita M. Quality indicators for palliative care in intensive care units: a systematic review. Ann Palliat Med. 2023;12:584–99. https://0-doi-org.brum.beds.ac.uk/10.21037/apm-22-1005.

    Article  PubMed  Google Scholar 

  27. Aslakson R, Spronk P. Tasking the tailor to cut the coat: how to optimize individualized ICU-based palliative care? Intensive Care Med. 2016;42:119–21. https://0-doi-org.brum.beds.ac.uk/10.1007/s00134-015-4107-4.

    Article  PubMed  Google Scholar 

  28. Keeney S, McKenna HA, Hasson F. The Delphi technique in nursing and Health Research. Hoboken: Wiley; 2011. p. 208.

    Book  Google Scholar 

  29. Fitch K, Bernstein SJ, Aguilar MD, Aguilar MS, Burnand B, Lazaro P, et al. The rand/UCLA appropriateness method User’s manual. Rand. 2001. 109 p

  30. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31:684–706. https://0-doi-org.brum.beds.ac.uk/10.1177/0269216317690685.

    Article  PubMed  Google Scholar 

  31. National Consensus Project for Quality Palliative Care; 2018. National Coalition for Hospice and Palliative Care Clinical Practice Guidelines for Quality Palliative Care. Clinical practice guidelines for quality palliative care. 4th ed. http://nationalcoalitionhpc.org/ncp. Richmond, VA: National Coalition for Hospice and Palliative Care

  32. Clarke EB, Curtis JR, Luce JM, Levy M, Danis M, Nelson J, et al. Quality indicators for end-of-life care in the intensive care unit. Crit Care Med. 2003;31:2255–62.

    Article  PubMed  Google Scholar 

  33. Mularski RA, Curtis JR, Billings JA, Burt R, Byock I, Fuhrman C, et al. Proposed quality measures for palliative care in the critically ill: a consensus from the Robert Wood Johnson Foundation Critical Care Workgroup. Crit Care Med. 2006;34(11 Suppl):S404–11.

    Article  PubMed  Google Scholar 

  34. Kruser JM, Aaby DA, Stevenson DG, Pun BT, Balas MC, Barnes-Daly MA, et al. Assessment of variability in end-of-life care delivery in intensive care units in the United States. JAMA Netw Open. 2019;2: e1917344. https://0-doi-org.brum.beds.ac.uk/10.1001/jamanetworkopen.2019.17344.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Aslakson RA, Kweku J, Kinnison M, Singh S, Crowe TY 2nd, AAHPM Writing Group. Operationalizing the measuring what matters spirituality quality metric in a population of hospitalized, critically ill patients and their family members. J Pain Symptom Manage. 2017;53:650–5. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jpainsymman.2016.12.323.

    Article  PubMed  Google Scholar 

  36. Dy SM, Kiley KB, Ast K, Lupu D, Norton SA, McMillan SC, et al. Measuring what matters: top-ranked quality indicators for hospice and palliative care from the American Academy of Hospice and Palliative Medicine and Hospice and Palliative Nurses Association. J Pain Symptom Manage. 2015;49:773–81. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jpainsymman.2015.01.012.

    Article  PubMed  Google Scholar 

  37. Dy SM, Lorenz KA, O’Neill SM, Asch SM, Walling AM, Tisnado D, et al. Cancer Quality-ASSIST supportive oncology quality indicator set: feasibility, reliability, and validity testing. Cancer. 2010;116:3267–75. https://0-doi-org.brum.beds.ac.uk/10.1002/cncr.25109.

    Article  PubMed  Google Scholar 

  38. Walling AM, Tisnado D, Asch SM, Malin JM, Pantoja P, Dy SM, et al. The quality of supportive cancer care in the veterans affairs health system and targets for improvement. JAMA Intern Med. 2013;173(22):2071–9.

    Article  PubMed  Google Scholar 

  39. Chan A, Chien I, Moseley E, Salman S, Kaminer Bourland S, Lamas D, et al. Deep learning algorithms to identify documentation of serious illness conversations during intensive care unit admissions. Palliat Med. 2019;33:187–96. https://0-doi-org.brum.beds.ac.uk/10.1177/0269216318810421.

    Article  PubMed  Google Scholar 

  40. Lindvall C, Deng CY, Moseley E, Agaronnik N, El-Jawahri A, Paasche-Orlow MK, et al. Natural language processing to identify advance care planning documentation in a multisite pragmatic clinical trial. J Pain Symptom Manage. 2022;63:e29-36. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jpainsymman.2021.06.025.

    Article  PubMed  Google Scholar 

  41. Masukawa K, Aoyama M, Yokota S, Nakamura J, Ishida R, Nakayama M, et al. Machine learning models to detect social distress, spiritual pain, and severe physical psychological symptoms in terminally ill patients with cancer from unstructured text data in electronic medical records. Palliat Med. 2022;2692163221105595(36):1207–16. https://0-doi-org.brum.beds.ac.uk/10.1177/02692163221105595.

    Article  Google Scholar 

  42. Devlin JW, Skrobik Y, Gélinas C, Needham DM, Slooter AJC, Pandharipande PP, et al. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU. Crit Care Med. 2018;46:e825–73. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000003299.

    Article  PubMed  Google Scholar 

  43. Durán-Crane A, Laserna A, López-Olivo MA, Cuenca JA, Díaz DP, Cardenas YR, et al. Clinical practice guidelines and consensus statements about pain management in critically ill end-of-life patients: a systematic review. Crit Care Med. 2019;47:1619–26. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000003975.

    Article  PubMed  Google Scholar 

  44. Igarashi Y, Tanaka Y, Ito K, Miyashita M, Kinoshita S, Kato A, et al. Current status of palliative care delivery and self-reported practice in ICUs in Japan: a nationwide cross-sectional survey of physician directors. J Intensive Care. 2022;10:18. https://0-doi-org.brum.beds.ac.uk/10.1186/s40560-022-00605-8.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Curtis JR, White DB. Practical guidance for evidence-based ICU family conferences. Chest. 2008;134:835–43. https://0-doi-org.brum.beds.ac.uk/10.1378/chest.08-0235.

    Article  PubMed  Google Scholar 

  46. Kerckhoffs MC, Senekal J, van Dijk D, Artigas A, Butler J, Michalsen A, et al. Framework to support the process of decision-making on life-sustaining treatments in the ICU: results of a Delphi study. Crit Care Med. 2020;48:645–53. https://0-doi-org.brum.beds.ac.uk/10.1097/CCM.0000000000004221.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Moss M, Good VS, Gozal D, Kleinpell R, Sessler CN. An official critical care societies collaborative statement-burnout syndrome in critical care health-care professionals: a call for action. Chest. 2016;150:17–26. https://0-doi-org.brum.beds.ac.uk/10.1016/j.chest.2016.02.649.

    Article  PubMed  Google Scholar 

  48. Kroenke K, Spitzer RL, Williams JBW. The Patient Health Questionnaire-2: validity of a two-item depression screener. Med Care. 2003;41:1284–92.

    Article  PubMed  Google Scholar 

  49. Wall RJ, Engelberg RA, Downey L, Heyland DK, Curtis JR. Refinement, scoring, and validation of the Family Satisfaction in the Intensive Care Unit (FS-ICU) survey. Crit Care Med. 2007;35:271–9.

    Article  PubMed  Google Scholar 

  50. Huber MT, Ling DY, Rozen AS, Terauchi SY, Sharma P, Fleischer-Black J, et al. Top ten tips palliative care clinicians should know about leveraging the electronic health record for data collection and quality improvement. J Palliat Med. 2023;26:849–55. https://0-doi-org.brum.beds.ac.uk/10.1089/jpm.2022.0536.

    Article  PubMed  Google Scholar 

  51. Ito K, George N, Wilson J, Bowman J, Aaronson E, Ouchi K. Primary palliative care recommendations for critical care clinicians. J Intensive Care. 2022;10:20. https://0-doi-org.brum.beds.ac.uk/10.1186/s40560-022-00612-9.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to thank all 16 panelists who took part in the Delphi study: Yasuhiro Norisue, Yuko Igarashi, Keita Tagami, Takashi Ohmori, Teppei Torisaki, Takeshi Unoki, Satomi Kinoshita, Asami Tado, Yusuke Ohyama, Eriko Hayashi, Chikako Hashiguchi, Noriko Matsui, Naomi Akiyama, Yuichiro Machida, Noriatsu Tatematsu, Toshiyasu Tsujii. This work was supported by JST SPRING, Grant Number JPMJSP2114. We would like to thank Editage (www.editage.com) for English language editing.

Funding

This work was supported by JST SPRING, Grant Number JPMJSP2114.

Author information

Authors and Affiliations

  1. Department of Palliative Nursing, Health Sciences, Tohoku University Graduate School of Medicine, 2-1 Seiryo-Machi, Aoba-ku, Sendai, Miyagi, 980-8575, Japan

    Yuta Tanaka, Kento Masukawa & Mitsunori Miyashita

  2. Department of Critical Care and Disaster Nursing, Japanese Red Cross Kyushu International College of Nursing, Munakata, Fukuoka, Japan

    Hideaki Sakuramoto

  3. Department of Adult and Geriatric Nursing, School of Health Science, Shinshu University, Matsumoto, Nagano, Japan

    Akane Kato

  4. Department of Transitional and Palliative Care, Aso Iizuka Hospital, Fukuoka, Japan

    Yuichiro Ishigami

  5. Nursing Department, Kokura Memorial Hospital, Fukuoka, Japan

    Junko Tatsuno

  6. Department of Surgery, Division of Acute Care Surgery, Teikyo University School of Medicine, Tokyo, Japan

    Kaori Ito

  7. Department of Palliative and Supportive Care, Institute of Medicine, University of Tsukuba, Tsukuba, Japan

    Yoshiyuki Kizawa

Authors
  1. Yuta Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kento Masukawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hideaki Sakuramoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Akane Kato
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuichiro Ishigami
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Junko Tatsuno
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kaori Ito
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yoshiyuki Kizawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mitsunori Miyashita
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YT formed the study team and designed and conducted all stages of the research, designed and administered the questionnaires, collated and analyzed the data, and drafted the manuscript. MM supervised the work. YT drafted the manuscript in consultation with KM and MM. All authors discussed the results and commented on the manuscripts. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yuta Tanaka.

Ethics declarations

Ethics approval and consent to participate

This study was conducted in compliance with the Declaration of Helsinki and the Ethical Guidelines for Medical Research Involving Human Subjects of the Japanese Government. Ethical approval for this study was granted by the Institutional Review Boards of Tohoku University (No. 2022-1-1023).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. 

The modified Delphi method.

Additional file 2. 

QI measurement manual.

Additional file 3. 

Table. Inter-rater reliability verification.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tanaka, Y., Masukawa, K., Sakuramoto, H. et al. Development of quality indicators for palliative care in intensive care units and pilot testing them via electronic medical record review. j intensive care 12, 1 (2024). https://0-doi-org.brum.beds.ac.uk/10.1186/s40560-023-00713-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s40560-023-00713-z

Keywords

Journal of Intensive Care

ISSN: 2052-0492