Advertisement

A Comparison of Self-evaluated Survey and Work Sampling Approach for Estimating Patient-care Unit Cost Multiplier in Genetic Nursing Activities

  • Khairu Hazwan Mustaffa
    Affiliations
    Discipline of Social and Administrative Pharmacy, School of Pharmaceutical Science, Universiti Sains Malaysia, Minden, Penang, Malaysia

    Department of Pharmacy, Hospital Sultanah Nur Zahirah, Kuala Terengganu, Terengganu, Malaysia
    Search for articles by this author
  • Asrul Akmal Shafie
    Correspondence
    Correspondence to: Discipline of Social and Administrative Pharmacy, School of Pharmaceutical Science, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia.
    Affiliations
    Discipline of Social and Administrative Pharmacy, School of Pharmaceutical Science, Universiti Sains Malaysia, Minden, Penang, Malaysia
    Search for articles by this author
  • Lock-Hock Ngu
    Affiliations
    Department of Genetics, Hospital Kuala Lumpur, Kuala Lumpur, Malaysia
    Search for articles by this author
Open AccessPublished:June 14, 2022DOI:https://doi.org/10.1016/j.anr.2022.06.001

      Summary

      Purpose

      To compare patient care multipliers estimated from subjective evaluation against work sampling (WS) techniques in genetic nursing activities.

      Methods

      An observational WS technique was conducted from November to December 2019 with nine genetic nurses in a tertiary referral center in Malaysia. The WS activity instrument was devised, validated, and pilot tested. All care- and non-care-related activities were sampled at 10-minute intervals within 8 hours of working over 14 days, followed by a subjective evaluation of activities survey over the same period. Bonferroni correction was undertaken for multiple testing with a p value of 0.0025.

      Results

      The two techniques produced significant differences in genetic nurses’ activities categorization. The WS showed that compared with subjective evaluation, direct care (19.3% vs. 45.0%; p < .001) was estimated to be significantly lower, and indirect care (40.4% vs. 25.6%; p < .001) and unit-related care (28.5% vs. 16.9%; p < .001) were higher. Both techniques produced a similar proportion of time spent in other non-7care activities (12.0%) but differed in genetic meetings and information-gathering activities. While the multipliers for patient face-to-face contact were significantly larger between WS (4.57) and the survey (1.94), the multipliers for patient care time were smaller between WS (1.47) and the survey (1.24), indicating that caution should be taken when multiplying for patient contact time compared to patient care activity to determine the cost of care provision.

      Conclusion

      A considerable proportion of time spent away from the patient needs to be allocated to patient-related care time. Thus, estimating the paid cost solely based on direct time with patients considerably underestimates the cost per hour of nurses' care. It is recommended to employ ‘patient-related activity’ instead of the ‘face-to-face contact’ multiplier because the former did not significantly differ from the one estimated using WS.

      Keywords

      Introduction

      Rare disease (RD) is a debilitating hereditary disease that affects only a small fraction of the population and is extremely difficult to treat. There are between 7000 and 8000 RDs worldwide. Approximately 80% of them are genetic disorders affecting young children, and the remainders are rare cancers, autoimmune diseases, congenital malformations, and the rare manifestation of common diseases. Unfortunately, due to a lack of attention from health care providers (HCPs), health system leaders, and health policy-makers, these patients are at a risk of missing life-saving treatment. The issue stems from gaps in disease knowledge among HCPs, diagnostic difficulties, and high treatment costs [
      • Cai X.
      • Yang H.
      • Genchev G.Z.
      • Lu H.
      • Yu G.
      Analysis of economic burden and its associated factors of twenty-three rare diseases in Shanghai.
      ]. In Malaysia, patients with RD are managed by a tertiary referral genetic unit. Geneticists, paediatricians, genetic counsellors, and nurses are the front-line providers of medical genetics services in the country [
      • Shafie A.A.
      • Supian A.
      • Ahmad Hassali M.A.
      • Ngu L.-H.
      • Thong M.-K.
      • Ayob H.
      • et al.
      Rare disease in Malaysia: challenges and solutions.
      ].
      Nurses remain the largest group of personnel resources employed by the Ministry of Health in the labor force [
      ]. In today's economic climate, HCPs, including nurses, must constantly perform at the highest level by re-evaluating their quality of care. As a result, there is considerable interest in measuring nurses' activities and/or workload via a professional judgement or subjective evaluation [
      • Twigg D.
      • Duffield C.
      A review of workload measures: a context for a new staffing methodology in Western Australia.
      ], clinical work indicators (or productivity data) [
      • Baernholdt M.
      • Cox K.
      • Scully K.
      Using clinical data to capture nurse workload: implications for staffing and safety.
      ], and time and motion study (TMS) [
      • Griffiths P.
      • Saville C.
      • Ball J.
      • Jones J.
      • Pattison N.
      • Monks T.
      Nursing workload, nurse staffing methodologies and tools: a systematic scoping review and discussion.
      ,
      • Lim M.L.
      • Ang S.Y.
      A time-motion observation study to measure and analyze clinical nursing workload in an acute care hospital in Singapore.
      ], for instance, in intensive care units (ICUs) [
      • Abbey M.
      • Chaboyer W.
      • Mitchell M.
      Understanding the work of intensive care nurses: a time and motion study.
      ,
      • Kakushi L.E.
      • Évora Y.D.M.
      Direct and indirect nursing care time in an intensive care unit.
      ,
      • Ahmadishad M.
      • Adib-Hajbaghery M.
      • Rezaei M.
      • Atoof F.
      • Munyisia E.
      Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
      ], emergency departments (Eds) [
      • Gholizadeh M.
      • Janati A.
      • Nadimi B.
      • Kabiri N.
      • Abri S.
      How do nurses spend their time in the hospital?.
      ], medical wards and surgical wards [
      • Desjardins F.
      • Cardinal L.
      • Belzile E.
      • McCusker J.
      Reorganizing nursing work on surgical units: a time-and-motion study.
      ], ambulatory care [
      • Hollingworth W.
      • Devine E.B.
      • Hansen R.N.
      • Lawless N.M.
      • Comstock B.A.
      • Wilson-Norton J.L.
      • et al.
      The impact of e-prescribing on prescriber and staff time in ambulatory care clinics: a time–motion study.
      ], general wards [
      • Westbrook J.I.
      • Li L.
      • Georgiou A.
      • Paoloni R.
      • Cullen J.
      Impact of an electronic medication management system on hospital doctors' and nurses' work: a controlled pre–post, time and motion study.
      ], and community nurses [
      • Gardner G.
      • Gardner A.
      • Middleton S.
      • Gibb M.
      • Della P.
      • Duffield C.
      Development and validation of a novel approach to work sampling: a study of nurse practitioner work patterns.
      ]. In recent years, outpatient genetic care has played a significant role in medical genetic management [
      • Cai X.
      • Yang H.
      • Genchev G.Z.
      • Lu H.
      • Yu G.
      Analysis of economic burden and its associated factors of twenty-three rare diseases in Shanghai.
      ]. However, there have been no works published thus far to the best of the author's knowledge about the work activity of genetic nurses in comparison to geneticists [
      • McPherson E.
      • Zaleski C.
      • Benishek K.
      • McCarty C.A.
      • Giampietro P.F.
      • Reynolds K.
      • et al.
      Clinical genetics provider real-time workflow study.
      ] and genetic counsellors [
      • Heald B.
      • Rybicki L.
      • Clements D.
      • Marquard J.
      • Mester J.
      • Noss R.
      • et al.
      Assessment of clinical workload for general and specialty genetic counsellors at an academic medical center: a tool for evaluating genetic counselling practices.
      ,
      • Attard C.A.
      • Carmany E.P.
      • Trepanier A.M.
      Genetic counselor workflow study: the times are they a-changin'?.
      ]. Thus, there is an obvious need to comprehend and disseminate information regarding their progressive role as the patient's primary point of contact in this largely unexplored area of RD.
      Patient care is a time-consuming [
      • Attard C.A.
      • Carmany E.P.
      • Trepanier A.M.
      Genetic counselor workflow study: the times are they a-changin'?.
      ,
      • Iosa M.
      • Grasso M.G.
      • Dandi R.
      • Carusi D.
      • Bacci A.
      • Marra R.
      • et al.
      Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
      ] and complex activity [
      • Ahmadishad M.
      • Adib-Hajbaghery M.
      • Rezaei M.
      • Atoof F.
      • Munyisia E.
      Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
      ,
      • McPherson E.
      • Zaleski C.
      • Benishek K.
      • McCarty C.A.
      • Giampietro P.F.
      • Reynolds K.
      • et al.
      Clinical genetics provider real-time workflow study.
      ] because of the lengthy process and skill mix [
      • Duffield C.
      • Forbes J.
      • Fallon A.
      • Roche M.
      • Wise W.
      • Merrick E.
      Nursing skill mix and nursing time: the roles of registered nurses and clinical nurse specialists.
      ] that nurses engage in while performing clinical activities, patient-related tasks, administrative work, communication, and personal tasks. Generally, these multitasking activities are categorized as either direct or indirect patient care activities, depending on whether they are performed in front of or away from the patient [
      • Abbey M.
      • Chaboyer W.
      • Mitchell M.
      Understanding the work of intensive care nurses: a time and motion study.
      ,
      • Urden L.D.
      • Roode J.L.
      Work sampling: a decision-making tool for determining resources and work redesign.
      ,
      • Kilpatrick K.
      Development and validation of a time and motion tool to measure cardiology acute care nurse practitioner activities.
      ]. There are significant disparities in the proportion of time spent by nurses on direct and indirect care activities that have been reported in other areas of practice [
      • Gholizadeh M.
      • Janati A.
      • Nadimi B.
      • Kabiri N.
      • Abri S.
      How do nurses spend their time in the hospital?.
      ]. In addition, previous studies over the last decade have demonstrated that the workload and the interplay between direct and indirect patient-related activity significantly impact the quality-of-care delivery [
      • Ahmadishad M.
      • Adib-Hajbaghery M.
      • Rezaei M.
      • Atoof F.
      • Munyisia E.
      Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
      ,
      • McPherson E.
      • Zaleski C.
      • Benishek K.
      • McCarty C.A.
      • Giampietro P.F.
      • Reynolds K.
      • et al.
      Clinical genetics provider real-time workflow study.
      ,
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ]. Thus, estimating direct contact time alone will underestimate the time required to provide care.
      Cost and cost-effectiveness (CE) intervention is designed to achieve the maximum possible health care delivery through the optimum use of limited resources. However, despite the wealth of literature available in this field [
      • Mogyorosy Z.
      • Smith P.C.
      The main methodological issues in costing health care services - a literature review.
      ,
      • Jacobs J.C.
      • Barnett P.G.
      Emergent challenges in determining costs for economic evaluations.
      ], there is a dearth of discussion of how ‘hidden’ or indirect patient-related activities should be handled and how they ultimately affect costs. The cost of care accounts for a sizeable amount of HCPs' time. Previously, the problem was addressed with the hidden-to-observed cost ratio [
      • Knapp M.
      • Baines B.
      Hidden cost multipliers for residential child care.
      ] or ‘overhead’ associated with patient care delivery [
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ]. While a recent study [
      • Yen P.-Y.
      • Kellye M.
      • Lopetegui M.
      • Saha A.
      • Loversidge J.
      • Chipps E.M.
      • et al.
      Nurses' time allocation and multitasking of nursing activities: a time motion study.
      ] focuses more on cost-effective care, less attention is placed on how to account for this overhead that was needed for the provision of care-related activity [
      • Ahmadishad M.
      • Adib-Hajbaghery M.
      • Rezaei M.
      • Atoof F.
      • Munyisia E.
      Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
      ]. In addition, the CE guidelines indicate that all costs that are likely to be impacted by intervention should be included in the analysis [
      • Drummond M.F.
      • Sculpher M.J.
      • Claxton K.
      • Stoddart G.L.
      • Torrance G.W.
      Methods for the economic evaluation of health care programme.
      ]. Thus, this study extends and generalizes the concept of an adjustment factor or multiplier (e.g., the ratio of direct to indirect time spent on care activity) as a metric to allow for the higher costs associated with providing patient-related care activity than the direct care cost [
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ,
      • Knapp M.
      • Baines B.
      Hidden cost multipliers for residential child care.
      ].
      The multiplier is typically approximated via time diaries, sheet surveys, focus groups, and verbal reports [
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ,
      • Oddone E.
      • Guarisco S.
      • Simel D.
      Comparison of housestaff's estimates of their workday activities with results of a random work-sampling study.
      ,
      • Sinsky C.
      • Colligan L.
      • Li L.
      • Prgomet M.
      • Reynolds S.
      • Goeders L.
      • et al.
      Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.
      ], or by explicitly requesting the percentage (or minutes) of time spent in the actual practice from the HCP [
      • Curtis L.A.
      How do professionals spend their time?.
      ]. The given estimate is then converted to a ratio of direct to indirect time spent with patients or performing patient-related activities. When calculating the total time required to deliver patient care, the ratio's components are added together and multiplied by the total direct contact time and unit cost component. This time allocation more accurately reflects actual care activities, resulting in a more equitable valuation of the resources required to hire an HCP. These approaches have become the incumbent method for obtaining information on the proportion of direct and indirect time spent on different locations (e.g., patient, clinic) or activities due to practical and cost reasons. This is the approach employed by the Personal Social Services Research Unit (PSSRU) [
      • Curtis L.A.
      • Burns A.
      Unit cost of health and social care 2017.
      ] to reflect current care practices in their health and social care services unit cost estimates.
      In contrast, other work pattern estimation methods exist, such as the gold-standard TMS [
      • Westbrook J.I.
      • Li L.
      • Georgiou A.
      • Paoloni R.
      • Cullen J.
      Impact of an electronic medication management system on hospital doctors' and nurses' work: a controlled pre–post, time and motion study.
      ], activity log [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ], video recording [
      • Gardner G.
      • Gardner A.
      • Middleton S.
      • Gibb M.
      • Della P.
      • Duffield C.
      Development and validation of a novel approach to work sampling: a study of nurse practitioner work patterns.
      ], and work sampling (WS) [
      • Finkler S.A.
      • Knickman J.R.
      • Hendrickson G.
      • Lipkin Jr., M.
      • Thompson W.G.
      A comparison of work-sampling and time-and-motion techniques for studies in health services research.
      ], which have been used in health care service research. For instance, WS is widely used as an experimental and practical technique for measuring work in nursing and non-nursing research [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ,
      • Rutter P.M.
      • Brown D.
      • Jones I.F.
      Pharmacy research: the place of work measurement.
      ]. However, it was not used to estimate the multiplier because it required more time and effort than the previous methods. Compared to TMS, WS measures activity frequencies that can easily be proportioned to determine the percentage of total time spent and the ratio of direct to indirect time [
      • Sittig D.F.
      Work-sampling: a statistical approach to evaluation of the effect of computers on work patterns in the healthcare.
      ]. Moreover, WS is less prone to personal and recall bias than the typical methods, e.g., focus groups, surveys, and verbal reports, which rely heavily on subjective judgement. The WS method not only provides an estimate of the amount of time spent in each activity but also accounts for actual work patterns in circumscribed working areas [
      • Gardner G.
      • Gardner A.
      • Middleton S.
      • Gibb M.
      • Della P.
      • Duffield C.
      Development and validation of a novel approach to work sampling: a study of nurse practitioner work patterns.
      ,
      • Duffield C.
      • Forbes J.
      • Fallon A.
      • Roche M.
      • Wise W.
      • Merrick E.
      Nursing skill mix and nursing time: the roles of registered nurses and clinical nurse specialists.
      ]. Nonetheless, the direct observational technique may introduce other instances of bias, known as the Hawthorne effect [
      • Finkler S.A.
      • Knickman J.R.
      • Hendrickson G.
      • Lipkin Jr., M.
      • Thompson W.G.
      A comparison of work-sampling and time-and-motion techniques for studies in health services research.
      ,
      • Sittig D.F.
      Work-sampling: a statistical approach to evaluation of the effect of computers on work patterns in the healthcare.
      ]. To our knowledge, no prior studies have explicitly compared the generation of multipliers for nurses’ paid care hours unit cost between WS and the survey.
      Therefore, it is imperative to investigate nursing care time allocation, as the provision of medical genetics management has shifted dramatically over the past two decades. Clearly, additional evidence is required for the conference, continuous medical education, and general administrative tasks that we assumed to be overheads on patient activity to accurately reflect the actual cost of care provision [
      • Curtis L.A.
      • Netten A.
      The costs of training a nurse practitioner in primary care: the importance of allowing for the cost of education and training when making decisions about changing the professional-mix.
      ].
      The purpose of this study was to compare the proportions of time spent on each activity in the total activity using two methods: WS and the survey. The following goals are of interest: (1) estimate the percentage of time required for patient-related care activities; (2) calculate the multipliers for future EE's human resource cost estimations in RDs.

      Methods

      Study design

      The study was conducted in two stages. First, an observational WS technique was used to describe the work activities of genetic nurses for two weeks within an eight-hour working period, immediately followed by a subjective evaluation of activities over two weeks. Nurses were unaware that the subjective evaluation technique would subsequently be employed.

      Setting and participants

      The observations were taken at a tertiary referral genetic centre at a government hospital in Malaysia between November and December 2019. The centre is overseen by a head nurse and 11 registered nurses who work during office hours (from 8.00 pm to 5.00 pm). The centre provides care to approximately 735 patients with RD and receives approximately 300 new referrals monthly. A total of nine registered nurses were recruited. The nurses were purposively sampled with job scopes focusing on patient care rather than managerial roles.

      Sample size and power

      The study was designed to capture the least often occurring activity for indirect care activity (ICA) in obtaining a representative number of observations needed to allow for acceptable statistical inferences to be made in line with earlier studies [
      • Iosa M.
      • Grasso M.G.
      • Dandi R.
      • Carusi D.
      • Bacci A.
      • Marra R.
      • et al.
      Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
      ,
      • Finkler S.A.
      • Knickman J.R.
      • Hendrickson G.
      • Lipkin Jr., M.
      • Thompson W.G.
      A comparison of work-sampling and time-and-motion techniques for studies in health services research.
      ]. The following formulas are used to determine the minimum: (1) sample size, e.g., N = z2P(1 − P)/d2 [
      • Lwanga S.K.
      • Lemeshow S.
      Sample size determination in health studies: a practical manual.
      ]; (2) sampling period, e.g., T = τ/480.p.ηʄ [
      • Bordin L.C.
      • Fugulin F.M.
      Distribuição do tempo das enfermeiras: identificação e análise em Unidade Médico-Cirúrgica = Nurses' time distribution: identification and analysis in a medical-surgical unit.
      ]. First, the preliminary sample estimates are obtained from the trial runs. Then, the minimum number of observations for each activity category is determined (Table 1). For instance, ICA was expected to occur 43% of the time, and the minimum number of observations needed was 600, so we have 95% confidence that the activity is within the ±5% level of accuracy (i.e., the minimum acceptable error, d, was set to 0.04 as presented in Table 1). To determine the sample period, we have τ = time interval between observations (10 minutes), p = minimum probability of occurrence for an activity (0.001), ηʄ = number of nurses per shift, and 480 = 8 h × 60 minutes; hence, the minimum sample period corresponded to 11 days. With observations every 10 minutes, the maximum working time is 8 hours per day, and the number of nurses who can be observed simultaneously is 2; the daily data points required are 28 (e.g., 600 observations ÷ 11 days ÷ 2 nurses).
      Table 1Activity categories’ frequency and sampling required.
      % Category (no of observations)
      DCAICAURAOA
      Preliminary trial 114.5 (17)47.0 (55)28.2 (33)10.3 (12)
      Preliminary trial 217.5 (32)38.8 (71)21.3 (39)22.4 (41)
      Estimated proportion16.0 (25)42.9 (63)24.8 (36)16.3 (27)
      Minimum observations needed, n350600400350
      Limit of error0.040.040.040.04
      Note. n = number of data points; DCA = direct care activity; ICA = indirect care activity; URA = unit-related activity; OA = ‘others’ activity.

      Instruments and validation

      A literature review identifies the list of potential nurses' activities attributable to care delivery. Then, interviews with practising nurses and leaders were conducted to elicit the normal work activities. These lists were contrasted and grouped. Then, a single list of activities was discussed for inclusion or exclusion from the final WS instrument. We adopted a categorization of work activity from the established sources [
      • Iosa M.
      • Grasso M.G.
      • Dandi R.
      • Carusi D.
      • Bacci A.
      • Marra R.
      • et al.
      Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
      ,
      • Urden L.D.
      • Roode J.L.
      Work sampling: a decision-making tool for determining resources and work redesign.
      ,
      • Kilpatrick K.
      Development and validation of a time and motion tool to measure cardiology acute care nurse practitioner activities.
      ,
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ,
      • Bordin L.C.
      • Fugulin F.M.
      Distribuição do tempo das enfermeiras: identificação e análise em Unidade Médico-Cirúrgica = Nurses' time distribution: identification and analysis in a medical-surgical unit.
      ]: (1) direct care activity, DCA (e.g., care activity requires patient's interaction or is performed directly with patient and family); (2) indirect care activity, ICA (e.g., care activity that is performed from a distance of patient and family but benefitting them); (3) unit-related activity, URA (non-care activity pertaining to the normal maintenance of the unit and its organization); (4) personal activity, PA (including rest periods and personal requirements unrelated to the professional task). Alternatively, we classified PA, idle time, and breaks as ‘other’ activities (OA).
      Additionally, the activity descriptions are developed and refined during the testing period by the author. Next, the author and nurse manager validated the instrument's content by standardizing the terminology and comparing the listed activities to local clinical practices [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ]. This process informed the refinement of techniques and methods for the trial runs. The runs help the author become familiar with the criteria and activity descriptions that will be used to identify the dominant activity occurring concurrently during the observations [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ]. The emphasis was to document what had taken place and not to define which activity had the greater priority. Following this procedure, the final data collection form organized 26 nurse activities (Figure 1) that were mutually exclusive, defined, and precoded into four categories of work: DCA, ICA, URA, and OA. The list of work activity descriptions with inclusion and exclusion criteria is available from the corresponding author upon request.
      Figure 1
      Figure 1Genetic nurse activities and their categorization identified from the study. Note. CME = continuous medical education. The normal activities of genetic nurses obtained from the work sampling. The insert shows the mean proportions for the category. The error bar represents the standard error.

      Procedure

      A pilot WS study was initially undertaken with seven nurses for three working days at random. The trial runs and study observations were made from 8.00 am to 4.30 pm and from 8.00 am to 5.00 pm, respectively, at 10-min fixed intervals by the first author. The interval set time was considered less demanding than the randomly distributed time sampling and a standard method in health services research [
      • Finkler S.A.
      • Knickman J.R.
      • Hendrickson G.
      • Lipkin Jr., M.
      • Thompson W.G.
      A comparison of work-sampling and time-and-motion techniques for studies in health services research.
      ]. In addition, it allows the sample to be randomized adequately since the nature of the health care activity follows some to no patterns [
      • Abdellah F.G.
      • Levine E.
      Work-sampling applied to the study of nursing personnel.
      ]. The pilot study provided an opportunity to test the data tool and gain insight into the following observation techniques. The observer moved to an exact spot from eight observation locations at random and positioned within earshot of the nurses prior to observing the activity while remaining as unobtrusive as possible. Then, two observations were made at each time interval instantaneously. The observed activities were logged on a tabular data collection form using a unique numbering assigned to each task adopted from a previous study [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ]. The observer documented all unforeseeable events that occurred during the period.
      Following that, the same author provided the nurses with similar activity descriptions and forms to evaluate and reflect on over the same period. Then, they were required to give estimates on the amount of time they spent on the specific activities and rate their confidence for those estimates via a survey form.

      Data analysis

      First, data from WS were aggregated by individual activity and grouped into categories. Then, the descriptive statistics were used to compute the average proportion of time for each activity and category. All values are reported as the mean and standard error of the mean. Second, the summarized data are contrasted with the estimates we obtained from the survey. The survey data are presented as the mean and standard deviation (SD) of the mean. The Bonferroni correction was undertaken for multiple testing and adopted a p value of 0.0028 for the significance test [
      • Ampt A.
      • Westbrook J.
      • Creswick N.
      • Mallock N.
      A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks.
      ]. Finally, the multiplier for the respective unit cost is calculated following the published method [
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ,
      • Curtis L.A.
      • Burns A.
      Unit cost of health and social care 2017.
      ]. All data were entered into Microsoft Excel 2013 (Microsoft Corporation, Redmond, Wa) and then transferred electronically to R version 4.0.2 (R Foundation for Statistical Computing, Vienna, Austria) for data wrangling and statistical analyses.

      Ethical considerations

      Ethical approval was granted by the Medical and Research Ethics Committee of the Ministry of Health , Malaysia ( NMRR-19-610-46077 ). This study poses no risk to patients and nurses, and the observations did not disrupt the patient's visit or nurse's routine care. The participants volunteered to participate in the study.

      Results

      Demographic characteristics

      The participants’ age ranged from 28 to 35 years, with a median age of approximately 30 years [IQR = 3.51]. The majority were female (77.8%, n = 7), and most had obtained a 3-year diploma in nursing and were staff nurses (75.5%, n = 6). In addition, the vast majority (75.5%, n = 6) had between 5 and 10 years of clinical experience. The nurses qualified as nurses as early as 2001 (the latest in 2016).

      Work sampling

      A total of 2,333 observations were recorded (including 300 trial run observations). The minimum daily observations required were not achieved at three periods for category OA due to one nurse being on emergency leave and meeting the week after. No significant changes to the WS instrument were made except for one activity, i.e., ‘hygiene/cleaning’ was excluded from the finalized instrument because it was not observed during trials. There are 26 types of activity descriptions that have been identified; of these, the 7, 9, 6, and 4 activities are categorized into DCA, ICA, URA, and OA, respectively (see Table 2 for list of activities).
      Table 2Genetic nurses’ time distribution during clinical operation hours.
      Category, by activityObs, n‘0’ obs, n%
      Percentage of weighted proportion.
      (SE)
      Standard error of percentage
      Max %
      (Total %)TotalBy ActivityBy Category
      Direct care453(14.3)19.3 (3.8)
       Assisting Dr14115.9 (1.3)31.1 (0.5)14.3
       Pt. interaction12005.1 (0.8)26.5 (0.3)10.8
       Pt. assessment5812.6 (0.5)12.8 (0.2)7.4
       Clinical procedures5412.3 (0.3)12.0 (0.2)4.3
       Registration3821.6 (0.3)8.4 (0.1)4.0
       Meds. admin2531.2 (0.3)5.5 (0.1)3.2
       Pt. mobility1760.8 (0.2)3.8 (0.1)3.2
      Indirect care944(15.1)40.4 (8.3)
       Pt. progress18108.1 (1.1)19.2 (0.2)15.1
       Communication17007.3 (0.9)18.0 (0.2)11.6
       Genetic meetings15785.8 (2.6)16.6 (0.6)31.3
       Data entry12305.6 (0.9)13.0 (0.2)13.5
       Meds. order8833.8 (0.8)9.3 (0.1)8.4
       Treat. preparation8613.4 (0.7)9.1 (0.2)7.3
       Phone follow-up7023.1 (0.4)7.4 (0.1)6.0
       Care coordination4922.4 (0.6)5.2 (0.1)6.4
       Gather information2030.9 (0.2)2.1 (0.0)2.6
      Unit-related656(22.6)28.5 (5.9)
       Clerical237010.2 (0.9)36.1 (0.3)16.8
       Meeting9865.0 (1.6)14.9 (0.4)19.1
       Transit10604.6 (0.5)16.1 (0.1)7.7
       Education105103.8 (2.1)16.0 (0.7)26.2
       Errands units6703.0 (0.4)10.2 (0.1)7.1
       Stocking4332.0 (0.4)6.6 (0.1)4.4
      Others280(10.7)11.8 (2.8)
       Idle time15906.7 (1.6)56.8 (1.1)18.5
       Personal5122.2 (0.5)18.2 (0.3)5.7
       Pantry4031.6 (0.5)14.3 (0.3)5.4
       Restroom3011.3 (0.2)10.7 (0.1)2.6
      Note. ‘0’ = zero; Admin. = administration; Dr = doctor; Max = maximum; Meds. = medication; n = number of data points; Obs. = observations; Pt. = patient; Treat. = treatment.
      a Percentage of weighted proportion.
      b Standard error of percentage
      Nurses in our study spent 19.3% of their clinical operational time on direct patient contact performing DCA. Most of the activities were spent away from the patient (80.7%); 40.4% were on ICA, 28.5% on URA, and 11.8% on OA. Together, DCA (19.3%) and ICA (40.4%) equate to 59.7% of patient-related care activities. On the other hand, time spent on non-patient care activities accounted for 40.3% of the activities, with 28.5% and 11.8% on URA and OA, respectively. The complete time distribution of the activities is presented in Table 2 below.
      Overall category analysis indicates that nurses spend the most time on ICA (40.4%), followed by URA (28.5%), DCA (19.3%), and OA (11.8%) (see insert in Figure 1). The three activities for ICA with the most significant frequency were ‘daily patient progress documentation’ (19.2%), ‘professional communication’ (18.0%), and ‘attending genetic meetings’ (16.6%). These three combined equal 53.8% of ICA (508 observations). In the URA category, ‘clerical work’ dominated with 36.1%, followed by ‘patient transit’ and ‘education’ at 16.1% and 16.0%, respectively. These three combined equal 68.2% of URA (448 observations). In the DCA category, the three activities with the highest frequency of observed activities were ‘assisting the doctor during review’ (31.1%), ‘patient/family interaction’ (26.5%), and ‘patient assessment’ (12.8%). These combined activities equal 70.4% of DCA (319 activities). Conversely, in OA, nurses spend more than half of the time in ‘idling’ (56.8%) activity alone, more than in ‘personal’ (18.2%) and ‘pantry’ (14.3%) combined. These three combined equal almost 90.0% of OA (251 observations) (see Table 2).
      Overall activity analysis shows that nurses spend most of their time on ‘clerical work’ (10.2%), ‘patient progress documentation’ (8.1%), ‘professional communication’ (7.3%), ‘idle time’ (6.7%), ‘assisting doctor during review’ (5.9%), and ‘genetic meetings’ (5.8%). In contrast, the lowest proportion of time was spent on ‘mobilizing patient’ (0.8%), ‘gathering information’ (0.9%), ‘medication administration’ (1.2%), and ‘using the restroom’ (1.3%). From Table 2, we can see that, although ‘education,’ ‘administrative meetings,’ ‘genetic meetings,’ and ‘mobilization of patient’ activities were not observable (higher number of zero observations) daily, the activities had a significant impact on the overall proportion of the time spent. Collectively, URA (22.6%) had the highest percentage of zero observations, followed by ICA (15.1%), DCA (14.3%), and OA (10.7%).

      Subjective evaluation

      The nurses had estimated that they spent 45.0% of their time on DCA [SD = 14.4], 25.6% (SD = 8.21) on ICA, 16.8% (SD = 5.94) on URA, and 12.5% (SD = 7.07) on OA. The nurses were 67% (SD = 13.35) confident with their estimates (on a 0–100 scale). Regarding specific care activities, the nurses approximated a minimum of 30–60 minutes and a maximum of 60–90 minutes weekly for ‘gathering information.’ In contrast, ‘genetic meetings’ had twice that weekly time spent, ranging from 60 to 120 minutes to a maximum of 120 to 210 minutes. We divided these activity estimates (minutes per week) by the total minutes per week (2,400 minutes) to obtain the expected proportion of time for the survey method (see Table 3). The chi-square tests showed that the given estimates differed significantly from the WS data for DCA, ICA, and URA, with p < .001 except for OA, for which p = .578 indicated no significance. Additionally, the two techniques showed no significant difference in the proportion of time spent on ‘genetic meetings’ (5.8% vs. 7.3%, p = .578), whereas ‘gathering information’ was significant (0.9% vs. 3.4%, p < .001).
      Table 3Comparison frequency of activities derived from work sampling and subjective evaluation.
      Category/activityObserved (%)Expected (%)d.f.χ2pModified Bonferroni adjusted alphaAdjusted significance
      WSSEV
      Direct453 (19.3)1,049 (45.0)1616.94<.001∗∗∗<.001 ∗∗∗No
      Indirect944 (40.4)596 (25.6)1594.55<.001∗∗∗<.001 ∗∗∗No
       Genetic meetings157 (5.8)170 (7.3)11.12.289.578No
       Gather information20 (0.9)79 (3.4)145.92<.001∗∗∗<.001 ∗∗∗No
      Unit656 (28.5)394 (16.9)1209.06<.001∗∗∗<.001 ∗∗∗No
      Others280 (11.8)291 (12.5)10.53.467.578No
      Note. SEV = subjective evaluation; WS = work sampling; d.f. = degree of freedom; ∗∗∗p < .001.

      Multiplier

      We have identified that the ‘others’ activity can be removed from the proportion of time used for patient care. Therefore, this overhead should not be included in the ratio and multiplier. Our results can be interpreted likewise (Table 4). Since we have a complete breakdown of time use, time spent on OA is classified outside of these care and non-care activities that did not generate outputs to the patient care delivery. The procedure for estimating multipliers to allocate time is illustrated using the three steps below [
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ].
      Table 4Care and contact types of activity with their associated time use.
      Category (notation in equation)Care typeContact type% of time
      Care activity (C)All patient care59.7
       Direct care activity (DCA)Direct careFace-to-face19.3
       Indirect care activity (ICA)Indirect careAway40.4
      Noncare activity (NC)All noncare40.3
       Unit-related activity (URA)Unit-relatedAway28.5
       ‘Others’ activity (OA)OthersAway11.8
      Total time (T)TotalTotal100
      Step 1. To calculate loading, we require a multiple of patient care-providing hours that reflect the relationship between care (C) and non-care (NC) activities and the total number of hours worked (T). The basic equation for this multiple (q) is represented as follows:
      T=qC


      T=C+NC


      Table 4 shows the breakdown of time spent delivering patient care at 59.7%. The proportion of time is:
      C=0.597


      T=1.0


      Thus, NC=0.403
      Step 2. The multiplier, q, is then calculated as follows:
      1=0.597+0.403


      1=q0.597


      q=10.597


      Step 3. However, we identified the need to include the information above to ensure that OA is not allocated to patient care delivery. To calculate the time spent on patient care, we must subtract the time spent on ‘others’: ‘n 'others':
      qC=TOA


      q=TOAC=0.8820.597


      =1.47


      Therefore, when estimating the cost per hour of a nurse in the genetic clinic, cost = 1.47 × ‘cost per hour.’ In comparison, the multiplier calculated from the survey data was 1.24. Similarly, other multipliers, e.g., patient face-to-face contact, can be approximated as 4.57 and 1.94 for WS and the survey, respectively (calculation not shown).

      Discussion

      This study explicitly compares two techniques for estimating unit cost multipliers of nurses in medical genetics. The WS and the survey indicated that nurses spend more than half of their time working away from patients, at 80.7% and 55.0%, respectively. The nurses spent half (40.4%) and less than half (25.6%) of that away time on patient care activities in WS and the survey, respectively. In addition, the nurses reported that they spent more time on DCA (45.0%) than on ICA (25.6%) in the survey. However, we discovered from WS that the nurses spent significantly less time on DCA (19.3%) than on ICA (40.4%). This finding is consistent with nurses who work in the ICU [
      • Abbey M.
      • Chaboyer W.
      • Mitchell M.
      Understanding the work of intensive care nurses: a time and motion study.
      ], ED [
      • Gholizadeh M.
      • Janati A.
      • Nadimi B.
      • Kabiri N.
      • Abri S.
      How do nurses spend their time in the hospital?.
      ], surgical ward [
      • Desjardins F.
      • Cardinal L.
      • Belzile E.
      • McCusker J.
      Reorganizing nursing work on surgical units: a time-and-motion study.
      ], and as research nurses [
      • Oddone E.
      • Guarisco S.
      • Simel D.
      Comparison of housestaff's estimates of their workday activities with results of a random work-sampling study.
      ,
      • Burke T.A.
      • McKee J.R.
      • Wilson H.C.
      • Donahue R.M.J.
      • Batenhorst A.S.
      • Pathak D.S.
      A comparison of time-and-motion and self-reporting methods of work measurement.
      ], all of which indicate that HCPs tend to overestimate the time spent on the direct patient-related activity. While surveys are beneficial in providing insights into work patterns, their accuracy might be impacted by recall error and desirability bias [
      • Podsakoff P.M.
      • MacKenzie S.B.
      • Lee J.-Y.
      • Podsakoff N.P.
      Common method biases in behavioral research: a critical review of the literature and recommended remedies.
      ].
      We demonstrated that OA was excluded from the multiplier, with idle (6.7%) and personal (2.2%) activity accounting for the majority (75.0%) of the category. Studies have shown that PA varied widely among nurses in surgical wards (2.5%), home nursing (5.1%), medical–surgical clinics (12–18%), neurorehabilitation units (19%), ICUs (24%), critical care units, and EDs (42%) [
      • Gholizadeh M.
      • Janati A.
      • Nadimi B.
      • Kabiri N.
      • Abri S.
      How do nurses spend their time in the hospital?.
      ,
      • Desjardins F.
      • Cardinal L.
      • Belzile E.
      • McCusker J.
      Reorganizing nursing work on surgical units: a time-and-motion study.
      ,
      • Bordin L.C.
      • Fugulin F.M.
      Distribuição do tempo das enfermeiras: identificação e análise em Unidade Médico-Cirúrgica = Nurses' time distribution: identification and analysis in a medical-surgical unit.
      ,
      • Munyisia E.N.
      • Yu P.
      • Hailey D.
      How nursing staff spend their time on activities in a nursing home: an observational study.
      ,
      • Williams H.
      • Harris R.
      • Turner-Stokes L.
      Work sampling: a quantitative analysis of nursing activity in a neuro-rehabilitation setting.
      ,
      • Tang Z.
      • Weavind L.
      • Mazabob J.
      • Thomas E.J.
      • Chu-Weininger M.Y.L.
      • Johnson T.R.
      Workflow in intensive care unit remote monitoring: a time-and-motion study.
      ]. Our nurses’ personal time was lower due to further subdividing OA into smaller activities to help us identify the activity unrelated to services, patient care, and professional development. Additionally, some authors classified education and training as personal tasks [
      • Iosa M.
      • Grasso M.G.
      • Dandi R.
      • Carusi D.
      • Bacci A.
      • Marra R.
      • et al.
      Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
      ], contrary to our approach, which categorized them as an overhead of patient care. Furthermore, variations also existed in the major categorization of the activities. In an Australian study [
      • Abbey M.
      • Chaboyer W.
      • Mitchell M.
      Understanding the work of intensive care nurses: a time and motion study.
      ], medication preparation was defined as DCA for ICU nurses, whereas we categorized it as ICA. These variations in definitions and categorization are well accepted to accommodate the particular objective of a study [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ,
      • Hendrich A.
      • Chow M.P.
      • Skierczynski B.A.
      • Lu Z.
      A 36-hospital time and motion study: how do medical-surgical nurses spend their time?.
      ]. Thus, the specific definition and categorization of the activities should be further inspected whenever a multiplier is to be estimated from the published study.
      There are significant disparities in the proportions of time spent by nurses on direct and indirect care activities that have been reported in other areas of practice [
      • Gholizadeh M.
      • Janati A.
      • Nadimi B.
      • Kabiri N.
      • Abri S.
      How do nurses spend their time in the hospital?.
      ]. In comparison, research indicates that geneticists spend more time directly contacting patients [
      • Heald B.
      • Rybicki L.
      • Clements D.
      • Marquard J.
      • Mester J.
      • Noss R.
      • et al.
      Assessment of clinical workload for general and specialty genetic counsellors at an academic medical center: a tool for evaluating genetic counselling practices.
      ], and genetic counsellors spend less time directly contacting patients than they do indirectly [
      • Attard C.A.
      • Carmany E.P.
      • Trepanier A.M.
      Genetic counselor workflow study: the times are they a-changin'?.
      ]. Moreover, the considerable time spent on indirect patient-related activity distinguishes genetic management from many other disciplines. These include gathering disease information, case conferencing, and continuing education of HCPs [
      • McPherson E.
      • Zaleski C.
      • Benishek K.
      • McCarty C.A.
      • Giampietro P.F.
      • Reynolds K.
      • et al.
      Clinical genetics provider real-time workflow study.
      ,
      • Heald B.
      • Rybicki L.
      • Clements D.
      • Marquard J.
      • Mester J.
      • Noss R.
      • et al.
      Assessment of clinical workload for general and specialty genetic counsellors at an academic medical center: a tool for evaluating genetic counselling practices.
      ]. Thus, estimating direct contact time alone will underestimate the time required to provide care.
      The study highlighted ‘genetic meetings’ as the activity that significantly contributed to the provision of patient care. Specialists led the meeting, and nurses took turns attending to discuss patient treatment. This is crucial because most patient cases are complicated and require collective input from team members to make pragmatic treatment and care management decisions. These are the most prominent features of genetic nurses compared to nurses in other fields. It is important to note that documentation activities classified as ‘clerical’ and ‘patient progress documentation’ are the most time-consuming tasks for nurses. The timely documentation was congruent with medical-surgical units, acute care, and nursing home nurses [
      • Lim M.L.
      • Ang S.Y.
      A time-motion observation study to measure and analyze clinical nursing workload in an acute care hospital in Singapore.
      ,
      • Munyisia E.N.
      • Yu P.
      • Hailey D.
      How nursing staff spend their time on activities in a nursing home: an observational study.
      ,
      • Hendrich A.
      • Chow M.P.
      • Skierczynski B.A.
      • Lu Z.
      A 36-hospital time and motion study: how do medical-surgical nurses spend their time?.
      ]. The required documentation is not only necessary for compliance with local health-related policies and regulations but also the maintenance of care quality.
      Many studies consistently report that nurses can perform multiple activities simultaneously [
      • Abbey M.
      • Chaboyer W.
      • Mitchell M.
      Understanding the work of intensive care nurses: a time and motion study.
      ,
      • Yen P.-Y.
      • Kellye M.
      • Lopetegui M.
      • Saha A.
      • Loversidge J.
      • Chipps E.M.
      • et al.
      Nurses' time allocation and multitasking of nursing activities: a time motion study.
      ], with communication being the most common multitasking activity among DCAs in high-care nursing homes (40%) and medical-surgical units (25%) [
      • Yen P.-Y.
      • Kellye M.
      • Lopetegui M.
      • Saha A.
      • Loversidge J.
      • Chipps E.M.
      • et al.
      Nurses' time allocation and multitasking of nursing activities: a time motion study.
      ,
      • Munyisia E.N.
      • Yu P.
      • Hailey D.
      How nursing staff spend their time on activities in a nursing home: an observational study.
      ]. Overall, ‘professional communications’ ranks third according to our study and recent findings of neurorehabilitation and medical-surgical unit nurses [
      • Iosa M.
      • Grasso M.G.
      • Dandi R.
      • Carusi D.
      • Bacci A.
      • Marra R.
      • et al.
      Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
      ,
      • Yen P.-Y.
      • Kellye M.
      • Lopetegui M.
      • Saha A.
      • Loversidge J.
      • Chipps E.M.
      • et al.
      Nurses' time allocation and multitasking of nursing activities: a time motion study.
      ]. Moreover, communication with patients represents the second largest proportion of the nurses' direct care time, contrary to a previous study that indicated nurses devoted little time to interact with their patients [
      • Ahmadishad M.
      • Adib-Hajbaghery M.
      • Rezaei M.
      • Atoof F.
      • Munyisia E.
      Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
      ]. The finding demonstrates that two modes of communication are essential for maintaining the organization's effectiveness and ensuring the delivery of quality genetic care.

      Limitations

      Our study has some limitations. First, it was conducted in a single genetic clinic. Then, the observation period happened to be two weeks full of unit briefings, audits, and renovation activities to meet the end of the current year. This may partly explain why activities such as ‘education’ were recorded at a slightly higher percentage (4%) than for nurses in the rehabilitation unit (<1%) [
      • Iosa M.
      • Grasso M.G.
      • Dandi R.
      • Carusi D.
      • Bacci A.
      • Marra R.
      • et al.
      Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
      ,
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ]. A previous report suggested that HCPs [
      • McPherson E.
      • Zaleski C.
      • Benishek K.
      • McCarty C.A.
      • Giampietro P.F.
      • Reynolds K.
      • et al.
      Clinical genetics provider real-time workflow study.
      ] also use a significant amount of preclinic and postclinic time to research, read, and prepare cases for patients. We have limited evidence of how this might impact nurses' total care activities. This may limit the generalizability of the findings.
      Second, the differences arising from the two techniques could also be attributed to the observer not practicing in nursing and potentially having a different interpretation of the activity definitions. However, we believe the impact was minimal because we rectified any discrepancies with the nurse leaders during the trial runs. In addition, we conducted a short communication with the nurses at gaps between time intervals to resolve any discrepancies immediately. Moreover, we have tested both techniques to differ significantly, prompting the methods themselves to influence the results.
      The observer was not permitted to enter observation rooms during patient assessment in certain circumstances. In this instance (unobserved activity), we made a reasonable assumption based on the information provided to the observer about the tasks the nurses were performing. Additionally, we left the observation unrecorded when participants failed to disclose their planned destination (e.g., outside the clinic) to the observer. Occasionally, participants engage in an activity that lasts longer than the activity being recorded. Nonetheless, the pattern identified in this study was obtained from a primary referral centre in Malaysia, and these limitations were found to be minimal.

      Advantages

      There were activities that we recorded at a distance from the participant to reduce the Hawthorne effect. Previous reports suggested that the effect was likely to affect time engaged in personal time, meal breaks, and idle time [
      • Ampt A.
      • Westbrook J.
      • Creswick N.
      • Mallock N.
      A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks.
      ]. However, we did not find any significant difference in the OA category between the two techniques. Moreover, while the previous study showed that the HCP tends to give estimations in favour of PRA [
      • Oddone E.
      • Guarisco S.
      • Simel D.
      Comparison of housestaff's estimates of their workday activities with results of a random work-sampling study.
      ], we found that the estimated proportion of PRA was lower for WS than that of the survey, suggesting that there was no significant Hawthorne effect in this study.
      Whilst surveys are relatively less expensive and take less time to administer, they lack the degree of detail required in assessing the overhead. In this instance, surveys may further reduce the representativeness of the multipliers. It is worth mentioning that WS is incomparable to the ‘gold standard’ TMS study [
      • Lopetegui M.
      • Yen P.-Y.
      • Lai A.
      • Jeffries J.
      • Embi P.
      • Payne P.
      Time motion studies in healthcare: what are we talking about?.
      ] in terms of the sequence of activities and actual duration of time [
      • Pelletier D.
      • Duffield C.
      Work sampling: valuable methodology to define nursing practice patterns.
      ,
      • Shearer J.
      • McCrone P.
      • Romeo R.
      Economic evaluation of mental health interventions: a guide to costing approaches.
      ]. However, the level of information obtained in this study was sufficient for the purpose of describing the working pattern. Despite its inferiority to TMS, WS enables a more objective description of time distribution while potentially avoiding the recall error and personal bias associated with surveys or self-reports. This suggests that WS appears to be a straightforward and cost-effective technique for estimating the multiplier.

      Implications and recommendations

      The two techniques produced high variability in the overall categorization of nurses' activities except for OA. While the multipliers for patient face-to-face contact were significantly larger between WS (4.57) and the survey (1.94), the multipliers for patient care time were smaller between WS (1.47) and the survey (1.24). Consequently, when comparing the true cost of patient care delivery between WS (1.47 × $20/h = $29.40) and the survey (1.24 × $20/h = $24.80), the gap in the true cost of patient face-to-face contact was 2.5 times greater between WS (4.57 × $20/h = $91.40) and the survey (1.94 × $20 = $38.80). Thus, estimating the paid cost solely based on direct time with patients considerably underestimates the cost per hour of nurses’ care delivery. Additionally, caution should be taken when multiplying the patient contact time by the patient-related activity to determine the cost of care provision, as demonstrated in this study.
      The study demonstrates the importance of adequately allocating overhead to ensure that the activities unrelated to patient care (e.g., PA) are not included as overhead on patient care costs, particularly when evaluating human resource expenses. However, numerous CE studies have ignored this critical component entirely from the analyses [
      • Jacobs J.C.
      • Barnett P.G.
      Emergent challenges in determining costs for economic evaluations.
      ]. One possible explanation for this is the complexity and difficulty of quantifying overhead. Cost estimation is a fundamental component of economic analysis. Despite the wealth of literature available in this field [
      • Mogyorosy Z.
      • Smith P.C.
      The main methodological issues in costing health care services - a literature review.
      ,
      • Jacobs J.C.
      • Barnett P.G.
      Emergent challenges in determining costs for economic evaluations.
      ], there is a dearth of discussion of how ‘hidden’ or indirect patient-related activities should be handled and how they ultimately affect costs. This study sheds light on quantifying the actual cost of nursing care intervention for resource prioritization, as highlighted by previous CE studies in nursing [
      • Oddone E.
      • Guarisco S.
      • Simel D.
      Comparison of housestaff's estimates of their workday activities with results of a random work-sampling study.
      ,
      • Lee M.
      • Moorhead S.
      • Clancy T.
      Determining the cost-effectiveness of hospital nursing interventions for patients undergoing a total hip replacement.
      ]. The differences in cost and unit cost estimates can alter the final CE conclusion of whether a new intervention is effective. While the sensitivity analysis [
      • Lee M.
      • Moorhead S.
      • Clancy T.
      Determining the cost-effectiveness of hospital nursing interventions for patients undergoing a total hip replacement.
      ] can be used to account for the uncertainty around the estimates, it is crucial to have a realistic range of parameter values (e.g., a multiplier) rather than being utterly arbitrary during the model simulation [
      • Oddone E.
      • Guarisco S.
      • Simel D.
      Comparison of housestaff's estimates of their workday activities with results of a random work-sampling study.
      ].
      A critical first step is to raise awareness to bridge gaps in knowledge and attitudes regarding the role of economic analysis in minimizing harm to the patient by recognizing the ‘value’ of nursing interventions for a cost-effective care strategy. This emphasizes the theoretical basis for characterizing genetic nursing care workloads and work patterns in estimating the actual cost of care delivery [
      • Lee M.
      • Moorhead S.
      • Clancy T.
      Determining the cost-effectiveness of hospital nursing interventions for patients undergoing a total hip replacement.
      ]. However, a conservative approach can be used by assuming that all time is spent on patient care [
      • Netten A.
      • Knight J.
      • Dennett J.
      • Cooley R.
      • Slight A.
      Development of a ready reckoner for staff costs in the NHS.
      ] when there is no detailed breakdown of time; for instance, limited information is available from standard workload measurements (e.g., patient turnover) for non-care-related activity [
      • Ahmadishad M.
      • Adib-Hajbaghery M.
      • Rezaei M.
      • Atoof F.
      • Munyisia E.
      Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
      ]. This necessitates the use of alternative (e.g., WS) or combined-with-standard methods to reconcile the actual pattern of care provision, provided that the findings and sensitivity of the results are explicitly stated as demonstrated by this study.
      The overarching process proves that the multiplier is beneficial in converting other forms of unit cost into terms that are also important compared to the cost per hour, e.g., cost per bed. Alternatively, the term ‘patient-related activity’ found in previous studies can be used in reference to patient care delivery, as in our study, for instance, when estimating geneticist [
      • McPherson E.
      • Zaleski C.
      • Benishek K.
      • McCarty C.A.
      • Giampietro P.F.
      • Reynolds K.
      • et al.
      Clinical genetics provider real-time workflow study.
      ] and genetic counsellor [
      • Heald B.
      • Rybicki L.
      • Clements D.
      • Marquard J.
      • Mester J.
      • Noss R.
      • et al.
      Assessment of clinical workload for general and specialty genetic counsellors at an academic medical center: a tool for evaluating genetic counselling practices.
      ,
      • Attard C.A.
      • Carmany E.P.
      • Trepanier A.M.
      Genetic counselor workflow study: the times are they a-changin'?.
      ] multipliers from published sources. We recommend employing ‘patient-related activity’ instead of the ‘face-to-face contact’ multiplier because the former did not significantly differ from the one estimated using WS in this study. The actual cost will include care overheads in the cost per hour of care delivery.

      Conclusion

      The observational data reflect current genetic nurses’ work practices. The ratio of time on patient care activities to time spent on non-care activities is 1:0.47. This means that for every hour nurses spend on care activity, e.g., assisting doctors, an additional 28 minutes are spent on non-care activity, e.g., clerical time, to deliver appropriate patient care. Hence, every hour spent with a patient requires 1.47 paid hours. We will utilize the adjusted total paid hours per nurse attributable to patient care delivery in a future CE model. This ensures that the total cost approximately reflects our current local practices.

      Funding

      This work was supported by the Universiti Sains Malaysia's Research University Grant ( 1001/PFARMASI/8011119 ).

      Conflict of interest

      The authors declared no conflict of interest.

      Acknowledgments

      The authors thank all participants for their generous participation in the project. The authors also thank Hospital Kuala Lumpur for providing the facilities for this study. Lastly, the authors would like to thank the Director General of Health Malaysia for his permission to publish the article.

      Appendix

      For editor and reviewers' reference only.
      Quantitative work-sampling activity definitions.
      The definitions of normal work activities performed with inclusion and exclusion of examples.
      Tabled 1
      CodeActivityActivity definitionInclusions and exclusions
      Direct Care: Performed in the presence of the patient and/or family
      1.Registration and new admissionThis involves taking registration at the counter.Includes: Telephone registration, clearing patient appointment routine. New and returning patients at the main counter.

      Excludes: Rewriting/re-entering ER in computer
      2.Patient assessmentThis involves nursing assessing patient overall health status. Additionally, readings taken from medical devicesIncludes: Vital signs, objective and subjective findings, measurement, weighing. Obtaining temperature.

      Excludes: Other clinical procedures specified at Activity 4.
      3.Medication administrationAdministration of medication during clinic hoursIncludes: ERT and non-ERT injections such as premedications.

      Excludes: Activity 2 and 4
      4.Clinical proceduresThis involves all medical procedures conducted in the treatment room.Includes: Branula insertion, poking, equipment attaches to patient, blood and urine samples, emergency procedures

      Excludes: Procedure conducted in the doctor's room will be classified under 6. Performing patient assessments prespecified under 2.
      5.Patient mobilityThis involves moving and directing patient to another location within the clinic.Includes: Infusion area. Treatment area.

      Excludes: Activity 7
      6.Assisting doctor during a reviewThis involves any clinical and administrative procedures done in the presence of the doctor.Includes: Conducted in the assessment and treatment room. Filling out other forms.

      Excludes: Filing conducted outside prespecified rooms
      7.Patient interactionsSpend time communicating and addressing the needs of the patient and family physically.Includes: Child's handling, instruction or counselling, and conversation with patient or family.

      Excludes: Over the phone patient contact is classified under Activity 15. Moving and directing patients is under Activity 5.
      Indirect Care: Performed away from the patient, but specifically on the patient's behalf
      8.Professional communicationsThis involves asking, reporting patient results and other required information for patient care.Includes: Reporting results of patient to other colleagues within the clinics. Seeking consultations, specialist, other hospital staff.

      Excludes: Activity 14
      9.Room and equipment setup, cleaningThis involves preparatory time before and after seeing the patient.Includes: Treatment and assessment room. Gathering supplies. Preparing equipment.

      Excludes: Medication-related is classified under 3
      10.Medication tasksIndent, send Rx, and collect medication at the pharmacy. Prepare IV administration and dilute.Includes: Present the prescription to the pharmacy and wait for the medication to be dispensed. Prepare and check medication.
      11.Progress and discharge notes documentationThis involves working on making discharge summaries and progress notes on the observation day.Includes: Yellow files only

      Excludes: Long overdue files management. This will be classified under Activity 20.
      12.Data entry and retrievalThis is specific to the activity conducted using the computer.Includes: White computer. Entry into the SMS system

      Excludes: Black computer
      13.Gathering information from phones, computersThis includes information searches conducted during patient assessment.Includes: Reviewing images, diseases information, journals

      Excludes:
      14.Telephone contact for follow-upThis includes getting information of the lab results and other relevant units ready prior to the assessment and arranging the collection of samples.Includes: Pediatrics, MRI via phones.

      Excludes: Activity 15
      15.Coordination of careThis involves the planning of care over the phone and the clinic's mobile phone. Referring to support groupsIncludes: Call and text messages via clinic's mobile phone (iPhone4 unit's phone)

      Excludes: Personal mobile phone, Activity 14
      16.Genetic meetingsThis includes general metabolic meetings specific to patients.Includes: Thursday and Friday

      Excludes: Other administrative meetings are classified under Activity 21
      17.Education and in-serviceThis involves participation in teaching and learning activities such as CME to meet learning needs.Includes: Departmental audits, continuous medical education

      Excludes:
      18.Supplies check (stocking)This involves checking medical equipment stocks and ordering supplies.Includes: Storage area

      Excludes:
      Unit-related task: Non-care activity related to general maintenance of the genetic unit
      19.Errands of unitThis involves getting necessary tasks completed as soon as possible.Includes: Lab results, obtain urgent stock.

      Excludes:
      20.Clerical workThis involves unit-related work and records. Nurse will spend time in counselling room after noon.Includes: Brown files

      Excludes: Yellow files classified under 11
      21.Administrative meetingsThis involves meetings and administrative work purposes.Includes: Ad hoc and planned meetings

      Excludes: Genetic and metabolic meetings
      22.TransitThis involves time spent between tasks for work-related activitiesIncludes: Getting equipment from one patient to another. Movement from patient to equipment

      Excludes: 5
      Others: Not patient- and unit-specific
      23.PersonalThis involves using a personal phone during working hours.Includes: Web surfing, praying, talking, sleeping

      Excludes: Use of phones for gathering information is classified under Item 13.
      24.Idle timeWaiting for the end of some activities. This involves no prespecified activity being conducted when the observation is made.Includes: Inactive time

      Excludes: Personal, pantry, and praying
      25.PantryTime spent in pantry during the observation timeIncludes: Middle pantry. Extra room in the unit.

      Excludes:
      26.RestroomToilet useIncludes: Built-in toilet

      References

        • Cai X.
        • Yang H.
        • Genchev G.Z.
        • Lu H.
        • Yu G.
        Analysis of economic burden and its associated factors of twenty-three rare diseases in Shanghai.
        Orphanet J Rare Dis. 2019; 14: 233https://doi.org/10.1186/s13023-019-1168-4
        • Shafie A.A.
        • Supian A.
        • Ahmad Hassali M.A.
        • Ngu L.-H.
        • Thong M.-K.
        • Ayob H.
        • et al.
        Rare disease in Malaysia: challenges and solutions.
        PLoS One. 2020; 15e0230850https://doi.org/10.1371/journal.pone.0230850
      1. Human resources for health country profiles Malaysia. Planning Division, Ministry of Health Malaysia, 2015-2018 (Available from:)
        • Twigg D.
        • Duffield C.
        A review of workload measures: a context for a new staffing methodology in Western Australia.
        Int J Nurs Stud. 2009; 46: 132-140https://doi.org/10.1016/j.ijnurstu.2008.08.005
        • Baernholdt M.
        • Cox K.
        • Scully K.
        Using clinical data to capture nurse workload: implications for staffing and safety.
        Comput Inform Nurs. 2010; 28: 229-234https://doi.org/10.1097/NCN.0b013e3181e1e57d
        • Griffiths P.
        • Saville C.
        • Ball J.
        • Jones J.
        • Pattison N.
        • Monks T.
        Nursing workload, nurse staffing methodologies and tools: a systematic scoping review and discussion.
        Int J Nurs Stud. 2020; 103103487https://doi.org/10.1016/j.ijnurstu.2019.103487
        • Lim M.L.
        • Ang S.Y.
        A time-motion observation study to measure and analyze clinical nursing workload in an acute care hospital in Singapore.
        Proc Singap Healthcare. 2019; 28: 124-128https://doi.org/10.1177/2010105819834569
        • Abbey M.
        • Chaboyer W.
        • Mitchell M.
        Understanding the work of intensive care nurses: a time and motion study.
        Aust Crit Care. 2012; 25: 13-22https://doi.org/10.1016/j.aucc.2011.08.002
        • Kakushi L.E.
        • Évora Y.D.M.
        Direct and indirect nursing care time in an intensive care unit.
        Rev Lat Am Enfermagem. 2014; 22: 150-157https://doi.org/10.1590/0104-1169.3032.2381
        • Ahmadishad M.
        • Adib-Hajbaghery M.
        • Rezaei M.
        • Atoof F.
        • Munyisia E.
        Care and noncare-related activities among critical care nurses: a cross-sectional observational time and motion study.
        Nurs Midwifery Stud. 2019; 8: 40-47https://doi.org/10.4103/nms.nms_60_18
        • Gholizadeh M.
        • Janati A.
        • Nadimi B.
        • Kabiri N.
        • Abri S.
        How do nurses spend their time in the hospital?.
        J Clin Res Govern. 2014; 3: 27-33https://doi.org/10.13183/jcrg.v3i1.52
        • Desjardins F.
        • Cardinal L.
        • Belzile E.
        • McCusker J.
        Reorganizing nursing work on surgical units: a time-and-motion study.
        Nurs Leadersh. 2008; 21: 26-38https://doi.org/10.12927/cjnl.2008.20057
        • Hollingworth W.
        • Devine E.B.
        • Hansen R.N.
        • Lawless N.M.
        • Comstock B.A.
        • Wilson-Norton J.L.
        • et al.
        The impact of e-prescribing on prescriber and staff time in ambulatory care clinics: a time–motion study.
        JAMIA Open. 2007; 14: 722-730https://doi.org/10.1197/jamia.M2377
        • Westbrook J.I.
        • Li L.
        • Georgiou A.
        • Paoloni R.
        • Cullen J.
        Impact of an electronic medication management system on hospital doctors' and nurses' work: a controlled pre–post, time and motion study.
        JAMIA Open. 2013; 20: 1150-1158https://doi.org/10.1136/amiajnl-2012-001414
        • Gardner G.
        • Gardner A.
        • Middleton S.
        • Gibb M.
        • Della P.
        • Duffield C.
        Development and validation of a novel approach to work sampling: a study of nurse practitioner work patterns.
        Aust J Adv Nurs. 2010; (Internet) ([cited 2020 Jun 20];27(4):4-12. Available from:)
        • McPherson E.
        • Zaleski C.
        • Benishek K.
        • McCarty C.A.
        • Giampietro P.F.
        • Reynolds K.
        • et al.
        Clinical genetics provider real-time workflow study.
        Genet Med. 2008; 10 (https://doi.org/10.1097GIM.0b013e318182206f): 699-706
        • Heald B.
        • Rybicki L.
        • Clements D.
        • Marquard J.
        • Mester J.
        • Noss R.
        • et al.
        Assessment of clinical workload for general and specialty genetic counsellors at an academic medical center: a tool for evaluating genetic counselling practices.
        NPJ Genom Med. 2016; 1: 1-8https://doi.org/10.1038/npjgenmed.2016.10
        • Attard C.A.
        • Carmany E.P.
        • Trepanier A.M.
        Genetic counselor workflow study: the times are they a-changin'?.
        J Genet Counsel. 2019; 28: 130-140https://doi.org/10.1002/jgc4.1041
        • Iosa M.
        • Grasso M.G.
        • Dandi R.
        • Carusi D.
        • Bacci A.
        • Marra R.
        • et al.
        Clinical staff work sampling in a neurorehabilitation hospital and its relationship to severity of disease.
        J Nurs Manag. 2019; 27: 179-189https://doi.org/10.1111/jonm.12663
        • Duffield C.
        • Forbes J.
        • Fallon A.
        • Roche M.
        • Wise W.
        • Merrick E.
        Nursing skill mix and nursing time: the roles of registered nurses and clinical nurse specialists.
        Aust J Adv Nurs. 2005; (Internet) ([cited 2021 Jun 23];23(2):14–21. Available from:)
        • Urden L.D.
        • Roode J.L.
        Work sampling: a decision-making tool for determining resources and work redesign.
        J Nurs Adm. 1997; 27: 34-41https://doi.org/10.1097/00005110-199709000-00009
        • Kilpatrick K.
        Development and validation of a time and motion tool to measure cardiology acute care nurse practitioner activities.
        Can J Cardiovasc Nurs. 2011; 21: 18-26
        • Netten A.
        • Knight J.
        • Dennett J.
        • Cooley R.
        • Slight A.
        Development of a ready reckoner for staff costs in the NHS.
        Personal Social Services Research Unit, Kent (UK)1998: 73
        • Mogyorosy Z.
        • Smith P.C.
        The main methodological issues in costing health care services - a literature review.
        Centre for Health Economics, York (UK)2005: 232
        • Jacobs J.C.
        • Barnett P.G.
        Emergent challenges in determining costs for economic evaluations.
        Pharmacoeconomics. 2017; 35: 129-139https://doi.org/10.1007/s40273-016-0465-1
        • Knapp M.
        • Baines B.
        Hidden cost multipliers for residential child care.
        Local Govern Stud. 1987; 13: 53-73https://doi.org/10.1080/03003938708433354
        • Yen P.-Y.
        • Kellye M.
        • Lopetegui M.
        • Saha A.
        • Loversidge J.
        • Chipps E.M.
        • et al.
        Nurses' time allocation and multitasking of nursing activities: a time motion study.
        AMIA Annu Symp Proc. 2018; (Internet) ([cited 2021 Jun 12];2018(2018):1137-1146. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6371290/)
        • Drummond M.F.
        • Sculpher M.J.
        • Claxton K.
        • Stoddart G.L.
        • Torrance G.W.
        Methods for the economic evaluation of health care programme.
        4th ed. Oxford University Press, Oxford, England2015: 445
        • Oddone E.
        • Guarisco S.
        • Simel D.
        Comparison of housestaff's estimates of their workday activities with results of a random work-sampling study.
        Acad Med. 1993; 68: 859-861
        • Sinsky C.
        • Colligan L.
        • Li L.
        • Prgomet M.
        • Reynolds S.
        • Goeders L.
        • et al.
        Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.
        Ann Intern Med. 2016; 165: 753-760https://doi.org/10.7326/m16-0961
        • Curtis L.A.
        How do professionals spend their time?.
        (Internet) University of Kent, Canterbury: Personal Social Services Research Unit, Kent2012 ([cited 2021 May 21]. Available from:)
        • Curtis L.A.
        • Burns A.
        Unit cost of health and social care 2017.
        Personal Social Services Research Unit, Kent (UK)2017: 247
        • Pelletier D.
        • Duffield C.
        Work sampling: valuable methodology to define nursing practice patterns.
        Nurs Health Sci. 2003; 5: 31-38https://doi.org/10.1046/j.1442-2018.2003.00132.x
        • Finkler S.A.
        • Knickman J.R.
        • Hendrickson G.
        • Lipkin Jr., M.
        • Thompson W.G.
        A comparison of work-sampling and time-and-motion techniques for studies in health services research.
        Health Serv Res. 1993; 28: 577-597
        • Rutter P.M.
        • Brown D.
        • Jones I.F.
        Pharmacy research: the place of work measurement.
        Int J Pharm Pract. 1998; 6: 46-58https://doi.org/10.1111/j.2042-7174.1998.tb00915.x
        • Sittig D.F.
        Work-sampling: a statistical approach to evaluation of the effect of computers on work patterns in the healthcare.
        Methods Inf Med. 1993; 32: 167-174https://doi.org/10.1055/s-0038-1634912
        • Curtis L.A.
        • Netten A.
        The costs of training a nurse practitioner in primary care: the importance of allowing for the cost of education and training when making decisions about changing the professional-mix.
        J Nurs Manag. 2007; 15: 449-457https://doi.org/10.1111/j.1365-2834.2007.00668.x
        • Lwanga S.K.
        • Lemeshow S.
        Sample size determination in health studies: a practical manual.
        1st ed. World Health Organization, England1991: 80
        • Bordin L.C.
        • Fugulin F.M.
        Distribuição do tempo das enfermeiras: identificação e análise em Unidade Médico-Cirúrgica = Nurses' time distribution: identification and analysis in a medical-surgical unit.
        Rev Esc Enferm USP. 2009; (Internet) ([cited 2021 Dec 14];43(4):833-837. Available from:) (Portuguese, English)
        • Abdellah F.G.
        • Levine E.
        Work-sampling applied to the study of nursing personnel.
        Nurs Res. 1954; 3: 11-16
        • Ampt A.
        • Westbrook J.
        • Creswick N.
        • Mallock N.
        A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks.
        J Health Serv Res Pol. 2007; 12: 18-24https://doi.org/10.1258/135581907779497576
        • Burke T.A.
        • McKee J.R.
        • Wilson H.C.
        • Donahue R.M.J.
        • Batenhorst A.S.
        • Pathak D.S.
        A comparison of time-and-motion and self-reporting methods of work measurement.
        J Nurs Adm. 2000; 30: 118-125https://doi.org/10.1097/00005110-200003000-00003
        • Podsakoff P.M.
        • MacKenzie S.B.
        • Lee J.-Y.
        • Podsakoff N.P.
        Common method biases in behavioral research: a critical review of the literature and recommended remedies.
        J Appl Psychol. 2003; 88: 879https://doi.org/10.1037/0021-9010.88.5.879
        • Munyisia E.N.
        • Yu P.
        • Hailey D.
        How nursing staff spend their time on activities in a nursing home: an observational study.
        J Adv Nurs. 2011; 67: 1908-1917https://doi.org/10.1111/j.1365-2648.2011.05633.x
        • Williams H.
        • Harris R.
        • Turner-Stokes L.
        Work sampling: a quantitative analysis of nursing activity in a neuro-rehabilitation setting.
        J Adv Nurs. 2009; 65: 2097-2107https://doi.org/10.1111/j.1365-2648.2009.05073.x
        • Tang Z.
        • Weavind L.
        • Mazabob J.
        • Thomas E.J.
        • Chu-Weininger M.Y.L.
        • Johnson T.R.
        Workflow in intensive care unit remote monitoring: a time-and-motion study.
        Crit Care Med. 2007; 35: 2057-2063https://doi.org/10.1097/01.ccm.0000281516.84767.96
        • Hendrich A.
        • Chow M.P.
        • Skierczynski B.A.
        • Lu Z.
        A 36-hospital time and motion study: how do medical-surgical nurses spend their time?.
        Perm J. 2008; 12: 25https://doi.org/10.7812/tpp/08-021
        • Lopetegui M.
        • Yen P.-Y.
        • Lai A.
        • Jeffries J.
        • Embi P.
        • Payne P.
        Time motion studies in healthcare: what are we talking about?.
        J Biomed Inf. 2014; 49: 292-299https://doi.org/10.1016/j.jbi.2014.02.017
        • Shearer J.
        • McCrone P.
        • Romeo R.
        Economic evaluation of mental health interventions: a guide to costing approaches.
        Pharmacoeconomics. 2016; 34: 651-664https://doi.org/10.1007/s40273-016-0390-3
        • Lee M.
        • Moorhead S.
        • Clancy T.
        Determining the cost-effectiveness of hospital nursing interventions for patients undergoing a total hip replacement.
        J Nurs Manag. 2014; 22: 825-836https://doi.org/10.1111/jonm.12022