Comparing VA and Non-VA Medical Centers

Background

Veteran Health Issues

The June 2019 implementation of the MISSION Act of 2018 precipitated a notable shift in the way veterans access health care in the United States. Conceptualized in an effort to improve veteran-provider relationships and ameliorate long VA wait times, this legislation built on previous VA attempts to streamline and combine VA and non-VA services by broadening the qualifying criteria by which a veteran could seek outside care.⁴ And unlike previous frameworks, the MISSION Act allowed qualifying veterans to seek primary care at a VA medical center while receiving sometimes difficult-to-obtain specialized care at a non-VA medical center.

Under the MISSION Act, eligibility for non-VA care is based on VA assessment that: (a) the veteran requires a medical service that is not available from the VA (e.g., labor and delivery); (b) the veteran lives in a state without a full-service VA (e.g., Alaska); (c) the veteran is already grandfathered into the previous program, the Choice program, and does not wish to change health care providers; (d) the VA cannot provide health care services within the designated access standards (i.e., 30-minute drive time for primary care or 20 days’ appointment wait time); (e) it is in the veteran’s best medical interest; or (f) the service provided by the VA does not meet quality standards, such as those analyzed for this study. In addition to creating a larger umbrella under which outside primary and (especially) specialty care can be sought, the MISSION Act provided for increased physician recruitment for both underserved specialties and underserved areas. It also allowed for extended telehealth services to further facilitate medical services for rural veterans, although it should be noted that limited practitioner access to patients (doctors but not nurses or nurse practitioners) is one critique of this aspect of the legislation.

Although opponents of the MISSION Act as enacted have argued that broader criteria for non-VA care would result in increased privatization and a consequent exodus of veterans from the VA system, VA Executive-in-Charge Richard Stone reported that the early months of its implementation have shown cause for optimism. According to Stone’s testimony to the U.S. House Committee on Veterans’ Affairs Subcommittee on Health, as of September 25, 2019, the VA had facilitated 1 million community care consultations since the act’s implementation in June 2019.⁵ He testified that the surge was due in large part to 80,000 veterans accessing VA care for the first time as a result of MISSION Act changes. While exact data are not yet available as to the distribution of appointments between VA and non-VA providers, Stone reported that the VA scheduled 1.6 million more ambulatory appointments in fiscal 2019 than in 2018, possibly necessitating additional legislation to accommodate growth.

It is clear from the little data available since the MISSION Act’s implementation that (a) the health system changes precipitated by the act have increased demand for VA services; and (b) veterans will likely seek more information about their health care options as a result of these changes. This report documents the findings of a study conducted by CNAS with MFRI at Purdue University to compare VA and non-VA health care using clinical and patient experience measures. Unlike other studies that have utilized expanded propensity score matching or aggregate hospital data, this study used 1-to-1 comparisons of VA medical centers and similar non-VA medical centers nearby.

Previous Research

Veterans as Health Care Recipients

The veteran population presents unique challenges in health care delivery, as these individuals tend to be older and at greater risk for many conditions, including mental health and substance use disorders, traumatic injuries, hazardous exposures, and homelessness.⁶ Because these medical conditions are not observed in the general population at the same rates, connecting veterans to appropriate care is often the first challenge in health care delivery.⁷ Female veterans in particular exhibit a unique set of behaviors when it comes to seeking and choosing medical care, and much of the recent academic literature consequently focuses on them. For example, for female veterans who are poor, inconsistent access to food is associated with delaying access to health care and general poorer health outcomes.⁸ Delaying care for women veterans has been found to be fairly commonplace, with the primary barriers identified as cost, inability to seek care because of work, and issues with transportation.⁹ Female veterans who only use the VA have been shown to have worse overall physical and mental health than female veterans who only use non-VA sources of medical care; however, female veterans who use only the VA for health care have the most positive perceptions of the VA when compared with non-VA and mixed (VA and non-VA) female users in some studies.¹⁰ And a high number of rural veterans report receiving care from both VA and non-VA medical centers, mainly because they have a preferred provider prior to joining VA or because of a lack of VA providers in their area.¹¹

Hospital Comparison

Multiple studies have attempted to compare the quality of VA and non-VA medical centers using myriad measures: wait times, clinical indicators (i.e., mortality rates, readmission rates), and quality indicators (i.e., patient satisfaction, patient recommendation). As the MISSION Act was recently implemented and government data often take up to a year to be released, clinical and quality results post-MISSION Act are not yet available for analysis, but some recent studies can be used to gauge how the systems compare.

Although analytic approaches vary, it is clear from the literature that in seeking to compare the two sets of hospitals, demographic parameters must first be established for comparison. William G. Lehrman and colleagues conducted one of the most oft-cited studies comparing U.S. hospitals in terms of quality and clinical measures.¹² They compared 2,583 U.S. hospitals using six demographic variables as predictors (bed size, urban/rural, Census division, ownership, teaching status, and proportion of Medicare patient days), with HCAHPS quality and Hospital Quality Alliance (HQA) clinical measures as outcomes. Rebecca Anhang Price and colleagues chose four of these variables (bed size, Census division, rural/urban location, and teaching status) in comparing VA and non-VA medical centers on numerous clinical and patient experience outcomes.¹³ In keeping with both these studies, this CNAS study used bed size (small, medium, and large) and teaching status (major, minor, not) as predictors.¹⁴ It also used ownership (VA, nonprofit, and government), with private and proprietary hospitals excluded from analysis. Finally, it used Rural/Urban Continuum Code in order to apply data to MFRI’s existing mapping tool.

Findings from these comparisons are inconclusive, presenting few clear generalizations about how VA medical centers compare with their non-VA counterparts. Lehrman and colleagues, who did not report VA status separately, found that top performers in both clinical and patient experience outcomes were small, rural, in New England or West North Central Census division, and nonprofit.¹⁵ Top performers in patient experience measures alone were small, rural, East and South Central divisions, and government-owned. Top performers in clinical care alone were medium-to-large and urban, West North Central, and non-government-owned. Anhang Price and colleagues found that VA medical centers performed similar to or better than non-VA medical centers for most of the analyzed measures but that high variation, especially among VA centers, indicated a need for targeted quality improvement.¹⁶ Eddie Blay Jr. and colleagues found that the VA had better outcomes for six out of nine patient safety indicators (PSIs) but that non-VA hospitals performed better on patient experience and behavioral measures.¹⁷ And finally, Claire O’Hanlon’s research team conducted a systematic review and found that the VA often, but not always, performed better on quality measures.¹⁸

It is clear from the variation in these studies’ findings that the differences in clinical and patient experience outcomes between VA and non-VA medical centers deserve additional examination. This is especially salient given that national clinical and quality data are released intermittently, putting the comparisons in constant flux. Data released in summer 2019 may provide a strong picture of hospital quality immediately prior to implementation of the MISSION Act. The aim of this study was to provide a 1-to-1 comparison of VA and non-VA medical centers, with the goal of offering broader transparency to veterans and their families.

Methods

Demographic Measures

This research started with constructing a master hospital database using the Hospital Service Area File from the U.S. Centers for Medicare and Medicaid Services (CMS) website and the VA medical centers database from the U.S. Department of Homeland Security (DHS). To the best of our knowledge, the geographic and contact information for all U.S. hospitals was included in these two files. Additional demographic data were obtained from a variety of websites and databases as described in the following section. Data sets obtained from the American Hospital Association (AHA) were later used to confirm demographic data.

Coding

Proximity

Possible comparators were identified using the American Enterprise Institute’s (AEI) VA MISSION Act access map.¹⁹ Veterans use this online tool to determine drive time to the VA. Under the MISSION Act of 2018, veterans are able to access non-VA community care health care providers if they live more than 30 minutes’ driving time from a VA facility that offers similar care. AEI created this tool for veterans to determine whether they live outside or inside of that 30-minute driving time radius. The AEI tool uses OpenStreetMap and Open Source Routing Machine to plot polygons that delineate these areas. Hospitals within that 30-minute driving time radius were considered for comparison, unless a medical center was not identifiable in the 30-minute radius, in which case we coded medical centers that were closest to the diameter created by AEI’s program.

Rural-Urban Continuum

Next, hospitals were coded for their place on the Rural-Urban Continuum, a classification that further divides the two Office of Management and Budget (OMB) metro and nonmetro (metropolitan/micropolitan) categories into nine population-based codes. Codes were determined using the Measuring Communities mapping tool from MFRI and the Purdue Center for Regional Development.²⁰ The tool identifies the Rural-Urban Continuum Code (RUCC) for all counties in the United States using the Department of Veterans Affairs Rural Veterans Health Care Atlas.

Ownership

Hospitals were also coded for ownership, which was included in the DHS data and confirmed using AHA data. Because of the similarity in organizational structure to VA-run medical centers, we considered only hospitals that were run by governments or nonprofits. Government-run hospitals (operated by counties or municipalities) and nonprofit hospitals were expected to be more similar to VA medical centers than those run by for-profit companies or physician groups. Proprietary and private hospitals were ultimately excluded from the search for comparators as they were anticipated to be disproportionately out-of-network for community care per MISSION Act regulations.

Teaching Status

Next, hospitals were coded for whether or not they were a teaching hospital. This information was collected from an array of online outlets, including contacting the hospitals themselves, and verified using AHA databases. Hospitals were coded as nonteaching, minor teaching, or major teaching.

Bed Size

Finally, bed size was coded per the AHA’s classification system: Hospitals with fewer than 100 acute care beds were coded as small, hospitals with between 100 and 399 beds were coded as medium, and hospitals with 400 or more beds were coded as large.

Comparator selection

As proximity was the most important factor in identifying VA comparators, hospitals in the same ZIP code as their comparable VA were automatically considered as possible comparators. If a comparable hospital could not be located in the same ZIP code, the closest hospital was used as long as it was still within the 30-minute driving time radius identified with the AEI mapping tool. A total of 119 pairs of records were identified that were either in the same ZIP code or less than 30 minutes’ driving time from each other; each pair consisted of the VA and its comparator.

This left 17 VA medical centers without a clear comparator. In most of these cases, there were multiple possible comparators to be analyzed that fell within the 30-minute radius. In order to narrow those choices to one comparator, the four comparison categories of proximity, Rural-Urban Continuum Code, teaching hospital status, and bed size were tallied and the medical center with the most similarities was chosen as the comparator. In the case of a tie, categories were assigned a hierarchy in the following order: Rural-Urban Continuum Code, teaching hospital, and bed size (all possible comparators were either within the 30-minute radius or all were outside; there were no cases in which one was out and one was in). Medical centers with the same Rural-Urban Continuum Code were chosen first as comparators. Non-VA medical centers were considered as comparators if they had the same Rural-Urban Continuum Code as their closest VA medical center, were also teaching hospitals, and/or fell in the same bed size category as the VA. In the case of more than one possible comparator being different from the VA medical center on all fronts, proximity was the tiebreaker, with the closest medical center chosen as the comparator.

Quality Measures

The authors met with relevant stakeholders for a roundtable discussion to decide which clinical and quality measures would be best used to compare hospitals; stakeholders included policy think tanks, VA administrators, and researchers. The goal was to select measures that would be important regardless of the characteristics of the patient population or care specialties. Based on recommendations from these field experts, the authors initially selected the following measures: IMM-2, PSI 90, PSI 04, HCAHPS #18 and #19, and CTM-3.

IMM-2. IMM-2 is a measure of influenza immunization. It describes the percentage of patients screened for prior immunization upon admission and administered the flu vaccine if needed. The numerator for this measure is patients screened and administered the flu vaccine, and the denominator is patients 6 months and older who were discharged from the hospital. The authors were able to locate recent (2018) data for IMM-2 for both VA and non-VA medical centers through Hospital Compare on the Centers for Medicare and Medicaid Services website.

MRSA/CAUTI. PSI 90, the original measure suggested by stakeholders, is a composite that includes rates of 10 indicators related to hospital safety, including hospital falls with hip fracture, peri-operative hemorrhages or hematomas, and postoperative sepsis. We were unable to locate current VA data for these indicators. Because PSI 90 is described as providing “an overview of hospital-level quality as it relates to a set of potentially preventable hospital-related events associated with harmful outcomes for patients,” we instead chose the readily available measures of MRSA and CAUTI found in Hospital Compare that also describe quality related to potentially preventable hospital-related events.²¹ The MRSA measure describes the number of methicillin-resistant Staphylococcus aureus infections per 1,000 bed days. The CAUTI measure describes the number of catheter-associated urinary tract infections per 1,000 device days (days of catheter use).

PSI 04. PSI 04 describes deaths among surgical inpatients with serious treatable complications; in other words, if a patient goes into surgery, what is the chance that the patient dies of a serious treatable complication? We located both VA and non-VA data for PSI 04 from 2018. PSI 04 is expressed as the number of deaths per 1,000 patients who develop specific complications while hospitalized and is calculated by dividing the number of hospital deaths (numerator) by the number of discharges that fit certain inclusion and exclusion criteria (denominator).

HCAHPS #18 and #19 and CTM-3. HCAHPS is a CMS survey that measures the quality of patients’ hospital experience.²² It is a national survey, administered by individual hospitals and reported to CMS. Hospitals survey discharged patients between 48 hours and six weeks after discharge. Patients are asked 29 questions on a range of experience measures, including care from doctors and nurses, hospital experiences and environment, and understanding of care after returning home. HCAHPS #18 measures the patient’s overall rating of the hospital and describes the percentage of patients who assess the hospital as a 9 or 10 on a 10-point scale. HCAHPS #19 measures the percentage of patients who are “very likely” to recommend the hospital to friends or family. VA medical centers do not use HCAHPS, but the VA Survey of Healthcare Experiences of Patients (SHEP) used to evaluate VA medical center quality is also a national, standardized survey and contains the same items as the HCAHPS survey.²³ The CTM-3 is a subscale of HCAHPS. It is a care transition composite consisting of three items measuring a patient’s understanding of the patient’s postoperative care and describes the percentage of patients who responded “strongly agree” to composite questions. HCAHPS #18 and #19 for non-VA medical centers were available for 2018, and the comparable SHEP data were available for VA medical centers for 2018 as well. CTM-3 data for both VA and non-VA medical centers from 2018 were used for this study.

After collecting quality data, 11 additional paired records were eliminated for a lack of reported VA data (i.e., no values for any clinical or patient experience measures). This left 125 pairs, or 250 medical centers, for analysis.

Results

Descriptive statistics for all variables can be found in Table 1. Demographic and quality variables were assessed for normality, and heavily skewed (-1.0 > z > 1.0) variables were examined for influential outliers. Variables with extreme scores were Winsorized to limit their influence by replacing scores lower than the 5th percentile of each quality variable with the value of the 5th percentile and replacing scores greater than the 95th percentile with the scores of the 95th percentile value.

Table 1: Descriptive Statistics

Correlation coefficients were calculated using Spearman’s rho and are presented in Table 2. Correlations can tell us how the different demographic and quality variables are associated. In the combined VA and non-VA sample, rurality was moderately negatively associated with size (r_s = -0.450; p<0.01) and teaching status (r_s = -0.423; p<0.01), meaning that the more rural the medical center, the more likely it was to be small or nonteaching. Rural code was also moderately positively associated with the three HCAHPS variables (HCAHPS #18, HCAHPS #19, CTM-3; r_s = 0.429 - 0.515; p<0.05) in the VA sample, meaning that the more rural the medical center, the higher the score on those three patient experience measures. Rurality was also moderately negatively associated with MRSA (r_s = -0.317; p<0.01), meaning that being more rural was associated with lower MRSA scores. Size had a moderate, positive association with teaching status (r_s = 0.485; p<0.01), meaning that larger hospitals were more likely to be teaching hospitals. Size also had a small positive association with IMM-2 (r_s = 0.172), PSI 04 (r_s = 0.209), and MRSA (r_s = 0.268); p<0.01 and a small negative association with the CTM-3 (r_s = -0.298; p<0.01), meaning that being a larger hospital was associated with an increase in immunization, surgical deaths, and MRSA, and a decrease in patient-reported care transition. Finally, all three of the HCAHPS measures were strongly to very strongly intercorrelated (r_s = 0.664 - 0.920; p<0.01), as expected.

Table 2: Correlation Coefficients

** = significant at the 0.01 level;
* = significant at the 0.05 level

Results of independent samples t-tests are presented in Table 3. VA medical centers as a whole fared significantly better in measures of PSI 04/surgical deaths (t = -7.96 , df = 66.7, p < .01, 95 percent CI for mean difference -60.80 to -36.43) and CTM-3/care transition (t = 4.73 , df = 237, p < .01, 95 percent CI for mean difference 1.83 to 4.45). The VA fared worse in IMM-2/flu immunization (t = -6.91, df = 220.24, p < .01, 95 percent CI for mean difference -9.71 to -5.40) and HCAHPS #18/overall hospital rating (t = -2.16, df = 240, p < 0.05, 95 percent CI for mean difference -4.27 to -0.19). There was no significant difference between the two groups for CAUTI, MRSA, or HCAHPS #19/willingness to recommend.

Table 3: T-Test Results Comparing VA and Non-VA Hospital Quality Indicators

** significant at the 0.01 level;
* significant at the 0.05 level;
† Satterthwaite approximation employed due to unequal group variances

To compare medical centers at a granular level, z-scores were calculated using raw scores for the seven quality measures for the combined sample. These were performed using the grand mean for each measure. VA medical centers were coded relative to their comparator, with individual VA medical center standardized scores coded as greater than their civilian comparator, similar to their civilian comparator (0.5>z>-0.5), and lower than their comparator. For three measures (PSI 04, CAUTI, and MRSA), more than half of the pairwise data was missing because of one or more missing values per pair.

Table 4 includes frequency results for standardization. For IMM-2, 46.6 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 118 pairs with valid data). For PSI 04, 87.5 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 56 pairs with valid data). For CAUTI, 71.3 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 73 pairs with valid data). For MRSA, 72.4 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 47 pairs with valid data). For CTM-3, 82.6 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 115 pairs with valid data). For HCAHPS #18, 61.6 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 117 pairs with valid data). And for HCAHPS #19, 61.1 percent of VA hospitals performed similar to or better than their non-VA comparator (n = 118 pairs with valid data).

Table 4: Frequency Results for Standardization

Conclusions

The purpose of this study was to compare Veterans Affairs and civilian medical centers on various quality indicators related to clinical and patient experience outcomes. Each VA medical center was matched with a similar comparator, and outcome data were analyzed in aggregate and standardized scores. Consistent with other studies, results were mixed and thus generalizations about quality in the two populations cannot be made. The VA fared significantly better on measures of surgical deaths and patient hospital rating, although the surgical death data had a high percentage of missing data (34 percent) and equal group variances could not be assumed. The VA fared significantly worse on measures of in-hospital influenza vaccination and care transition, with equal group variances not assumed for the IMM-2. Results of this study mirrored those of Anhang Price et al. (2018), who found that the VA performed the same or better on measures of inpatient safety and mortality, and non-VA performed better on measures of readmission and some effectiveness measures.²⁴

Pairwise analysis offers some insight into how the two categories of medical centers compare at a granular level; for all but one measure (IMM-2), a majority of VA medical centers performed the same as or better than their civilian comparator when comparing standardized scores. This suggests that low opinion of the VA system, particularly by nonusers, could be misplaced when comparing VA centers to a medical center that bears close resemblance.

This study used 2018 data, collected pre-MISSION Act implementation. We recommend that additional analysis be conducted with 2019 data, the last collection period before implementation, so trends post-implementation can be analyzed with appropriate referents.