Understanding Consumer Engagement with VR Services

June 2016

Introduction

The Vocational Rehabilitation (VR) program assists people with disabilities in finding and maintaining employment.  However, approximately half of VR consumers leave the VR system prematurely when VR loses contact with a consumer (19%); the consumer refuses to continue services (17%); or the consumer fails to cooperate (14.2%; RSA 911, 2012).  Premature exit is costly for both VR agencies and consumers.  In 2012, premature exists cost VR approximately $365 million dollars in direct expenses related to evaluation, eligibility determination, and service provision prior to dropout.  For consumers, lack of follow-through with VR services was associated with lower employment rates compared to those who stayed attached to the program (Hayward & Schmidt-Davis, 2003).

Methods

To better understand factors leading to premature exit, we conducted a prospective study that tracked VR consumer movement through the VR process.  We recruited study participants when they enrolled in the VR program and collected data at six month intervals over 24 months. Surveys assessed barriers and facilitators to keeping consumers engaged over time. We hypothesized that consumer engagement and satisfaction with VR services would improve based on the counselor/consumer relationship, the pacing of VR service delivery, and the frequency of counselor/consumer contact early in the VR process.  At each six month interval, study participants indicated if they were still engaged with VR, and if not, why they had exited the program. We classified exit reasons into three categories including personal reasons, dissatisfied with services, and met goals.  For consumers still in the program, we assessed satisfaction with VR based on a single item question rated on a 4-point scale from “dissatisfied” to “satisfied”.

Participants

Three hundred and fifty-five consumers provided T1 (time 1) data, 226 (64%) provided T2 data, 212 (60%) provided T3 data, and 188 (53%) provided T4 data.  Decreasing sample size represents a combination of attrition from the study and exit from the VR program. There were no statistical differences between responders and non-responders in terms of rural/urban location, gender, race, employment status, or severity of disability.  However, on average, responders were older and more-educated than non-responders.

Counselor/consumer relationship

We measured the counselor/consumer relationship with a 12 item scale (CSS-12) that measured relationship, meeting effectiveness, professionalism, and responsiveness dimensions.  Consumers rated statements on a 4-point scale from dissatisfied to satisfied.  Mean values for individual statements fell between somewhat satisfied and satisfied on all items.  Overall satisfaction with VR services was significantly correlated to the CSS-12 at all time periods (p≤.000).

Delivery pacing

Consumers rated delivery pacing by indicating if services were “too slow”, “at a good pace” or “too fast”. Almost half felt VR pacing was “too slow” and results were consistent over time (46% at T1, 47% at T2, 43% at T3, and 49% at T4).  Only 1-3% felt that pacing was “too fast” at any time period.

We explored how delivery pacing was associated with a consumer’s stated exit reason from the program (collapsed across all time periods).

Table 1 shows exit reasons by delivery pace ratings.

Reason for Exit Services rated as “too slow” Services rated as “about the right pace”
Personal reasons (e.g. concerns about losing benefits, health issues, family issues, substance use issues, transportation barriers, moved) 50% 50%
Dissatisfied with services (e.g. not receiving desired services, VR stopped contacting me, problems with counselor, process taking too long) * 71% 29%
Met goals (e.g. got a job, received desired services * 28% 72%

* Column proportions significantly different (p ≤ .05)

Description. Fifty percent of consumers who exited the program for personal reasons rated service delivery as “too slow” as compared to 50 percent who rated it as “about the right pace”. Seventy-one percent of consumers who exited the program for dissatisfied with services reasons rated service delivery as “too slow” as compared to 29 percent who rated it as “about the right pace”. Twenty-eight percent of consumers who exited the program for met goals reasons rated service delivery as “too slow” as compared to 72 percent who rated it as “about the right pace”.  These differences were statistically significant.  Personal reasons including concerns about losing benefits, health issues, family issues, substance use issues, transportation barriers and moving.  Dissatisfied with services reasons included not receiving desired services, VR stopped contacting me, problems with counselor, and process taking too long. Met goals reasons including getting a job and receiving desired services.

Frequency of Counselor/Consumer Contact

We examined the rate of face to face and phone/email contact between counselors and clients at each data collection point for those still engaged with the program.  In the first six months of VR services, the typical consumer met with their counselor 2.8 times and communicated remotely 3.1 times.  These contact rates decreased over time.

Figure 1 shows the average contact rates over time.
Average Rate of Contact Over Time - description below image

Description.  This chart shows a decrease on counselor consumer contact over time.  The average number of face to face contacts decreased from 2.8 at Time 1 to 2.3 at Time 2, 1.7 at Time 3, and 1.5 at Time 4.  The average number of email/phone contacts changed from 3.1 at Time 1 to 2.6 at Time 2, 1.8 at Time 3, and 1.8 at Time 4.

We hypothesized that contact rates would predict satisfaction with VR services.  This hypothesis was supported in the data.  Tables 2 and 3 show contact rates by VR satisfaction.

Table 2: VR Satisfaction by Contact Rates at Time 1

Satisfaction with VR Face to Face Visits Phone/Email Visits
Dissatisfied µ = 1.89 µ = 1.73
Somewhat dissatisfied µ = 2.14 µ = 2.41
Somewhat satisfied µ = 2.62 µ = 3.52
Satisfied µ = 3.49 µ = 3.71
Significance p ≤ .000 p ≤ .001

Description.  Satisfaction with VR services was associated with increases in face to face visits (p less than or equal to .000) and phone/email visits (p less than or equal to .001) at Time 1.  The average number of face to face visits by satisfaction ratings were 1.89 visits for dissatisfied, 2.14 visits for somewhat dissatisfied, 2.62 visits for somewhat satisfied and 3.49 visits for satisfied.  The average number of phone/email visits by satisfaction ratings were 1.73 visits for dissatisfied, 2.41 visits for somewhat dissatisfied, 3.52 visits for somewhat satisfied and 3.71 visits for satisfied.  

Table 3: VR Satisfaction by Contact Rates at Time 3

Satisfaction with VR Face to Face Visits Phone/Email Visits
Dissatisfied µ = .58 µ = .79
Somewhat dissatisfied µ = 1.95 µ = 1.90
Somewhat satisfied µ = 1.41 µ = 2.24
Satisfied µ = 2.53 µ = 2.12
Significance p ≤ .012 p ≤ .007

Description.  Satisfaction with VR services was associated with increases in face to face visits (p less than or equal to .012) and phone/email visits (p less than or equal to .007) at Time 2.  The average number of face to face visits by satisfaction ratings were .58 visits for dissatisfied, 1.95 visits for somewhat dissatisfied, 1.41 visits for somewhat satisfied and 2.53 visits for satisfied.  The average number of phone/email visits by satisfaction ratings were .79 visits for dissatisfied, 1.90 visits for somewhat dissatisfied, 2.24 visits for somewhat satisfied and 2.12 visits for satisfied.  

Conclusions

The data suggest that more counselor/consumer engagement during the VR process and faster service delivery pacing improves satisfaction with VR services and reduces the number of clients exiting for “dissatisfied with services” reasons.  Premature exit might be reduced by increasing the number of opportunities for counselor/consumer contact using cost neutral communication channels (such as email and phone) and compressing upfront services to help consumers become more engaged in the VR process.  Even a small increase in consumer engagement could result in significant financial and employment outcomes.  Using 2012 data, a 5% increase in retention rates could save approximately $16.3 million and result in an additional 6,650 employment outcomes.

These results are part of the Premature Exit from VR Project.

Contributing Authors: Catherine Ipsen and Rebecca Goe