Written by Kimberly Silva, PhD, Senior R&D Consultant
As innovative assessment designers and users, we are constantly adapting the way we think about, build, and deliver our tools. Although our products and strategy may change, our primary goals remain the same – create measures that are relevant, psychometrically sound, and fair. Unfortunately, sometimes these goals are at odds, and we may feel stuck prioritising one objective over another. In this insight piece, I describe my recent experience with this challenge, share lessons learned, and offer practical advice for addressing a growing Diversity, Equity, and Inclusion (DEI) concern.
Designing a valid hiring assessment
In 2021, my team was asked to create a new measure to help screen and select manufacturing job candidates. The measure needed to be highly engaging, accessible on any device, and effectively measure the ability to ‘work accurately at a fast and steady pace’ – a competency we named Work Tempo. After ensuring Work Tempo was job related, my team got to work designing the measure. We created a prototype and thoroughly evaluated it according to industry standards. By and large, the measure was a success. It was accurate, consistent, and mirrored true manufacturing tasks. But there was one outstanding limitation – we found age-related score differences. Younger people were outperforming older people.
It’s common for ability tests to demonstrate age-related score differences (Wee, Newman, & Joseph, 2014), especially if the test has a speed component. As we age, our reaction time slows, and with it, the pace or rhythm at which we work (Dickerson et al., 2021; Gilles et al., 2022; Welford, 1988). Knowing this, we anticipated a similar relationship between age and Work Tempo and were prepared to modify the prototype.
DEI and fair hiring practices
There are cases where a measure demonstrates subgroup score differences but is still used in the hiring process (Outtz, 2002). For example, under many equal opportunity guidelines, tests that are job related and consistent with business necessity can be used for screening and selection, even if they disproportionally exclude persons from a protected class.
In other words, a measure can be legally permissible or ‘fair’ but not advance or align with DEI goals. DEI is about building a workforce that is representative of the population as a whole. It’s a genuine commitment to hiring people from different backgrounds, with different experiences, and with different perspectives. In the selection space, this means building measures that not only meet legal standards, but ethical ones, too. In this scenario, our challenge was to minimise age-related score differences without sacrificing measurement quality.
What is causing score differences?
While exploring the data, we made an interesting discovery. Another factor had magnified the score differences – accessory use. In the pilot studies, testers were allowed to use any accessory to complete the measure such as a touchscreen, mouse, or trackpad. Those who used a touchscreen performed more accurately and quickly than individuals who used a mouse or trackpad. Few older individuals used a touchscreen.
Upon further analysis, we found that older individuals (those 40+ years of age) who used a touchscreen performed about 36% better than their peers who used a trackpad and 9% better than peers who used a mouse. This held true even if the touchscreen user expressed little confidence with technology. Also, the score differences between individuals under 40 years old and 40+ were smaller when both age groups used a touchscreen, rather than a mouse or trackpad.
This was not something we expected to find. While many technology researchers agree that touchscreens are the fastest and most comfortable input device in general (Ulrich et al., 2015), age specific research says that older individuals tend to struggle with touchscreens and perform computer tasks better with a mouse, especially when the task has a moving display like the one in our prototype (Wood et al., 2005; Guedes, et al., 2020). Still, we found the reverse to be true – older individuals did very well with a touchscreen.
Hiring and age-related considerations
There is an important lesson to be learned from this research: Avoid making assumptions about age-groups (or any subgroup) that may hinder further exploration of score differences. Initially assuming that any measure with a speed component would lead to age-related score differences distracted us from investigating the impact of accessories. After discovering the joint influence of accessory and age on scores, we realised a new way for handling this age-related DEI concern. The best way to minimise score differences may require you to not only address the measure itself, but also the way people interact with the measure.
For those designing and using technologically advanced assessments, I recommend the following:
- Be cautious when creating or utilising measures with a speed component, as it may inadvertently exclude older job candidates. Use these types of tools when speed-related abilities are truly job related.
- Aim to standardise accessories used for testing, as differences could impact scores. Consider providing or recommending all candidates use (or don’t use) a particular accessory.
- Ensure job candidates are well-prepared and comfortable with their chosen accessory by offering ample practice opportunities and instruction before the assessment.
Sources:
Dickerson, A., Taylor, R. G., Register, J., & Miller, M. (2023). The Impact of Age, Sex, and Position on Visual-Motor Processing Speed and Reaction Time as Measured by the Vision Coach. Occupational Therapy In Health Care. doi: 10.1080/07380577.2023.2176965
Gilles, M. A., Gaudez, C., Savin, J., Remy, A., Remy, O., & Wild, P. (2022). Do age and work pace affect variability when performing a repetitive light assembly task?. Applied Ergonomics, 98, 103601.
Outtz, J. L. (2002). The role of cognitive ability tests in employment selection. Human performance, 15(1-2), 161-171.
Soares Guedes, L., CA Ribeiro, C., & Ounkhir, S. (2020, December). How Can We Improve the Interaction of Older Users With Devices?. In Proceedings of the 9th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (pp. 158-162).
Ulrich, T. A., Boring, R. L., & Lew, R. (2015). Control board digital interface input devices – touchscreen, trackpad, or mouse? Resilience Week (RWS) (pp. 1-6). Philadelphia, PA, USA. doi: 10.1109/RWEEK.2015.7287438.
U.S. Equal Employment Opportunity Commission. (2023, March 8). Employment tests and selection procedures. Retrieved from https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures
Welford, A. T. (1988). Reaction time, speed of performance, and age. Ann NY Acad Sci, 515, 1-17.
Wood, E., Willoughby, T., Rushing, A., Bechtel, L., & Gilbert, J. (2005). Use of computer input devices by older adults. Journal of Applied Gerontology, 24(5), 419-438.
