Response to ImPACT Reliability Study

by Dr. Michael Collins

Director, University of Pittsburgh Sports Medicine Program, ImPACT co-founder

PITTSBURGH -- We commend Dr. Resch and colleagues for researching computerized neurocognitive testing, and, in fact, many other independent, peer-reviewed studies have already been published examining this important psychometric issue. For example, in a large independent study of college athletes, Schatz (2009) reported that good stability occurred with the ImPACT test battery and that the study “documented the stability of baseline pre-season cognitive assessments over a period of two years” and that ‘mean ImPACT composite scores and symptoms showed little variation between the two assessments”. Schatz concluded that ImPACT is a reliable measure of cognitive function in assessing sports related concussion. Moreover, another, significantly larger independent study involving nearly 400 athletes recently was accepted for publication and is forthcoming. In this study, test-retest correlations for the online version of ImPACT have been found to be good to excellent. In fact, reliability of the online version of the program was found to be significantly improved relative to the 2009 study, where the desktop version of ImPACT was utilized.

Miller and colleagues (2007) also tested athletes with ImPACT at pre-season, mid-season and post-season and found stability of these scores throughout the season in non-concussed athletes:“ImPACT scores are not significantly altered by a season of repetitive contact in collegiate football athletes who have not sustained a concussion”. In short, these and numerous others among the nearly 100 published studies utilizing ImPACT, have systematically showed the stability, validity, sensitivity and added value of this system in assessing sports-related concussion.


Although Dr. Resch’s study has yet to be peer reviewed or published and we are not privy to the data or results, it is important to note that the study was conducted in a smaller group of athletes than previous studies (n = 45), and perhaps, most important, it appears these researchers conducted memory testing with an alternative measure just prior to administering ImPACT. This is similar to the referenced work conducted by Broglio and colleagues, where they administered ImPACT, as well as two other computerized and paper-and-pencil neurocognitive measures. In short, it is not surprising at all to see less than adequate reliability with these methodologies. As ImPACT is a cognitive tool, and these other measures utilize alternative/interfering memory stimuli, there are significant interference effects with the data. We would not expect ImPACT to be reliable in conjunction with other memory tests administered, as interference effects are fully expected to occur with such a methodology. This likely explains why the ImPACT memory reliability scores in the Resch and Broglio studies are suboptimal. In clinical practice, we would never administer competing memory stimuli just prior to administering ImPACT, as this would surely affect the integrity of the scores.

Within the context of the outlined comments, however, we completely concur with Dr. Resch in stating that multiple assessment modalities should be utilized in evaluating and treating sports concussion. Dr. Resch reaches the same overall conclusion as ImPACT has practiced for years, and he summarizes our approach almost to the letter. We advocate the use of the ImPACT test as a tool -- one of several in the clinician's toolbox for assessing and managing concussion. We always conduct a very detailed clinical evaluation, including an extensive clinical interview, vestibular and balance testing, as well as computerized (and sometimes paper and pencil) neurocognitive testing that help to elucidate the overall clinical picture. With all of these data, we attempt to make responsible decisions regarding injury severity, considerations for academic accommodation during recovery, appropriate physical exertion levels for the concussed athlete, and for ultimately determining when athletes may return to play in a safe manner. Concussion management deserves a detailed, comprehensive and well informed evaluation -- and ImPACT is one tool to help determine these issues. We agree wholeheartedly with his comments in this respect.

Questions/comments?  Contact Jean Rickerson at jean@sportsconcussions.org