# PsyRAT: Extensible Open-Source Software for Applying Generalizability Theory to Assess Psychometric Reliability of Trial-Wise Scores and Optimize Tasks for RDoC

> **NIH NIH R01** · UNIVERSITY OF SOUTH FLORIDA · 2024 · $589,164

## Abstract

PROJECT SUMMARY/ABSTRACT
Neural biomarkers are an important focus of the NIMH Research Domain Criteria (RDoC) initiative, and they
are increasingly used in the context of genomic studies and clinical trials. The use of biomarkers with strong
psychometric reliability increases the likelihood of finding replicable effects, improves the validity of their
interpretation, and decreases the likelihood of missing real phenomena. Although fundamental psychometric
principles have long been a prominent concern among studies that use self-report measures, these principles
are underappreciated in studies of psychopathology that use biological measures. This lack of attention to
reliability limits the more widespread application of biomarkers in psychopathology and likely contributes to
replication problems. Generalizability theory is a multifaceted framework for identifying sources of
measurement error, and this framework is uniquely suited to assessing the reliability of biological measures
and to optimizing tasks for reliability. A critical need exists for tractable software to facilitate the application of
generalizability theory to time-frequency electroencephalography (EEG), event-related potentials (ERPs), facial
electromyography (EMG), and electrodermal activity (EDA). The objective of this project in response to PAR-
18-930 on measurement tool development for RDoC is to (i) develop an extensive treatment of generalizability
theory for psychopathology researchers, (ii) develop accessible software to implement it, (iii) show how to
apply these resources to optimize paradigms for individual-differences research, and (iv) disseminate the
software with a user-friendly guide. This project will facilitate the routine evaluation of reliability through these
specific aims: 1) Design and implement generalizability theory formulas for evaluating group- and subject-level
reliability for paradigm optimization; 2) Develop software to implement these formulas with data from widely
used psychophysiological software; 3) Apply results to optimize three commonly studied tasks; and 4) Develop
online educational material on the application of these resources to novel paradigms and measures. This
research project is innovative, because it represents a substantive departure from standard practice by shifting
the focus to the reliability of data from individuals, rather than groups, to identify sources of measurement error
and minimize their impact. This work promotes best practices in reporting psychometric properties of biological
measures and is applicable to data from any task with trial-wise scores. The resulting open-source toolbox, the
Psychophysiologist’s Reliability Analysis Toolbox (PsyRAT), can facilitate guidelines for optimizing paradigms,
making decisions about individual-subject data, and grounding individual-differences questions (central to
clinical research, especially for applications in precision medicine) in measures of reliability. The proposed
process...

## Key facts

- **NIH application ID:** 10873191
- **Project number:** 5R01MH128208-03
- **Recipient organization:** UNIVERSITY OF SOUTH FLORIDA
- **Principal Investigator:** Peter Eugene Clayson
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2024
- **Award amount:** $589,164
- **Award type:** 5
- **Project period:** 2022-08-04 → 2027-06-30

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10873191

## Citation

> US National Institutes of Health, RePORTER application 10873191, PsyRAT: Extensible Open-Source Software for Applying Generalizability Theory to Assess Psychometric Reliability of Trial-Wise Scores and Optimize Tasks for RDoC (5R01MH128208-03). Retrieved via AI Analytics 2026-05-23 from https://api.ai-analytics.org/grant/nih/10873191. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*