LDP Technical Brief 164006: Psychometric Reliability

LDP Technical Brief 164006: Psychometric Reliability

In the field of personnel psychology, reliability refers to the consistency and stability of a psychometric assessment, both within the assessment (meaning, its items relate to a similar underlying construct) and between administrations of the same assessment (meaning, its results are fairly consistent over a period of time). While less important than validity in evaluating an assessment’s effectiveness, reliability is more readily measurable (Cooper & Emory, 1995) and worthy of consideration. A reliable assessment is one that offers consistent results, providing users with confidence in the decisions made based on those results (Werner & DeSimone, 2009).  This is especially important when decisions involve workplace implications.

The psychometric reliability of the Leading Dimension Profile (LDP) was examined using three methods: Cronbach’s Alpha, Guttman’s Split-Half, and Test-Retest. The Cronbach’s Alpha coefficient focuses on content sampling and content heterogeneity. The Split-Half coefficient focuses on content sampling and internal consistency. The Test-retest coefficient focuses on time sampling (Anastasi  & Urbina, 1997). Each method offers both benefits and limitations for evaluating reliability.  As such, results for all three methods are presented below.

Cronbach’s Alpha Method

Arguably the most common analysis utilized for psychometric assessments, Cronbach’s Alpa provides an accurate evaluation regarding the consistency among participants’ responses (Biddle, 2005). This method utilizes specialized correlation formulas to specifically measure the homogeneity (sameness) of the questions asked by the assessment (that is, the degree to which questions point to a single construct or dimension) (Cooper & Emory, 1995). For example, Cronbach’s Alpha would evaluate the extent to which questions within the LDP’s Assertiveness dimension point to the personality construct of confidence in social and workplace settings.  The Cronbach’s Alpha method was applied to all LDP scorings collected between 2012 and 2016 (n=17,843), the results of which are reported in Table 1 below.

Table 1: Reliability Coefficients for Cronbach’s Alpha Method

Constructs (Factors and Dimensions) Items Coefficient
Achievement Drive Factor (Composite) 27 .87
Primary Achieving Dimensions (Facets)
    Intensity 8 .78
    Assertiveness 11 .80
    Risk Tolerance 8 .81
Secondary Achieving Dimensions
    Adaptability 8 .68
    Decision-making 8 .61
Relational Drive Factor (Composite) 29 .84
Primary Relating Dimensions (Facets)
    Affiliation 10 .73
    Consideration 10 .77
    Openness 9 .77
Secondary Relating Dimensions
    Status Motivation 14 .68
    Self-protection 9 .63
n = 17,843

Cronbach’s Alpha coefficients for the LDP’s composite factors ranged from .84 to .87, while the primary dimensions ranged from .73 to .81. For the secondary dimensions, this coefficient ranged from .61 to .68.

Guttman’s Split-Half Method

Like Cronbach’s Alpha method, Guttman’s Split-Half method provides an indication of the assessment’s reliability using content sampling. Participants’ responses within each dimension are split, resulting in two groups of questions for each dimension. The groups are then evaluated for correlation, within their respective dimension. Differences between participants’ scores on both groups may indicate random error variance, whereas the reliability coefficient would indicate the true variance for the given dimension (Anastasi & Urbina, 1997). The Split-Half method was applied to all LDP scorings collected between 2012 and 2016 (n=17,843), the results of which are reported in Table 2 below.

Table 2: Reliability Coefficients for Guttman’s Split Half Method

Constructs (Factors and Dimensions) Items Coefficient
Achievement Drive Factor (Composite) 27 .75
Primary Achieving Dimensions (Facets)
    Intensity 8 .77
    Assertiveness 11 .81
    Risk Tolerance 8 .82
Secondary Achieving Dimensions
    Adaptability 8 .54
    Decision-making 8 .55
Relational Drive Factor (Composite) 29 .84
Primary Relating Dimensions (Facets)
Affiliation 10 .76
    Consideration 10 .72
    Openness 9 .78
Secondary Relating Dimensions
    Status Motivation 14 .71
    Self-protection 9 .66
n = 17,843

Guttman’s Split-Half coefficients for the LDP’s composite factors ranged from .75 to .84, while the primary dimensions ranged from .72 to .82. For the secondary dimensions, this coefficient ranged from .54 to .71.

Test-Retest Method

Perhaps a more obvious approach to evaluating reliability is to administer the assessment on two occasions, and then measure the correlation coefficient between the occasions. This method, referred to as Test-Retest Analysis, measures the correlation between dimension scores between two administrations of the same questions. The length of the elapsed time between administrations may differ based on a variety of factors, including logistics and sample availability. In some cases, only a few minutes may elapse between administrations, while in other cases, several weeks may elapse. Practically speaking, the appropriateness of a given period between administrations will be influenced by the length of time required for the participant to forget the details of their prior responses (Rogelberg, 2002). Test-retest correlations will tend to decline progressively as the elapsed time between administrations grows (Anastasi & Urbina, 1997), especially as external and uncontrolled factors may contaminate or distort reliability. In any case, the upper limit on the elapsed time rarely should surpass 180 days (Cascio, 1998; Cooper & Emory, 1995).

Despite the relative simplicity of the procedure, some argue that Test-Retest procedures may not be an appropriate technique for evaluating assessments of a psychological nature (Anastasi & Urbina, 1997). A participant’s experiences, training, fatigue, or contextual conditions may impact the consistency of responses from one administration to the next. Such impacts result in potential sources of error within the measurement of specified dimensions. For example, a reliability coefficient of .87 suggests that nearly 90% of the differences in dimensions results can be attributed to systematic variance in the dimension measured, with slightly more than 10% being attributed to error variance. Practically speaking, reliability coefficients are meant to provide an estimate of the proportion of total variance that is systematic or “true” variance (Cascio, 1998). With such limitation noted, the Test-Retest method was applied to LDP scorings collected as part of training and development programs, during or subsequent to which a second LDP had been administered to the same participants (n=251).  Results of the Test-Retest method are reported in Table 3 below.

Table 3: Reliability Coefficients for Test-Retest Method

Constructs (Factors and Dimensions) Items Coefficient
Achievement Drive Factor (Composite) 27 .81
Primary Achieving Dimensions (Facets)
    Intensity 8 .76
   Assertiveness 11 .81
    Risk Tolerance 8 .72
Secondary Achieving Dimensions
    Adaptability 8 .73
    Decision-making 8 .67
Relational Drive Factor (Composite) 29 .78
Primary Relating Dimensions (Facets)
    Affiliation 10 .66
    Consideration 10 .70
    Openness 9 .76
Secondary Relating Dimensions
    Status Motivation 14 .75
    Self-protection 9 .74
n = 251

Average elapsed days = 134.18 days

Table 4: Reliability Coefficients for Test-Retest Method by Period (Elapsed Days between LDP administrations)

Constructs (Factors and Dimensions) Items ≤ 90 days ≤ 180 days ≤ 270 days ≤ 360 days
Achievement Drive Factor (Composite) 27 .84 .82 .81 .83
Primary Achieving Dimensions (Facets)
    Intensity 8 .79 .75 .73 .75
    Assertiveness 11 .83 .83 .81 .84
    Risk Tolerance 8 .77 .75 .74 .74
Secondary Achieving Dimensions
    Adaptability 8 .78 .72 .72 .73
    Decision-making 8 .69 .72 .71 .70
Relational Drive Factor (Composite) 29 .81 .80 .80 .79
Primary Relating Dimensions (Facets)
    Affiliation 10 .72 .69 .69 .67
    Consideration 10 .70 .72 .71 .71
    Openness 9 .77 .76 .77 .77
Secondary Relating Dimensions
    Status Motivation 14 .80 .77 .77 .76
    Self-protection 9 .74 .73 .73 .73
Average Elapsed Days 18.34 31.98 39.31 57.48
Sample Size (n) 173 197 205 219
n = 251

Test-Retest coefficients for the LDP’s primary (composite) factors ranged from .79 to .84, while the primary dimensions ranged from .67 to .84. For the secondary dimensions, this coefficient ranged from .69 to .80.

Geographic Reliability Comparison

The Cronbach’s Alpha Method was repeated to compare reliability coefficients among the three major geographic regions within which the LDP is distributed. These regions are: ASIA-PAC, including Australia, Hong Kong, Indonesia, Malaysia, New Zealand, Singapore; EU, including Ireland and the United Kingdom; and, the United States.

Table 5: Reliability Coefficients for Cronbach’s Alpha Method by Geographic Region

Constructs (Factors and Dimensions) Items ASIA-PAC EU US
Achievement Drive Factor (Composite) 27 .87 .85 .85
Primary Achieving Dimensions (Facets)
    Intensity 8 .77 .76 .75
    Assertiveness 11 .80 .77 .77
    Risk Tolerance 8 .82 .80 .80
Secondary Achieving Dimensions
    Adaptability 8 .70 .63 .66
    Decision-making 8 .63 .62 .59
Relational Drive Factor (Composite) 29 .85 .85 .83
Primary Relating Dimensions (Facets)
    Affiliation 10 .72 .73 .73
    Consideration 10 .78 .77 .76
    Openness 9 .78 .76 .75
Secondary Relating Dimensions
    Status Motivation 14 .72 .68 .65
    Self-protection 9 .60 .59 .64
Sample Size (n)

8,107

740

8,959

Across all geographic samples, Test-Retest coefficients for the LDP’s primary (composite) factors ranged from .83 to .87, while the primary dimensions ranged from .72 to .82. For the secondary dimensions, this coefficient ranged from .59 to .72.

Conclusion

While the Uniform Guidelines for Employment Selection Procedures do not provide a minimum acceptable level for reliability, the US Department of Labor (DOL) suggests that reliability coefficients of .70 and higher generally offer adequate reliability for employment assessment purposes (Biddle, 2005; DOL, 1999). The DOL indicates that assessments should not be accepted or rejected based on reliability coefficients, although coefficient values of below .70 may offer limited applicability (DOL, 1999). According to Cascio (1998), reliabilities as low as .70 are proven useful, with lower reliabilities offering potential applications for research purposes.

Reliability is an important characteristic in evaluating the effectiveness and suitability of employment assessments, in that it creates an upper limit on validity estimates (SIOP, 2003). Since these estimates impact inferences made regarding how the assessment relates to specific behaviors (and their consequences for job performance), it is important to measure and report reliability within the context of psychometric considerations (SIOP, 2003).  As such, the psychometric reliability of the LDP has been reported in this technical brief using Cronbach’s Alpha, Guttman’s Split Half, and Test-Retest procedures.  Questions regarding these findings may be directed to Dr. Douglas Waldo at doug.waldo@leadingdimensions.com.

References

Anastasi, A. & Urbina, S. (1997). Psychological Testing, 7th ed., Upper Saddle River: Prentice Hall.

Biddle, D. (2005). Adverse Impact and Test Validation: A Practitioner’s Guide to Valid and Defensible Employment Testing. Burlington: Gower Publishing Ltd.

Cascio, W. F. (1998). Applied Psychology in Human Resource Management, 5th ed., Upper Saddle River: Prentice Hall.

Copper, D. R. & Emory, C W. (1995). Business Research Methods, 5th ed., Chicago: Irwin.

Rogelberg, S. G. (2002). Handbook of Research Methods in Industrial and Organizational Psychology. Malden: Blackwell Publishers.

Society for Industrial and Organizational Psychology (2003). Principles for the Validation and Use of Personnel Selection Procedures. Bowling Green: Society for Industrial and Organizational Psychology, Inc.

United States Department of Labor Employment and Training Administration (1999). Testing and Assessment: An Employer’s Guide to Good Practices. Washington, DC: US Department of Labor.

United States Equal Employment Opportunity Commission (1978). Uniform Guidelines on Employee Selection Procedures. Washington, DC: EEOC.

Werner, J. M. & DeSimone, R. L. (2009). Human Resource Development, 5th ed., Mason: South-Western Cengage Learning.