Test-Retest Reliability Study of MAPP
August 11, 1997
Executive Summary for Reliability Study of the Motivational Appraisal of Personal Potential (MAPP)
Test-retest reliability of the Motivational Appraisal of Personal Potential (MAPP) instrument was established by a
scientific study completed in July, 1997. While .50 is an acceptable reliability coefficient, MAPP exceeds standards with larger
numbers showing consistency over time. The median test-retest correlation for the worker trait outcomes was .95, and for job
ratings was .90. These high correlations for 72 outcomes in nine categories were based on the weighted outcomes that reflected
individual characteristics. Worker trait outcomes and job ratings make up the information that workers, employers, and career
consultants use when working with MAPP in the world of work. The nine categories of worker trait outcomes and job ratings are:
aptitude for the job; capacity for mathematical applications; capacity for usage of language; interest in job contents; relationship
to data; relationship to people; relationship to things; relationship to reasoning; and temperament for the job. The median
test-retest correlation for item responses was .71. This high correlation was based on people tending to give the same
answers to 213 choices on 71 questions each time the test was taken. These responses are the basis for all the interpretations
given to workers and employers.
The study used a sample of 32 people who were employed full time. They took the MAPP test twice, with an interval of
approximately 9 months between tests (279.5 median number of days, ranging from 152 to 496 days). The gender composition of the
sample was 12 women (37.5%) and 20 men (62.5%). The overall median age was 49, with ages ranging from 26 to 67 years. Those in
the study sample had diverse occupations representing a full range of professional work. Evidence for the stability of MAPP is
based on the fact that the time interval between tests contained major life changes for 11 (34.38%) of the participants in the
study.
Reliability Study of the Motivational Appraisal of Personal Potential (MAPP)
A research study to establish test-restest reliability of the MAPP instrument was completed in July, 1997. This type
of reliability was chosen for the study because an important feature of MAPP is that it be stable over time. Stability over time
allows the test taker to use the results confidently for pursuing vocational choices as well as developing personal characteristics
that would upgrade job qualifications. For employers, stability over time provides confidence that the test results can be used to
carefully select candidates for job positions after the test is on file. Moreover, employers can use the test results with certainty
for employee development in the workplace over time after the test has been taken.
Correlations between First and Second Tests
While .50 is an acceptable reliability coefficient for true score variability and measurement error, the MAPP
instrument exceeds standards with larger numbers as explained below, thus showing great consistency over time.
Reliability of MAPP was established in three different ways using the test-retest method. First, the worker trait
outcomes based on the responses to the test items were compared. Second, the ratings for jobs based on the responses to the test
items were compared. Third, answers to the test items were used to determine how people answered the questions from one time to
the next. Each of these ways of establishing reliability is discussed in the following paragraphs.
The median correlation for the worker trait outcomes based on the first and second responses to MAPP items was .95.
The individual correlations for those included in the sample ranged from .66 to .99. The calculations were based on 72 outcomes
in nine categories. The interpretation of this correlation about the differences in worker trait outcomes is that 95% reflects
true variability, while only 5% of the differences can be attributed to measurement error. A perfect reliability would be a
coefficient of 1.0, showing 100% true score consistency, and 0% measurement error. Total lack of reliability would be a coefficient
of .0, showing 0% true score consistency, and 100% measurement error.
The median correlation for the job ratings based on the first and second responses to MAPP items was .90.
The individual correlations for those included in the sample ranged from .57 to .98. The calculations were based on 72 ratings
in nine categories. The interpretation of this correlation is the same as for the first correlation. In other words, 90% reflects
true variability in the differences in job ratings, while only 10% is due to measurement error.
The median correlation between first and second responses to MAPP items was .71. The individual correlations
for the 32 people in the sample ranged from .52 to .92. The calculations were based on responses to MAPP's 213 choices for 71 items.
The interpretation of the correlation is the same as for the other two. In other words, 71% reflects true score variability in the
differences in responses, while only 29% is due to measurement error.
The understanding of the differences among correlations is based on the structure of MAPP. The first and
second correlations are quite similar because they are both referring to the outcomes of MAPP and reflect the consistency of the
test to describe individuals. The first correlation (.95) is based on the worker trait outcomes that weights the responses to the
test items for 72 different trait descriptions. The values of the worker trait outcomes can vary from 0 to 100. The second correlation
(.90) is based on job ratings that directly correspond to the worker trait outcomes, but the ratings only have possible values
ranging from 1 through 5. Thus, the smaller range of possible values accounts for the reduction in the correlation from .95 to .90.
The third correlation (.71) is based on item responses and reflects the consistency of the person taking the test to
answer the same way at both times when taking the test. The other two correlations are higher because they refer to the interpretation
of item responses through a weighting process that reflects characteristics of the individuals taking the test. This third
correlation pertains to repeating the same responses to the 213 choices for the 71 items each time the test was taken.
All three types of information are valuable for the research study to establish reliability. However, the first two
correlations are especially pertinent, because the worker trait outcomes and job ratings make up the information that workers,
employers, and career consultants use when working with MAPP in the world of work. The nine categories of worker trait outcomes
and job ratings are: aptitude for the job; capacity for mathematical applications; capacity for usage of language; interest in
job contents; relationship to data; relationship to people; relationship to things; relationship to reasoning; and temperament
for the job. The third correlation is also critical because the responses that people make to individual items form the basis
for all of the interpretations of the test.
Sample for the Test-Retest Study
The study used a sample of 32 people who were employed full time. They took the MAPP test twice, with an interval of
approximately 9 months between tests (279.5 median number of days, ranging from 152 to 496 days). The gender composition of the
sample was 12 women (37.5%) and 20 men (62.5%). The overall median age was 49, with ages ranging from 26 to 67 years. This range
indicates that the ages typically found in workers has been represented in the sample. In addition, this broad range was found for
both men (from 26 to 67 years) and women (from 26 to 54). Table 1 shows the data by individual person for the study of reliability.
Those in the study sample had diverse occupations representing a full range of professional work. They were selected
because most of them were involved in some way with the test developers, as employees, distributors, employees of customers, or
acquaintances of employees. The occupations represented by those taking the tests included; account executive; attorney;career
consultant; company president; consultant; corporate consultant; director of training; executive coach; finance and insurance
manager; finance lease manager; finance manager; independent sales agent; investment broker; marketing manager; multi-media developer
for television; network administrator; office administrator; personnel consultant; private investor; promotional consultant; sales
agent; self-employed; sales consultant; seller of insurance; student; teacher; vice-president of professional services; and
vice-president of sales and marketing.
The reliability of MAPP remains high even though major changes happen in a person's life. Evidence for the stability
of MAPP is based on the fact that the time interval between tests contained major life changes for 11 (34.38%) of the participants
in the study. Changes that those people experienced included changing types of work; changing locations of work; changing from
searching for jobs to working at them; and broken relationships. However, these life changes were not related to the strength of
the correlations between the first and second times that the tests were taken. Rather, changes in life experiences were found as
often for those with higher correlations between test administrations as for those with the lower correlations.
Continuing Research about the Reliability of MAPP
Research on MAPP is continuous to maintain high levels of reliability. Other areas of outcomes to be studied are
vocational traits, personal traits, and learning styles. Within vocational traits are 19 areas with 111 outcomes. Personal traits
are categorized into nine areas with 109 outcomes. Learning styles have six areas with 51 outcomes.
Studies will be done to look at reliability within specific occupations to make sure that all kinds of workers
respond to items consistently over time, with consistent outcomes in worker traits, job ratings, vocational traits, personal traits,
and learning styles. Not only the types of workers, but also the types of work experience and the length of time on the job will
be considered in terms of reliability for test items and interpretations. Research on reliability will also consider the stability
of responses within different cultures and geographic locations.
View Data Study Table
|
Information about the Assessment