Examine individual changes

''''Construct validity''' is "the degree to which a test measures what it claims, or purports, to be measuring."<ref>{{cite book |last=Brown |first=J. D. |year=1996 |title=Testing in language programs |publisher=Upper Saddle River, NJ: Prentice Hall Regents |url=http://jalt.org/test/bro_8.htm}}</ref><ref name="Cronbach55">{{cite journal|last=Cronbach |first=L. J. |last2= Meehl |first2=P.E. |year=1955 |title=Construct Validity in Psychological Tests |journal=Psychological Bulletin |volume=52 |pages=281–302. |url=http://psychclassics.yorku.ca/Cronbach/construct.htm |doi=10.1037/h0040957 |pmid=13245896 |issue=4}}</ref><ref name="Polit">Polit DF Beck CT (2012). Nursing Research: Generating and Assessing Evidence for Nursing Practice, 9th ed. Philadelphia, USA: Wolters Klower Health, Lippincott Williams & Wilkins</ref> In the classical model of [[test validity]], construct validity is one of three main types of validity evidence, alongside [[content validity]] and [[criterion validity]].<ref name="guion1980">{{cite journal |last=Guion |first=R. M. |year=1980 |title=On trinitarian doctrines of validity |journal=[[Professional Psychology]] |volume= 11 |pages=385–398 |doi=10.1037/0735-7028.11.3.385}}</ref><ref name="Brown86">{{cite book|last= Brown |first=J. D. |year=1996 |title=Testing in language programs |publisher=Upper Saddle River, NJ: Prentice Hall Regents}}</ref> Modern validity theory defines construct validity as the overarching concern of validity research, subsuming all other types of validity evidence.<ref name="messick1995">{{cite journal |last=Messick |first=S. |year=1995 |title=Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning |journal=American Psychologist |volume=50 |pages=741–749 |doi=10.1037/0003-066x.50.9.741}}</ref><ref name="Schotte97">{{cite journal |last=Schotte |first=C. K. W. |last2=Maes |first2=M. |last3=Cluydts |first3=R. |last4=De Doncker |first4=D. |last5=Cosyns |first5=P. |year=1997 |title= Construct validity of the Beck Depression Inventory in a depressive population |journal=Journal of Affective Disorders |volume=46 |issue=2 |pages=115–125. |doi=10.1016/s0165-0327(97)00094-3}}</ref> Construct validity is the appropriateness of inferences made on the basis of observations or measurements (often test scores), specifically whether a test measures the intended construct. Constructs are abstractions that are deliberately created by researchers in order to conceptualize the [[latent variable]], which is correlated with scores on a given measure (although it is not directly observable). Construct validity examines the question: Does the measure behave like the theory says a measure of that construct should behave? Construct validity is essential to the perceived overall validity of the test. Construct validity is particularly important in the [[social sciences]], [[psychology]], [[psychometrics]] and language studies. Psychologists such as [[Samuel Messick]] (1998) have pushed for a unified view of construct validity "...as an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores..."<ref name="Messick">{{cite journal |last=Messick |first=Samuel |year=1998 |title=Test validity: A matter of consequence |journal= Social Indicators Research |volume= 45 |issue=1-3 |pages= 35–44|doi= 10.1023/a:1006964925094 }}</ref> Key to construct validity are the theoretical ideas behind the trait under consideration, i.e. the concepts that organize how aspects of [[personality]], [[intelligence]], etc. are viewed.<ref name="Pennington">{{cite book |last=Pennington |first=Donald |year=2003 |title=Essential Personality. |publisher=Arnold. |ISBN= 0-340-76118-0}}</ref> [[Paul Meehl]] states that, "The best construct is the one around which we can build the greatest number of inferences, in the most direct fashion."<ref name="Cronbach55" /> Scale purification, i.e. "the process of eliminating items from multi-item scales" (Wieland et al., 2017) can influence construct validity. A framework presented by Wieland et al. (2017) highlights that both statistical and judgmental criteria need to be taken under consideration when making scale purification decision.<ref>Wieland, A., Durach, C.F., Kembro, J. & Treiblmaier, H. (2017), Statistical and judgmental criteria for scale purification, Supply Chain Management: An International Journal, Vol. 22, No. 4, https://doi.org/10.1108/SCM-07-2016-0230</ref> == History == Throughout the 1940s scientists had been trying to come up with ways to validate experiments prior to publishing them. The result of this was a myriad of different validities ([[intrinsic validity]], [[face validity]], [[logical validity]], [[empirical validity]], etc.). This made it difficult to tell which ones were actually the same and which ones were not useful at all. Until the middle of the 1950s there were very few universally accepted methods to validate psychological experiments. The main reason for this was because no one had figured out exactly which qualities of the experiments should be looked at before publishing. Between 1950 and 1954 the APA Committee on Psychological Tests met and discussed the issues surrounding the validation of psychological experiments.<ref name="Cronbach55"/> Around this time the term construct validity was first coined by [[Paul Meehl]] and [[Lee Cronbach]] in their seminal article " [[Construct Validity In Psychological Tests|Validity In Psychological Tests]]". They noted the idea of construct validity was not new at that point. Rather, it was a combinations of many different types of validity dealing with theoretical concepts. They proposed the following three steps to evaluate construct validity: # articulating a set of theoretical concepts and their interrelations # developing ways to measure the hypothetical constructs proposed by the theory # empirically testing the hypothesized relations<ref name="Cronbach55"/> Many psychologists note that an important role of construct validation in [[psychometrics]] was that it place more emphasis on theory as opposed to validation. The core issue with validation was that a test could be validated, but that did not necessarily show that it measured the theoretical construct it purported to measure. Construct validity has three aspects or components: the substantive component, structural component, and external component.<ref name= "Loevinger">{{cite journal | author = Loevinger J | year = 1957 | title = Objective Tests As Instruments Of Psychological Theory: Monograph Supplement 9 | url = | journal = Psychological reports | volume = 3 | issue = 3| pages = 635–694 | doi=10.2466/pr0.1957.3.3.635}}</ref> They are related close to three stages in the test construction process: constitution of the pool of items, analysis and selection of the internal structure of the pool of items, and correlation of test scores with criteria and other variables. In the 1970s there was growing debate between theorist who began to see construct validity as the dominant model pushing towards a more unified theory of validity and those who continued to work from multiple validity frameworks.<ref name="Kane06">{{cite journal |last=Kane |first=M. T. |year=2006 |title= Validation. |journal=Educational measurement |volume=4 |pages=17–64.}}</ref> Many psychologists and education researchers saw "predictive, concurrent, and content validities as essentially ''ad hoc'', construct validity was the whole of validity from a scientific point of view"<ref name="Loevinger"/> In the 1974 version of ''The [[Standards for Educational and Psychological Testing]]'' the inter-relatedness of the three different aspects of validity was recognized: "These aspects of validity can be discussed independently, but only for convenience. They are interrelated operationally and logically; only rarely is one of them alone important in a particular situation". In 1989 Messick presented a new conceptualization of construct validity as a unified and multi-faceted concept.<ref name="Messick89">{{cite book |last=Messick, |first=S. |year=1989 | chapter=Validity. |editor=R. L. Linn (Ed.), |title=Educational Measurement (3rd ed., pp. 13-103). |publisher=New York: American Council on Education/Macmillan}}</ref> Under this framework, all forms of validity are connected to and are dependent on the quality of the construct. He noted that a unified theory was not his own idea, but rather the culmination of debate and discussion within the scientific community over the preceding decades. There are six aspects of construct validity in Messick's unified theory of construct validity.<ref name="Messick95">{{cite journal |last=Messick, |first=S. |year=1995 |title=Standards of validity and the validity of standards in performance assessment. |journal=Educational Measurement: Issues and Practice |volume=14 |issue=4, |pages=5–8. |doi=10.1111/j.1745-3992.1995.tb00881.x}}</ref> They examine six items that measure the quality of a test's construct validity: #'''Consequential''' – What are the potential risks if the scores are, in actuality, invalid or inappropriately interpreted? Is the test still worthwhile given the risks? #'''Content''' – Do test items appear to be measuring the construct of interest? #'''Substantive''' – Is the theoretical foundation underlying the construct of interest sound? #'''Structural''' – Do the interrelationships of dimensions measured by the test correlate with the construct of interest and test scores? #'''External''' – Does the test have convergent, discriminant, and predictive qualities? #'''Generalizability''' – Does the test generalize across different groups, settings and tasks? How construct validity should be properly viewed is still a subject of debate for validity theorists. The core of the difference lies in an [[epistemology|epistemological]] difference between [[positivist]] and [[postpositivist]] theorists. == Evaluation == Evaluation of construct validity requires that the correlations of the measure be examined in regard to variables that are known to be related to the construct (purportedly measured by the instrument being evaluated or for which there are theoretical grounds for expecting it to be related). This is consistent with the [[multitrait-multimethod matrix]] (MTMM) of examining construct validity described in Campbell and Fiske's landmark paper (1959).<ref name="Campbell"/> There are other methods to evaluate construct validity besides MTMM. It can be evaluated through different forms of [[factor analysis]], [[structural equation modeling]] (SEM), and other statistical evaluations.<ref name="Hammond96">Hammond, K. R., Hamm, R. M., & Grassia, J. (1986). Generalizing over conditions by combining the multitrait multimethod matrix and the representative design of experiments (No. CRJP-255A). Colorado University At Boulder Center For Research On Judgment And Policy.</ref><ref>{{cite journal |author1=Westen Drew |author2=Rosenthal Robert | year = 2003 | title = Quantifying construct validity: Two simple measures | url = | journal = Journal of Personality and Social Psychology | volume = 84 | issue = 3| pages = 608–618 | doi=10.1037/0022-3514.84.3.608}}</ref> It is important to note that a single study does not prove construct validity. Rather it is a continuous process of evaluation, reevaluation, refinement, and development. Correlations that fit the expected pattern contribute evidence of construct validity. Construct validity is a judgment based on the accumulation of correlations from numerous studies using the instrument being evaluated.<ref>Peter, J. P. (1981). Construct validity: a review of basic issues and marketing practices. Journal of Marketing Research, 133-145.</ref> Most researchers attempt to test the construct validity before the main research. To do this [[pilot studies]] may be utilized. Pilot studies are small scale preliminary studies aimed at testing the feasibility of a full-scale test. These pilot studies establish the strength of their research and allow them to make any necessary adjustments. Another method is the known-groups technique, which involves administering the measurement instrument to groups expected to differ due to known characteristics. Hypothesized relationship testing involves logical analysis based on theory or prior research.<ref name="Polit"/> [[Intervention studies]] are yet another method of evaluating construct validity. Intervention studies where a group with low scores in the construct is tested, taught the construct, and then re-measured can demonstrate a test's construct validity. If there is a significant difference pre-test and post-test, which are analyzed by statistical tests, then this may demonstrate good construct validity.<ref>{{cite journal |author1=Dimitrov D. M. |author2=Rumrill Jr P. D. | year = 2003 | title = Pretest-posttest designs and measurement of change | url = | journal = Work: A Journal of Prevention, Assessment and Rehabilitation | volume = 20 | issue = 2| pages = 159–165 }}</ref> ===Convergent and discriminant validity=== {{Main| convergent validity| discriminant validity }} Convergent and discriminant validity are the two subtypes of validity that make up construct validity. Convergent validity refers to the degree to which two measures of constructs that theoretically should be related, are in fact related. In contrast discriminant validity tests whether concepts or measurements that are supposed to be unrelated are, in fact, unrelated.<ref name="Campbell">{{cite journal | author = Campbell D. T. | year = 1959 | title = Convergent and discriminant validation by the multitrait-multimethod matrix | url = | journal = Psychological Bulletin | volume = 56 | issue = | pages = 81–105 | doi=10.1037/h0046016}}</ref> Take, for example, a construct of general happiness. If a measure of general happiness had convergent validity, then constructs similar to happiness (satisfaction, contentment, cheerfulness, etc.) should relate closely to the measure of general happiness. If this measure has discriminate validity, then constructs that are not supposed to be related to general happiness (sadness, depression, despair, etc.) should not relate to the measure of general happiness. Measures can have one of the subtypes of construct validity and not the other. Using the example of general happiness, a researcher could create an inventory where there is a very high correlation between general happiness and contentment, but if there is also a significant correlation between happiness and depression, then the measure's construct validity is called into question. The test has convergent validity but not discriminant validity. === Nomological network === {{Main|nomological network}} Lee Cronbach and Paul Meehl (1955)<ref name="Cronbach55"/> proposed that the development of a nomological net was essential to measurement of a test's construct validity. A [[nomological network]] defines a construct by illustrating its relation to other constructs and behaviors. It is a representation of the concepts (constructs) of interest in a study, their observable manifestations and the interrelationship among them. It examines whether the relationships between similar construct are considered with relationships between the observed measures of the constructs. Thorough observation of constructs relationships to each other it can generate new constructs. For example, [[intelligence]] and [[working memory]] are considered highly related constructs. Through the observation of their underlying components psychologists developed new theoretical constructs such as: controlled attention<ref>Engle, R. W., Kane, M. J., & Tuholski, S. W. (1999). Individual differences in working memory capacity and what they tell us about controlled attention, general fluid intelligence, and functions of the prefrontal cortex. In A. Miyake, & P. Shah (Eds.),Models of working memory (pp. 102−134). Cambridge: Cambridge University Press.</ref> and short term loading.<ref>{{cite journal |author1=Ackerman P. L. |author2=Beier M. E. |author3=Boyle M. O. | year = 2002 | title = Individual differences in working memory within a nomological network of cognitive and perceptual speed abilities | url = | journal = Journal of Experimental Psychology-General | volume = 131 | issue = | pages = 567–589 | doi=10.1037/0096-3445.131.4.567}}</ref> Creating a nomological net can also make the observation and measurement of existing constructs more efficient by pinpointing errors.<ref name="Cronbach55"/> Researchers have found that studying the bumps on the human skull ([[phrenology]]) are not indicators of intelligence, but volume of the brain is. Removing the theory of phrenology from the nomological net of intelligence and adding the theory of brain mass evolution, constructs of intelligence are made more efficient and more powerful. The weaving of all of these interrelated concepts and their observable traits creates a "net" that supports their theoretical concept. For example, in the nomological network for academic achievement, we would expect observable traits of academic achievement (i.e. GPA, SAT, and ACT scores) to relate to the observable traits for studiousness (hours spent studying, attentiveness in class, detail of notes). If they do not then there is a problem with measurement (of [[academic achievement]] or studiousness), or with the purported theory of achievement. If they are indicators of one another then the nomological network, and therefore the constructed theory, of academic achievement is strengthened. Although the nomological network proposed a theory of how to strengthen constructs, it doesn't tell us how we can assess the construct validity in a study. === Multitrait-multimethod matrix === {{Main|Multitrait-multimethod matrix}} The [[multitrait-multimethod matrix]] (MTMM) is an approach to examining construct validity developed by Campbell and Fiske (1959).<ref name="Campbell"/> This model examines convergence (evidence that different measurement methods of a construct give similar results) and discriminability (ability to differentiate the construct from other related constructs). It measures six traits: the evaluation of convergent validity, the evaluation of discriminant (divergent) validity, trait-method units, multitrait-multimethods, truly different methodologies, and trait characteristics. This design allows investigators to test for: "convergence across different measures...of the same ‘thing’...and for divergence between measures...of related but conceptually distinct 'things'.<ref>{{cite book |author1=Cook T. D. |author2=Campbell D. T. | year = 1979 | title = Quasi-experimentation. |location=Boston|publisher= Houghton Mifflin. }}</ref><ref>{{cite journal | author = Edgington, E. S. |date=1974 | title = A new tabulation of statistical procedures used in APA journals | url = | journal = American Psychologist | volume = 29 | issue = | page = 61 | doi = 10.1037/h0035846 }}</ref> ==Threats to construct validity== Apparent construct validity can be misleading due to a range of problems in hypothesis formulation and experimental design. * <u>Hypothesis guessing</u>: If the participant knows, or guesses, the desired end-result, the participant's actions may change.<ref>McCroskey, J. C., Richmond, V. P., & McCroskey, L. L. (2006). An introduction to communication in the classroom: The role of communication in teaching and training. Boston: Allyn & Bacon</ref> An example is the [[Hawthorne effect]]: in a 1925 industrial ergonomics study conducted at the Hawthorne Works factory outside Chicago, experimenters observed that both lowering <u>and</u> brightening the ambient light levels improved worker productivity. They eventually determined the basis for this paradoxical result: workers who were aware of being observed worked harder no matter what the change in the environment. *<u>Bias in experimental design</u> (intentional or unintentional). An example of this is provided in [[Stephen Jay Gould]]'s 1981 book, "[[The Mismeasure of Man]]".<ref>Gould, S. J. (1996). The Mismeasure of Man. 2nd edition. New York: W. W. Norton & Company.</ref> Among the questions used around the time of World War I in the battery used to measure intelligence was, "In which city do the Dodgers play?" (they were then based in Brooklyn). Recent immigrants to the USA from Eastern Europe unfamiliar with the sport of baseball got the answer wrong, and this was used to infer that Eastern Europeans had lower intelligence. The question did not measure intelligence: it only measured how long one had lived in the USA and become accultured to a popular pastime. *<u>Researcher expectations </u> may be communicated unintentionally to the participants non-verbally, eliciting the desired effect. To control for this possibility, [[double-blind]] experimental designs should be used where possible. That is, the evaluator of a particular participant should be unaware of what intervention has been performed on that particular participant, or should be independent of the experimenter. * <u>Defining predicted outcome too narrowly</u>.<ref>{{cite journal | author = MacKenzie S. B. | year = 2003 | title = The dangers of poor construct conceptualization | url = | journal = Journal of the Academy of Marketing Science | volume = 31 | issue = 3| pages = 323–326 | doi=10.1177/0092070303031003011}}</ref> For instance, using only [[job satisfaction]] to measure happiness will exclude relevant information from outside the workplace. * <u>[[Confounding|Confounding variables]]</u> (covariates): The root cause for the observed effects may be due to variables that have not been considered or measured.<ref>{{cite journal |author1=White D. |author2=Hultquist R. A. | year = 1965 | title = Construction of confounding plans for mixed factorial designs | url = | journal = The Annals of Mathematical Statistics | volume = 36| issue = | pages = 1256–1271 | doi=10.1214/aoms/1177699997}}</ref> An in-depth exploration of the threats to construct validity is presented in Trochim.<ref name="Trochim, William M.">[http://www.socialresearchmethods.net/kb/consthre.php Threats to Construct Validity], Trochim, William M. The Research Methods Knowledge Base, 2nd Edition.</ref> == See also == *[[Statistical conclusion validity]] *[[Internal validity]] *[[Ecological validity]] *[[Content validity]] *[[External validity]] *[[Reliability (psychometrics)]] *[[Face validity]] *[[Logical validity]] *[[Lee J. Cronbach]] *[[Paul E. Meehl]] == References == {{Reflist}} == External links == * [http://art.unt.edu/designresearchcenter/sites/default/files/articles/research_v2_ryan_gupta_hermosillo.pdf/ Useful reference guide for research terms] * [http://www.socialresearchmethods.net/kb/nomonet.php/ Provides a visual representation of the nomological network] [[Category:Validity (statistics)]]'