(research methods)(validity)(reliability)(Validity vs. Reliability)(experimental design) (Internal threats) (external threats)

Research Methods: (top)

Qualitative: qualified descriptions; case studies, exploration for explanation, verbal description, unstructured and spontaneous data collection in natural settings
Quantitative: describing in numbers, by using measures, by validating hypothesis, pre-planning and structuring

The higher number indicates the greater the emphasis . . .	Description	Explanation	Validation
Qualitative methods	5	5	1
theory building	5	5	1
classification schemes and theory building (same ass model building except used to describe phenomena and events such as classifying new plants)	4	3	2
Survey methods	4	3	2
Historical research	4	3	2
evaluation study	4	3	2
single-subject studies	4	3	2
longitudinal studies	4	3	3
meta-analysis
model building (such as supply and demand models used in prediction studies)	1	3	5
experimental methods	1	3	5

Links to previous studies and explanations, rationale, theory, point of view:

literature review – TOC, citation index, ERIC, primary sources, secondary sources,
conceptual analysis; examples of constructs, defining conditions, checking completeness, chain of reasoning

Observation and measurements: (top)

Validity - test measures what we want to measure (Krathwohl, p.205)

Kind of evidence of Validity	Purpose	Question Posed	How Obtained
Construct validity - evidence to show that the measure behaves as we would expect it to represent the concept	To show that the test is a valid measure of the construct	Does the test behave as would be expected of a valid test of the construct?	Determine how the test would behave if valid and demonstrate that it so behaves (correlate with other measures on the construct; show an internal structure as predicted from the construct)
Criterion-related validity: Predictive	To demonstrate how the test would work when used to predict success or failure.	Does the test predict success and failure of entrants?	Correlate test scores obtained at or *prior to* entrance to a program with a criterion measure of success obtained at end of program.
Concurrent	To estimate predictive validity	Does the test predict success and failure of survivors?	Correlate test scores obtained at the *same time* as the criterion measures of success is obtained
Content Validity - competency or domain validity, represents the content of an area	To show how well the test covers the domain of a subject it is designed to test	Does the test sample the content and behaviors of its domain with adequate coverage and proper emphasis?	Compare test items with a table of specifications that describe the intersections of the content covered and the behaviors taught and indicates the relative emphasis of each intersection
Face validity - measure looks as though it would be valid	To show that the test meets our expectations of a valid test, especially important for a test used for public policy purposes with laypersons untutored in measurement	Does the test look like a valid measure?	Examine the test, usually by an expert in the field being measured.

(top)

Kind of Reliability	Question posed	Area of consistency of responses	Purpose	How estimated
Internal Consistency – consistency of a test in measuring whatever it measures	Are all items measuring the same thing – a single construct or dimensions? Is the score Interpretable?	Items of varying difficulty and content	To determine the homogeneity of the test and the interpretability of scores	Use random split halves or the Kuder-Richardson or Cronbach’s alpha for tests with weighted responses
Equivalence - consistency of scores over two different samples of test items, such as two forms of the same test	Are different forms of the test equivalent?	Different forms of a test	To determine whether the forms can be used interchangeably	Correlate the scores of form A with those of form B
Stability - how consistent are the measures over time	Is the behavior stable over time?	Different testing times (intervals should approximate that of intended use)	To determine the stability of the construct over time	Correlate the scores at one testing with those of the second testing
Stability and Equivalence – interaction of different tests taken as different times	Will one form of the test produce the same results as another form at a later time?	Different forms of a test given at different times	To determine the stability of the construct over time and across test forms	Correlate the score at one testing with those on the other form at a later testing

Can you have a test that is reliable and not validity OR valid and not reliability?(top)

If it is Reliable it can be "not valid": measure consistently, not the right thing
If it is Valid must be reliable: measure the right things
If not reliable then not valid

Reliable	Yes	Yes	No
Valid	No	Yes	No

Experimental Design and Internal and External Threats to Validity:(top)

Internal threats to validity (Krathwohl, p 474-475)

History: event occur before post testing that effects results
Testing: 2 or more tests or closely related instruments, earlier may effect later
Regression: causes the mean of the split-off group to regress of move toward the mean of the parent group on retest – comparison of control and experimental groups controls for this threat
Mortality – drop out, control group subject to same – acts as a control for this threat
Maturation: bio and psycho process that vary with the passage of time – control group provides control for this threat
Instrument decay: changes in the way an instrument is used, re-interpreting, multiple usage over time
Selection – occurs when subjects are present in a group as the result of a selective factor that causes the same effect as the treatment – random assignment to groups usually avoids this threat
Interaction effects can make treatment appear more or less effective.
- Testing-treatment interaction – treatment effect is strengthened or weakened by pre-testing
- Selection-treatment interaction – when treatment effects who is selected for experiment or control group

The Threats to Internal Validity chart below indicates the following:(top)

Research design: X=treatment, O=observation/data collection, C=control group, R=randomization

Research design	History	Maturation	Testing	Instrum	Regression	Selection	Mortaility	select X mature	test X treatment	selection X treatment
XO	-	-				-	-
OXO	-	-	-	-	?	+	+	-	-	-
OXO OCO	+	+	+	+	?	+	+	-	-	?
R OXO ...OCO	+	+	+	+	+	+	+	+	-	?
R XO ...CO	+	+	+	+	+	+	+	+	+	?
...OXO ...OCO R XO ...CO	+	+	+	+	+	+	+	+	+	?
000X00	-	+	+	?	+	+	+	-	+	+

Interpreting the chart:

The symbols mean:

" - " this design does not control for this threat;

"+" this design controls for this threat;

"?" design may or may not control for the threat

For example . . .

The pre-test, post-test, treatment-control group is threatened by selection-maturity interaction and testing-treatment interaction (most likely because of the lack of randomization); also, depending on the treatment it may or may not control for regression and selection-treatment interaction. By collecting observation before the treatment from the treatment and control groups it accounts for history and maturation threats; by having a treatment and control group it accounts for threats based on testing, regression (potentially), selection, and mortality.

On the other hand, the randomized pre-test, post-test, treatment-control group is only threatened by the testing-treatment interaction and perhaps by the selection-treatment interaction.

External threats to validity(top)

Obtrusiveness:

Hawthorne effect – I’m special, giving the subject attention causes the effect rather than the treatment
Hypothesis guessing – What do you expect from me . . . I want to help meet expectations, subject’s perceptions of what is going on during the treatment facilitate the effect.
Diffusion – treatment is communicated on to others during study
Demoralization – others not involved in the treatment my cause different effects as compared to the treatment group, because of the perceived "more attention" on the other side ("the other guys are getting special help, we should quit")
Novelty effect – the treatment is new and exciting for the subject, after the novelty wears, the effects of the treatment may also decrease

Addressing external, obtrusive threats:

Unobtrusiveness . . . become part of the everyday activities with treatment and control groups prior to treatment, when possible and ethical, observe without subject’s awareness

Researcher Expectancy Threats:

Researcher expectancy effects – inadvertently tip the scales in favor of hypothesis

Addressing external, researcher expectancy threats:

Implement research using double-blind procedures (no one can detect control and treatment groups)

(top)