Research Methods guide, Unit 2, step 7 of 16Guide overview Next step

Step 7: Measurement, Validity and Reliability

By the EconTutorials team. 12-minute read with 6 practice questions.

MatchesMethodology chapter: measures and instruments · psychology and social science research methods modules

In 30 seconds

Many things you want to study (stress, motivation, satisfaction) are constructs: they cannot be seen directly, so you need an operational definition that turns them into something measurable.
Variables are measured on four levels: nominal, ordinal, interval and ratio. The level decides which statistics you can use.
Reliability = consistency (do you get the same result again?). Validity = accuracy (are you measuring the right thing?).
A measure can be reliable without being valid, but not valid without being reliable.
Cronbach’s alpha checks whether items in a scale hang together; about 0.7 or above is usually acceptable.

Video lesson coming soonA short animated lesson for this step will appear here.

From concept to measure

Hours of paid work can be counted. “Academic stress” cannot. To study it you must decide exactly what counts as evidence of it, for example the average of five agree–disagree statements such as “I feel overwhelmed by my coursework”. That decision is your operational definition, and the whole study depends on it being sensible.

Key termsA construct is an abstract idea you want to measure. An operational definition states the precise procedure used to measure it. An indicator or item is one observable piece of that measure.

Levels of measurement

Level	Property	Example	Average to use
Nominal	Categories with no order	Type of job (retail, hospitality, tutoring)	Mode
Ordinal	Ordered categories, gaps not equal	Year of study; “never / sometimes / often”	Median
Interval	Equal gaps, no true zero	Temperature in °C; many scale scores	Mean
Ratio	Equal gaps and a true zero	Hours worked; income; marks out of 100	Mean

The level matters later: you cannot calculate a meaningful mean of job types, and the choice of statistical test depends on it (Step 12).

Chapter 7 of Research Methods Made Simple covers this in full, with Cronbach’s alpha run in Python and R on real scale data.Get the textbook · $15

Reliability: is it consistent?

Type	Question	How checked
Test–retest	Same result if repeated later?	Correlate scores from two occasions
Inter-rater	Do two coders or observers agree?	Percentage agreement, Cohen’s kappa (Step 10)
Internal consistency	Do the items in a scale measure the same thing?	Cronbach’s alpha

Cronbach’s alpha

\[ \alpha = \frac{k}{k-1}\left(1 – \frac{\sum s_i^2}{s_T^2}\right) \]

where $ k $ is the number of items, $ s_i^2 $ is the variance of each item and $ s_T^2 $ is the variance of the total score. If the items move together, the total varies much more than the items separately, and alpha is high.

Worked example

A five-item stress scale has item variances that add up to 5.0, and the variance of the total score is 15.0. Calculate alpha.

Step 1$ k/(k-1) = 5/4 = 1.25 $.

Step 2$ 1 – 5.0/15.0 = 0.667 $.

Step 3$ \alpha = 1.25 \times 0.667 = 0.833 $.

Step 4Interpret: 0.83 is good internal consistency, so it is reasonable to add the items into one stress score.

A rough guide: below 0.6 poor, 0.6–0.7 questionable, 0.7–0.8 acceptable, 0.8–0.9 good. Very high values (above about 0.95) can mean items are near-duplicates. Remember to reverse-code negatively worded items before calculating alpha.

Validity: is it accurate?

Type	Question
Face validity	Does it look like it measures the construct?
Content validity	Does it cover all parts of the construct?
Construct validity	Does it relate to other measures as theory predicts (convergent) and not to unrelated ones (discriminant)?
Criterion validity	Does it agree with an established measure now (concurrent) or predict a later outcome (predictive)?

A target picture helps. Shots clustered tightly in the wrong corner are reliable but not valid. Shots scattered around the bullseye are valid on average but unreliable. You want a tight cluster on the bullseye.

In your dissertationUse an established, published scale where one exists (for example a recognised stress or wellbeing scale), cite it, and report its alpha in your own sample. This is far stronger than inventing your own items, which you would then have to validate yourself.

Common mistakeMeasuring a construct with a single vague question (“Are you stressed? Yes/No”). One item cannot cover the content of a construct, its reliability cannot be checked with alpha, and a yes/no answer throws away information.

Practise until it sticks: Research Methods Made Simple Practice Questions has 1,004 questions, each with a worked answer.Get the practice book · $9

Practise: 6 questions

Try each one on paper first, then check the worked answer.

EasyQ1. Give the level of measurement: (a) degree subject; (b) weekly income in pounds; (c) satisfaction rated 1–5; (d) year of birth. Show worked answer

(a) Nominal. (b) Ratio. (c) Ordinal (often treated as interval when several items are averaged). (d) Interval: equal gaps but year 0 is not a true zero.

EasyQ2. In one sentence each, define reliability and validity. Show worked answer

Reliability: the measure gives consistent results. Validity: the measure captures the construct it is intended to measure.

MediumQ3. A four-item scale has item variances summing to 3.2 and a total-score variance of 8.0. Calculate Cronbach’s alpha. Show worked answer

$ \alpha = \tfrac{4}{3}(1 – 3.2/8.0) = 1.333 \times 0.6 = 0.80 $. Good internal consistency.

MediumQ4. A bathroom scale always reads 2 kg too heavy. Is it reliable? Valid? Show worked answer

Reliable, because it gives the same reading each time; not valid, because the readings are systematically wrong.

MediumQ5. Write an operational definition of “study engagement” for a survey. Show worked answer

For example: the mean of agreement (1 = strongly disagree to 5 = strongly agree) with four statements: “I attend most of my scheduled classes”, “I complete reading before seminars”, “I take part in discussions” and “I study outside class at least 10 hours a week”, with alpha checked in the sample.

ExamQ6. Explain how you would check the validity and reliability of a new “financial worry” scale. [6] Show worked answer

Content validity: ask experts and students whether the items cover all aspects of financial worry [1]. Construct validity: check it correlates with related measures such as stress (convergent) [1] and not with unrelated ones (discriminant) [1]. Internal consistency: calculate Cronbach’s alpha, aiming for 0.7 or more [1]. Test–retest: give it twice two weeks apart and correlate the scores [1]. Pilot first and revise weak items [1].

Check yourself

1. Cronbach’s alpha measures

Alpha checks whether items in a scale measure the same thing.

2. Hours worked per week is

Equal intervals and a true zero (zero hours means no work).

3. Which is true?

Consistency does not ensure accuracy; validity requires reliability.

Your score: 0 out of 3

The textbook and the practice book together: the Research Methods Made Simple bundle for $19 (was $24).Both books · $19

The short version

Turn constructs into measures with clear operational definitions.
Know the level of measurement of every variable.
Reliability is consistency; validity is accuracy; check both.
Use established scales and report Cronbach’s alpha in your sample.

Studying more than one subject? The Complete Book Bundle includes every Economics, Statistics, Mathematics, Econometrics and Research Methods book.

Want more questions like these, with worked answers?

The practice book has 1,004 more, from quick recall to exam style.

Practice book, 1,004 questions: $9 See the bundle: $19

Next: step 8 of 16Designing Questionnaires and Surveys