Appendix E — Exercises

Additional problems and exercises for each chapter are provided below. For some of the exercises, hints and/or solutions are offered on the book web page. Exercises with hints available are marked with an asterisk (\(^\ast\)), while exercises with solutions are marked with a dagger (\(^\dagger\)).

E.1 Exercises for Chapter 1

Exercise 1.1

Run the ShinyItemAnalysis application and try basic data exploration. Using the default dataset, answer the following questions:

What is the name of the dataset?
Of which type (binary, ordinal, or nominal) and of how many items does the dataset consist?
How many observations does the dataset contain?
How many observations come from the focal group (females, coded as 1) and the reference group (males, coded as 0)?
What are the maximum and minimum values of the criterion variable?

Exercise 1.2

The LSAT7 dataset contains 5 dichotomously scored items obtained from the Law School Admissions Test, section 7 (Bock & Lieberman, 1970). Upload the LSAT7 dataset into the ShinyItemAnalysis application and explore it. The csv file is available in the datasets directory on the book web page.

What are the mean and the standard deviation of the total scores?
Calculate and interpret the Z-score and T-score for a respondent with a total score of 3 points.
How many points did a respondent in the 69th percentile receive?

Exercise 1.3

Create a short R script including the following tasks:

Upload data from Exercise 1.2.
Compute the total scores for all students. Calculate mean, median, standard deviation, skewness, and kurtosis of the computed total scores.
Draw a histogram of the total scores.
Are the total scores approximately normally distributed? Why/Why not?
Compute the Z-scores and T-scores for all respondents.

E.2 Exercises for Chapter 2

Exercise 2.1\(^\ast\)

Prove that if \(X\) and \(Y\), as well as \(X + Y\), are normally distributed, then the pair \((X, Y)\) has a bivariate normal distribution.

Exercise 2.2

Derive the two-sided confidence interval for the mean (see Chapter 2 for the formula).

Exercise 2.3

Derive the one-sided confidence interval for the mean.

Exercise 2.4

The theory suggests that the students majoring in science should obtain higher scores than non-majors on the HCI test.

Use the HCI dataset and the major variable to provide a source of evidence about validity of the HCI with two sample \(t\) test.
Calculate the Pearson correlation coefficient between the score and major.
Run the ShinyItemAnalysis application (online or locally), and change data to the HCI. The HCI dataset includes the criterion variable – an indicator of whether a student plans to major in the life sciences. Briefly interpret the results of the offered analysis.

Exercise 2.5

Derive formulae of the OLS estimates (see Chapter 2).

Exercise 2.6

Set and solve likelihood equations to obtain MLE using the likelihood function (see Chapter 2).

Exercise 2.7\(^\ast\)\(^\dagger\)

The TheoryPractice dataset includes responses of 2,254 examinees who were given a theoretical test with 65 dichotomous items. Those examinees who passed the theoretical test (had a score at least 52 points) were given a practical test which consists of five parts. To pass the practical test, the examinee needed to succeed in all five parts (i.e., gain a score of 5 points).

Download data available in the data folder at the course web page, and create an R script to answer the following questions:

How many examinees passed the theoretical test? What is the pass rate of the theoretical test?
How many examinees passed the practical test? What is the pass rate of the practical test in the restricted sample (i.e., examinees who passed theoretical test)?
Calculate the correlation between scores of the theoretical and practical tests in the restricted sample.
Use the correction formula for restriction of range to estimate correlation between scores of theoretical and practical tests in the unrestricted sample. What would the correlation be approximately in an unrestricted sample?

Exercise 2.8\(^\dagger\)

Assume that the score of the first test \(X \sim \mathcal{N}(5, 1)\). The score of the second test \(Y\) is linearly dependent on the score of the first test by formula \(Y = 2X - 1 + e\), where \(e \sim \mathcal{N}(0, 1)\), and \(X\) and \(e\) are independent. Theoretical correlation between two scores \(X\) and \(Y\) is \(\frac{2}{\sqrt{5}} = 0.894\).

Create an R script to generate \(X\) and \(Y\) of sample size 1,000. Use set.seed(1) for reproducibility.
Calculate an estimate of correlation between the generated scores \(X\) and \(Y\). Compare to the theoretical correlation.
Consider only those respondents who achieved:
1. at least in the 85th percentile in the first test,
2. at most in the 15th percentile in the first test,
3. at most in the 15th percentile or at least in the 85th percentile in the first test.
Recalculate the estimate of correlation, apply a formula for correction for restriction of the range, compare results, and try to explain.

Exercise 2.9

Prove the formula for correction of range restriction (see Chapter 2).

E.3 Exercises for Chapter 3

Exercise 3.1

Run the ShinyItemAnalysis application (online or locally), change data to the HCI dataset, and answer the following questions:

Which two items correlate with item 9 the most?
Read the wording of these items in the supplement of the paper and try to explain why the correlation between them is increased?

Exercise 3.2\(^\dagger\)

Consider all items in the BFI2 dataset used in Chapter 3. Fit a single factor model, a 5-factor CFA model, and a hierarchical 5-factor CFA according to the data description. Compare the three models. Estimate factor scores and provide interpretation for the first respondent.

E.4 Exercises for Chapter 4

Exercise 4.1\(^\dagger\)

Assume a true score \(T\) following a normal distribution with a mean of 5 and a variance of 1, i.e., \(T \sim \mathcal{N}(5, 1)\), and an error term following a normal distribution with a zero mean and variance of 1, i.e., \(e \sim \mathcal{N}(0, 1)\), where \(T\) and \(e\) are independent. The observed score \(X\) is assumed to be composed of the true score and the error term only, i.e., \(X = T + e\).

Calculate reliability using formula \(\rho_X = \frac{\text{Var}(T)}{\text{Var}(X)}\). Provide the whole calculation.
How does the reliability change when the error term variance is lowered by a more precise measurement to \(\text{Var}(e) = 0.1\) and distribution of the true score \(T\) remains unchanged?
How does the reliability change when the more heterogeneous group is considered and the variance of the true score is increased to \(\text{Var}(T) = 2\) while the distribution of the error term \(e\) remains the same?

Exercise 4.2\(^\dagger\)

Follow the assignment of Exercise 4.1 and create an R script with the following parts:

Generate data containing the true scores \(T \sim \mathcal{N}(5, 1)\) and error terms \(e \sim \mathcal{N}(0, 1)\) for a sample of size 100. For the reproducibility of your results, use set.seed(123) before generating.
Calculate the observed score \(X\) and estimate its reliability using both formulae \(\rho_X = \frac{\text{Var}(T)}{\text{Var}(X)}\) and \(\rho_X = \text{Cor}(X, T)^2\); compare with theoretical results from Exercise 4.1.
How do the results change when you increase the sample size to 1,000 and 10,000?

Exercise 4.3

Show that the reliability formula also holds for the reliability of the average rating \(\bar{X} = \frac{1}{m}\sum_{i = 1}^{m} X_i\), of \(m > 1\) parallel or essentially \(\tau\)-equivalent measurements.

Exercise 4.4\(^\ast\)

Provide derivation of attenuation of correlation with unreliable measurements (see Chapter 4).

Exercise 4.5

An instrument consists of 5 items and has a reliability 0.7. Use the Spearman-Brown formula and answer the following questions:

What would be the reliability if we used 10 items (equally precise, measuring the same construct)? Provide the whole calculation.
How many items would be necessary to increase the reliability to 0.9? Provide the whole calculation.
Provide an R script for your calculations.

Exercise 4.6\(^\ast\)

Prove the Spearman-Brown formula (see Chapter 4).

Exercise 4.7\(^\ast\)

Prove the generalized Spearman-Brown formula (see Chapter 4).

Exercise 4.8\(^\ast\)\(^\dagger\)

Show that under assumptions of essentially \(\tau\)-equivalent items, and when true scores are uncorrelated with error terms \(\text{Cov}(T, e) = 0\), Cronbach’s alpha is equal to the reliability of a test’s total score.

Exercise 4.9\(^\ast\)\(^\dagger\)

Assume the ANOVA model from Exercise 4.8. Use the knowledge about distribution of the \(F\) statistic to derive confidence intervals for Cronbach’s alpha under the assumptions of normality.

Exercise 4.10\(^\dagger\)

Assume the one-way random effect model with the balanced design. Derive the formula for the likelihood function (see Chapter 4).

Exercise 4.11\(^\dagger\)

Use the AIBS dataset and the one-way random effect model fitted in Chapter 4. Prepare an R code to fulfill the following tasks:

Provide the bootstrap confidence interval for the inter-rater reliability.
Provide the Bayesian estimate of inter-rater reliability and the credible intervals for your estimate.

Exercise 4.12

Consider the data.ratings2 dataset from the sirt package, which consists of ratings by 16 raters of 178 students on 5 items (see Chapter 4). A dataset is available in the data folder at the book web page. Run a two-facet G-study and D-study with the gstudy() and dstudy() functions from the gtheory package presented in Chapter 4. Check your results with those obtained with the lmer() function.

Exercise 4.13

Use the ShinyItemAnalysis application. Explore reliability of the LearningToLearn dataset for Grade 6 and Grade 9.

For both Grades 6 and 9, compare the reliability estimates for the first-second and for the even-odd split-half approach, and provide reasoning for your results.
For both Grades 6 and 9, provide an estimate of internal consistency with Cronbach’s alpha also considering the 95% confidence interval. How would you categorize the reliability by Cicchetti’s rules of thumb?
Using the estimate of Cronbach’s alpha and the Spearman-Brown formula, estimate what would be the reliability if the instrument length was doubled? What would be the reliability if it only contained 10 items? What number of items is needed for reliability of 0.9?

E.5 Exercises for Chapter 5

Exercise 5.1

For the Learning to Learn 9 toy dataset in the ShinyItemAnalysis interactive application answer the following questions:

Which item is the most difficult?
Which item is the easiest?
Which item has the largest discrimination? (Use ULI, RIT, RIR.)
Which item has the smallest discrimination? (Use ULI, RIT, RIR.)

Exercise 5.2

Describe properties of items from Exercise 5.1 using a traditional item analysis table.

What is the correlation between each item and the total test score?
What is the correlation between each item and the rest of the items?
What is Cronbach’s alpha of the test without given items (alpha drop)? Compare with Cronbach’s alpha of the test and briefly comment.

Exercise 5.3\(^\ast\)

Try different settings for a number of groups to compare for generalized ULI for the Learning to Learn 9 dataset. Answer following questions:

What happens with generalized ULI when considering 5 groups (comparing the first and the fifth) instead of 3 groups (comparing the first and the third)? Try to explain.
What happens with generalized ULI when considering 5 groups and comparing the fourth and the fifth group instead of the first and the fifth group? Try to explain.
When would it be desired to compare the fourth and the fifth group instead of the first and the fifth (and vice versa)?

Exercise 5.4\(^\ast\)

Using Selected R code from the ShinyItemAnalysis interactive application, create an R script which loads the HCI datasets (you will need HCI, HCItest, and HCIkey) and include the following items:

Discrimination-difficulty plot for 5 groups comparing the weakest group to the second-weakest group.
Traditional item analysis table including generalized ULI with the same setting as previously used.
Distractor plots for the easiest and the most difficult item, and the best and the worst discriminating item in terms of RIT.

E.6 Exercises for Section 9.4

Exercise 6.1

Show that the formulae for the logistic regression model and the link logit are equivalent.

Exercise 6.2

Consider the logistic regression model for the probability of answering an item correctly given the observed ability \(X\):

\[ P(Y = 1|X) = \frac{\exp(\beta_0 + \beta_1 X)}{1 + \exp(\beta_0 + \beta_1 X)} \]

Consider an item with parameters \(\beta_0 = -5\) and \(\beta_1 = 0.5\).

How do you interpret these parameters?
Fill in the table below with probabilities of the correct answer:

Total score \(X\)	0	5	10	15	20
\(P(Y = 1\|X)\)

Using the table above, sketch a curve describing the probability of the correct answer.

Exercise 6.3

Derive the formula for category probabilities in an adjacent category logit model (see Section 9.4).

Exercise 6.4

Derive the formula for category probabilities in a multinomial regression model (see Section 9.4).

E.7 Exercises for Chapter 7

Exercise 7.1

Finish Exercises 1, 2, and 3 in the IRT/Training/Dichotomous models tab of the ShinyItemAnalysis application.

Exercise 7.2\(^\ast\)\(^\dagger\)

Originally the Rasch model was defined as:

\[ \pi_{pi} = \frac{\tau_p}{\tau_p + \xi_i} \]

for \(\tau_p > 0\) and \(\xi_i > 0\). Show that this is equivalent to the standard notation (see Chapter 7).

Exercise 7.3\(^\ast\)\(^\dagger\)

Prove formulae for the Item Information Function (IIF) of the 4PL IRT model. For which ability level is the information the highest?

Exercise 7.4\(^\ast\)\(^\dagger\)

Use the HeightInventory dataset and the ggWrightMap() function to plot the person-item maps under the regression model when the actual height is known, and under the 1PL IRT model when the actual height is not available (see Chapter 7).

Exercise 7.5

The EPI is a very frequently administered personality test with 57 items measuring two broad dimensions, Extraversion-Introversion and Stability-Neuroticism, with an additional Lie scale. Download the epi_escore dataset which includes 24 items on E-score; it is available in the data folder at the book web page. Upload the epi_escore dataset into the ShinyItemAnalysis application, and answer the following questions. Note that the epi_escore dataset is binary.

How many observations do data consist of and which items are included in the E-score part?
Fit the Rasch model.
1. Which is the easiest item? Which is the most difficult one?
2. What is the correlation between standardized total scores and factor scores?
Fit the 1PL IRT model.
1. How does the estimate of discrimination in this model differ from the Rasch model? Briefly comment.
2. Which item provides the most information, and for which level of ability?
Fit the 2PL IRT model.
1. Which items have a negative discrimination? Read their wording and try to explain.
2. Which item provides the most information and for which level of ability is the maximal information reached?
3. What is the correlation between standardized total scores and factor scores? Compare with the Rasch model.

Exercise 7.6\(^\dagger\)

In Exercise 7.5 using the 2PL IRT model we observed that some items have negative discrimination. Upload data into R; prepare an R script using the mirt package to complete the following tasks and to answer the following questions:

Reverse items with negative discrimination.
Fit the 2PL IRT model with the mirt() function.
Compare estimated item parameters in original data and in data with reversed items. How did the discrimination parameters change? How did the difficulty parameters change?
Use the fscores() function to extract factor scores from the 2PL IRT model on original data and on data with reversed items. Compare estimates. Briefly comment.

E.8 Exercises for Chapter 8

Exercise 8.1

Finish the Exercise in the IRT/Training/Polytomous models tab of the ShinyItemAnalysis application for the GRM and GPCM.

Exercise 8.2\(^\ast\)

Derive the equation for category probabilities in GPCM from the logit formulation (see Chapter 8).

Exercise 8.3\(^\dagger\)

Consider the neuroticism data, neuroticism500.csv, available in the data folder at the book webpage. Prepare an R code to answer the following questions:

How many items and how many respondents are in the data? How are the items rated?
Fit the GRM with the same discrimination parameter for all items. Provide item parameter estimates for each item and interpret them.
Fit the GRM with no constraints. Provide item parameter estimates for each item and interpret them.
Use criteria to decide between these two models; explain your reasoning. For the better fitting model, plot Item Response Curves (IRC) for all items, Item Information Curves (IIC) for all items in one plot, and a Test Information Curve (TIC). For respondents of what levels of the latent trait is the instrument most informative?

Exercise 8.4\(^\dagger\)

Show that threshold parameters \(b_{i1} = -1.5, b_{i2} = -0.5\), and \(b_{i3} = 1.4\) for item \(i\) under the PCM, and the item parameters \(a_{i1} = 1, b_{i1} = -1.5, a_{i2} = 2, b_{i2} = -1\), and \(a_{i3} = 3, b_{i3} = -0.2\) under the NRM define the same model, i.e., the same IRCs.

Exercise 8.5\(^\dagger\)

Fit a confirmatory multidimensional IRT GRM considering all items of the BFI2 dataset used in Chapter 8 and 5 factors according to the data description.

E.9 Exercises for Chapter 9

Exercise 9.1

Give an example of how the Mantel-Haenszel test is unable to detect DIF with an item which functions in a non-uniform way.

Exercise 9.2

Show that the relationship between the intercept-slope parametrization and the IRT notation holds for the logistic regression method (see Chapter 9).

Exercise 9.3

Derive the relationship between the intercept-slope parametrization and the IRT parametrization in the group-specific cumulative logit model for DIF detection (see Chapter 9).

Exercise 9.4\(^\dagger\)

Consider the group-specific cumulative logit model for DIF detection fitted on item R10 “I had sudden feelings of panic” of the Anxiety dataset. What are the estimated item parameters? Does the item function differently in uniform or non-uniform way? Plot its cumulative and category probabilities. What is the probability of having a sudden feeling of panic rarely for females and males with an average level of anxiety? And what is the probability for them to have a sudden feeling of panic at least sometimes?

Exercise 9.5\(^\ast\)

Using a recursive formula, show that the formulae for category probabilities hold in the adjacent category logit model for DIF detection. Show also formulae in the IRT parametrization.

Exercise 9.6

Derive the relationship between the intercept-slope parametrization and the IRT parametrization in the group-specific adjacent category logit model for DIF detection. Show also the relationship for category probabilities.

Exercise 9.7\(^\dagger\)

Consider the group-specific adjacent category logit model for DIF detection fitted on item R19 “I found it hard to focus on anything other than my Anxiety” of the Anxiety dataset. What are the estimated item parameters? Does the item function differently in uniform or non-uniform way? Plot its category probabilities. What is the probability of finding it hard to focus on anything other than their Anxiety sometimes or often for females and males with a level of anxiety of two standard deviations above average?

Exercise 9.8\(^\ast\)

Using a recursive formula, show that the formulae for category probabilities hold in the multinomial logit model for DDF detection. Show also formulae in the IRT parametrization.

Exercise 9.9

Use the ShinyItemAnalysis interactive application, and finish Exercises in DIF/Fairness/Exercises.

Exercise 9.10

Use the ShinyItemAnalysis interactive application, and consider the GMAT dataset. With the DIF/Fairness tab answer the following questions:

Using the Delta plot method, which items are detected as functioning differently when using a fixed threshold? Which items are detected as functioning differently when using a threshold based upon a normal approximation? What is the value of the threshold now?
Using the Mantel-Haenszel test, which items are detected as functioning differently? What are the values of DIF effect sizes for DIF items? What is the odds ratio for item 1 and item 7? Interpret the result. What is the odds ratio considering only respondents with the total score of 12?
Using the logistic regression method, which items are detected as functioning differently? Which items favor males and which females? Is there any item performing non-uniform DIF? How do the results change when the Benjamini-Hochberg correction for multiple comparison is applied?
Using the Lord’s test with the 2PL IRT model, which items are detected as functioning differently? How do the results change when using item purification? How many iterations of item purification were run until convergence?

Exercise 9.11

Consider the verbal dataset from the difR package. Follow sample R code from the ShinyItemAnalysis interactive application in the DIF/Fairness tab to create R code answering the following questions:

How many items and how many respondents are in the data?
Explain the names of items (e.g., what do S1WantCurse and S3DoShout mean?)
Fit the Mantel-Haenszel test with the difMH() function using item purification. Which items are detected as DIF? Explain why some items perform a positive effect size deltaMH while some items perform a negative effect size.
Create a table describing the purification process.
Fit the logistic regression method with the difLogistic() function using item purification. Which items are detected as functioning differently?
Plot characteristic curves for DIF items using the plot() method and argument plot = "itemCurve". Check plots with Item Characteristic Curves (ICCs). Which items favor males (reference) and which favor females (focal)? Is there any item showing non-uniform DIF? Briefly describe your conclusion – which items are DIF, which items are unfair, and what would be the next step?

E.10 Exercises for Chapter 10

Exercise 10.1

Check the Supplementary R code and explain the reasoning behind the definition of difficulty parameters in the generated item bank for the multiplication tables from 1 to 10 described in Chapter 10. Adapt the code to create an item bank for the multiplication tables from 1 to 20.

Exercise 10.2

Using the item bank created in Exercise 10.1, follow Chapter 10 to prepare an analogous interactive CAT.

Exercise 10.3

In the post-hoc analysis presented in Chapter 10, we have confirmed that the final estimated ability of 0.4210 (SE = 0.2947) fulfilled the termination criterion (90% confidence interval not covering the threshold of −0.1) was met. Use analogous calculations to show that the estimate of ability of 0.3609 (SE = 0.2976) obtained in the step preceding the final step did not fulfill the termination criterion.

Exercise 10.4\(^\ast\)

Using the data generated in Chapter 10, extend the MCMC simulation for the following two cases:

A CAT with stopping criterion SE < 0.2
A linear test with stopping criterion SE < 0.3

For each simulated respondent, extract the ability estimate, standard error, and number of items. Add the extracted values into the figure in blue and red colors. Discuss your results.