Theoretical foundations of computational psychometrics

Theoretical foundations of computational psychometrics was a Czech Science Foundation grant awarded to Patrícia Martinková for period 2021—2023. It focused on theoretical and computational aspects of psychometrics with the aim to propose estimation and detection methods superior to traditional ones, as well as extensions to more complex designs. Project also provided software implementations together with simulated and real data examples demonstrating usefulness and superiority of the proposed methods.

Team members

Manuscripts stemming from the project

  • Martinková, P. & Hladká, A. (2023). Computational Aspects of Psychometric Methods: With R. Chapman and Hall/CRC. doi:10.1201/9781003054313

  • Bartoš, F., & Martinková, P. (2024). Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability. British Journal of Mathematical and Statistical Psychology. doi:10.1111/bmsp.12343
  • Hladká, A. & Martinková, P. & Magis, D. (2024). Combining item purification and multiple comparison adjustment methods in detection of differential item functioning. Multivariate Behavioral Research, 59(1), 46-61. doi:10.1080/00273171.2023.2205393
  • Kolek, L., Martinková, P., Vařejková, M., Šisler, V., & Brom, C. (2024). Is video games' effect on attitudes universal? Results from an empirical study comparing video games' impact on the attitude change of players with different backgrounds. Journal of Computer Assisted Learning, 40(2), 667—684. doi:10.1111/jcal.12911
  • Martinková, P., Bartoš, F., & Brabec, M. (2023). Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables. Journal of Educational and Behavioral Statistics. doi:10.3102/10769986221150517. arXiv:2207.02071
  • Štěpánek, L., & Dlouhá, J., & Martinková, P. (2023). Item Difficulty Prediction Using Item Text Features: Comparison of Predictive Performance across Machine-Learning Algorithms. Mathematics, 11(9) doi:10.3390/math11194104
  • Holub, M. & Martinková, P. (2023). Supervised Machine Learning for Text Analysis in R, Journal of the American Statistical Association, 118(543), 2207-2209. doi:10.1080/01621459.2023.2231224
  • Pérez, I., Vomlel, J. (2023). On Identifiability of BN2A Networks. In Bouraoui, Z.; Vesic, S. (Eds.): Symbolic and Quantitative Approaches to Reasoning with Uncertainty. ECSQARU 2023. Cham: Springer, 136-148. Lecture Notes in Computer Science, Lecture Notes in Artificial Intelligence, 14294. ISBN 978-3-031-45607-7.
  • Hladká, A., Martinková, P., & Brabec, M. (2024). New iterative algorithms for estimation of item functioning. arXiv. doi:10.48550/arXiv.2302.12648
  • Bartoš, F. & Martinková, P. (2023). Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability. arXiv. doi:10.48550/arXiv.2207.09101
  • Hrba, M., Maciak, M., Peštová, B., Pešta, M. (2022). Bootstrapping Not Independent and Not Identically Distributed Data. Mathematics. 10(24):4671. doi:10.3390/math10244671
  • Erosheva, E. A., Martinková, P., & Lee, C. J. (2021). When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society — Series A, doi:10.1111/rssa.12681
  • Goldhaber, D., Grout, C., Wolff, M., & Martinková, P. (2021). Evidence on the dimensionality and reliability of professional references' ratings of teacher applicants. Economics of Education Review, 83, 102130. doi:10.1016/j.econedurev.2021.102130
  • Kolek, L., Šisler, V., Martinková, P., & Brom, C. (2021). Can video games change attitudes towards history? Results from a laboratory experiment measuring short- and long-term effects. Journal of Computer Assisted Learning. In print. doi:10.1111/jcal.12575

Software

Selected conference presentations

  • Martinková, P., Hladká, A. (2023). Computational aspects of modelling item responses. IMPS 2023, College Park, Maryland.
  • Bartoš, F., Martinková, P., Brabec, M. (2023). Bayesian location-scale model for assessing reliability differences with ordinal ratings. IMPS 2023, College Park, Maryland
  • Martinková, P., Netík, J. (2023). SIAmodules: Modules for ShinyItemAnalysis. Psychoco 2023, International Workshop on Psychometric Computing, June 8-9, 2023, Universität Zürich, Switzerland
  • Martinková, P. (2023). Generalized linear and nonlinear regression models for DIF detection in longitudinal designs with binary and polytomous responses. International Meeting on Detecting Differential Item Functioning in Polytomous IRT Models and/or Multiple Groups, invited talk. Zurich, 2023, Universität Zürich, Switzerland
  • Štěpánek, L., Dlouhá, J., Martinková, P. (2023). Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance. 10th European Congress of Methodology (EAM 2023), July 11-13, Ghent University, Belgium.
  • Martinková, P., Pavlech, J. (2023). Measurement invariance in factor analytic and item response theory framework. 10th European Congress of Methodology (EAM 2023), July 11-13, Ghent University, Belgium.
  • Martinková, P., Hladká, A. (2023). Modeling item responses under different frameworks. 10th European Congress of Methodology (EAM 2023), July 11-13, Ghent University, Belgium.
  • Netík, J., Martinková, P. (2023). Enhancing Psychometrics with Interactive ShinyItemAnalysis Modules. 10th European Congress of Methodology (EAM 2023), July 11-13, Ghent University, Belgium.
  • Vařejková, M., Martinková, P., Potužníková, E. (2023). A simulation study of repeated covariate equating. 10th European Congress of Methodology (EAM 2023), July 11-13, Ghent University, Belgium.
  • Martinková, P., Vařejková, M., Potužníková, E. (2023). Obtaining comparable scores from multiple test forms in case of non-equivalent groups via repeated covariate equating. FREMO 2023, Oslo, Norway
  • Martinková, P., Bartoš, F., & Brabec, M. (2022). Computational aspects of reliability estimation, IMPS 2022 Spotlight talk
  • Štěpánek, L., Dlouhá, J., & Martinková, P. (2022). Machine-learning methods for item difficulty prediction using item text features, IMPS 2022, Bologna, Italy
  • Dlouhá, J., Štěpánek, L., & Martinková, P. (2022). Item difficulty prediction using computational psychometrics and linguistic algorithms, IMPS 2022, Bologna, Italy
  • Netík, J., & Martinková, P. (2022). Revisiting parametrizations for the nominal response model, IMPS 2022, Bologna, Italy
  • Martinková, P. (2021). Computational aspects of psychometrics taught with R and Shiny, useR!2021, virtual.
  • Martinková, P., Bartoš, F., Brabec, M. (2021). Inter-rater reliability in complex situations, IMPS 2021, virtual
  • Hladká, A. Martinková, P., Brabec, M. (2021). Estimation in generalized logistic regression models for DIF detection, IMPS 2021, virtual
  • Martinková, P. (2021). Does a zero inter-rater reliability mean grant peer review is arbitrary? Metascience 2021, virtual

Previous work we were building on

  • Hladká, A., & Martinková, P. (2020). difNLR: Generalized logistic regression models for DIF and DDF detection. The R Journal, 12(1), 300—323, doi:10.32614/RJ-2020-014
  • Martinková, P., Hladká, A., & Potužníková, E. (2020). Is academic tracking related to gains in learning competence? Using propensity score matching and differential item change functioning analysis for better understanding of tracking implications. Learning and Instruction, 66, 101286, doi:10.1016/j.learninstruc.2019.101286
  • Bartoš, F., Martinková, P., & Brabec, M. (2020). Testing heterogeneity in inter-rater reliability. In M. Wiberg, D. Molenaar, J. González, U. Böckenholt, & J.-S. Kim (Eds.), Quantitative psychology (pp. 347—364). Cham: Springer International Publishing, doi:10.1007/978-3-030-43469-4_26
  • Štěpánek, L., Martinková, P. (2020). Feasibility of computerized adaptive testing evaluated by Monte-Carlo and post-hoc simulations. In Proceedings of the 2020 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 359-367, doi:10.15439/2020F197
  • Martinková, P., Goldhaber, D., & Erosheva, E. (2018) Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability. PLoS ONE, 13(10), e0203002, doi:10.1371/journal.pone.0203002
  • Martinková, P., & Drabinová, A. (2018) ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal, 10(2), 503—515, doi:10.32614/RJ-2018-074
  • Drabinová, A., & Martinková, P. (2017) Detection of differential item functioning with non-linear regression: Non-IRT approach accounting for guessing. Journal of Educational Measurement, 54(4), 498—517, doi:10.1111/jedm.12158
  • Martinková, P., & Drabinová, A., Liaw, Y.-L., Sanders, E. A., McFarland, J. L., & Price, R. M. (2017) Checking equity: Why DIF analysis should be a routine part of developing conceptual assessments. CBE—Life Sciences Education, 16(2), rm2, doi:10.1187/cbe.16-10-0307