COMPS Group Meeting Invited Talk

COMPS seminar main page

Tree-Based Modeling of Item Difficulty and Time Intensity: Evidence from the INVALSI Digital Competence Assessment

Speaker: Tommaso Giorgi, Department of Statistical Sciences, University of Bologna
Date and time: Wednesday, July 8, 2026 (10:00 AM CET)
Place ICS CAS room 318, Pod Vodárenskou věží 2, Prague 8, also on Zoom.

Abstract:

Digital competence is becoming a key educational outcome, especially as students increasingly interact with artificial intelligence, online information, and digital environments. This study investigates which item features are associated with item difficulty and response time in the first large-scale INVALSI assessment of digital competences, administered to almost 20,000 Grade 10 students in Italy. The test was developed within the DigComp 2.2 framework and included 246 multiple-choice items covering Information and Data Literacy, Communication and Collaboration, Digital Content Creation, and Safety. Item difficulty and time intensity are first estimated through a Log-Normal Item Response Theory model, which jointly models response accuracy and response times. These estimates are then analysed using multivariate regression trees and univariate classification trees to identify interpretable item profiles. Results show that textual features play a central role in shaping both psychometric and temporal item behaviour. In particular, answer-option length, stem length, distractor length, and the presence of technical terms emerge as key variables distinguishing item profiles. The findings also indicate that difficulty and time intensity are related but not equivalent: some items may be easy but time-consuming, while others may be more difficult but relatively fast. Overall, the proposed tree-based approach provides interpretable evidence to support item design, helping balance expected difficulty and response-time burden in digital competence assessments. Future developments include enriching item descriptions with additional linguistic features and extending the approach to LLM-assisted item generation and revision.

References:

Larsen, D.R., Speckman, P.L.: Multivariate regression trees for analysis of abundance data. Biometrics 60, 543--549 (2004). https://www-jstor-org.ezproxy.unibo.it/stable/pdf/3695785.pdf?refreqid=fastly-default%3A9cbdbea9ed9083f55f4f4d0c034746e1&ab_segments=&initiator=&acceptTC=1

Fox, J.-P., Klotzke, K., Simsek, A.S.: R-package LNIRT for joint modeling of response accuracy and times. PeerJ Computer Science 9, e1232 (2023). https://pmc.ncbi.nlm.nih.gov/articles/PMC10280685/pdf/peerj-cs-09-1232.pdf

Matteucci, M., Mignani, S., Veldkamp, B.P.: The use of predicted values for item parameters in item response theory models: an application in intelligence tests. Journal of Applied Statistics 39(12), 2665--2683 (2012) https://www.tandfonline.com/doi/epdf/10.1080/02664763.2012.725034?needAccess=true

Štěpánek, L., Dlouhá, J., Martinková, P.: Item difficulty prediction using item text features: comparison of predictive performance across machine-learning algorithms. Mathematics 11, 4104 (2023). https://doi.org/10.3390/math11194104

Vuorikari, R., Kluzer, S., Punie, Y.: DigComp 2.2: The Digital Competence Framework for Citizens. Publications Office of the European Union (2022). https://publications.jrc.ec.europa.eu/repository/bitstream/JRC128415/JRC128415_01.pdf