New development of cognitive diagnosis models

doi:10.1007/s11704-022-1128-3

Front. Comput. Sci.

2023, Vol. 17

Issue (1) : 171604 https://doi.org/10.1007/s11704-022-1128-3

REVIEW ARTICLE

New development of cognitive diagnosis models

Yingjie LIU, Tiancheng ZHANG(

), Xuecen WANG, Ge YU, Tao LI

School of Computer Science and Engineering, Northeastern University, Shenyang 110169, China

Download: PDF(1078 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Cognitive diagnosis is the judgment of the student’s cognitive ability, is a wide-spread concern in educational science. The cognitive diagnosis model (CDM) is an essential method to realize cognitive diagnosis measurement. This paper presents new research on the cognitive diagnosis model and introduces four individual aspects of probability-based CDM and deep learning-based CDM. These four aspects are higher-order latent trait, polytomous responses, polytomous attributes, and multilevel latent traits. The paper also sorts on the contained ideas, model structures and respective characteristics, and provides direction for developing cognitive diagnosis in the future.

Keywords higher-order latent traits polytomous responses polytomous attributes multilevel latent traits cognitive diagnosis

Corresponding Author(s): Tiancheng ZHANG

Issue Date: 17 March 2022

Cite this article:

Yingjie LIU,Tiancheng ZHANG,Xuecen WANG, et al. New development of cognitive diagnosis models[J]. Front. Comput. Sci., 2023, 17(1): 171604.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-022-1128-3
https://academic.hep.com.cn/fcs/EN/Y2023/V17/I1/171604

Model	Mathematical expression	Feature
Linear Logistic Trait Model (LLTM,[3])	$P (Y i j = 1 θ i) = exp ? (θ i ? b j ?) / [1 + exp ? (θ i ? b j ?)]$ $b j ? = ∑ η k q j k + d$ represents the difficulty of the item.	The model depicts item difficulty through the linear combination of attribute difficulty, thereby realizing the combination of cognition and measurement.
Rule Space Model (RSM,[4])	Computes a set of order pairs ( $θ$ , $ζ$ ) where $ζ = f (x) / [Var ? f (x)] 12$ and $f (x) = [P (θ) ? T (θ)] ′ [P (θ) ? X]$ . $θ$ : is a variable that represents students. $ζ$ : indicates the degree to which the reaction mode of the actual test items of a students with ability deviates from his or her ability levels. $P (θ)$ : represents the probability of correct answers with $θ$ ability. $X$ : represents the binary response of the student’s answers for the items. $T (θ)$ : represents the mean vector of the probability with ability $θ$ of correct answer to the items.	The model evaluates a student’s ability, creatively puts forward the Q-matrix theory, and can also distinguish and diagnose a student’s attribute mastery mode.
Unified Model [5]	$P (Y i j = 1 α i, θ i) = (1 ? s j) {d j ∏ k = 1 K [π j k α i k q j k$ $r j k (1 ? α i k) q j k] P c j (θ i) + (1 ? d j)$ $P b j (θ i)}$ . $s j$ : represents that probability of errors on item $j$ . $d j$ : selects the strategy described by the Q-matrix to solve item $j$ . $p c (θ)$ : represents the probility of correctly using the external attributes of the Q-matrix when the ability is $θ$ .	The attributes are divided into Q-matrix inner and outer attributes, and the mastery mode and problem-solving strategy of the Q-matrix inner attributes and the latent residual ability of items aiming at the outer attributes $θ$ are considered. However, the model is too complex, and some parameters in the model can at times not be identified or estimated.
Fusion Model [6]	$P (x i j = 1 α i, θ i) = π j ? ∏ k = 1 K r j k ? (1 ? α i k) q j k P c j (θ i)$ . $π j ?$ : represents the item difficulty parameter of item $j$ based on the Q-matrix. $r j k ?$ : represents the probability ratio of a student lacking attribute $k$ and mastering attribute $k$ but answering item $j$ correctly.	The model is a simplified and unified model, which conforms to three basic and important characteristics: 1) estimating the mastery mode of students, 2) describing the relationship between items and various attributes, and 3) identifying the model parameters.

Tab.1 Four classical cognitive diagnostic models

Tab.2 Classification of TIMSS mathematics problems in terms of content and cognitive domains

Fig.1 PISA 2015 scientific literacy theory

Step	Score	Attribute
Step	Score	A1	A2	A3
$398 ? 38$	1	1	0	0
$368$	0	1	0
$334$	3	0	0	1

Tab.3 Restricted Q-matrix

Tab.4 Polytomous attributes

	K=1	K=2	$?$	$?$	K=K
G=1	$π 1 1$	$π 1 2$	$?$	$?$	$π 1 K$
G=2	$π 2 1$	$π 2 2$	$?$	$?$	$π 2 K$
$?$	$?$	$?$	$?$	$?$	$?$
$?$	$?$	$?$	$?$	$?$	$?$
G=G	$π G 1$	$π G 1$	$?$	$?$	$π G K$
Sum	$∑ g = 1 G π g 1 = 1$	$∑ g = 1 G π g 2 = 1$	$?$	$?$	$∑ g = 1 G π g K = 1$

Tab.5 The mixing ratio of mmixIRT

Tab.6 Comparison of seven new advanced cognitive diagnosis models

1	J P Leighton, M J Gierl. Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge: Cambridge University Press, 2007
2	D B Tu , Y Cai , H Q Dai , S Q Qi . A review on cognitive diagnostic models under modern test theory. Psychological Exploration, 2008, 28( 2): 64– 68
3	G H Fischer . The linear logistic test model as an instrument in educational research. Acta Psychologica, 1973, 37( 6): 359– 374
4	K K Tatsuoka . Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 1983, 20( 4): 345– 354
5	L V DiBello, W F Stout, L A Roussos. Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In: Nichols P D, Chipman S F, Brennan R L, eds. Cognitively Diagnostic Assessment. Hillsdale: Erlbaum, 1995, 361– 389
6	S M Hartz. A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. University of Illinois at Urbana-Champaign, Dissertation, 2002
7	J Schmid , J M Leiman . The development of hierarchical factor solutions. Psychometrika, 1957, 22( 1): 53– 61
8	A A Rupp, J Templin, R A Henson. Diagnostic Measurement: Theory, Methods, and Applications. New York: Guilford Press, 2010
9	Y Y Sheng , C K Wikle . Bayesian multidimensional IRT models with a hierarchical structure. Educational and Psychological Measurement, 2008, 68( 3): 413– 430
10	F Rijmen , M Jeon , M Von Davier , S Rabe-Hesketh . A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 2014, 39( 4): 235– 256
11	P D Zhan , Z H Yu , F M Li , L J Wang . Using a multi-order cognitive diagnosis model to assess scientific literacy. Acta Psychologica Sinica, 2019, 51( 6): 734– 746
12	H Y Huang , W C Wang , P H Chen , C M Su . Higher-order item response models for hierarchical latent traits. Applied Psychological Measurement, 2013, 37( 8): 619– 637
13	for Economic Co-operation Organisation. Technical report of the survey of adult skills (PIAAC). Paris: OECD, 2013
14	H Y Huang , W C Wang . Higher order testlet response models for hierarchical latent traits and testlet-based items. Educational and Psychological Measurement, 2013, 73( 3): 491– 511
15	Y Huo , J De La Torre , E Y Mun , S Y Kim , A E Ray , Y Jiao , H R White . A hierarchical multi-unidimensional IRT approach for analyzing sparse, multi-group data for integrative data analysis. Psychometrika, 2015, 80( 3): 834– 855
16	H Y Huang . A multilevel higher order item response theory model for measuring latent growth in longitudinal data. Applied Psychological Measurement, 2015, 39( 5): 362– 372
17	X Zhang , C Wang , J Tao . Assessing item-level fit for higher order item response theory models. Applied Psychological Measurement, 2018, 42( 8): 644– 659
18	Z H Fu , X Zhang , J Tao . Gibbs sampling using the data augmentation scheme for higher-order item response models. Physica A: Statistical Mechanics and its Applications, 2020, 541: 123696
19	D B Tu. Advanced Cognitive Diagnosis. Beijing: Beijing Normal University Publishing House, 2019
20	F Samejima . Estimation of latent ability using a response pattern of graded scores. Psychometrika, 1969, 34( 1): 1– 97
21	G N Masters . A rasch model for partial credit scoring. Psychometrika, 1982, 47( 2): 149– 174
22	E Muraki . A generalized partial credit model: application of an EM algorithm. Applied Psychological Measurement, 1992, 16( 2): 159– 176
23	D B Tu , Y Cai , H Q Dai , S L Ding . A polytomous cognitive diagnosis model: P-DINA model. Acta Psychologica Sinica, 2010, 42( 10): 1011– 1020
24	J S Chen , J De La Torre . Introducing the general polytomous diagnosis modeling framework. Frontiers in Psychology, 2018, 9: 1474
25	W C Ma , J De La Torre . A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 2016, 69( 3): 253– 275
26	J De La Torre . The generalized DINA model framework. Psychometrika, 2011, 76( 2): 179– 199
27	J Templin, R Henson, A Rupp, E Jang, M Ahmed. Cognitive diagnosis models for nominal response data. See researchgate.net/profile/Robert-Henson/publication/228894528_Cognitive_diagnosis_models_for_nominal_response_data/links/0a85e5332fadc2ef60000000/Cognitive-diagnosis-models-for-nominal-response-data.pdf website, 2008
28	W C Ma . A diagnostic tree model for polytomous responses with multiple strategies. British Journal of Mathematical and Statistical Psychology, 2019, 72( 1): 61– 82
29	J S Chen , J De La Torre . A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 2013, 37( 6): 419– 437
30	H Tjoe , J De La Torre . Designing cognitively-based proportional reasoning problems as an application of modern psychological measurement models. Journal of Mathematics Education, 2013, 6( 2): 17– 26
31	Y Cai , D B Tu . Extension of cognitive diagnosis models based on the polytomous attributes framework and their Q-matrices designs. Acta Psychologica Sinica, 2015, 47( 10): 1300– 1308
32	S Y Zhao, W Chang, L J Wang, P D Zhan. A polytomous extension of reparametrized polytomous attributes DINA. CNKI (in Chinese), 2019
33	J Rost . Rasch models in latent classes: an integration of two approaches to item analysis. Applied Psychological Measurement, 1990, 14( 3): 271– 282
34	S J Cho , A S Cohen . A multilevel mixture IRT model with an application to DIF. Journal of Educational and Behavioral Statistics, 2010, 35( 3): 336– 370
35	A M M D Meij , H Kelderman , H Van Der Flier . Fitting a mixture item response theory model to personality questionnaire data: characterizing latent classes and investigating possibilities for improving prediction. Applied Psychological Measurement, 2008, 32( 8): 611– 631
36	L Tay , D A Newman , J K Vermunt . Using mixed-measurement item response theory with covariates (MM-IRT-C) to ascertain observed and unobserved measurement equivalence. Organizational Research Methods, 2011, 14( 1): 147– 176
37	E S Kim , S H Joo , P Lee , Y Wang , S Stark . Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 2016, 23( 6): 870– 887
38	X Wang , G H Tan , X Wang , M Q Zhang , C Luo . The mixture item response theory models and its application traces. Advances in Psychological Science, 2014, 22( 3): 540– 548
39	S Cheng, Q Liu, E H Chen, Z Huang, Z Y Huang, Y Y Chen, H P Ma, G P Hu. DIRT: deep learning enhanced item response theory for cognitive diagnosis. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019, 2397-2400
40	L H Yao , R D Schwarz . A multidimensional partial credit model with associated item and test statistics: an application to mixed-format tests. Applied Psychological Measurement, 2006, 30( 6): 469– 492
41	Z Y Huang, Q Liu, E H Chen, H K Zhao, M Y Gao, S Wei, Y Su, G P Hu. Question difficulty prediction for READING problems in standard tests. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 2017, 1352−1359
42	Z Shu , R Henson , J Willse . Using neural network analysis to define methods of DINA model estimation for small sample sizes. Journal of Classification, 2013, 30( 2): 173– 194
43	R L Lamb , L Annetta , D B Vallett , T D Sadler . Cognitive diagnostic like approaches using neural-network analysis of serious educational videogames. Computers & Education, 2014, 70: 92– 104
44	F Wang, Q Liu, E H Chen, Z Y Huang, Y Y Chen, Y Yin, Z Huang, S J Wang. Neural cognitive diagnosis for intelligent education systems. In: Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. 2019, 6153−6161
45	Q Liu , R Z Wu , E H Chen , G D Xu , Y Su , Z G Chen , G P Hu . Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology, 2018, 9( 4): 48
46	E A Boyle , T Hainey , T M Connolly , G Gray , J Earp , M Ott , T Lim , M Ninaus , C Ribeiro , J Pereira . An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, 2016, 94: 178– 192
47	D Andrich , C Hagquist . Real and artificial differential item functioning. Journal of Educational and Behavioral Statistics, 2012, 37( 3): 387– 416
48	X F Yu, Y Cheng, H H Chang. Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): a comprehensive review. In: Von Davier M, Lee Y S, eds. Handbook of Diagnostic Classification Models. Cham: Springer, 2019, 307– 331
49	H Y Liu , T C Zhang , P W Wu , G Yu . A review of knowledge tracking. Journal of East China Normal University: Natural Science, 2019, ( 5): 1– 15

Viewed

Full text

Abstract

Cited

Shared

Discussed