Why do many candidates find their TSA thinking assessment scores not as expected?

This year is the second year that Hanoi University of Science and Technology has organized the TSA thinking assessment exam in a new way, meaning that the exam structure has been adjusted compared to 2022 and earlier. In particular, in addition to adjusting the exam structure, Hanoi University of Science and Technology has applied a new assessment theory, which is the question response theory (IRT) in assessing the quality of questions as well as calculating exam scores for candidates.

This has caused candidates to not be able to grade themselves after completing the test, but have to wait for the grading results from the analysis of the candidate's test data. Therefore, after receiving the result announcement, many candidates are confused because the scores they received are not as they imagined.

Many students are very interested in the TSA thinking assessment test organized by Hanoi University of Science and Technology.

Test scores are a quantified model of candidate's ability estimation.

To relieve this concern for candidates, Thanh Nien Newspaper had an interview with Dr. Dang Xuan Cuong, an independent scientist with many years of research and implementation in educational assessment and measurement, one of the people who consulted Hanoi University of Science and Technology to implement the TSA thinking assessment test for the past 2 years.

According to Dr. Dang Xuan Cuong, the question response theory is built on mathematical models to explain the relationship between the candidate's ability and the questions that measure that ability. With this theoretical approach, the parameters of the questions (such as difficulty, discrimination, guessing, etc.) and the candidate's ability will be quantified based on the test data of the group of candidates participating in the test. Therefore, the candidate's ability estimated from the question response theory will be different from the traditional scoring method.

Candidates are tested before taking the TSA thinking assessment test.

With traditional grading, the test score is simply the mechanical addition of the scores of the questions in the test according to a specific grading guide. Experts call this the raw score. With the raw score, most candidates can rely on the answers to grade their own test, and even if the test is only objective multiple-choice questions, candidates can grade themselves after taking the test.

With the approach of question response theory, the test score is a result announced to the candidate after a process of analyzing the candidate's test data. This is a rather complicated step in terms of expertise, performed through specialized software and carefully considered through statistical indicators. In essence, the test score is a quantified result, serving as a basis for helping candidates know their abilities as well as having a basis for comparing their test results with the results of all other candidates in the exam.

Dr. Dang Xuan Cuong explained: "In principle, after the candidates finish the test, the system will rely on the data of each question in the test to estimate the candidates' ability, including information about how many candidates can answer each corresponding question and which candidates they are, how many questions each candidate can answer and which questions they are.

The test taker's test result data will help the system generate a model to estimate the test taker's ability. From this estimated ability, the thinking assessment score will be calculated by converting it to the scale used by the test (usually from 0 to 100).

Even if you get 70 questions correct, your test scores may be different.

Dr. Dang Xuan Cuong also said that the current TSA thinking assessment exam of Hanoi University of Science and Technology is an exam that combines the application of a one-parameter question response model (only concerned with the difficulty of the question) and two parameters (concerned with the difficulty and discrimination of the question). With the principle stated above, a group of candidates have the same raw score but the TSA test scores of the students are different.

Dr. Dang Xuan Cuong gave an example: "Suppose the test has 100 questions. According to the traditional approach, each correct answer is worth 1 point. Then, if 2 candidates can do 70 questions, each candidate will get 70 points. However, with the approach of question response theory, each question in the test will have a different level of difficulty and the candidate's ability will be estimated based on which specific questions the candidate has answered correctly. And with the 2 candidates mentioned here, if candidate A can do more difficult questions than candidate B, the estimated ability of candidate A will be higher than that of candidate B. In addition, the distinction of the questions is also a factor used in estimating the candidates' abilities. From there, the result is that the announced score of student A is higher than that of student B".

To put it more simply, with the traditional approach based on raw scores, the scores of questions are calculated with the same "weight", meaning that the difficulty and ease of those questions are not calculated. For the question response theory, the scores of questions will have different "weights", however, this weight is not due to the subjective will of the test designer but is calculated based on mathematical models from the test taker's test data.

In the process of applying the theory of question response, the analysis of test data also uses the technique of comparison to bring the candidates' scores to the same common scale. Each year, the thinking assessment test has many rounds, and candidates are notified of the results shortly after taking the test (not waiting until the annual test is held).

To ensure fairness in the admission process between candidates taking the exam in different rounds, the exam organizer will use the scale of the first round as the original scale. For the following rounds, during the analysis of the exam results, the equalization technique is applied to bring the scores of the candidates taking the exam in the following round back to the original scale, and then the official results of the candidates are given. This will make it easier to compare the exam results between rounds.

According to Dr. Dang Xuan Cuong, each theory has its advantages and disadvantages. "With the application of question-answering theory at Hanoi University of Science and Technology for the thinking assessment exam, we are also very cautious, through a specific roadmap to complete the exam each year," said Mr. Cuong.

Dr. Dang Xuan Cuong received his PhD in Education with a thesis on educational measurement and evaluation from Flinders University, Australia. He has over 18 years of experience in this field.

Source: https://thanhnien.vn/vi-sao-nhieu-thi-sinh-thay-diem-thi-danh-gia-tu-duy-tsa-khong-nhu-ky-vong-185240525095542657.htm