A02

Development of Economics Students’ Domain-Specific Critical Online Reasoning (DOM-COR) Skills over the Course of Their Bachelor Studies and Key Factors Influencing DOM-COR

Background and Study Focus

Recent research and practice uniformly show that today’s students of economics use the Internet as their main source of information for their studies. Students and lecturers rate the skills of searching for and evaluating online information as crucial for successfully studying economics. There is growing evidence that students often acquire subject-related misconceptions in Internet-based learning, resulting in a distorted understanding of economics.

Current developments in economics and societal fields highlight that teaching critical online reasoning (COR) should be anchored more deeply in economics education to reduce the substantial deficits in corresponding skills among students and even graduates, as shown in recent studies. Today’s educators need valid assessments for valid diagnostic that capture students’ COR skills to be able to design effective instructional programs that prepare students for competent (self-directed) learning while using the Internet.

Concept and Research Objective

Based on prior research, A02 defines domain-specific (DOM)-COR as the necessary skillset to critically select, evaluate, and use economics-related information from the Internet to derive warranted knowledge and reasoned decisions.

The DOM-COR construct is modeled in accordance with the three facets:

• online information acquisition,
• critical information evaluation,
• reasoning with evidence, argumentation, and synthesis.

A02 further differentiates between three practical contexts, in which DOM-COR can be applied:

• theoretical/fundamental reasoning for determining correct meanings of concepts (e.g., the break-even point),
• applied reasoning required in economic decision-making under real business conditions (e.g., preparing a financial plan based on conflicting parameters),
• and transdisciplinary reasoning for complex problem-solving (e.g., making decisions about costly life-saving measures with limited resources).

A02 aims to provide an objective and reliable assessment of students’ DOM-COR development that allows for valid test score interpretations and meets diagnostic and prognostic needs in and outside of university contexts.

Measurement and Analyses

To validly assess DOM-COR skills, A02 uses scenario-based tasks which immerse students in a realistic economics problem, employing newly developed and validated performance assessments in an open format with unrestricted Internet access and a closed format with Internet-like simulations with preselected sources.

Following the conceptional basis, every task in the DOM-COR assessment is designed to cover

• all three DOM-COR skill facets and
• all three reasoning contexts.

All DOM-COR tasks focus on eight problem scenarios in economics in these three contexts related to reasoning derived from found (open task format) or given online information (closed task format).

Both the response texts and task solving processes (incl. search history, clicks, scrolling) are recorded in the digital DOM-COR assessment environment in real time. These multimodal student performance and response process data are then evaluated by multiple independent, trained raters using a newly developed rating scheme based on the COR construct definition.

The various DOM-COR measures are then analyzed in relation to additional surveyed information such as socio-demographic data, media use habits, professional knowledge, intelligence as well as characteristics of online sources students used to gain a comprehensive understanding of students' COR skills and their determinants.

The effects of curricula and learning opportunities on the development of DOM-COR skills as well as other key personal and contextual theoretically assumed covariates are longitudinally analyzed in A02 based on data from two panel cohorts with students of economics (and social sciences students serving as a comparison group) assessed over the course of their bachelor studies at three measurement points.

At the first measurement in winter term 2023/24, overall 250 economics and social sciences students took part in the DOM-COR assessment. In addition, the test participants answered the economics knowledge test and the intelligence test which were assessed as key personal covariates.

For the second measurement in winter term 2024/25, 253 economics and social sciences students overall took part in the DOM-COR assessment as well as the economics knowledge test and the intelligence test. Of these, 126 students participated in the survey for the second time and 127 students participated for the first time.