Development of Economics Students’ Domain-Specific Critical Online Reasoning (DOM-COR) Skills over the Course of Their Bachelor Studies and Key Factors Influencing DOM-COR
Background and Study Focus
Current research and practice uniformly show that today’s students of economics use the Internet as the main source of information for their studies. Students and lecturers rate the skills of searching for and evaluating online information as crucial for successfully studying economics. There is growing evidence that students often acquire subject-related misconceptions in Internet-based learning, resulting in a distorted understanding of economics. Current developments in economics and societal fields highlight that teaching critical online reasoning (COR) should be anchored more deeply in economics education to reduce the substantial deficits in corresponding skills among students and even graduates, as shown in recent studies. Today’s educators need valid assessments for valid diagnostic that capture students’ COR skills to be able to design effective instructional programs that prepare students for competent (self-directed) learning using the Internet.
Concept and Research Objective
Based on prior research, A02 defines domain-specific (DOM)-COR as the necessary skillset to critically select, evaluate and use economics-related information from the Internet to derive warranted knowledge and reasoned decisions. The DOM-COR construct is modeled in accordance with the three facets: online information acquisition, critical information evaluation, reasoning with evidence, argumentation and synthesis. A02 further differentiates between three practical contexts, in which DOM-COR can be applied. These include theoretical/fundamental reasoning for determining correct meanings of concepts (e.g., the break-even point), applied reasoning required in economic decision-making under real business conditions (e.g., preparing a financial plan based on conflicting parameters), and transdisciplinary reasoning for complex problem-solving (e.g., making decisions about costly life-saving measures with limited resources). A02 aims to provide an objective and reliable assessment of students’ DOM-COR development that allows for valid test score interpretations and meets diagnostic and prognostic needs in and outside of university contexts.
Measurement and Analyses
To validly assess DOM-COR skills, A02 uses scenario-based tasks, employing newly developed and validated performance assessments in an open format with unrestricted Internet access and a closed format with Internet-like simulations with preselected sources. DOM-COR tasks immerse students in a realistic problem that they have to solve based on the freely available Internet resources (open formant) that they increasingly use in their studies, or on preselected sources (closed format). Following the above-mentioned conceptional basis, every task in the DOM-COR assessment is designed to cover (i) all three DOM-COR skill facets and (ii) all three reasoning contexts. Thus, DOM-COR tasks focus on problem scenarios in economics in the three contexts related to reasoning derived from found or given online information.
The response texts and task solving processes (incl. search history, clicks, scrolling) are recorded in the digital DOM-COR assessment environment in real time. These multimodal performance and response process data are then evaluated by multiple independent, trained raters using a newly developed rating scheme based on the COR construct definition. The various indicators of DOM-COR skills are then analyzed in relation to additional surveyed information such as socio-demographic data, media use habits, professional knowledge, intelligence and characteristics of used online sources in order to gain a comprehensive understanding of students' COR skills.
The effects of curricula and learning opportunities on the development of DOM-COR skills and other key personal and contextual covariates that have not been researched thus far are longitudinally analyzed in A02 based on data from two panel cohorts with students of economics (and social sciences students serving as a comparison group) assessed over the course of their bachelor studies at four measurement points.
At the first measurement in winter term 2023/24, overall 250 economics and social sciences students took part in the DOM-COR assessment. In addition, a total of 260 students answered the economics knowledge test and overall 473 students answered the intelligence test.
A02 generates unique data and findings,
- on the development of students’ DOM-COR in economics,
- on key personal and institutional covariates (such as attended courses) and
- on the relation to key learning outcomes (evaluation of academic performance).
In this way, A02 provides a necessary basis for the conceptual development of instructional interventions to effectively foster DOM-COR (in the 2nd research phase).
Contribution within the research unit CORE
Data and findings from A02 will be used in subsequent analyses in all research unit projects and beyond; for instance, analyses of the online information and learning materials, together with B-projects, allow for critical appraisals of fit, and professional integration of ‘real-world’ online sources into higher education to help to improve students’ learning outcomes and study success.
