To investigating a key skillset for online learning, i.e., COR, and the sources students access and use when learning and solving COR tasks (i.e., the online IL), the research unit (FOR) draws on a novel interdisciplinary conceptual and methodological approach. To determine students’ COR levels, their development over university studies, and key COR covariates, the FOR examines qualitative and quantitative multimodal data, including:
(1) students’ sociodemographic background and learning characteristics (e.g., prior domain knowledge, general intellectual skills, reading skills, epistemological beliefs),
(2) students’ Web behavior and information processing (log data including all search activities collected during COR task-solving process),
(3) online sources and information/content the students access, select and use during COR task-solving,
(4) learning opportunities the students have taken (e.g., lectures) and general (online) media use, and
(5) the learning success indicators including domain knowledge development and grades in their university studies.
We aim to investigate COR as well as online information environments in representative study disciplines that are both large and at the same time exemplary. For this purpose, we chose study domains that are in high demand: medicine and economics. Correspondingly, we assess COR performance on GEN- and DOM-COR tasks in economics (Econ) and medicine (Med) among students of these domains. To control for expected domain-specific effects, we included students of sociology (Soc) and physics (Phys) as comparison groups respectively.
To validly measure COR skills in open (i.e., real Internet-based) and closed (i.e., Internet-like simulations) scenario-based tasks, students are immersed in a realistic situation and asked to solve a constructed or real domain-specific course assignment or task using the Internet or the Internet-like simulation. Each COR task is based on the joint conceptual and assessment taxonomy and encompasses a multitude of behavioral indicators classified in a taxonomy of
(a) the three COR facets (OIA, CIE, REAS);
(b) three developmental levels (basic, advanced, proficient, based on the Model of domain learning (MDL by Alexander, 2004), and specific to the DOM-COR assessment
(c) the three requirement areas/contexts (i.e., fundamental Econ/Med reasoning, practical Econ/Med reasoning, transdisciplinary reasoning).
The COR skillset was assessed among beginning students at the start of the winter semester 2023. In total 2,500 students (2,200 in 1st semester) participated in the full survey between 10 October and 4 November across seven universities in Germany. Of the 2,500 students, 1,400 students were enrolled in economics and sociology degrees, 800 in medicine and physics, and 300 in other subjects. Students were invited to complete the 15-minute survey during introductory courses. The survey was administered online using the Unipark survey tool. Students accessed it through their own devices and, in some cases, also as survey on site.
In the longitudinal study design, assessments (booklets) are being administered at eight participating universities throughout Germany over four annual measurement points. The first measurement took place in the winter term of 2023 (t0) with first-year students in the four domains: economics, medicine, sociology, and physics. The study follows them throughout their undergraduate studies, concluding in 2026. The selected sample size (accounting for expected drop-out and panel mortality) is N=500 students. Fine-grained analyses of COR task solving processes are explored in experimental studies with smaller samples (N=20 per domain) and in additional studies (e.g., to account for AI chatbot use, N=200).
Invitations to the t0 survey were sent in early December to all first-year students (who had expressed an interest) and the assessment concluded at the end of January. Students were provided with access to a virtual machine that was used to record the process data while they were completing the Internet-based assessmnt task. Participants were free to work from home on their own computers or laptops, and were permitted to take breaks between tasks in the multi-hour assessment. A total of 455 students participated in the study (250 from economics and sociology, and 170 from medicine and physics).
Analyses in the FOR focus not only on students’ COR process and performance, but also on the information landscape (IL) students encounter online. When solving COR tasks on the Internet, all sources visited and used are recorded and stored in a common data corpus. The FOR focuses on the characteristics of the (main) logged sources that students select and use as basis in their responses to COR tasks, in particular identifying features that are expected to contribute to the students’ (decision of) relying on those sources and pieces of information. For example, they may promote/claim or diminish/contest qualities such as comprehensibility, (non-)distraction, accuracy, authority, competence, imparted author confidence, certainty, positive or negative stance, cues for (high/low) credibility, loadedness or bias, etc. Coding manuals are compiled for the topic areas and range of logged sources at hand (as excerpts of the ‘IL’ online). Research approaches range from qualitative content coding of media products, to qualitative typological and quantitative computational linguistic analyses, to narrative structures analyses such as metaphors and reconstruction of latent meanings.
Process analyses during task-solving from think-alouds to eye-tracking on specific websites to logfile analyses are aimed at uncovering process indicators of (un)successful COR performance. Integrative cross-project analyses are expected to reveal overarching patterns of (un)successful performance and relations of COR development with influence variables as well as with study outcomes in the educational big data set. This provides a necessary basis for the improvement of learning materials and practitioner training in higher education and design learning interfaces to consider the affordances and risks of learning in an online environment.
Findings from the 1st research phase (2023–2027) will be used during the 2nd phase to devise innovative instructional interventions to help university students develop the COR skills needed to search for, critically assess, and use online information in higher education. These interventions will help universities adapt to the rapidly growing trend of Internet-based learning and fulfil their educational mandate in the Information Age.