A01

Generic Critical Online Reasoning Skills – Measurement, Development, and Comparative Analyses across Academic Domains

Background and Study Focus

Students learn in very diverse learning environments, with the internet becoming increasingly important in information acquisition. Students are confronted with a constantly increasing number of materials and media from which they can freely choose. Internet-based learning is one of the most important components of studying, which creates new opportunities and challenges such as inadequate sources, contradictory information, or a lack of transparency concerning the underlying interests of a source. Therefore, students need special skills to recognize and use high-quality online information in their studies. They need to be able to evaluate relevance, credibility and accuracy to determine the quality of the information used while avoiding distorting or misleading sources and protecting themselves from deception, error, or bias. Nevertheless, there has been little research on the medium to long-term relationship between individual use of online information in studies and study-related learning outcomes.

 

Concept and Research Objective 

Critical Online Reasoning (COR) combines three skill facets required to deal with information on the internet in one construct: online information acquisition (OIA), critical information evaluation (CIE) and reasoning with evidence, argumentation and synthesis (REAS). The DFG-funded CORE research group is investigating these skills and their development in a longitudinal study in several study domains. Within the CORE research group, a cooperation between the DIPF, Goethe University Frankfurt, Johannes Gutenberg University Mainz, and Ludwig-Maximilian University Munich, the GEN-CORE project deals with the generic part of COR skills, for which no domain-specific professional knowledge is required. The students selected from four study domains (medicine, sociology, economics, and physics) take part in three measurement sessions during their studies, where they solve generic and domain-specific information problems using the Internet. During their research the students’ behavior is recorded and they formulate written answers - this makes it possible to track the student’s profiles and the types of visited websites, why they select or reject certain sources and which criteria they use to assess the credibility of a website. CORE is designed as a longitudinal study to reliably measure the level of development of generic and domain-specific COR skills over the course of undergraduate study and their relationship to higher education learning outcomes.

For the assessment of generic COR skills, the project aims to develop indicators for COR skills based on the collected log data and written text responses and to design a measurement model for them. In addition, their relationship with other cognitive abilities will be investigated. The project compares the generic COR skills of assessed students from the four domains and investigates their changes during their undergraduate studies to obtain insight into the development of COR skills.

 

Research Outcomes

Major outcomes of A01 so far:

  • successful assessment design in terms of task development, technical implementation, and collecting response and process data that can be used to infer GEN-COR skills (Hartig et al., 2026; Illmann et al., in review)
  • validation of indicators for the OIA facet based on process data with eye-tracking data (Ilmann et al., 2026)
  • insights into the dimensional structure of the CIE facet and in its relationship to general cognitive abilities and reading abilities (Hahnel et al., submitted)

 

Contribution within the Research Unit CORE

A01 contributes data, assessment instruments as well as psychometric and measurement expertise to the research unit and cross-project analyses across all three CORE research areas:

  • A02/03: Joint samples and methods; joint GEN-DOM-COR analyses
  • B-projects: Input on information environments and text properties; provision of COR performance data
  • C-projects: Recording and joint analysis of behavioral data

Publications

Goldhammer, F., Engelhardt, & Zehner, F. (in press). Performance-Based Assessment of Digital Literacies. In J. Castek, J. Coiro, E. Forzani, M. Schira Hagerman, C. Kiili, & J. Sparks (Eds.), International Handbook of Research in Digital Literacies. Routledge.

Hahnel, C., Kiesler, N., Naumann, J., Macedo-Rouet, M., & Rouet, J.-F. (in press). Core Skills of Digital Literacies: Constants in an Ever-Changing Literacy Landscape. In J. Castek, J. Coiro, E. Forzani, M. Schira Hagerman, C. Kiili, & J. R. Sparks (Eds.), International Handbook of Research in Digital Literacies. Routledge.

Hartig, J., Drake, P., Illmann, J., Köhler, C., Kohmer, A., & Stroganova, A. (2026). Rating procedures to evaluate generic critical online reasoning in an open Internet environment. Zeitschrift für Erziehungswissenschaft, 29(1), 19-44. https://doi.org/10.1007/s11618-025-01365-5

Illmann, J., Baumartz, D., Drake, P., Hahnel, C., Hartig, J., Köhler, C., Schrickel, M., & Goldhammer, F. (in review). Defining log data indicators for the Measurement of Generic Critical Online Reasoning (GEN-COR). JCAL (Journal of Computer Assisted Learning).

Köhler, C., & Hartig, J. (2024). ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude. Contemporary Educational Technology, 16(4), ep528. https://doi.org/10.30935/cedtech/15144

 

Paper and Poster Presentations

Drake, P., Goldhammer, F., Hahnel, C., Hartig, J., Illmann, J., Köhler, C., & Schrickel, M. (2024, March). Erfassung der kritischen Informationsnutzung im Internet durch interaktive computerbasierte Aufgaben [Paper presentation]. 11th Conference of the Society for Empirical Educational Research (GEBF), Potsdam, Germany.

Drake, P., Goldhammer, F., Hahnel, C., Hartig, J., Illmann, J., Köhler, C., & Schrickel, M. (2024, September). Zur Bedeutung kognitiver Fähigkeiten bei der kritischen Nutzung von Online-Informationen [Symposium]. Congress of the German Psychological Society (DGPs), Vienna, Austria.

Drake, P., Hartig, J., Illmann, J., Koehler, C., Nagel, M. T., Kohmer, A., & Stroganova, A. (2025, January). Interaktive Testverfahren zur Bewertung der kritischen Informationsnutzung von Hochschulstudierenden im digitalen Zeitalter [Presentation]. 12th Conference of the Society for Empirical Educational Research (GEBF), Mannheim, Germany.

Hahnel, C. (2025, January). Wissen oder Annahmen? Beiträge zur Validierung von Interpretationen aus Logdaten [Invited symposium]. 12th Conference of the Society for Empirical Educational Research (GEBF), Mannheim, Germany].

Hahnel, C. & Köhler, C. (2024, September). Kritischer Umgang mit (Fehl-)Informationen: Herausforderungen in Messung und Intervention [Symposium]. Congress of the German Psychological Society (DGPs), Vienna, AT.

Hahnel, C., Philippi, K., Drake, P., Grobshäuser, N., Illmann, J., Goldhammer, F., Hartig, J., & Köhler, C. (submitted). Assessing Critical Online Evaluation Skills Despite Internet-Related Standardization Challenges [Presentation]. National Council on Measurement in Education (NCME), Toronto, Canada.

Hahnel, C., Schrickel, M., Drake, P., Illmann, J., Hartig, J., Goldhammer, F., & Köhler, C. (2024, September). How reading skills support students in their journey through the online information landscape [Paper presentation]. EARLI SIG 2 Conference, Valencia, Spain.

Hahnel, C., Scherer, T., Schrickel, M., Drake, P., Illmann, J., Hartig, J., Goldhammer, F., & Köhler, C. (2025, July). How reading skills support students in their journey through the online information landscape [Conference presentation]. 16th European Congress on Psychological Assessment (ECPA), Barcelona, Spain.

Hartig, J., Fischer, M. R., & Zlatkin-Troitschanskaia, O. (2023, September). Generische und fachspezifische Kompetenzen von Hochschulstudierenden im Umgang mit Online-Informationen – Erfassung und Entwicklung im Studium [Symposium presentation]. AEPF Conference, Potsdam, Germany.

Hartig, J., Kohmer, A., Drake, P., Illmann, J., Köhler, C., Stroganova, A., & Zottmann, J. (2025, January). Bewertungsverfahren zur Beurteilung offener Antworttexte zum kritisch-reflektierten Umgang mit Online-Medien [Presentation]. 12th Conference of the Society for Empirical Educational Research (GEBF), Mannheim, Germany.

Hartig, J., Drake, P., Goldhammer, F., Hahnel, C., Illmann, J., & Schrickel, M. (2026, March). Vergleich von Skalierungsmethoden für Mehrfach-Ratings von Statements zur Erfassung von Critical Online Reasoning [Presentation]. 13th Conference of the Society for Empirical Educational Research (GEBF), Munich, DE.

Illmann, J., Drake, P., Goldhammer, F., Hahnel, C., Hartig, J., Köhler, C., & Schrickel, M. (2024, September). Wie können Logdaten für die Messung des kritischen Umgangs mit Online-Informationen genutzt werden? [Symposium]. Congress of the German Psychological Society (DGPs), Vienna, Austria.

Illmann, J., Drake, P. Hahnel, C., Hartig, J., Köhler, C., Schrickel, M., & Goldhammer, F. (2024, November). How can log data contribute to measuring skills in the critical use of online information? [Symposium presentation]. EARLI SIG 27 Conference, Trondheim, Norway.

Illmann, J., Baumartz, D., Drake, P., Hahnel, C., Hartig, J., Köhler, C., Schrickel, M., Stroganova, A., & Goldhammer, F. (2025, January). Die Definition und Validierung von Logdaten-Indikatoren für die Messung von Critical Online Reasoning [Symposium]. 12th Conference of the Society for Empirical Educational Research (GEBF), Mannheim, Germany.

Illmann, J., Goldhammer, F., Hahnel, C., Hartig, J., Köhler, C., Schrickel, M., & Drake, P. (2025, August). Evaluating critical information usage among university students in the digital age [Presentation]. EARLI 2025 Conference, Graz, Austria.

Illmann, J., Dinc, Y., Hahnel, C., Küchemann, S., Ruf, V., & Goldhammer, F. (2026, March). Blickbewegungsdaten zur Validierung von Logdatenindikatoren der Online-Suchkompetenz [Presentation]. 13th Conference of the Society for Empirical Educational Research (GEBF), Munich, DE.

Illmann, J., Baumartz, D., Drake, P., Hahnel, C., Hartig, J., Köhler, C., Schrickel, M., & Goldhammer, F. (2026, April). Using Browser Log Data to Measure Critical Online Reasoning [Presentation]. National Council on Measurement in Education (NCME), Los Angeles, US.

Naumann, J., Hahnel, C., Kiesler, N., Macedo-Rouet, M., & Rouet, J.-F. (2024, April). Core skills of digital literacies: Constants in an ever-changing literacy landscape [Poster]. Annual Meeting of the American Educational Research Association (AERA), Philadelphia, US. https://doi.org/10.3102/IP.24.2103339