Registration for advanced groups

Modeling Cognition with Deep Convolutional Neural Networks

Project details

Number of members in the team

2-8

Requirements

• A computer with Python installed
• Advanced knowledge of machine learning techniques (Participants should be able to create, train and evaluate artificial neural networks in TensorFlow or PyTorch.)

Project Description

Deep learning has become increasingly prevalent in scientific research, and cognitive neuroscience stands to benefit from its expanding capabilities. Artificial neural networks can now achieve human‑level performance on certain cognitive tasks, suggesting that analyzing how these models solve such tasks may offer insights into the computational mechanisms underlying human cognition. Convolutional neural networks (CNNs), in particular, provide a useful—though highly simplified—framework for modeling visual information processing. While CNNs remain limited as biological models of the nervous system, they can serve as valuable tools for exploration and hypothesis generation.

This project aims to investigate the mechanisms of visuospatial attention by training CNNs on established cognitive psychology tasks and applying machine‑learning interpretability methods to identify mechanisms that emerge through learning rather than being specified a priori. Interpretable machine learning is all about explaining the decisions of complex nonlinear statistical model in ways understandable to humans. To disentangle the causal contributions of individual units and classes of neurons, we will focus on developing and refining relatively underexplored approaches for creating artificial lesions. In the second part of the project, we will test the limits of these methods by attempting to simulate hemispatial neglect – a common deficit of spatial attention most commonly present in patients after right hemisphere strokes.

Attention

Registering for the event means accepting the rules of the event

SemCoupling: Linking brain and behavior to cognitive states during naturalistic experience

Project Details

Number of members in the team

3-10

Requirements

• A computer with Python installed
• Basic programming skills in Python

Project description

Modern neuroscience can continuously record brain activity while people listen to narratives, watch movies, navigate virtual environments, or interact with others. These naturalistic paradigms are increasingly used to study cognition in healthy individuals and clinical populations because they capture brain function under conditions closer to everyday life than conventional laboratory tasks.

However, they create a fundamental measurement problem: we can continuously measure what the brain is doing and what information is presented, but we usually do not know what the participant is cognitively processing at each moment.

Attention naturally fluctuates during continuous experience. At some moments cognition closely follows incoming information; at others it shifts toward previous information, memories, plans, or internally generated thoughts. This process can be described in terms of cognitive or semantic coupling: the degree to which ongoing cognition is related to the current external information stream. Currently, cognitive state is typically measured using intermittent thought probes, behavioral responses, or retrospective reports. Such measurements are sparse and interrupt the very cognitive and neural dynamics that naturalistic experiments aim to study.

SemCoupling is a proof-of-concept for an open-source neuroscience framework designed to address this problem. The long-term goal is to combine sparse measurements of cognitive state with continuous behavioral and physiological signals – such as eye movements, pupil size, facial dynamics, and eventually EEG, intracranial EEG, fMRI, or other neural recordings – to generate time-resolved annotations of cognitive coupling during naturalistic experience.

Such annotations could provide a missing intermediate level between stimulus and neural activity:

STIMULUS → COGNITIVE STATE → BRAIN ACTIVITY

rather than assuming that identical stimulus presentation implies identical cognitive processing.

During BrainHack, we will develop and test core components required for such a framework using existing naturalistic behavioral and neuroscience datasets. We will focus on four complementary questions:

1. Can semantic coupling be quantified?
   We will develop methods for comparing open-ended thought reports with time-resolved semantic representations of naturalistic stimuli using natural language processing techniques.
2. Can cognitive coupling be inferred from continuous behavior?
   We will test whether multivariate dynamics of pupil size, blinks, eye movements, facial movements, head pose, and other available signals predict cognitive state at thought probes.
3. Can these cognitive-state estimates improve neuroscience analyses?
   Where suitable open neural datasets are available, we will explore whether accounting for fluctuations in cognitive coupling explains neural variability beyond stimulus properties alone.
4. Can the approach generalize across participants and datasets?
   We will test whether behavioral representations associated with cognitive engagement generalize across individuals and different naturalistic tasks.

The immediate objective is not to claim that cognition can already be reconstructed continuously. Instead, BrainHack will establish the computational and empirical foundations required to determine whether sparse cognitive measurements can eventually be transformed into continuous cognitive annotations. It will serve as the starting point for an open-source methodological initiative that can subsequently be validated using behavioral and neural datasets and extended to naturalistic and clinical neuroscience. SemCoupling is therefore conceived as a potential measurement framework for studying latent cognitive-state dynamics in naturalistic neuroscience. The project is developed within the Brain and Cognition Lab at Jagiellonian University, which studies the neural and behavioral dynamics of cognition using electrophysiology, eye tracking, and naturalistic experimental paradigms.

Attention

Registering for the event means accepting the rules of the event