Piek Vossen - 4 avril 2024
Titre : Referential Grounding
Résumé :
LLMs or “Foundation models” are good at generalizing from observations but are they also good at individuation, reference and remembering? Grounding is often interpreted as an association across modalities. Multimodal models learn through fusion and co-attention from paired signals such as images and textual descriptions. But if the representation of each modality is a generalization what does that tell us about the referential grounding of individual people and objects in specific situations? Explicit extensional individuation of things and situations is a fundamental problem for LLMs because they are continuous and not discrete. In my research, I focus on identity, reference and perspective by analyzing different ways of framing in text that describe the same referentially grounded events and by developing embodied conversational AI models that create an extensional memory by observation and communication within real world environments.
Biographie :
Piek Vossen is Professor of Computational Lexicology at the Vrije Universiteit Amsterdam, where he directs the Computational Linguistics and Text Mining Lab. His research focuses on modeling understanding of language by machines. Within the Hybrid Intelligence program, he currently investigates how human and AI memories can be aligned through communication and their differences can be leveraged for collaborative tasks.
Références :
L. Remijnse, P. Vossen, A. Fokkens, and S. Titarsolej, Introducing frege to fillmore: a framenet dataset that captures both sense and reference, 2022, Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 39–50
P. Vossen, F. Ilievski, M. Postma, A. Fokkens, G. Minnema, and L. Remijnse, “Large-scale cross-lingual language resources for referencing and framing,” in Proceedings of the 12th language resources and evaluation conference, 2020, p. 3162–3171
S. B. Santamaría, T. Baier, T. Kim, L. Krause, J. Kruijt, and P. Vossen, “EMISSOR: A platform for capturing multimodal interactions as episodic memories and interpretations with situated scenario-based ontological references,” Proceedings of the first workshop beyond language: multimodal semantic representations, in conjunction with iwcs2022, 2021.
P. Vossen, L. Bajčetić, S. Báez Santamaria, S. Basić, and B. Kraaijeveld, “Modelling context awareness for a situated semantic agent,” in Proceedings of 11th international and interdisciplinary conference on modeling and using context, context 2019, 2019