Natural Language Processing

We address the controllability of large language models (LLMs) by giving them interpretable beliefs and programmable knowledge through leveraging the PI's work on understanding and improving transformer embeddings. Transformers' empirical success comes from the attention function's ability to induce graphs of relations from text. Our recent work has extended this ability to knowledge graphs, and to inducing the nodes of the graph as well, known as entity induction, with the first variational-Bayesian generalisation of the attention mechanism. This project will further develop this information-theoretic understanding of transformer embeddings and its sparsity-inducing regulariser, for learning graphs of higher-level abstract entities. The resulting Bayesian beliefs over generalised transformer embeddings of texts and graphs will give us the more interpretable, more programmable and more learnable abstract representations which are the core of this proposed project.

To leverage and extend these fundamental advances in representation learning, we will develop LLM architectures with a memory. Motivated by the success of Retrieval Augmented LLMs, our Belief Augmented Language Models (BALMs) will move knowledge extracted from training data out of large uninterpretable weight matrices into our interpretable Bayesian beliefs over large transformer embeddings. These beliefs will then be: augmented with human-editable knowledge graphs and selected new texts, refined with control objectives and multi-hop reasoning, and combined with inference of concensus beliefs and opinion summarisation. BALMs will be developed both to evaluate these beliefs and as a chat interface for specifying, accessing and editing the beliefs themselves, including the collaborative specification of shared beliefs. These fundamental advances in deep learning theory and architectures will allow us to control what an LLM says by controlling what it believes, thereby unlocking the power of AI for society.

In recent years, attention-based models like Transformers have radically improved the performance of natural language understanding (NLU), demonstrating the appropriateness of attention-based representation for language. In (Henderson, 2020) we show that these representation share many characteristics with those found in traditional computational linguistics (e.g. graph structure), except that they do not automatically learn multiple levels of representation nor their entities (morphemes, phrases, discourse entities, etc). Motivated by this challenge of entity induction, our recent work has discovered a very non-traditional perspective, which characterises attention-based models like Transformers as doing nonparametric Bayesian inference. Given an input text, our Nonparametric Variational Information Bottleneck (NVIB) Transformer infers distributions over nonparametric mixture distributions (Henderson and Fehr, 2023). We have even shown that pretrained Transformers can be converted into equivalent NVIB Transformers, and regularised post-training (Fehr and Henderson, 2023).

This reinterpretation of Transformers, combined with their unprecedented empirical success, leads us to postulate the hypothesis that natural language understanding is nonparametric variational Bayesian inference over mixture distributions. This claim of the adequacy of NVIB leads to two fundamental challenges which are not currently being addressed, each with an associated technological aim:

1. How can NVIB support inducing graph-structured representations at multiple levels of representation?
   Making deep learning representations interpretable.

2. How can NVIB enable controlling the information in representations?
   Making deep learning representations controllable.

For the first challenge, we will extend our previous structure processing methods (Mohammadshahi and Henderson, 2020, 2021, 2023; Miculicich and Henderson, 2022), developed for set-of-vector representations, to mixture-of-component distributions. And we will focus on unsupervised learning methods, rather than our previous supervised learning methods. To extend these models to multiple levels, we will take the approach of embedding all levels in one big mixture of non-homogeneous components, which are computed with iterative refinement. This extends our previous work on iterative graph refinement (Mohammadshahi and Henderson, 2021; Miculicich and Henderson, 2022), adding the induction of the nodes of the graph and the induction of multiple levels of representation. Learning representations which are interpretable as linguistic structures will be a testbed for the general aim of deep learning of interpretable representations.

For the second challenge, we will leverage the information theory behind NVIB to model both inferring implicit information and removing private information. We will investigate the use of KL divergence as a measure of entailment in semantic inference. We will apply the framework of Rényi differential privacy (Mironov, 2017) to provide privacy guarantees by adding noise which removes targeted information from Transformer embeddings. This method extends differential privacy to anything that can be embedded with a Transformer (especially text), with many important applications. These methods address the general aim of controlling the information in deep learning representations.

Addressing these challenges will lead to fundamental advances in machine learning, including novel deep learning architectures and fundamental insights into Transformers and their pretraining. We will do both intrinsic and extrinsic evaluations of our induced representations, expecting to show improvements on core NLP tasks, including privacy-preserving sharing of textual data. Given the current level of interest in the AI research and development community for Transformers and variational Bayesian methods, we expect the proposed research to have a profound impact on the field.

All speakers can understand a sentence never heard before, or derive the meaning of a word or a sentence from its parts. Children can learn any language they are exposed to. And yet, these basic linguistic skills have proven very hard to reach by computational models. The current reported success of machine learning architectures is based on computationally expensive algorithms and prohibitively large amounts of data that are available for only a few, non-representative languages. This limits the access to natural language processing technology to a few dominant languages and modalities and leads to the development of systems that are not human-like, with great potential for unfairness and bias. To reach better, possibly human-like, abilities in neural networks' abstraction and generalization, we need to move beyond the simple language modelling tasks currently used and develop tasks and data that train the networks to more complex and compositional linguistic skills.In this project, we set the challenging goals of achieving higher-level linguistic abilities in machines, while training in more realistic settings. We identify these abilities as (i) the intelligent ability to infer patterns of regularities in unstructured data, (ii) generalise from few examples, using (iii) abstractions that are valid across possibly very different languages.We study if current neural network architectures have the same properties of learning, generalization, and abstraction when processing language. Specifically, we ask (i) Can they learn the underlying generative structure of complex data? (ii) Under what conditions can they learn from zero or few examples? (iii) Do their learning patterns exhibit cross-linguistically valid abstractions?
To achieve these goals, we concentrate on one of the core building blocks of any language: verbs and their argument structure, the 'who did what to whom' expressing core events and actions. Argument structure is defined by specific combinations of elements (the arguments and the predicate) in different templates (the subcategorization frames), that form higher-order patterns of similarity across sentences (the alternations) and are defined at a high level of abstraction across languages (the semantic roles).We aim to learn disentangled representations of these components of argument structure. A disentangled representation encodes information about the salient factors of variation in the data independently. We have developed, and demonstrated in pilot work, a new set of progressive matrix tasks, inspired by IQ intelligence tests. These tasks are developed specifically for language and learning disentangled linguistic representations of underlying linguistic rules of grammar.
We apply this novel method to the three main questions of our investigation: (i) To demonstrate learning of the generative components of argument structure from data and facets of verb meaning, we develop new data sets and new versions of the progressive matrix task for argument structure concentrating specifically on learning argument alternations (John loaded the truck with hay/John loaded Hay onto the truck), on the gradation of compositionality (obligatory arguments vs optional adjuncts) and the compositionality of complex clauses. (ii) To investigate the conditions that make learning from few examples possible, we hypothesize that structured categorization reduces the data sample size needed for learning. We also take inspiration from human learning biases and develop models of increasing complexity and size to study how structure and data size interact, and in so doing learn solutions for low-resource languages. (iii) To study if truly abstract representations of verb argument structure and semantic roles emerge, we create novel progressive matrix learning tasks and novel cross-lingual artificial complex data of complex (non-concatenative) morphological paradigms in typologically different languages.(iv) We develop novel computational architectures and novel evaluation metrics adapted to these problems that exhibit a mixture of linguistic abilities and logical coherence.This kind of investigation, based on tasks that require a mixture of linguistic knowledge and higher-level linguistic reasoning has never been tried before for natural language processing. Current pilot studies for simple linguistic problems by the PI and her team show that the method is promising. We plan to extend the method here to two very challenging areas of natural language processing. On the one hand, we tackle complex linguistic data that tap into the core semantics of clauses, a well-studied, but so far unsolved, core problem for natural language understanding, for which the PI is a leading expert. On the other hand, we study complex morphological paradigms and systems, a very little studied cross-linguistically important problem.If successful, this research could lead to a significant methodological shift. These investigations can lead to three beneficial improvements of methods and practices: (i) deep, compositional representations would be learned, thus reducing needs in data size; (ii) current machine learning methods would be extended to low-resources languages or low-resource modalities and scenarios; (iii) higher-level abstractions would be learned, avoiding the use of superficial, associative cues that are the cause of so much bias and potential harm in the representations learned by current artificial intelligence and natural language processing systems.

This project aims to understand how evolution shapes animal physiology in response to prolonged exposure to juvenile undernutrition, whereby juvenile animals are forced to grow and develop in spite of chronic nutrient shortage. It will address four general questions:(1) In what way can evolution modify metabolism of a growing juvenile animal to alleviate consequences of undernutrition for Darwinian fitness?(2) What changes in allocation of metabolic resources does this adaptation entail?(3) To what degree are these changes mediated by genetic variants in 'master' genes with large effects on multiple aspects of the adaptation?(4) To what degree is evolution of tolerance to poor diet mediated by genetic variants in genes that regulate growth (i.e., the 'demand' for biomass building blocks) versus genes whose products catalyze and regulate their acquisition and allocation (i.e., the 'supply' side of growth)? The project will use lines of Drosophila melanogaster characterized by extraordinary genetically-based tolerance to larval undernutrition, a unique resource generated through >17 years of laboratory experimental evolution. To address the above questions, we will elucidate causal changes in growth regulation and metabolism underlying this highly polygenic and phenotypically complex evolutionary adaptation. First, we will quantify the rate of amino acid turnover and the allocation of resources to different components of biomass (proteins, triglycerides, glycogen, etc.) in the 'Selected' (malnutrition-tolerant) and 'Control' (unselected) larvae. Together with gene expression data, these data will be used to model the core metabolic network of Drosophila. This will generate testable predictions about differences in metabolic fluxes underlying malnutrition tolerance and identify nodes of the metabolic network critical for differential fluxes and resulting allocation patterns (question 1 and 2). It will also be used to evaluate the roles of nutrient supply versus demand on metabolic output in shaping the patterns of metabolic flux (question 4).Second, we will verify the contribution of a cis-regulatory variant in an ecdysone oxidase gene fiz to enhanced tolerance to undernutrition, and test whether the adaptive value of this variant is contingent on the presence of other elements of this complex adaptation. fiz emerged as a candidate with potential large effects on this adaptation (questions 3); it is thought to regulate growth by deactivating ecdysone, but this hypothesis appears incompatible with the direction of its effects on growth. To elucidate the effect of variation in fiz on ecdysteroid signaling we will study its effect on the abundance of different ecdysteroid species, identifying those most likely to mediate its growth-regulating effect. Third, we will combine the two above threads by investigating how fiz expression affects the rate of nutrient acquisition, metabolite abundance and the allocation of metabolic resources. Given that fiz is thought to act by modulating the demand for key metabolites, we will be able to ascertain to what degree increased demand for biomass building blocks has effects that propagate through metabolism and affect resource allocation question 4). To achieve these aims, we will combine experimental evolution with state-of-the art approaches, including genome editing, metabolomics, isotope tracing, high resolution mass spectroscopy and genome-scale flux balance analysis. This project will advance our understanding of a poorly understood and ecologically important evolutionary adaptation. It will also throw light on the broader fundamental question of how changes indifferent metabolic and regulatory elements interact to generate complex adaptations that enhance Darwinian fitness under conditions of environmental stress. Through novel application of recent experimental techniques and system-scale computational models the project will expand the boundaries of dissecting this complexity.

In previous years, we have focused our automatic speech recognition (ASR) research with Samsung on accents and on multi-linguality. In this year, we propose to focus on “Natural” user interfaces. By natural, we mean that the interface should function in such a way that the user should not have to behave differently from when he or she interacts with a person. Of course, there are many facets to this; however, two are pertinent: conversational/spontaneous speech and recognition exploiting natural sensors. Speech user interfaces typically rely on being able to place a microphone close to the user’s mouth. This maximizes the volume and clarity of the speech signal, whilst minimizing the effect of other noise in the vicinity. Such an interface is natural for, say, a telephone. However, many applications do not lend themselves to this type of interface. Examples include most home electronics, where the user might typically be in the center of a room, but the device is near a wall. In the case of televisions, a useful intermediate device is the remote control. Nevertheless, it is still inconvenient to hold a remote control like a telephone in order to talk to it.

Construction of a Natural Language Processing (NLP) infrastructure to support domain experts in biomedical discovery over large scientific textual bases (papers and patents). ABRoad aims to develop an NLP software infrastructure which will support biomedical discovery using large-scale textual interpretation over scientific text (papers and patents). The project will use state-of-the-art methods in Deep Learning based text representation, such as transformers and graph neural networks, to support specialised inferences over large-scale corpora. The project aims to provide a universal (embeddings-based) textual interpretation platform to support the identification of new hypotheses in the life science space. The platform will integrate two main data modalities: textual and molecular representations. More specific target scenarios include support for drug discovery (e.g. drug repurposing), the determination of bioequivalent substances and the identification of novel antibiotics.

Background and Motivation. The HEP-VS is a key academic institution for the education of children, adolescents, and young adults in Valais. The potential transformation that AI will enable in education is unprecedented, as AI technologies open opportunities but also involve significant risks. To incorporate AI as part of present and future educational programs, and effectively serve the multiple actors in the cantonal educational ecosystem – including HEPVS students, HEPVS faculty and instructors, teachers and students in public schools - a systematic institutional approach to acquire AI literacy (knowledge, skills, and abilities needed to effectively interact with AI technologies) is paramount.

In parallel, Idiap, as a prominent institution in AI research, has been committed to inventing core advanced technologies, but also to design methods to facilitate the widespread understanding and use of these technologies, aligned with its “AI for Society’ mission. In particular, Idiap has recently developed a conceptual framework for AI Literacy, based on twelve competencies that span a range of needed knowledge, skills, and abilities, and focused on last-generation Generative AI technologies.

HEP-VS and Idiap started discussions in 2024 to envision a long-term collaboration as key cantonal institutions in education and AI, with the ambitious objective of devising a systematic approach to support the HEP-VS reflect upon, acquire, and spread AI Literacy, based upon the HEP-VS’s specific needs and priorities, and contextualized with different actors of the cantonal educational ecosystem.

Objective and methodology. The proposed one-year project presented here represents both a launching pad and a proof-of-concept of the Idiap/HEP-VS long-term collaboration. We will develop the systematic approach based on the twelve-competency Generative AI Literacy Framework and an “AI Literacy & Literacy with AI” approach that will integrate pedagogical, participatory, and technological methods. The project includes specific research activities to achieve 3 research objectives: (O1) assessment of the state of AI literacy for a population of HEP-VS students; (O2) co-design of customized educational modules for key components of the Generative AI Literacy Framework; and (O3) prototyping and initial testing of such modules. Importantly, some of the prototyping will use state-of-the-art AI as a tool to support AI literacy acquisition and assessment, thus integrating the expertise of Idiap in NLP and social computing, while HEP-VS will contribute to the research framework through its expertise, particularly in the areas of methodological design (educational sciences and disciplinary didactics), pedagogical design and experimentation (digital learning engineering and pedagogical facilitation), as well as in the analysis of the effectiveness and efficiency of classroom interventions and the evaluation of their implementation (statistics, cognitive psychology, epistemology of scientific inquiry, etc.). The conceptual framework proposed by Idiap for Gen AI literacy is directly aligned with the issue of effectiveness (learning outcomes) and efficiency (resource optimization) in the use of AI for learning across distinct disciplinary domains.

Expected outcomes and impact. As research outputs, the project will produce an assessment of the state of AI Literacy of a sample of HEP-VS students; educational resources for Generative AI Literacy (both consolidated from existing ones, as well as novel resources developed in the project); and joint scientific publications. Furthermore, the impact beyond concrete research outputs includes the consolidation of the bi-institutional partnership about this important priority theme for both HEP-VS and Idiap, and the establishment of first joint scientific results, which are needed to apply for larger research and educational projects.

This proposal aims to support a visit by Dr. Damien Teney, head of the Machine Learning group at the Idiap Research Institute, to the Australian Institute for Machine Learning (AIML) in Adelaide. Dr. Teney has an extensive history of successful collaborations with several scientists from the AIML. This visit will enable rapid progress on two key projects requiring intense collaboration due to the combination of multiple skillsets and domains of expertise. More specifically, these projects aim to improve our scientific understanding of the capabilities, limitations, and reliability characteristcs of large machine learning models. These topics are increasingly relevant on scientific, societal, and economical levels due to the growing importance and adoption of machine learning and AI at large. This proposal is strongly supported by the AIML's director since it addresses topics of mutual interest. The visit will benefit the two parties through the accelerated production of high-impact scientific knowledge. It will also contribute to international visibility of Swiss research capacity. The candidate additionally plans to prepare joint grant proposals with AIML scientists, as well as to promote future opportunities for Australian scientists to visit Swiss institutions. These activities will help sustain the partnership and ensure that its benefits extend beyond the duration of the visit.