Machine Learning

Artificial Intelligence (AI) has become a powerful and pervasive technology in recent years, influencing numerous aspects of our daily lives. We encounter AI through recommendation algorithms in online stores, voice-activated smartphone assistants, or the widespread use of technologies like ChatGPT. However, its rapid growth and integration into society raise complex questions and concerns among the public. Public opinions on AI vary widely; while some people are enthusiastic about its potential to revolutionize industries and enable breakthroughs in fields such as medicine, others fear that AI could lead to undesirable outcomes, such as a loss of human control or privacy. These views are often shaped by media narratives and the competing interests of different stakeholders, which play a significant role in influencing both public opinion and policy decisions.

Our team of scientists and communication experts aims to enhance the understanding of AI technologies among the Swiss people, with a particular focus on teenagers and female students, to create a positive societal impact. Building on the foundations of our previous project, NewsOnAI, we will expand beyond traditional media such as newspapers and employ diverse methods, including artistic performances and interactive exhibitions. We will design these activities to be highly interactive, encouraging active participation and dialogue. Activities will include themed theater plays that explore AI’s impact on everyday life, exhibitions where participants can interact with AI tools, and workshops specifically designed for teenagers and female students to discuss AI’s future role in society. Feedback collected will include real-time audience reactions, structured questionnaires, and focus group discussions, which will be analyzed to continuously refine and adapt our engagement strategies.

Our primary audience includes Swiss citizens interested in cultural activities, particularly teenagers who are keen to follow new trends. Additionally, we are committed to addressing gender aspects by designing content and activities that specifically appeal to female students. We aim to inspire and empower young women to take on more prominent roles in shaping the digital world, acknowledging that they have historically been underrepresented in these fields. As societal attention shifts toward greater inclusion, our project will contribute to fostering a more balanced and equitable digital future.

While many individuals in our target groups may lack in-depth technical knowledge of AI, they often encounter new AI products, companies, and social issues through various media channels, including newspapers and science fiction movies. As a result, they may be aware of recent developments but also susceptible to misunderstandings and controversies related to technologies such as ChatGPT, Elon Musk's brain-chip startup, and other emerging AI applications. It is crucial to recognize that media portrayal significantly influences public opinion on AI, both positively and negatively. Media creators, even if they are not experts in AI, often produce content that captures public attention, which high-profile figures, including entrepreneurs, CEOs, and politicians, may leverage to advance their agendas. This can sometimes lead to skewed public perceptions, whether intentionally or unintentionally. Given this landscape, it is essential for AI scientists to collaborate with media creators, providing evidence-based insights to ensure accurate and balanced information is shared with the public. Our project fosters such collaboration, ensuring that both the potential and limitations of AI are clearly communicated. By sharing our findings through diverse media outlets, we aim to reach a broad audience, extending beyond Switzerland. Furthermore, our proactive engagement efforts will foster dynamic, two-way communication between scientists and the public, using interactive methods in exhibitions and theater plays to engage teenagers and female students specifically. Analyzing the feedback from these initiatives will provide invaluable insights into public perspectives on emerging technologies. This understanding will guide scientists in pursuing research directions that effectively address societal concerns, demonstrating the tangible benefits of our project for both scientific advancement and societal well-being. We anticipate that our efforts will have a multiplying social impact over time, promoting informed public discourse and a deeper understanding of AI technologies.

Large Language Models (LLMs) are trained on broad data, using self-supervision at scale, to complete a wide range of tasks. Wider use of LLMs has risen in recent months due to applications such as ChatGPT. Although LLMs bring many opportunities to improve our everyday lives, the impacts on humans and society have not yet been prioritized or fully understood. Given the rapid development of these tools, the risk of negative implications is significant if LLMs are not developed and deployed in a way that is aligned with human values and responds to individual needs and preferences. To mitigate any negative consequences, academia, in close collaboration with industry, needs to train the next generation of researchers to understand the complexities of the socio-technical implications surrounding the use of LLMs.
The alignAI Doctoral Network will train 17 doctoral candidates (DCs) to work in the international and highly interdisciplinary field of LLM research and development. The core of the project focuses on the alignment of LLMs with human values, identifying relevant values and methods for alignment implementation. Two principles provide a foundation for the approach. First, explainability is a key enabler for all aspects of trustworthiness, accelerating development, promoting usability, and facilitating human oversight and auditing of LLMs. Second, fairness is a key aspect of trustworthiness, facilitating access to AI applications and ensuring equal impact of AI-driven decision-making. The practical relevance of the project is ensured by three use cases in education, positive mental health, and news consumption. This approach allows us to develop specific guidelines and test prototypes and tools to promote value alignment. We follow a unique methodological approach, with DCs from social sciences and humanities “twinned” with DCs from technical disciplines for each use case (9 DCs in total), while the other 8 DCs carry out horizontal research across the use cases.

We address the controllability of large language models (LLMs) by giving them interpretable beliefs and programmable knowledge through leveraging the PI's work on understanding and improving transformer embeddings. Transformers' empirical success comes from the attention function's ability to induce graphs of relations from text. Our recent work has extended this ability to knowledge graphs, and to inducing the nodes of the graph as well, known as entity induction, with the first variational-Bayesian generalisation of the attention mechanism. This project will further develop this information-theoretic understanding of transformer embeddings and its sparsity-inducing regulariser, for learning graphs of higher-level abstract entities. The resulting Bayesian beliefs over generalised transformer embeddings of texts and graphs will give us the more interpretable, more programmable and more learnable abstract representations which are the core of this proposed project.

To leverage and extend these fundamental advances in representation learning, we will develop LLM architectures with a memory. Motivated by the success of Retrieval Augmented LLMs, our Belief Augmented Language Models (BALMs) will move knowledge extracted from training data out of large uninterpretable weight matrices into our interpretable Bayesian beliefs over large transformer embeddings. These beliefs will then be: augmented with human-editable knowledge graphs and selected new texts, refined with control objectives and multi-hop reasoning, and combined with inference of concensus beliefs and opinion summarisation. BALMs will be developed both to evaluate these beliefs and as a chat interface for specifying, accessing and editing the beliefs themselves, including the collaborative specification of shared beliefs. These fundamental advances in deep learning theory and architectures will allow us to control what an LLM says by controlling what it believes, thereby unlocking the power of AI for society.

In recent years, attention-based models like Transformers have radically improved the performance of natural language understanding (NLU), demonstrating the appropriateness of attention-based representation for language. In (Henderson, 2020) we show that these representation share many characteristics with those found in traditional computational linguistics (e.g. graph structure), except that they do not automatically learn multiple levels of representation nor their entities (morphemes, phrases, discourse entities, etc). Motivated by this challenge of entity induction, our recent work has discovered a very non-traditional perspective, which characterises attention-based models like Transformers as doing nonparametric Bayesian inference. Given an input text, our Nonparametric Variational Information Bottleneck (NVIB) Transformer infers distributions over nonparametric mixture distributions (Henderson and Fehr, 2023). We have even shown that pretrained Transformers can be converted into equivalent NVIB Transformers, and regularised post-training (Fehr and Henderson, 2023).

This reinterpretation of Transformers, combined with their unprecedented empirical success, leads us to postulate the hypothesis that natural language understanding is nonparametric variational Bayesian inference over mixture distributions. This claim of the adequacy of NVIB leads to two fundamental challenges which are not currently being addressed, each with an associated technological aim:

1. How can NVIB support inducing graph-structured representations at multiple levels of representation?
   Making deep learning representations interpretable.

2. How can NVIB enable controlling the information in representations?
   Making deep learning representations controllable.

For the first challenge, we will extend our previous structure processing methods (Mohammadshahi and Henderson, 2020, 2021, 2023; Miculicich and Henderson, 2022), developed for set-of-vector representations, to mixture-of-component distributions. And we will focus on unsupervised learning methods, rather than our previous supervised learning methods. To extend these models to multiple levels, we will take the approach of embedding all levels in one big mixture of non-homogeneous components, which are computed with iterative refinement. This extends our previous work on iterative graph refinement (Mohammadshahi and Henderson, 2021; Miculicich and Henderson, 2022), adding the induction of the nodes of the graph and the induction of multiple levels of representation. Learning representations which are interpretable as linguistic structures will be a testbed for the general aim of deep learning of interpretable representations.

For the second challenge, we will leverage the information theory behind NVIB to model both inferring implicit information and removing private information. We will investigate the use of KL divergence as a measure of entailment in semantic inference. We will apply the framework of Rényi differential privacy (Mironov, 2017) to provide privacy guarantees by adding noise which removes targeted information from Transformer embeddings. This method extends differential privacy to anything that can be embedded with a Transformer (especially text), with many important applications. These methods address the general aim of controlling the information in deep learning representations.

Addressing these challenges will lead to fundamental advances in machine learning, including novel deep learning architectures and fundamental insights into Transformers and their pretraining. We will do both intrinsic and extrinsic evaluations of our induced representations, expecting to show improvements on core NLP tasks, including privacy-preserving sharing of textual data. Given the current level of interest in the AI research and development community for Transformers and variational Bayesian methods, we expect the proposed research to have a profound impact on the field.

Alpine lakes (those located above the 2000 m tree line) are excellent sentinels of climate change as their chemistry and biology respond rapidly to environmental forcing. The Swiss alps are host to over 1500 alpine lakes, many of which have been newly mapped and thus never been studied2. Microorganisms play major ecological roles in these ecosystems, including primary production, cycling of elements, and attenuation of contaminants, but it is uncertain how physical climatic changes may affect microbial communities and their activities in alpine lakes. This project aims to: (i) record and monitor the unexplored microbial diversity in Swiss alpine lakes, and (ii) engage citizens in science and spread awareness about environmental conservation through participation in our field campaigns. In summary, 2000LAKES is a project of alpine citizen science aiming to understand the ecological impacts of climate change in alpine lakes and to promote the conservation of alpine microbial ecosystems joining forces between scientists and citizens.

In previous years, we have focused our automatic speech recognition (ASR) research with Samsung on accents and on multi-linguality. In this year, we propose to focus on “Natural” user interfaces. By natural, we mean that the interface should function in such a way that the user should not have to behave differently from when he or she interacts with a person. Of course, there are many facets to this; however, two are pertinent: conversational/spontaneous speech and recognition exploiting natural sensors. Speech user interfaces typically rely on being able to place a microphone close to the user’s mouth. This maximizes the volume and clarity of the speech signal, whilst minimizing the effect of other noise in the vicinity. Such an interface is natural for, say, a telephone. However, many applications do not lend themselves to this type of interface. Examples include most home electronics, where the user might typically be in the center of a room, but the device is near a wall. In the case of televisions, a useful intermediate device is the remote control. Nevertheless, it is still inconvenient to hold a remote control like a telephone in order to talk to it.

Construction of a Natural Language Processing (NLP) infrastructure to support domain experts in biomedical discovery over large scientific textual bases (papers and patents). ABRoad aims to develop an NLP software infrastructure which will support biomedical discovery using large-scale textual interpretation over scientific text (papers and patents). The project will use state-of-the-art methods in Deep Learning based text representation, such as transformers and graph neural networks, to support specialised inferences over large-scale corpora. The project aims to provide a universal (embeddings-based) textual interpretation platform to support the identification of new hypotheses in the life science space. The platform will integrate two main data modalities: textual and molecular representations. More specific target scenarios include support for drug discovery (e.g. drug repurposing), the determination of bioequivalent substances and the identification of novel antibiotics.

Current state-of-the-art automatic speech recognition (ASR) systems commonly use hidden Markov models (HMMs), where phonemes (phones) are assumed to be the intermediate subword units and each word to be recognized is explicitly modeled as a sequence of phonemes. Thus, despite availability of sophisticated statistical modeling or machine learning techniques, to develop an ASR system one requires prior knowledge, such as lexical resources (e.g., phoneme set, lexicon) and some minimum phonetic expertise. The lexicon in ASR system contains phonetic transcription of each word. One of the key aspect in lexicon development is learning the relation between graphemes/alphabets and phonemes. Often this is done by applying statistical methods such as, decision trees, conditional random fields which invariably rely on the availability of an initial lexicon that contains good quality pronunciations. Major languages such as, English, French, German, Spanish have well developed lexical resources. However, there are minority languages, such as Scottish Gaelic, Afrikaans that do not have such well developed lexical resources. Thus, development of ASR systems have mainly focussed towards major languages. Recently, at Idiap we have developed a novel approach which, with the aid of new statistical models, learns/captures probabilistic relation between graphemes and phonemes through acoustic data. This has opened up multiple opportunities for further development and research. For instance, this approach allows the possibility to exploit both lexical and acoustic resources from one or multiple languages to develop lexical resources for another language. In addition, it allows the possibility to develop an ASR system where instead of phonemes units automatically derived from acoustic data are used as subword units. Such systems are of utmost interest to all languages for rapid development and deployment of ASR systems. The goal of the present project is to exploit the novel approach to a) develop a framework for flexible development of lexical resources for both major and minority languages, and b) develop an ASR system that overcomes the need for linguistically motivated subword units (i.e., phonemes) or prior lexical resources, while yielding state-of-the-art performance.