Tag - Machine learning

Greg Yang: The unreasonable effectiveness of mathematics in large scale deep learning

23rd March, 2023

Recently, the theory of infinite-width neural networks led to the first technology, muTransfer, for tuning enormous neural networks that are too expensive to train more than once. For example, this allowed us to tune the 6.7 billion parameter version of GPT-3 using only 7% of its pretraining compute budget, and with some asterisks, we get a performance comparable to the original GPT-3 model with twice the parameter count. In this talk, I will explain the core insight behind this theory. In fact, this is an instance of what I call the Optimal Scaling Thesis, which connects infinite-size limits for general notions of 'size' to the optimal design of large models in practice, illustrating a way for theory to reliably guide the future of AI. I’ll end with several concrete key mathematical research questions whose resolutions will have incredible impact on how practitioners scale up their NNs.

Julián David Gutiérrez Pineda: Machine learning architectures for mean-field games models of price formation

29th September, 2022

In this talk, we approach the solution of mean-field game systems arising in price formation models employing machine learning. We use a min-max characterization of the optimal control and price variables. We guarantee the convergence of the training algorithm using first-order conditions of the underlying optimal control problem. Numerical results for linear-quadratic models illustrate our results.

Wenqing Li: Backdoor attack detection in deep neural networks: a coherence optimization based method

13th June, 2022

Backdoor attacks impose a new threat in Deep Neural Networks (DNNs), where a backdoor is inserted into the neural network by poisoning the training dataset, misclassifying inputs that contain the adversary trigger. The major challenge for defending against these attacks is that only the attacker knows the secret trigger and the target class. The problem is further exacerbated by the recent introduction of Hidden Triggers, where the triggers are carefully fused into the input, bypassing detection by human inspection and causing backdoor identification through anomaly detection to fail. To defend against such attacks, in this work we systematically analyze how representations, i.e., the set of neuron activations for a given DNN when using the training data as inputs, are affected by backdoor attacks. We propose PiDAn, an algorithm based on coherence optimization purifying the poisoned data. Our analysis shows that representations of poisoned data and authentic data in the target class are still embedded in different linear subspaces, which implies that they show different coherence with some latent spaces. Based on this observation, the proposed PiDAn algorithm learns a sample-wise weight vector to maximize the projected coherence of weighted samples, where we demonstrate that the learned weight vector has a natural grouping effect'' and is distinguishable between authentic data and poisoned data. This enables the systematic detection and mitigation of backdoor attacks. Based on our theoretical analysis and experimental results, we demonstrate the effectiveness of PiDAn in defending against backdoor attacks that use different settings of poisoned samples on GTSRB and ILSVRC2012 datasets in comparison with the state-of-the-art methods. Our PiDAn algorithm can detect more than 90% infected classes and identify 95% poisoned samples.

Julian Barreiro-Gomez: The role of stochastic differential games of mean-field type in smart cities applications

13th June, 2022

This brief talk aims to show how the stochastic differential games contribute to the optimal solution of large-scale engineering problems emerging in smart cities where several dynamical interactions occur, e.g., the water distribution system, the crowd management, the traffic flow, power systems, among many others. We show that the general simplest problem statement leads to a complex PIDE system involving a backward Hamilton-Jacobi-Bellman equation coupled with a forward Fokker-Plank-Kolmogorov equation. Then, we discuss how this complexity can be handled for specific cases pursuing to develop real implementation. As an example, we focus on the crowd evacuation problem. Finally, future directions we are currently working on involving machine learning and stability are presented.

Sylvain Lamprier: Spatio-temporal and Video Prediction with State-based Neural Models

1st February, 2022

An increasing number of works combining neural networks and differential equations for spatio-temporal forecasting have been proposed for the last few years. Some of them show substantial improvements for the prediction of dynamical systems or videos compared to standard RNNs by defining the dynamics using learned ODEs. In this talk, we first present our recent approach for adapting such works for stochastic data. We introduce a novel dynamic model for stochastic video prediction which, unlike prior image-autoregressive models, decouples frame synthesis and dynamics. The dynamics of the model are governed in a latent space by a residual update rule, which is motivated by discretization schemes of differential equations. This endows our method with several desirable properties, such as temporal efficiency and latent space interpretability. Then, we will present a second method, more specifically focused on learning disentangled spatial and temporal representations of spatio-temporal phenomena, with the aim of more accurately predicting future tendencies from initial observations. We propose to model the evolution of partially observed spatiotemporal phenomena with unknown dynamics by taking inspiration from a formal method for the analytical resolution of PDEs: the functional separation of variables. We experimentally demonstrate the performance and broad applicability of our method against prior state-of-the-art models on physical and synthetic video datasets.

Cynthia Rudin: Interpretable Machine Learning for High-Stakes Decisions

19th January, 2022

With widespread use of machine learning, there have been serious societal consequences from using black box models for high-stakes decisions in criminal justice, healthcare, financial lending, and beyond. Interpretability of machine learning models is critical when the cost of a wrong decision is high. Throughout my career, I have had the opportunity to work with power engineers, doctors, and police detectives. Using interpretable models has been the key to allowing me to help them with important high-stakes societal problems. Interpretability can bring us out of the 'dark' age of the black box into the age of insight and enlightenment.

Patrick Shafto: Mathematical foundations for human-level intelligence: Cooperative communication as belief transport

22nd November, 2021

Human learning outstrips modern machine learning and AI in at least three abilities: rapid robust learning, in effectively open worlds, in near-real time with very little energy. Mathematical formalization of signature human abilities has the possibility to advance machine learning theory and practice. In this talk, I will demonstrate the possibility of rigorous mathematical formalizations of rapid and robust learning via cooperative communication, and discuss generalizations and directions.

Sanjeev Arora: Toward Theoretical Understanding of Deep Learning

12th September, 2018

The empirical success of deep learning drives much of the excitement about machine learning today. This success vastly outstrips our mathematical understanding. This lecture surveys progress in recent years toward developing a theory of deep learning. Works have started addressing issues such as speed of optimization, sample requirements for training, effect of architecture choices, and properties of deep generative models.