Automatic Designs in Deep Neural Networks

Link: https://bluejeans.com/490786993

Abstract: With the big success of deep neural networks (DNNs) in vision, language, speech and many other Machine Learning (ML) fields, lots of efforts are put into designing better deep neural network models. Researchers have explored different dimensions of the design space for DNNs to improve its performance. However, the design is often limited by human decisions and requires expert knowledge. Broader space and novel aspects may be explored with the help of automation. This dissertation explores automated designs of deep neural networks, in three dimensions including data design, model design and loss design.

This dissertation presents several contributions towards automated designs in the three dimensions in deep neural network designs. In data design, to address the problem of difficulties to obtain training data in some challenging cases due to cost or rarity constraints, we propose a synthetic data generation method to automatically generate synthetic data for certain tasks. In model design, to explore larger architecture space and more flexible architecture choices, we make efforts towards architecture search and dynamic architectures. In loss design, to optimize for non-differentiable performance metrics, instead of manually proposing differentiable surrogate losses for every task with substantial effort, we propose a unified framework that can optimize for different tasks and metrics with little manual effort.