Vision, Learning, and Acceleration (VLA-Lab) Seminar Series - VLA-Lab Seminar | Finding Good, Cheap and Transferrable Sparsity

Sparsity is commonly produced from the model compression (i.e., pruning), which eliminates unnecessary parameters. However, identifying high-quality sparse patterns without sacrificing performance is extremely expensive. Fortunately, to amortize the cost, our recent work demonstrates one kind of sparsity capable of universally transferring across diverse downstream tasks. It provides an efficient and effective "copy" of its dense counterpart especially valuable for enormous pre-trained models. Beyond this, sparsity also serves as an important tool for modeling the underlying low dimensionality of NNs, for understanding their generalization, implicit regularization, expressivity, and robustness. In this talk, I will first discuss, practically, why the sparsity is treasured and how to find it. Then I will present, theoretically, how to understand the universal transferability of our located sparse NNs. Finally, I will describe what is the prospect of exploiting sparsity in future.