Research Output

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

  Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős–Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.

  • Type:

    Article

  • Date:

    19 June 2018

  • Publication Status:

    Published

  • Publisher

    Springer Nature

  • DOI:

    10.1038/s41467-018-04316-3

  • Cross Ref:

    4316

  • Library of Congress:

    QA75 Electronic computers. Computer science

  • Dewey Decimal Classification:

    004 Data processing & computer science

  • Funders:

    Historic Funder (pre-Worktribe)

Citation

Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., & Liotta, A. (2018). Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1), 1-12. https://doi.org/10.1038/s41467-018-04316-3

Authors

Keywords

General Biochemistry, Genetics and Molecular Biology; General Physics and Astronomy; General Chemistry

Monthly Views:

Available Documents

  • pdf

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

    1MB

    Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
    © The Author(s) 2018

  • Downloadable citations

    HTML BIB RTF