Pruning (artificial neural network): Difference between revisions
Citation bot (talk | contribs) Add: eprint, class, authors 1-1. Removed parameters. Some additions/deletions were parameter name changes. | Use this bot. Report bugs. | Suggested by Headbomb | Pages linked from cached Wikipedia:WikiProject_Academic_Journals/Journals_cited_by_Wikipedia/Sandbox | via #UCB_webform_linked 23/26 |
Elaborated further of the subject adding sections on types of pruning. Tags: Reverted Visual edit |
||
| Line 4: | Line 4: | ||
}} |
}} |
||
In the context of [[artificial neural network]], '''pruning''' is the practice of removing parameters (which may entail removing individual parameters, or parameters in groups such as by [[artificial neurons|neurons]]) from an existing network.<ref>{{cite arxiv|last1=Blalock|first1=Davis|last2=Ortiz|first2=Jose Javier Gonzalez|last3=Frankle|first3=Jonathan|last4=Guttag|first4=John|date=2020-03-06|title=What is the State of Neural Network Pruning?|class=cs.LG|eprint=2003.03033}}</ref> The goal of this process is to maintain accuracy of the network while increasing its efficiency. This can be done to reduce the computational resources required to run the neural network. |
In the context of [[artificial neural network]], '''pruning''' is the practice of removing parameters (which may entail removing individual parameters, or parameters in groups such as by [[artificial neurons|neurons]]) from an existing network.<ref>{{cite arxiv|last1=Blalock|first1=Davis|last2=Ortiz|first2=Jose Javier Gonzalez|last3=Frankle|first3=Jonathan|last4=Guttag|first4=John|date=2020-03-06|title=What is the State of Neural Network Pruning?|class=cs.LG|eprint=2003.03033}}</ref> The goal of this process is to maintain accuracy of the network while increasing its efficiency. This can be done to reduce the computational resources required to run the [[neural network]]. After a network is trained to a desired solution with the training data, units (hidden layer nodes or interconnections) are analysed to determine which are not contributing to the solution. There are several approaches described in the literature to determine non-contributing units. A widely used approach is that non-essential units can be determined by a form of sensitivity analysis.So far, few studies have been performed in the analysis of pruning algorithms for classification of remotely-sensed data. |
||
A basic algorithm for pruning is as follows:<ref>Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). ''Pruning convolutional neural networks for resource efficient inference''. arXiv preprint arXiv:1611.06440.</ref><ref>[https://jacobgil.github.io/deeplearning/pruning-deep-learning Pruning deep neural networks to make them fast and small].</ref> |
A basic algorithm for pruning is as follows:<ref>Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). ''Pruning convolutional neural networks for resource efficient inference''. arXiv preprint arXiv:1611.06440.</ref><ref>[https://jacobgil.github.io/deeplearning/pruning-deep-learning Pruning deep neural networks to make them fast and small].</ref> |
||
#Evaluate the importance of each neuron. |
#Evaluate the importance of each neuron. |
||
#Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance"). |
#Rank the [[neurons]] according to their importance (assuming there is a clearly defined measure for "importance"). |
||
#Remove the least important neuron. |
#Remove the least important neuron. |
||
#Check a termination condition (to be determined by the user) to see whether to continue pruning. |
#Check a termination condition (to be determined by the user) to see whether to continue pruning. |
||
== Types == |
|||
=== Magnitude based Pruning === |
|||
The [[Magnitude Based pruning technique]] (MB) is the simplest pruning [[algorithm]] . It is based on deleting interconnections with small ‘[[Salience (neuroscience)|saliency]]’, i.e. those whose deletion will have the least effect on the training error. Saliency corresponds to the magnitude (weight) value of the [[interconnections]]. It is assumed that the interconnections whose magnitude value is small will have minor effect on the performance of the network. After reasonable initial training, the interconnection having the smallest magnitude value is removed. The network is then retrained and the process is repeated in an iterative fashion until the training error reaches a certain limit. |
|||
=== Optimum Brain Damage === |
|||
The [https://nyuscholars.nyu.edu/en/publications/optimal-brain-damage Optimum Brain Damage pruning algorithm] (OBD), introduced by [http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf Le Cun, Denker and Solla] in 1990, is based on second order derivatives of the error function. The aim is to iteratively delete the weights whose deletion will result in the least increase of the error in the network. There is an important problem in the estimation of formula, is the size of the Hessian matrix. The calculation of the Hessian matrix is time-consuming, hence Le Cun et al. (1990) assume that the [[Hessian]] is diagonal. On the other hand, Hassibi and Stork (1993) argue that Hessian for every problem they have considered are strongly non-diagonal, and this leads OBD to eliminate the wrong weights |
|||
== References == |
== References == |
||
Revision as of 06:32, 19 May 2021
 | This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages)
|
In the context of artificial neural network, pruning is the practice of removing parameters (which may entail removing individual parameters, or parameters in groups such as by neurons) from an existing network.[1] The goal of this process is to maintain accuracy of the network while increasing its efficiency. This can be done to reduce the computational resources required to run the neural network. After a network is trained to a desired solution with the training data, units (hidden layer nodes or interconnections) are analysed to determine which are not contributing to the solution. There are several approaches described in the literature to determine non-contributing units. A widely used approach is that non-essential units can be determined by a form of sensitivity analysis.So far, few studies have been performed in the analysis of pruning algorithms for classification of remotely-sensed data.
A basic algorithm for pruning is as follows:[2][3]
- Evaluate the importance of each neuron.
- Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").
- Remove the least important neuron.
- Check a termination condition (to be determined by the user) to see whether to continue pruning.
Types
Magnitude based Pruning
The Magnitude Based pruning technique (MB) is the simplest pruning algorithm . It is based on deleting interconnections with small ‘saliency’, i.e. those whose deletion will have the least effect on the training error. Saliency corresponds to the magnitude (weight) value of the interconnections. It is assumed that the interconnections whose magnitude value is small will have minor effect on the performance of the network. After reasonable initial training, the interconnection having the smallest magnitude value is removed. The network is then retrained and the process is repeated in an iterative fashion until the training error reaches a certain limit.
Optimum Brain Damage
The Optimum Brain Damage pruning algorithm (OBD), introduced by Le Cun, Denker and Solla in 1990, is based on second order derivatives of the error function. The aim is to iteratively delete the weights whose deletion will result in the least increase of the error in the network. There is an important problem in the estimation of formula, is the size of the Hessian matrix. The calculation of the Hessian matrix is time-consuming, hence Le Cun et al. (1990) assume that the Hessian is diagonal. On the other hand, Hassibi and Stork (1993) argue that Hessian for every problem they have considered are strongly non-diagonal, and this leads OBD to eliminate the wrong weights
References
- ^ Blalock, Davis; Ortiz, Jose Javier Gonzalez; Frankle, Jonathan; Guttag, John (2020-03-06). "What is the State of Neural Network Pruning?". arXiv:2003.03033 [cs.LG].
- ^ Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
- ^ Pruning deep neural networks to make them fast and small.
 | This artificial intelligence-related article is a stub. You can help Wikipedia by expanding it. |