Net reclassification improvement: Difference between revisions

Content deleted Content added

Inline

Revision as of 20:35, 30 July 2019

Net reclassification improvement (NRI) is an index that attempts to quantify how well a new model reclassifies subjects - either appropriately or inappropriately - as compared to an old model.^[1] While c-statistics or AUC has been the standard metric for quantifying improvements over the last few decades, several studies have analyzed the limitations of this metric including lack of clinical relevance and difficulty in interpretation of small magnitude changes.^[2]^[3] This limitation can be best seen in the example of HDL and Framingham Risk Score (FRS). When a models with and without HDL were analyzed with AUC regarding effect of HDL of modifying FRS, HDL was found not to have a statistical significant effect. However, when analyzed in terms of outcomes, HDL was found to be a significant predictor of heart disease and thus should affect FRS.^[4] To overcome this limitation the concept of reclassification, that is how well a new model correctly reclassifies cases, was introduced through the metric of NRI.^[5]

Basic Concept

NRI attempts to quantify how well a new model correctly reclassifies subjects. Typically this comparison is between an original model (e.g. hip fractures as a function age and sex) and a new model which is the original model plus one additional component (e.g. hip fractures as a function of age, sex, and weight). NRI is composed of two components, subjects without events and subjects with events. Subject without (with) events who were correctly reclassified lower (higher) are assigned a +1. Subjects without (with) events who were incorrectly classified as higher (lower) are assigned a -1. Subjects not reassigned are assigned a 0. Sum the scores in each group and divide by the number of subjects in that group. The sum of these two values is the NRI.

Example

NRI example table
Event		Test 1		Total, split	Total
Non-event		Abnormal	Normal	Total, split	Total
Test 2	Abnormal	18	4	22	28
	Abnormal	2	4	6	28
	Normal	2	6	8	72
	Normal	8	56	64	72
Total, split		20	10	30
Total, split		10	60	70
Total		30	70		100

In a perfect test, all subjects with events would be classified as abnormal and all subjects without events would be classified as normal. Bold indicates subjects correctly classified by both tests. White indicates subjects incorrectly classified by both tests. Green indicates subjects correctly reclassified by test 2. Red indicates subjects incorrectly reclassified by test 2. NRI_e = (4-2)/30 = 0.067. NRI_ne = (8-4)/70 = 0.057. NRI is the sum which is approximately 0.12.

Limitations

NRI limitations include determining whether a subject has been "correctly" reclassified and issues when the models results are not binary (e.g. low, medium, and high risk). A paper by Pepe et al. states that despite its popularity NRI appears to give misleading results ^[6].

References

^ Leening MJG, Vedder MM, Witteman JCM, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med. 2014;160(2):122-131.
^ 1. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935.
^ Pencina MJ, D’Agostino RB, Pencina KM, Janssens ACJW, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176(6):473-481.
^ Steyerberg EW, Calster BV, Pencina MJ. Performance Measures for Prediction Models and Markers: Evaluation of Predictions and Classifications. Revista Española de Cardiología (English Edition). 2011;64(9):788-794
^ Pencina MJ, D’Agostino RB, D’Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-212.
^ Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J. (2015). The net reclassification index (NRI): a misleading measure of prediction improvement even with independent test data sets. Statistics in biosciences, 7(2), 282-295.

External links

http://en.citizendium.org/wiki/Sensitivity_and_specificity#Reclassification_tables

[1] Leening MJG, Vedder MM, Witteman JCM, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med. 2014;160(2):122-131.

[2] 1. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935.

[3] Pencina MJ, D’Agostino RB, Pencina KM, Janssens ACJW, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176(6):473-481.

[4] Steyerberg EW, Calster BV, Pencina MJ. Performance Measures for Prediction Models and Markers: Evaluation of Predictions and Classifications. Revista Española de Cardiología (English Edition). 2011;64(9):788-794

[5] Pencina MJ, D’Agostino RB, D’Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-212.

[6] Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J. (2015). The net reclassification index (NRI): a misleading measure of prediction improvement even with independent test data sets. Statistics in biosciences, 7(2), 282-295.

[1]

[2]

[3]

[4]

[5]

[6]

@@ Line 10: / Line 10: @@
 |+ NRI example table
 |-
-| colspan="2" style="background: yellow" | Event
+| colspan="2" style="background: green" | Event
 ! colspan="2" | Test 1
 | rowspan="2" | Total, split
 | rowspan="2" | Total
 |-
-| colspan="2" style="background: blue" | Non-event
+| colspan="2" style="background: lightblue" | Non-event
 | Abnormal
 | Normal
@@ Line 21: / Line 21: @@
 ! rowspan="4" | Test 2
 | rowspan="2" | Abnormal
-| style="background: yellow"| '''18'''
+| style="background: green"| '''18'''
-| style="background: yellow; color:green"|4
+| style="background: green; color:green"|4
-| style="background: yellow"|22
+| style="background: green"|22
 | rowspan="2" | 28
 |-
-| style="background: blue; color:white"| 2
+| style="background: lightblue; color:white"| 2
-| style="background: blue; color:red"| 4
+| style="background: lightblue; color:red"| 4
-| style="background: blue"| 6
+| style="background: lightblue"| 6
 |-
 | rowspan="2" | Normal
-| style="background: yellow; color:red"| 2
+| style="background: green; color:red"| 2
-| style="background: yellow; color:white"| 6
+| style="background: green; color:white"| 6
-| style="background: yellow"| 8
+| style="background: green"| 8
 | rowspan="2" | 72
 |-
-| style="background: blue; color:green"| 8
+| style="background: lightblue; color:green"| 8
-| style="background: blue"| '''56'''
+| style="background: lightblue"| '''56'''
-| style="background: blue"| 64
+| style="background: lightblue"| 64
 |-
 | rowspan="2" colspan="2" | Total, split
-| style="background: yellow"| 20
+| style="background: green"| 20
-| style="background: yellow"| 10
+| style="background: green"| 10
-| style="background: yellow"| 30
+| style="background: green"| 30
 | rowspan="2" |
 |-
-| style="background: blue"| 10
+| style="background: lightblue"| 10
-| style="background: blue"| 60
+| style="background: lightblue"| 60
-| style="background: blue"| 70
+| style="background: lightblue"| 70
 |-
 | colspan="2" | Total
@@ Line 57: / Line 57: @@
 |}
 In a perfect test, all subjects with events would be classified as abnormal and all subjects without events would be classified as normal. Bold indicates subjects correctly classified by both tests. White indicates subjects incorrectly classified by both tests. Green indicates subjects correctly reclassified by test 2. Red indicates subjects incorrectly reclassified by test 2. NRI<sub>e</sub> = ({{font color|green|4}}-{{font color|red|2}})/30 = 0.067. NRI<sub>ne</sub> = ({{font color|green|8}}-{{font color|red|4}})/70 = 0.057. NRI is the sum which is approximately 0.12.
 ==Limitations==