Calculates a cross-tabulation of observed and predicted classes with associated statistics.
confusionMatrix(data, ...) # S3 method for default confusionMatrix(data, reference, positive = NULL, dnn = c("Prediction", "Reference"), prevalence = NULL, mode = "sens_spec", ...) # S3 method for table confusionMatrix(data, positive = NULL, prevalence = NULL, mode = "sens_spec", ...)
| data | a factor of predicted classes (for the default method) or an
object of class |
|---|---|
| ... | options to be passed to |
| reference | a factor of classes to be used as the true results |
| positive | an optional character string for the factor level that
corresponds to a "positive" result (if that makes sense for your data). If
there are only two factor levels, the first level will be used as the
"positive" result. When |
| dnn | a character vector of dimnames for the table |
| prevalence | a numeric value or matrix for the rate of the "positive"
class of the data. When |
| mode | a single character string either "sens_spec", "prec_recall", or "everything" |
a list with elements
the results of table on
data and reference
the positive result level
a numeric vector with overall accuracy and Kappa statistic values
the sensitivity, specificity, positive predictive
value, negative predictive value, precision, recall, F1, prevalence,
detection rate, detection prevalence and balanced accuracy for each class.
For two class systems, this is calculated once using the positive
argument
The functions requires that the factors have exactly the same levels.
For two class problems, the sensitivity, specificity, positive predictive
value and negative predictive value is calculated using the positive
argument. Also, the prevalence of the "event" is computed from the data
(unless passed in as an argument), the detection rate (the rate of true
events also predicted to be events) and the detection prevalence (the
prevalence of predicted events).
Suppose a 2x2 table with notation
| Reference | ||
| Predicted | Event | No Event |
| Event | A | B |
| No Event | C | D |
The formulas used here are: $$Sensitivity = A/(A+C)$$ $$Specificity = D/(B+D)$$ $$Prevalence = (A+C)/(A+B+C+D)$$ $$PPV = (sensitivity * prevalence)/((sensitivity*prevalence) + ((1-specificity)*(1-prevalence)))$$ $$NPV = (specificity * (1-prevalence))/(((1-sensitivity)*prevalence) + ((specificity)*(1-prevalence)))$$ $$Detection Rate = A/(A+B+C+D)$$ $$Detection Prevalence = (A+B)/(A+B+C+D)$$ $$Balanced Accuracy = (sensitivity+specificity)/2$$
$$Precision = A/(A+B)$$ $$Recall = A/(A+C)$$ $$F1 = (1+beta^2)*precision*recall/((beta^2 * precision)+recall)$$
where beta = 1 for this function.
See the references for discussions of the first five formulas.
For more than two classes, these results are calculated comparing each factor level to the remaining levels (i.e. a "one versus all" approach).
The overall accuracy and unweighted Kappa statistic are calculated. A
p-value from McNemar's test is also computed using
mcnemar.test (which can produce NA values with
sparse tables).
The overall accuracy rate is computed along with a 95 percent confidence
interval for this rate (using binom.test) and a
one-sided test to see if the accuracy is better than the "no information
rate," which is taken to be the largest class percentage in the data.
If the reference and data factors have the same levels, but in the incorrect order, the function will reorder them to the order of the data and issue a warning.
Kuhn, M. (2008), ``Building predictive models in R using the caret package, '' Journal of Statistical Software, (http://www.jstatsoft.org/article/view/v028i05/v28i05.pdf).
Altman, D.G., Bland, J.M. (1994) ``Diagnostic tests 1: sensitivity and specificity,'' British Medical Journal, vol 308, 1552.
Altman, D.G., Bland, J.M. (1994) ``Diagnostic tests 2: predictive values,'' British Medical Journal, vol 309, 102.
Velez, D.R., et. al. (2008) ``A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction.,'' Genetic Epidemiology, vol 4, 306.
as.table.confusionMatrix,
as.matrix.confusionMatrix, sensitivity,
specificity, posPredValue,
negPredValue, print.confusionMatrix,
binom.test
################### ## 2 class example lvs <- c("normal", "abnormal") truth <- factor(rep(lvs, times = c(86, 258)), levels = rev(lvs)) pred <- factor( c( rep(lvs, times = c(54, 32)), rep(lvs, times = c(27, 231))), levels = rev(lvs)) xtab <- table(pred, truth) confusionMatrix(xtab)#> Confusion Matrix and Statistics #> #> truth #> pred abnormal normal #> abnormal 231 32 #> normal 27 54 #> #> Accuracy : 0.8285 #> 95% CI : (0.7844, 0.8668) #> No Information Rate : 0.75 #> P-Value [Acc > NIR] : 0.0003097 #> #> Kappa : 0.5336 #> #> Mcnemar's Test P-Value : 0.6025370 #> #> Sensitivity : 0.8953 #> Specificity : 0.6279 #> Pos Pred Value : 0.8783 #> Neg Pred Value : 0.6667 #> Prevalence : 0.7500 #> Detection Rate : 0.6715 #> Detection Prevalence : 0.7645 #> Balanced Accuracy : 0.7616 #> #> 'Positive' Class : abnormal #>confusionMatrix(pred, truth)#> Confusion Matrix and Statistics #> #> Reference #> Prediction abnormal normal #> abnormal 231 32 #> normal 27 54 #> #> Accuracy : 0.8285 #> 95% CI : (0.7844, 0.8668) #> No Information Rate : 0.75 #> P-Value [Acc > NIR] : 0.0003097 #> #> Kappa : 0.5336 #> #> Mcnemar's Test P-Value : 0.6025370 #> #> Sensitivity : 0.8953 #> Specificity : 0.6279 #> Pos Pred Value : 0.8783 #> Neg Pred Value : 0.6667 #> Prevalence : 0.7500 #> Detection Rate : 0.6715 #> Detection Prevalence : 0.7645 #> Balanced Accuracy : 0.7616 #> #> 'Positive' Class : abnormal #>confusionMatrix(xtab, prevalence = 0.25)#> Confusion Matrix and Statistics #> #> truth #> pred abnormal normal #> abnormal 231 32 #> normal 27 54 #> #> Accuracy : 0.8285 #> 95% CI : (0.7844, 0.8668) #> No Information Rate : 0.75 #> P-Value [Acc > NIR] : 0.0003097 #> #> Kappa : 0.5336 #> #> Mcnemar's Test P-Value : 0.6025370 #> #> Sensitivity : 0.8953 #> Specificity : 0.6279 #> Pos Pred Value : 0.4451 #> Neg Pred Value : 0.9474 #> Prevalence : 0.2500 #> Detection Rate : 0.6715 #> Detection Prevalence : 0.7645 #> Balanced Accuracy : 0.7616 #> #> 'Positive' Class : abnormal #>#> Confusion Matrix and Statistics #> #> Reference #> Prediction setosa versicolor virginica #> setosa 11 16 23 #> versicolor 19 18 13 #> virginica 20 16 14 #> #> Overall Statistics #> #> Accuracy : 0.2867 #> 95% CI : (0.2159, 0.3661) #> No Information Rate : 0.3333 #> P-Value [Acc > NIR] : 0.9043 #> #> Kappa : -0.07 #> #> Mcnemar's Test P-Value : 0.8550 #> #> Statistics by Class: #> #> Class: setosa Class: versicolor Class: virginica #> Sensitivity 0.22000 0.3600 0.28000 #> Specificity 0.61000 0.6800 0.64000 #> Pos Pred Value 0.22000 0.3600 0.28000 #> Neg Pred Value 0.61000 0.6800 0.64000 #> Prevalence 0.33333 0.3333 0.33333 #> Detection Rate 0.07333 0.1200 0.09333 #> Detection Prevalence 0.33333 0.3333 0.33333 #> Balanced Accuracy 0.41500 0.5200 0.46000newPrior <- c(.05, .8, .15) names(newPrior) <- levels(iris$Species) confusionMatrix(iris$Species, sample(iris$Species))#> Confusion Matrix and Statistics #> #> Reference #> Prediction setosa versicolor virginica #> setosa 12 19 19 #> versicolor 18 18 14 #> virginica 20 13 17 #> #> Overall Statistics #> #> Accuracy : 0.3133 #> 95% CI : (0.2402, 0.3941) #> No Information Rate : 0.3333 #> P-Value [Acc > NIR] : 0.7257 #> #> Kappa : -0.03 #> #> Mcnemar's Test P-Value : 0.9930 #> #> Statistics by Class: #> #> Class: setosa Class: versicolor Class: virginica #> Sensitivity 0.2400 0.3600 0.3400 #> Specificity 0.6200 0.6800 0.6700 #> Pos Pred Value 0.2400 0.3600 0.3400 #> Neg Pred Value 0.6200 0.6800 0.6700 #> Prevalence 0.3333 0.3333 0.3333 #> Detection Rate 0.0800 0.1200 0.1133 #> Detection Prevalence 0.3333 0.3333 0.3333 #> Balanced Accuracy 0.4300 0.5200 0.5050