Confusion matrices compare two classifications (usually one done automatically using a machine learning algorithm versus the true classification done by a specialist... but one can also compare two automatic or two manual classifications against each other).
Usage
confusion(x, ...)
# S3 method for default
confusion(
x,
y = NULL,
vars = c("Actual", "Predicted"),
labels = vars,
merge.by = "Id",
useNA = "ifany",
prior,
...
)
# S3 method for mlearning
confusion(
x,
y = response(x),
labels = c("Actual", "Predicted"),
useNA = "ifany",
prior,
...
)
# S3 method for confusion
print(x, sums = TRUE, error.col = sums, digits = 0, sort = "ward.D2", ...)
# S3 method for confusion
summary(object, type = "all", sort.by = "Fscore", decreasing = TRUE, ...)
# S3 method for summary.confusion
print(x, ...)Arguments
- x
an object with a
confusion()method implemented.- ...
further arguments passed to the method.
- y
another object, from which to extract the second classification, or
NULLif not used.- vars
the variables of interest in the first and second classification in the case the objects are lists or data frames. Otherwise, this argument is ignored and
xandymust be factors with same length and same levels.- labels
labels to use for the two classifications. By default, they are the same as
vars, or the one in the confusion matrix.- merge.by
a character string with the name of variables to use to merge the two data frames, or
NULL.- useNA
do we keep
NAs as a separate category? The default"ifany"creates this category only if there are missing values. Other possibilities are"no", or"always".- prior
class frequencies to use for first classifier that is tabulated in the rows of the confusion matrix. For its value, see here under, the
value=argument.- sums
is the confusion matrix printed with rows and columns sums?
- error.col
is a column with class error for first classifier added (equivalent to false negative rate of FNR)?
- digits
the number of digits after the decimal point to print in the confusion matrix. The default or zero leads to most compact presentation and is suitable for frequencies, but not for relative frequencies.
- sort
are rows and columns of the confusion matrix sorted so that classes with larger confusion are closer together? Sorting is done using a hierarchical clustering with
hclust(). The clustering method is"ward.D2"by default, but see thehclust()help for other options). IfFALSEorNULL, no sorting is done.- object
a confusion object
- type
either
"all"(by default), or consideringTPis the true positives,FPis the false positives,TNis the true negatives andFNis the false negatives, one can also specify:"Fscore"(F-score = F-measure = F1 score = harmonic mean of Precision and recall),"Recall"(TP / (TP + FN) = 1 - FNR),"Precision"(TP / (TP + FP) = 1 - FDR),"Specificity"(TN / (TN + FP) = 1 - FPR),"NPV"(Negative predicted value = TN / (TN + FN) = 1 - FOR),"FPR"(False positive rate = 1 - Specificity = FP / (FP + TN)),"FNR"(False negative rate = 1 - Recall = FN / (TP + FN)),"FDR"(False Discovery Rate = 1 - Precision = FP / (TP + FP)),"FOR"(False omission rate = 1 - NPV = FN / (FN + TN)),"LRPT"(Likelihood Ratio for Positive Tests = Recall / FPR = Recall / (1 - Specificity)),"LRNT"Likelihood Ratio for Negative Tests = FNR / Specificity = (1 - Recall) / Specificity,"LRPS"(Likelihood Ratio for Positive Subjects = Precision / FOR = Precision / (1 - NPV)),"LRNS"(Likelihood Ratio Negative Subjects = FDR / NPV = (1 - Precision) / (1 - FOR)),"BalAcc"(Balanced accuracy = (Sensitivity + Specificity) / 2),"MCC"(Matthews correlation coefficient),"Chisq"(Chisq metric), or"Bray"(Bray-Curtis metric)- sort.by
the statistics to use to sort the table (by default, Fmeasure, the F1 score for each class = 2 * recall * precision / (recall + precision)).
- decreasing
do we sort in increasing or decreasing order?
Examples
data("Glass", package = "mlbench")
# Use a little bit more informative labels for Type
Glass$Type <- as.factor(paste("Glass", Glass$Type))
# Use learning vector quantization to classify the glass types
# (using default parameters)
summary(glass_lvq <- ml_lvq(Type ~ ., data = Glass))
#> Codebook:
#> Class RI Na Mg Al Si K
#> 68 Glass 1 1.521578 12.95007 3.92060159 0.7609164 72.34307 0.13436937
#> 38 Glass 1 1.519355 13.44587 4.02549187 0.8598053 73.27975 -0.65459788
#> 29 Glass 1 1.520683 12.93587 3.98669073 1.5911967 73.08568 0.55901650
#> 59 Glass 1 1.518503 13.18995 3.80533210 1.0458458 73.14299 0.52751843
#> 66 Glass 1 1.520291 13.61389 3.90550140 1.0625164 71.96650 0.23594792
#> 33 Glass 1 1.517804 12.89570 3.60734181 1.2411209 73.05301 0.58662651
#> 70 Glass 1 1.521656 13.45540 3.95871610 0.6610944 72.16606 0.11827573
#> 48 Glass 1 1.523329 14.24472 4.01893496 0.4230803 71.93159 0.03217094
#> 115 Glass 2 1.518567 13.00966 3.88981210 1.2259895 72.48994 0.56046870
#> 117 Glass 2 1.517239 13.53893 3.97575235 1.2842881 72.38045 0.37785918
#> 88 Glass 2 1.516742 13.11040 4.06636711 1.7041803 73.03254 -0.15782039
#> 119 Glass 2 1.517708 13.27198 3.86833059 1.1580659 73.01953 0.42318193
#> 106 Glass 2 1.528194 11.98583 0.00000000 1.3295772 71.46347 0.31723656
#> 120 Glass 2 1.516509 13.29630 3.82530000 0.9039000 73.10880 0.39040000
#> 75 Glass 2 1.516204 13.03875 3.53608516 1.5833155 72.99700 0.57913827
#> 85 Glass 2 1.509780 14.48487 4.34526300 1.8453460 73.06978 -0.42379600
#> 162 Glass 3 1.517816 13.66737 3.61538989 0.6409221 72.92571 0.08737938
#> 160 Glass 3 1.511979 12.91493 3.99933120 1.6205488 72.48891 0.78271680
#> 166 Glass 5 1.520699 12.45088 0.80148788 1.7997257 72.95239 0.39569379
#> 178 Glass 6 1.517809 14.77600 1.58108551 1.3885427 73.11796 -0.35529741
#> 205 Glass 7 1.517259 14.49979 0.00000000 3.5128100 73.32193 -1.90200000
#> 196 Glass 7 1.516084 14.52729 0.05037736 2.6320693 73.21952 -0.11970653
#> 214 Glass 7 1.517816 14.54594 0.04773913 2.0046758 73.00199 0.05534753
#> Ca Ba Fe
#> 68 9.813214 -0.016079322 0.147024767
#> 38 9.042885 -0.082701208 -0.055399254
#> 29 8.530751 0.000000000 -0.219829500
#> 59 8.307269 -0.085080812 -0.031852888
#> 66 9.015965 0.052841224 -0.018509254
#> 33 8.541861 -0.003446127 0.051153231
#> 70 9.560170 0.000000000 0.086951807
#> 48 9.306804 -0.141330556 0.022025897
#> 115 8.619077 -0.066977972 0.173076048
#> 117 8.486520 -0.394739095 0.148651809
#> 88 8.099332 0.000000000 0.085611842
#> 119 8.442326 -0.412500000 0.294176050
#> 106 14.076300 0.583754180 0.160455896
#> 120 8.637700 -0.660000000 0.000000000
#> 75 8.154841 -0.126230585 -0.006288184
#> 85 8.430369 -1.900743000 0.000000000
#> 162 8.860970 0.076696295 0.122714072
#> 160 8.494849 -0.405381600 -0.053035200
#> 166 11.361974 0.015242174 0.080096215
#> 178 9.915830 -0.397097497 -0.134253854
#> 205 9.183520 1.471990000 -0.031200000
#> 196 8.897774 0.744367910 0.012255346
#> 214 8.385990 1.893292340 0.007916969
# Calculate cross-validated confusion matrix
(glass_conf <- confusion(cvpredict(glass_lvq), Glass$Type))
#> 214 items classified with 138 true positives (error rate = 35.5%)
#> Predicted
#> Actual 01 02 03 04 05 06 (sum) (FNR%)
#> 01 Glass 3 2 8 7 0 0 0 17 88
#> 02 Glass 1 2 56 12 0 0 0 70 20
#> 03 Glass 2 0 21 50 5 0 0 76 34
#> 04 Glass 5 0 1 3 7 0 2 13 46
#> 05 Glass 6 0 3 1 0 1 4 9 89
#> 06 Glass 7 0 2 3 1 1 22 29 24
#> (sum) 4 91 76 13 2 28 214 36
# Raw confusion matrix: no sort and no margins
print(glass_conf, sums = FALSE, sort = FALSE)
#> 214 items classified with 138 true positives (error rate = 35.5%)
#> Predicted
#> Actual 01 02 03 04 05 06
#> 01 Glass 1 56 12 2 0 0 0
#> 02 Glass 2 21 50 0 5 0 0
#> 03 Glass 3 8 7 2 0 0 0
#> 04 Glass 5 1 3 0 7 0 2
#> 05 Glass 6 3 1 0 0 1 4
#> 06 Glass 7 2 3 0 1 1 22
summary(glass_conf)
#> 214 items classified with 138 true positives (error = 35.5%)
#>
#> Global statistics on reweighted data:
#> Error rate: 35.5%, F(micro-average): 0.544, F(macro-average): 0.506
#>
#> Fscore Recall Precision Specificity NPV FPR
#> Glass 7 0.7719298 0.7586207 0.7857143 0.9675676 0.9623656 0.032432432
#> Glass 1 0.6956522 0.8000000 0.6153846 0.7569444 0.8861789 0.243055556
#> Glass 2 0.6578947 0.6578947 0.6578947 0.8115942 0.8115942 0.188405797
#> Glass 5 0.5384615 0.5384615 0.5384615 0.9701493 0.9701493 0.029850746
#> Glass 3 0.1904762 0.1176471 0.5000000 0.9898477 0.9285714 0.010152284
#> Glass 6 0.1818182 0.1111111 0.5000000 0.9951220 0.9622642 0.004878049
#> FNR FDR FOR LRPT LRNT LRPS LRNS
#> Glass 7 0.2413793 0.2142857 0.03763441 23.390805 0.2494702 20.877551 0.2226656
#> Glass 1 0.2000000 0.3846154 0.11382114 3.291429 0.2642202 5.406593 0.4340155
#> Glass 2 0.3421053 0.3421053 0.18840580 3.491903 0.4215226 3.491903 0.4215226
#> Glass 5 0.4615385 0.4615385 0.02985075 18.038462 0.4757396 18.038462 0.4757396
#> Glass 3 0.8823529 0.5000000 0.07142857 11.588235 0.8914027 7.000000 0.5384615
#> Glass 6 0.8888889 0.5000000 0.03773585 22.777778 0.8932462 13.250000 0.5196078
#> BalAcc MCC Chisq Bray Auto Manu A_M TP FP FN TN
#> Glass 7 0.8630941 0.7370528 116.254820 0.002336449 28 29 -1 22 6 7 179
#> Glass 1 0.7784722 0.5285291 59.779400 0.049065421 91 70 21 56 35 14 109
#> Glass 2 0.7347445 0.4694889 47.169851 0.000000000 76 76 0 50 26 26 112
#> Glass 5 0.7543054 0.5086108 55.358577 0.000000000 13 13 0 7 6 6 195
#> Glass 3 0.5537474 0.2146373 9.858806 0.030373832 4 17 -13 2 2 15 195
#> Glass 6 0.5531165 0.2216027 10.509056 0.016355140 2 9 -7 1 1 8 204
summary(glass_conf, type = "Fscore")
#> Glass 7 Glass 1 Glass 2 Glass 5 Glass 3 Glass 6
#> 0.7719298 0.6956522 0.6578947 0.5384615 0.1904762 0.1818182
#> attr(,"stat.type")
#> [1] "Fscore"