R/collect.R
collect_metrics.workflow_set.RdReturn a tibble of performance metrics for all models or submodels.
# S3 method for class 'workflow_set'
collect_metrics(x, ..., summarize = TRUE)
# S3 method for class 'workflow_set'
collect_predictions(
x,
...,
summarize = TRUE,
parameters = NULL,
select_best = FALSE,
metric = NULL
)
# S3 method for class 'workflow_set'
collect_notes(x, ...)
# S3 method for class 'workflow_set'
collect_extracts(x, ...)A workflow_set object that has been evaluated
with workflow_map().
Not currently used.
A logical for whether the performance estimates should be summarized via the mean (over resamples) or the raw performance values (per resample) should be returned along with the resampling identifiers. When collecting predictions, these are averaged if multiple assessment sets contain the same row.
An optional tibble of tuning parameter values that can be
used to filter the predicted values before processing. This tibble should
only have columns for each tuning parameter identifier (e.g. "my_param"
if tune("my_param") was used).
A single logical for whether the numerically best results
are retained. If TRUE, the parameters argument is ignored.
A character string for the metric that is used for
select_best.
A tibble.
When applied to a workflow set, the metrics and predictions that are returned do not contain the actual tuning parameter columns and values (unlike when these collect functions are run on other objects). The reason is that workflow sets can contain different types of models or models with different tuning parameters.
If the columns are needed, there are two options. First, the .config column
can be used to merge the tuning parameter columns into an appropriate object.
Alternatively, the map() function can be used to get the metrics from the
original objects (see the example below).
The package supplies two pre-generated workflow sets, two_class_set
and chi_features_set, and associated sets of model fits
two_class_res and chi_features_res.
The two_class_* objects are based on a binary classification problem
using the two_class_dat data from the modeldata package. The six
models utilize either a bare formula or a basic recipe utilizing
recipes::step_YeoJohnson() as a preprocessor, and a decision tree,
logistic regression, or MARS model specification. See ?two_class_set
for source code.
The chi_features_* objects are based on a regression problem using the
Chicago data from the modeldata package. Each of the three models
utilize a linear regression model specification, with three different
recipes of varying complexity. The objects are meant to approximate the
sequence of models built in Section 1.3 of Kuhn and Johnson (2019). See
?chi_features_set for source code.
library(dplyr)
#>
#> Attaching package: ‘dplyr’
#> The following objects are masked from ‘package:stats’:
#>
#> filter, lag
#> The following objects are masked from ‘package:base’:
#>
#> intersect, setdiff, setequal, union
library(purrr)
library(tidyr)
two_class_res
#> # A workflow set/tibble: 6 × 4
#> wflow_id info option result
#> <chr> <list> <list> <list>
#> 1 none_cart <tibble [1 × 4]> <opts[3]> <tune[+]>
#> 2 none_glm <tibble [1 × 4]> <opts[3]> <rsmp[+]>
#> 3 none_mars <tibble [1 × 4]> <opts[3]> <tune[+]>
#> 4 yj_trans_cart <tibble [1 × 4]> <opts[3]> <tune[+]>
#> 5 yj_trans_glm <tibble [1 × 4]> <opts[3]> <rsmp[+]>
#> 6 yj_trans_mars <tibble [1 × 4]> <opts[3]> <tune[+]>
# ------------------------------------------------------------------------------
# \donttest{
collect_metrics(two_class_res)
#> # A tibble: 52 × 9
#> wflow_id .config preproc model .metric .estimator mean n std_err
#> <chr> <chr> <chr> <chr> <chr> <chr> <dbl> <int> <dbl>
#> 1 none_cart Preprocessor1… formula deci… accura… binary 0.777 5 0.00517
#> 2 none_cart Preprocessor1… formula deci… roc_auc binary 0.839 5 0.0160
#> 3 none_cart Preprocessor1… formula deci… accura… binary 0.784 5 0.00812
#> 4 none_cart Preprocessor1… formula deci… roc_auc binary 0.841 5 0.0171
#> 5 none_cart Preprocessor1… formula deci… accura… binary 0.791 5 0.0111
#> 6 none_cart Preprocessor1… formula deci… roc_auc binary 0.853 5 0.00813
#> 7 none_cart Preprocessor1… formula deci… accura… binary 0.814 5 0.0104
#> 8 none_cart Preprocessor1… formula deci… roc_auc binary 0.829 5 0.0145
#> 9 none_cart Preprocessor1… formula deci… accura… binary 0.788 5 0.0133
#> 10 none_cart Preprocessor1… formula deci… roc_auc binary 0.852 5 0.0101
#> # ℹ 42 more rows
# Alternatively, if the tuning parameter values are needed:
two_class_res |>
dplyr::filter(grepl("cart", wflow_id)) |>
mutate(metrics = map(result, collect_metrics)) |>
dplyr::select(wflow_id, metrics) |>
tidyr::unnest(cols = metrics)
#> # A tibble: 40 × 9
#> wflow_id cost_complexity min_n .metric .estimator mean n std_err .config
#> <chr> <dbl> <int> <chr> <chr> <dbl> <int> <dbl> <chr>
#> 1 none_ca… 2.28e- 7 13 accura… binary 0.777 5 0.00517 Prepro…
#> 2 none_ca… 2.28e- 7 13 roc_auc binary 0.839 5 0.0160 Prepro…
#> 3 none_ca… 1.32e- 5 16 accura… binary 0.784 5 0.00812 Prepro…
#> 4 none_ca… 1.32e- 5 16 roc_auc binary 0.841 5 0.0171 Prepro…
#> 5 none_ca… 8.19e-10 26 accura… binary 0.791 5 0.0111 Prepro…
#> 6 none_ca… 8.19e-10 26 roc_auc binary 0.853 5 0.00813 Prepro…
#> 7 none_ca… 9.52e- 3 9 accura… binary 0.814 5 0.0104 Prepro…
#> 8 none_ca… 9.52e- 3 9 roc_auc binary 0.829 5 0.0145 Prepro…
#> 9 none_ca… 4.76e- 7 20 accura… binary 0.788 5 0.0133 Prepro…
#> 10 none_ca… 4.76e- 7 20 roc_auc binary 0.852 5 0.0101 Prepro…
#> # ℹ 30 more rows
# }
collect_metrics(two_class_res, summarize = FALSE)
#> # A tibble: 260 × 8
#> wflow_id .config preproc model id .metric .estimator .estimate
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <dbl>
#> 1 none_cart Preprocessor1_Mod… formula deci… Fold1 accura… binary 0.786
#> 2 none_cart Preprocessor1_Mod… formula deci… Fold1 roc_auc binary 0.875
#> 3 none_cart Preprocessor1_Mod… formula deci… Fold2 accura… binary 0.766
#> 4 none_cart Preprocessor1_Mod… formula deci… Fold2 roc_auc binary 0.851
#> 5 none_cart Preprocessor1_Mod… formula deci… Fold3 accura… binary 0.778
#> 6 none_cart Preprocessor1_Mod… formula deci… Fold3 roc_auc binary 0.848
#> 7 none_cart Preprocessor1_Mod… formula deci… Fold4 accura… binary 0.766
#> 8 none_cart Preprocessor1_Mod… formula deci… Fold4 roc_auc binary 0.779
#> 9 none_cart Preprocessor1_Mod… formula deci… Fold5 accura… binary 0.791
#> 10 none_cart Preprocessor1_Mod… formula deci… Fold5 roc_auc binary 0.841
#> # ℹ 250 more rows